The nasa-blue-marble source reads one pinned publication -- NASA Earth Observatory's Blue Marble Next Generation base map -- into a raster-only Earth volume: one world, one lens, the shared whole-sphere declarations with the body and its radius named, and nothing else. The whole-sphere equirectangular arithmetic NASA Trek carried moves to the interchange package as the shared convention it always was, and Trek reads through it unchanged. The crawler half is the one networked step and it is aimed at one URL, one digest, one recorded capture time. Bytes that do not hash to the pin are refused by name, the fetched source is kept content-addressed beside the capture so a re-run asks NASA nothing, and the reference level is cut through a fixed-point Catmull-Rom resampler built for the same property the deriver's box fold has: integer arithmetic wherever a pixel is decided, so the capture -- and everything stamped downstream of it -- reproduces byte for byte. The capture index takes a pinned capturedAt for exactly this shape of source; a live origin still gets the clock's first sight. A raster-only world also writes its empty collections array as a list rather than a null, which no existing bundle carried and no reader should have to guess about. make included-earth is the whole run, and docs/generate.md carries the source's section.
- commit
- 99549cf50fd53968687943edf8cdf1659b147f03
- parent
- 3f9d78a5bdd32456bf402cd552975cd5f4e826a1
- author
- Dami <dami.etoile@gmail.com>
- kind
- feat risk score 1.7 of 3, Jev's estimate
| M | Makefile | 14 | +- |
| M | docs/ | 2 | +- |
| M | docs/ | 44 | ++++- |
| M | internal/ | 12 | +- |
| A | internal/ | 312 | ++++++++++++++++++++++++++++++ |
| A | internal/ | 244 | +++++++++++++++++++++++ |
| M | internal/ | 1 | + |
| A | internal/ | 210 | ++++++++++++++++++++ |
| M | internal/ | 13 | +- |
| A | internal/ | 79 | ++++++++ |
| A | internal/ | 47 | +++++ |
| A | internal/ | 206 | ++++++++++++++++++++ |
| A | internal/ | 178 | +++++++++++++++++ |
| M | internal/ | 66 | ++----- |
| M | internal/ | 4 | +- |
| M | internal/ | 2 | + |
@@ -3,7 +3,7 @@
33 # These targets are the enforcement surface; the existing build recipes are44 # untouched and still live where they always did.55 6-6+77 88 # The one entrypoint: every required gate, and nothing that can silently99 # decline to judge. Go tests run through tools/testgate, which fails the run@@ -89,6 +89,18 @@ static: seam
8989 serve-static: static9090 go run ./cmd/atlas serve -static dist/static9191 92+# The included Earth volume: the pinned Blue Marble base map captured, derived93+# and composed into the committed bundle the desktop shell embeds94+# (included/README.md is the recipe). The crawl step is the one networked part95+# and fetches nothing when the pinned source is already in the staged archive;96+# everything after it is offline. The registry index compose derives beside the97+# bundle is deleted because included/ is an embedded asset, not a library.98+included-earth:99+ go run ./cmd/atlas crawl -archive crawl/blue-marble/fmg-archive -source nasa-blue-marble earth100+ go run ./cmd/atlas tiles -archive crawl/blue-marble/fmg-archive -output tiles/blue-marble101+ go run ./cmd/atlas compose -archive crawl/blue-marble/fmg-archive -tiles tiles/blue-marble/index.json -bundles included earth102+ rm -f included/index.json103+92104 # The desktop shell (issue #5 §3.4): the application in a window, one file,93105 # seam embedded. This is the macOS recipe -- the same `go build` .github/94106 # workflows/release.yml runs, plus the bundle directory macOS wants before it@@ -19,7 +19,7 @@ pictures it, and [`decisions/`](decisions/) is why any of it is shaped this way.
1919 |---|---|2020 | [`format.md`](format.md) | Normative `.atlas` v3 spec, sufficient alone to implement a reader: the container, the manifest, the world payload, the `ATLASLOC` bytes, stamps and ordering, validation, the registry directory, and a checklist for a new reader. |2121 | [`semconv/REGISTRY.md`](semconv/REGISTRY.md) | The attribute vocabulary, generated from `spec/registry.yaml`; `spec/gen`'s own test holds the committed copy up to date. |22-| [`generate.md`](generate.md) | The generate lane: the interchange document, the sources and their gates, the capture archive, tile derivation, composition, curation. §2.7 is the "add a source" walkthrough. |22+| [`generate.md`](generate.md) | The generate lane: the interchange document, the sources and their gates, the capture archive, tile derivation, composition, curation. §2.8 is the "add a source" walkthrough. |2323 | [`enrich.md`](enrich.md) | The enrich lane: the enricher interface, the contribution format, the ledger vocabulary, the curated queue, the maturity score. |2424 | [`analysis.md`](analysis.md) | The cell-system contract: the eighteen methods, coordinates and continuity, the `Ground` descriptor, the plan order, the style tokens. §9 is the "add a third system" checklist. |2525 | [`app.md`](app.md) | The hypermedia application: routes, the session record, regions and partials, the state island, the hostenv contract and the three-host story. |@@ -282,7 +282,41 @@ Nomenclature's feature list for the same body.
282282 - `idSpace: "derived"`: nothing in either publication numbers a mosaic or a283283 feature type, so every identity is minted from a stable name.284284 285-### 2.4 The IGN reader285+### 2.4 The Blue Marble reader286+287+`internal/generate/sources/nasabluemarble` reads the included Earth volume's288+one capture: NASA Earth Observatory's Blue Marble Next Generation base map, a289+single pinned publication rather than a live endpoint.290+291+- Capture kind `blue-marble-map`, one per capture of the pinned image.292+- The volume is its picture. One world (`earth`), one lens (`Blue Marble`), no293+ collections and no features: pins, layers and enrichment are later sources'294+ work, folded in by the enrich lane.295+- The coordinate design is the shared whole-sphere window NASA Trek also rides296+ (`doc.EquirectPx` and friends): a 2:1 global image filling the top half of297+ the world square, declared as registered conventions — surface, projection,298+ the equirect px/deg pair, the body, and the body's mean radius — so the cell299+ systems recognise the ground through the same declarations every spherical300+ world makes.301+- The capture body records the product's identity — its title, NASA's credit302+ line, the source digest and size — and the derivation policy its tiles were303+ cut under, so a policy change reads as a new capture rather than the same304+ one wearing new bytes.305+- The gates: another kind's capture, another source's name, another body, a306+ missing digest, a missing size, or a capture declaring no pyramid are307+ refused by name.308+- `idSpace: "derived"`: nothing in the publication numbers a base map.309+310+The crawler half (`internal/generate/crawl/bluemarble.go`) is the one place311+the pinned URL, digest and capture time are spelled. It fetches the source312+image exactly once into the staged archive, refuses any digest but the pinned313+one, cuts the reference level through the deterministic fixed-point314+Catmull-Rom resampler beside it, and records the capture under the pinned315+time, so the archive — and everything stamped downstream of it — reproduces on316+any machine. `included/README.md` is the recipe and the provenance, and317+`make included-earth` is the whole run.318+319+### 2.5 The IGN reader286320 287321 `internal/generate/sources/ign` reads a community wikimap: a flat image tiled288322 like a world, markers placed on it in image-relative coordinates, and a flat list@@ -307,7 +341,7 @@ of marker types that names a parent to make two levels of legend.
307341 the evidence has been thrown away.308342 - `idSpace: "derived"`: markers are opaque strings and types are slugs.309343 310-### 2.5 The Piggyback reader344+### 2.6 The Piggyback reader311345 312346 `internal/generate/sources/piggyback` reads the official guide house's maps,313347 which carry what a community wikimap does not: prose. Pins arrive with names and@@ -335,7 +369,7 @@ descriptions and both survive into a volume.
335369 against the published map; a capture whose numbers are not in it fails.336370 - `idSpace: "derived"`: pins, categories and types are opaque string ids.337371 338-### 2.6 The ArcGIS Hub reader372+### 2.7 The ArcGIS Hub reader339373 340374 `internal/generate/sources/arcgishub` reads a city. It is the only source whose341375 subject is a real place, and the only one where **a world is a date**: each crawl@@ -376,7 +410,7 @@ difference in the city.
376410 - `idSpace: "derived"`: a hub's object ids are load artifacts that churn between377411 refreshes, and nothing above a row is numbered at all.378412 379-### 2.7 Adding a source413+### 2.8 Adding a source380414 381415 A source is one directory, one constructor, and one line in382416 `internal/generate/sources/registry.go`. The walkthrough:@@ -541,6 +575,7 @@ somebody else's server is having a bad afternoon.
541575 | crawler | target | runnable here |542576 | --- | --- | --- |543577 | `ign` | `<objectSlug>/<mapSlug>` | code-complete, not run — the captures are archived |578+| `nasa-blue-marble` | `earth` | runnable — one pinned image, digest-verified, fetched once (§2.4) |544579 545580 The game sources' captures are archived and their endpoints are somebody else's546581 editorial work, so their crawlers are kept complete and are not run against live@@ -1072,6 +1107,7 @@ points at:
10721107 | --- | --- | --- |10731108 | the games corpus archive | `crawl/fmg-archive` | `ATLAS_ARCHIVE_DIR` |10741109 | the proof city's archive | `crawl/bend-or/fmg-archive` | `ATLAS_CITY_ARCHIVE_DIR` |1110+| the included Earth's archive | `crawl/blue-marble/fmg-archive` | none — `make included-earth` names it |10751111 | the derived tile sets | `tiles/`, register at `tiles/index.json` | `ATLAS_TILES_INDEX` |10761112 10771113 So the whole lane, over one volume, is three commands:@@ -180,10 +180,14 @@ func tally(collections []composedCollection) counts {
180180 // them, points before shapes.181181 func buildPayload(w composedWorld) (worldPayload, []byte, map[string]featureText) {182182 payload := worldPayload{183- Grid: w.Grid,184- Lenses: w.Lenses,185- Attrs: w.Attrs,186- Merged: w.Merged,183+ Grid: w.Grid,184+ // The collections array is always present, even empty: a raster-only185+ // world holds none, and a reader iterating the legend should meet a186+ // list rather than a null.187+ Collections: []wireCollection{},188+ Lenses: w.Lenses,189+ Attrs: w.Attrs,190+ Merged: w.Merged,187191 }188192 var locations []bundle.Location189193 text := make(map[string]featureText)@@ -0,0 +1,312 @@
1+package crawl2+3+import (4+ "bytes"5+ "context"6+ "encoding/json"7+ "fmt"8+ "image"9+ "image/jpeg"10+ "log/slog"11+ "os"12+ "path/filepath"13+ "strconv"14+15+ "github.com/FelineStateMachine/atlas/internal/generate/doc"16+ "github.com/FelineStateMachine/atlas/internal/generate/sources/nasabluemarble"17+ "github.com/FelineStateMachine/atlas/internal/logging"18+)19+20+// The Blue Marble crawler: the one networked step behind the included Earth21+// volume.22+//23+// Every other crawler follows a publisher wherever its endpoints lead. This one24+// is aimed at a single pinned publication -- NASA Earth Observatory's Blue25+// Marble Next Generation base map, July 2004, one image at one URL with one26+// digest -- and it refuses anything else. The digest is the whole contract:27+// bytes that do not hash to the pin are not the pinned publication, whatever28+// the URL served, and accepting them silently would put an unaccounted picture29+// into every library that ships the included volume.30+//31+// What lands in the archive is the ordinary shape every capture takes: a32+// capture body naming the product, and the reference-level tiles the deriver33+// folds down -- the source image cut to the shared whole-sphere window through34+// the deterministic resampler beside this file. The downloaded source itself is35+// kept beside the capture, content-addressed, so a re-run derives without36+// asking NASA again; it is a working file of the archive, never part of a37+// bundle and never read at runtime.38+39+// blueMarblePin is everything fixed about the capture: where the publication40+// is, what its bytes must hash to, when it was first captured for this41+// artifact, and the derivation policy the tiles are cut under. It is a struct42+// rather than constants so the write path can be exercised against a stated43+// pin; the shipped pin is blueMarble below.44+type blueMarblePin struct {45+ // Page is the product's own page, and Asset the exact image fetched.46+ Page string47+ Asset string48+ // SHA256 is the digest the fetched bytes must carry. A mismatch is a49+ // refusal, never a warning.50+ SHA256 string51+ // CapturedAt is the moment the pinned publication was first captured for52+ // this artifact. It is part of the pin so the archive -- and everything53+ // stamped downstream of it -- reproduces on any machine.54+ CapturedAt string55+ // Width and Height are the source image's declared pixels.56+ Width, Height int57+ // Quality is the JPEG setting the reference tiles are encoded at. It is58+ // spelled here, once, because two runs of one pipeline may not disagree.59+ Quality int60+}61+62+// blueMarble is the shipped pin: Blue Marble Next Generation, July 2004, as63+// published by NASA Earth Observatory. included/README.md is the recipe that64+// carries these same facts for a person.65+var blueMarble = blueMarblePin{66+ Page: "https://science.nasa.gov/earth/earth-observatory/blue-marble-next-generation/base-map/",67+ Asset: "https://assets.science.nasa.gov/content/dam/science/esd/eo/images/bmng/bmng-base/july/world.200407.3x21600x10800.jpg",68+ SHA256: "dea8b4dc8a4f93f5f8bce0c8c85a508a178e7901e9ed8e6bf86e6ce7ef6d61e2",69+ CapturedAt: "2026-08-03T14:30:39Z",70+ Width: 21600,71+ Height: 10800,72+ Quality: 90,73+}74+75+// blueMarbleResampler names the cut for the capture body: the resampler beside76+// this file, at its fixed-point precision. A change to how the reference level77+// is cut renames this, and the renamed policy reads as a new capture.78+const blueMarbleResampler = "catmull-rom-fixed15"79+80+// blueMarbleZoom is the reference level the source is cut to: the level whose81+// pixels are the world square's own, holding the whole 8192×4096 picture.82+const blueMarbleZoom = doc.SyntheticZoom83+84+// blueMarbleIDBit marks Blue Marble identities in the archive register, beside85+// the bits the other sources' identities carry.86+const blueMarbleIDBit = int64(1) << 3687+88+type blueMarbleCrawler struct{}89+90+func (blueMarbleCrawler) Name() string { return nasabluemarble.SourceName }91+92+func (blueMarbleCrawler) Usage() string {93+ return "the pinned Blue Marble base map; the target is \"earth\""94+}95+96+func (c blueMarbleCrawler) Crawl(ctx context.Context, run Run) error {97+ if run.Target != nasabluemarble.Body {98+ return fmt.Errorf("%s captures the pinned Earth base map; the target is %q, not %q",99+ c.Name(), nasabluemarble.Body, run.Target)100+ }101+ log := run.Logger().With(logging.Source(c.Name()), logging.Op("crawl"))102+103+ volume := Volume{104+ ID: blueMarbleArchiveID("bluemarble:" + nasabluemarble.Body),105+ Title: doc.Title(nasabluemarble.Body),106+ Locator: blueMarble.Page,107+ Source: c.Name(),108+ // The same-slug policy: any source describing the Earth registers the109+ // plain title while naming its directory after itself.110+ DirectoryTitle: "NASA Blue Marble Earth",111+ }112+ world := World{113+ ID: blueMarbleArchiveID("bluemarble:" + nasabluemarble.Body + "/" + nasabluemarble.Body),114+ Slug: nasabluemarble.Body,115+ Title: doc.Title(nasabluemarble.Body),116+ }117+ if run.DryRun {118+ log.Info("crawl would archive", logging.Volume(volume.Title), logging.World(world.Slug),119+ "asset", blueMarble.Asset, "sha256", blueMarble.SHA256)120+ return nil121+ }122+123+ volumeDir, err := run.Archive.RegisterVolume(volume)124+ if err != nil {125+ return err126+ }127+ worldDir, err := run.Archive.RegisterWorld(volume, volumeDir, world)128+ if err != nil {129+ return err130+ }131+ source, err := c.source(ctx, run, worldDir, log)132+ if err != nil {133+ return err134+ }135+ if err := writeBlueMarble(run.Archive, worldDir, world.ID, blueMarble, source, log); err != nil {136+ return err137+ }138+ return nil139+}140+141+// source answers the pinned image's bytes: the copy already kept beside the142+// capture when there is one, the origin exactly once when there is not. Either143+// way the bytes are held to the pinned digest before anything is derived from144+// them.145+func (c blueMarbleCrawler) source(ctx context.Context, run Run, worldDir string, log *slog.Logger) ([]byte, error) {146+ cached := blueMarbleSourcePath(worldDir, blueMarble)147+ if data, err := os.ReadFile(cached); err == nil {148+ if err := verifyBlueMarble(data, blueMarble); err != nil {149+ return nil, fmt.Errorf("cached source %s: %w", cached, err)150+ }151+ log.Info("source already captured", logging.Path(cached), "bytes", len(data))152+ return data, nil153+ }154+ data, _, err := run.Fetch.Get(ctx, blueMarble.Asset, nil)155+ if err != nil {156+ return nil, fmt.Errorf("fetch %s: %w", blueMarble.Asset, err)157+ }158+ if err := verifyBlueMarble(data, blueMarble); err != nil {159+ return nil, err160+ }161+ if err := writeFile(cached, data); err != nil {162+ return nil, err163+ }164+ log.Info("source captured", logging.Path(cached), "bytes", len(data))165+ return data, nil166+}167+168+// blueMarbleSourcePath is where the downloaded source sits: beside the world's169+// captures, content-addressed by the pinned digest.170+func blueMarbleSourcePath(worldDir string, pin blueMarblePin) string {171+ return filepath.Join(worldDir, "source", pin.SHA256+".jpg")172+}173+174+// verifyBlueMarble holds bytes to the pin. It runs before any decode: bytes175+// that are not the pinned publication are not decoded, not derived from, and176+// not archived.177+func verifyBlueMarble(data []byte, pin blueMarblePin) error {178+ if sum := Hash(data); sum != pin.SHA256 {179+ return fmt.Errorf(180+ "source digest mismatch: fetched bytes hash to %s where the pin requires %s; "+181+ "refusing changed upstream bytes", sum, pin.SHA256)182+ }183+ return nil184+}185+186+// writeBlueMarble derives the capture from verified source bytes: the image187+// decoded, held to its declared size, cut to the reference level through the188+// deterministic resampler, tiled, and recorded -- and the capture body written189+// last, carrying the product's identity and the policy the cut was made under.190+func writeBlueMarble(store *Archive, worldDir string, worldID int64, pin blueMarblePin, source []byte, log *slog.Logger) error {191+ decoded, kind, err := image.Decode(bytes.NewReader(source))192+ if err != nil {193+ return fmt.Errorf("decode source: %w", err)194+ }195+ bounds := decoded.Bounds()196+ if kind != "jpeg" || bounds.Dx() != pin.Width || bounds.Dy() != pin.Height {197+ return fmt.Errorf("source is a %d×%d %s where the pin requires a %d×%d jpeg",198+ bounds.Dx(), bounds.Dy(), kind, pin.Width, pin.Height)199+ }200+201+ maxX, maxY := doc.EquirectLevelExtent(blueMarbleZoom)202+ ground := resample(toNRGBA(decoded), (maxX+1)*doc.SyntheticTileSize, (maxY+1)*doc.SyntheticTileSize)203+204+ index, err := store.OpenTileIndex(worldDir, worldID)205+ if err != nil {206+ return err207+ }208+ setID := index.SetID(nasabluemarble.TileSet)209+ for y := 0; y <= maxY; y++ {210+ for x := 0; x <= maxX; x++ {211+ tile := ground.SubImage(image.Rect(212+ x*doc.SyntheticTileSize, y*doc.SyntheticTileSize,213+ (x+1)*doc.SyntheticTileSize, (y+1)*doc.SyntheticTileSize))214+ var encoded bytes.Buffer215+ if err := jpeg.Encode(&encoded, tile, &jpeg.Options{Quality: pin.Quality}); err != nil {216+ return fmt.Errorf("encode tile %d/%d: %w", x, y, err)217+ }218+ data := encoded.Bytes()219+ if err := WriteTile(TilePath(worldDir, setID, blueMarbleZoom, x, y, "jpg"), data); err != nil {220+ return err221+ }222+ index.Put(TileRecord{223+ ByteLength: len(data),224+ ContentHash: Hash(data),225+ ContentType: "image/jpeg",226+ CoverageKey: index.CoverageKey(setID, blueMarbleZoom, x, y),227+ Status: StatusCached,228+ TileSetID: setID,229+ URL: blueMarbleTileURL(blueMarbleZoom, x, y),230+ X: x,231+ Y: y,232+ Zoom: blueMarbleZoom,233+ })234+ }235+ }236+ if err := index.Close(worldDir); err != nil {237+ return err238+ }239+240+ body, err := json.MarshalIndent(blueMarbleCapture{241+ Source: nasabluemarble.SourceName,242+ Body: nasabluemarble.Body,243+ Product: "Blue Marble: Next Generation, July 2004",244+ Credit: "NASA Earth Observatory",245+ AssetSHA256: pin.SHA256,246+ Width: pin.Width,247+ Height: pin.Height,248+ Map: blueMarbleMosaic{249+ MaxZoom: blueMarbleZoom,250+ Extension: "jpg",251+ LayerTitle: "Blue Marble",252+ },253+ Derive: blueMarbleDerivation{254+ Resampler: blueMarbleResampler,255+ JPEGQuality: pin.Quality,256+ },257+ }, "", " ")258+ if err != nil {259+ return err260+ }261+ hash, fresh, err := store.WriteCapture(worldDir, Capture{262+ Kind: nasabluemarble.CaptureKind,263+ SourceID: worldID,264+ SourceURL: pin.Asset,265+ CapturedAt: pin.CapturedAt,266+ Body: body,267+ })268+ if err != nil {269+ return err270+ }271+ log.Info("capture archived", logging.World(nasabluemarble.Body),272+ logging.Stamp(hash[:12]), "fresh", fresh, "tiles", (maxX+1)*(maxY+1))273+ return nil274+}275+276+// blueMarbleTileURL is the address a reference tile is recorded under. These277+// tiles were derived here and never served anywhere, so the address is honest278+// about that -- an .invalid host, like the rendered basemaps' -- while carrying279+// the marker and pyramid path the archive's reader recovers the tile set from.280+func blueMarbleTileURL(zoom, x, y int) string {281+ return "https://blue-marble.invalid/tiles/" + nasabluemarble.TileSet +282+ "/" + strconv.Itoa(zoom) + "/" + strconv.Itoa(x) + "/" + strconv.Itoa(y) + ".jpg"283+}284+285+// blueMarbleCapture is the capture body as this crawler writes it, mirrored by286+// the reader's own declaration in sources/nasabluemarble.287+type blueMarbleCapture struct {288+ Source string `json:"source"`289+ Body string `json:"body"`290+ Product string `json:"product"`291+ Credit string `json:"credit"`292+ AssetSHA256 string `json:"assetSha256"`293+ Width int `json:"width"`294+ Height int `json:"height"`295+ Map blueMarbleMosaic `json:"map"`296+ Derive blueMarbleDerivation `json:"derive"`297+}298+299+type blueMarbleMosaic struct {300+ MaxZoom int `json:"maxZoom"`301+ Extension string `json:"extension"`302+ LayerTitle string `json:"layerTitle"`303+}304+305+type blueMarbleDerivation struct {306+ Resampler string `json:"resampler"`307+ JPEGQuality int `json:"jpegQuality"`308+}309+310+func blueMarbleArchiveID(name string) int64 {311+ return int64(fnv32a(name)&0x7fffffff) | blueMarbleIDBit312+}@@ -0,0 +1,244 @@
1+package crawl2+3+import (4+ "bytes"5+ "encoding/json"6+ "image"7+ "image/color"8+ "image/jpeg"9+ "log/slog"10+ "os"11+ "path/filepath"12+ "strings"13+ "testing"14+15+ "github.com/FelineStateMachine/atlas/internal/generate/archive"16+ "github.com/FelineStateMachine/atlas/internal/generate/doc"17+)18+19+// TestBlueMarblePinsTheSpecifiedPublication holds the shipped pin to the facts20+// the included volume is specified against. A typo here is not a style problem:21+// the digest decides which bytes are accepted, the capture time decides the22+// bundle's name, and the credit is what NASA asks a republisher to keep.23+func TestBlueMarblePinsTheSpecifiedPublication(t *testing.T) {24+ if blueMarble.Asset != "https://assets.science.nasa.gov/content/dam/science/esd/eo/images/bmng/bmng-base/july/world.200407.3x21600x10800.jpg" {25+ t.Errorf("asset URL %q is not the pinned publication", blueMarble.Asset)26+ }27+ if blueMarble.SHA256 != "dea8b4dc8a4f93f5f8bce0c8c85a508a178e7901e9ed8e6bf86e6ce7ef6d61e2" {28+ t.Errorf("digest %q is not the pinned digest", blueMarble.SHA256)29+ }30+ if blueMarble.CapturedAt != "2026-08-03T14:30:39Z" {31+ t.Errorf("capture time %q is not the recorded first capture", blueMarble.CapturedAt)32+ }33+ if blueMarble.Width != 21600 || blueMarble.Height != 10800 {34+ t.Errorf("source size %d×%d is not the published image's", blueMarble.Width, blueMarble.Height)35+ }36+}37+38+// TestBlueMarbleRefusesAWrongDigest is the gate the whole capture stands39+// behind: bytes that do not hash to the pin are refused by name, and bytes that40+// do pass.41+func TestBlueMarbleRefusesAWrongDigest(t *testing.T) {42+ err := verifyBlueMarble([]byte("not the publication"), blueMarble)43+ if err == nil {44+ t.Fatal("changed upstream bytes were accepted")45+ }46+ if !strings.Contains(err.Error(), blueMarble.SHA256) || !strings.Contains(err.Error(), "refusing") {47+ t.Errorf("the refusal does not name the pin: %v", err)48+ }49+50+ stated := []byte("a stated source")51+ pin := blueMarblePin{SHA256: Hash(stated)}52+ if err := verifyBlueMarble(stated, pin); err != nil {53+ t.Errorf("the pinned bytes themselves were refused: %v", err)54+ }55+}56+57+// statedSource paints a small stated source image -- a horizontal gradient58+// crossed by bands, so neighbouring tiles differ -- and encodes it the way the59+// pin expects.60+func statedSource(t *testing.T, width, height int) []byte {61+ t.Helper()62+ img := image.NewNRGBA(image.Rect(0, 0, width, height))63+ for y := range height {64+ for x := range width {65+ img.SetNRGBA(x, y, color.NRGBA{66+ R: uint8(255 * x / width),67+ G: uint8(255 * y / height),68+ B: uint8((x / 8) % 256),69+ A: 255,70+ })71+ }72+ }73+ var out bytes.Buffer74+ if err := jpeg.Encode(&out, img, &jpeg.Options{Quality: 95}); err != nil {75+ t.Fatal(err)76+ }77+ return out.Bytes()78+}79+80+// writeStated runs the capture's write path over a stated pin into a fresh81+// archive, and answers the world directory it wrote.82+func writeStated(t *testing.T, pin blueMarblePin, source []byte) (root, worldDir string) {83+ t.Helper()84+ root = t.TempDir()85+ store, err := Open(root)86+ if err != nil {87+ t.Fatal(err)88+ }89+ volumeDir, err := store.RegisterVolume(Volume{ID: 42, Title: "Earth", Source: "nasa-blue-marble"})90+ if err != nil {91+ t.Fatal(err)92+ }93+ worldDir, err = store.RegisterWorld(94+ Volume{ID: 42, Title: "Earth", Source: "nasa-blue-marble"}, volumeDir,95+ World{ID: 43, Slug: "earth", Title: "Earth"})96+ if err != nil {97+ t.Fatal(err)98+ }99+ if err := writeBlueMarble(store, worldDir, 43, pin, source, slog.New(slog.DiscardHandler)); err != nil {100+ t.Fatal(err)101+ }102+ return root, worldDir103+}104+105+// TestBlueMarbleCutsTheWholeSphereWindow: the write path cuts exactly the106+// reference level the deriver expects -- every tile of the half-height window107+// at the reference zoom, cached, recorded under the tile set the reader names,108+// and readable back through the archive.109+func TestBlueMarbleCutsTheWholeSphereWindow(t *testing.T) {110+ source := statedSource(t, 432, 216)111+ pin := blueMarblePin{112+ SHA256: Hash(source), CapturedAt: "2026-08-03T14:30:39Z",113+ Width: 432, Height: 216, Quality: 90,114+ }115+ root, _ := writeStated(t, pin, source)116+117+ store, err := archive.Open(root)118+ if err != nil {119+ t.Fatal(err)120+ }121+ worlds, err := store.Worlds(store.Volumes()[0])122+ if err != nil {123+ t.Fatal(err)124+ }125+ captured, err := store.Tiles(worlds[0])126+ if err != nil {127+ t.Fatal(err)128+ }129+ levels, held := captured["earth/blue-marble"]130+ if !held {131+ t.Fatalf("no tiles recorded under the reader's tile set; recorded sets: %v", keysOf(captured))132+ }133+ maxX, maxY := doc.EquirectLevelExtent(blueMarbleZoom)134+ if len(levels[blueMarbleZoom]) != (maxX+1)*(maxY+1) {135+ t.Fatalf("%d tiles at the reference level, want the whole %d×%d window",136+ len(levels[blueMarbleZoom]), maxX+1, maxY+1)137+ }138+ for _, ref := range levels[blueMarbleZoom] {139+ path, format, err := store.Raster(worlds[0], ref)140+ if err != nil || format != "jpg" {141+ t.Fatalf("tile %d/%d is not readable as jpg: %v", ref.X, ref.Y, err)142+ }143+ data, err := os.ReadFile(path)144+ if err != nil {145+ t.Fatal(err)146+ }147+ if Hash(data) != ref.ContentHash {148+ t.Fatalf("tile %d/%d does not carry the hash its record claims", ref.X, ref.Y)149+ }150+ }151+152+ captures, err := store.Captures(worlds[0])153+ if err != nil || len(captures) != 1 {154+ t.Fatalf("%d captures recorded: %v", len(captures), err)155+ }156+ if captures[0].Kind != "blue-marble-map" || captures[0].CapturedAt != pin.CapturedAt {157+ t.Errorf("capture records %s at %s; the pin decides both",158+ captures[0].Kind, captures[0].CapturedAt)159+ }160+}161+162+// TestBlueMarbleWritesTheSameCaptureTwice is the determinism the included163+// volume stands on: the same pinned source, cut on two runs, is the same164+// archive -- every tile byte for byte, and the capture body under the same165+// content hash.166+func TestBlueMarbleWritesTheSameCaptureTwice(t *testing.T) {167+ source := statedSource(t, 432, 216)168+ pin := blueMarblePin{169+ SHA256: Hash(source), CapturedAt: "2026-08-03T14:30:39Z",170+ Width: 432, Height: 216, Quality: 90,171+ }172+ firstRoot, firstWorld := writeStated(t, pin, source)173+ secondRoot, secondWorld := writeStated(t, pin, source)174+175+ first, err := os.ReadFile(filepath.Join(firstWorld, "tiles", "index.json"))176+ if err != nil {177+ t.Fatal(err)178+ }179+ second, err := os.ReadFile(filepath.Join(secondWorld, "tiles", "index.json"))180+ if err != nil {181+ t.Fatal(err)182+ }183+ if !bytes.Equal(first, second) {184+ t.Fatal("two runs over one pin recorded different tile indexes")185+ }186+ var records []TileRecord187+ if err := json.Unmarshal(first, &records); err != nil {188+ t.Fatal(err)189+ }190+ for _, record := range records {191+ a, err := os.ReadFile(filepath.Join(firstRoot, tileRelPath(firstWorld, firstRoot, record)))192+ if err != nil {193+ t.Fatal(err)194+ }195+ b, err := os.ReadFile(filepath.Join(secondRoot, tileRelPath(secondWorld, secondRoot, record)))196+ if err != nil {197+ t.Fatal(err)198+ }199+ if !bytes.Equal(a, b) {200+ t.Fatalf("tile %d/%d differs between two runs", record.X, record.Y)201+ }202+ }203+204+ firstIndex, err := os.ReadFile(filepath.Join(firstWorld, "snapshots", "index.json"))205+ if err != nil {206+ t.Fatal(err)207+ }208+ secondIndex, err := os.ReadFile(filepath.Join(secondWorld, "snapshots", "index.json"))209+ if err != nil {210+ t.Fatal(err)211+ }212+ if !bytes.Equal(firstIndex, secondIndex) {213+ t.Fatal("two runs over one pin recorded different captures")214+ }215+}216+217+// TestResamplePreservesFlatGround: the frozen weights of every tap window sum218+// exactly to one, so a flat picture resamples to exactly itself -- the property219+// that keeps the cut from shifting the whole planet's tone.220+func TestResamplePreservesFlatGround(t *testing.T) {221+ flat := image.NewNRGBA(image.Rect(0, 0, 270, 135))222+ for i := range flat.Pix {223+ flat.Pix[i] = 137224+ }225+ out := resample(flat, 128, 64)226+ for i, value := range out.Pix {227+ if value != 137 {228+ t.Fatalf("pixel byte %d resampled 137 to %d", i, value)229+ }230+ }231+}232+233+func tileRelPath(worldDir, root string, record TileRecord) string {234+ rel, _ := filepath.Rel(root, TilePath(worldDir, record.TileSetID, record.Zoom, record.X, record.Y, "jpg"))235+ return rel236+}237+238+func keysOf(m map[string]map[int][]archive.TileRef) []string {239+ out := make([]string, 0, len(m))240+ for key := range m {241+ out = append(out, key)242+ }243+ return out244+}@@ -84,6 +84,7 @@ var ErrNotRunnable = errors.New("this crawler is not run against live endpoints
8484 func Registry() []Crawler {8585 return []Crawler{8686 ignCrawler{},87+ blueMarbleCrawler{},8788 }8889 }8990 @@ -0,0 +1,210 @@
1+package crawl2+3+import (4+ "image"5+ "image/draw"6+ "math"7+)8+9+// A deterministic, high-quality downsampler.10+//11+// The Blue Marble capture cuts a pinned source image down to the reference12+// level the deriver folds, and the cut has to be the same bytes on every13+// machine: the tiles' content hashes feed the derivation stamp, and the stamp14+// feeds the bundle's name. The deriver's own 2×2 box fold cannot do this cut --15+// the ratio is not a power of two -- so this file is the one other resampler in16+// the lane, and it is built for the same property the fold has: integer17+// arithmetic wherever a pixel is decided.18+//19+// The kernel is Catmull-Rom, widened by the scale factor so a downsample20+// averages over the ground a destination pixel covers instead of aliasing21+// across it. Tap weights are computed once per output column and row -- the22+// only floating-point step -- with every operation explicitly rounded, so the23+// compiler may not fuse two of them into an differently-rounded one, and are24+// then frozen to 15-bit fixed point, normalized to sum exactly to one.25+// Everything per-pixel after that is integer multiply-accumulate.26+27+// resampleShift is the fixed-point precision a frozen weight carries.28+const resampleShift = 1529+30+// resampleTaps is the weights one output index takes over the source axis.31+type resampleTaps struct {32+ first int33+ weights []int3234+}35+36+// resample scales src to width×height through the widened Catmull-Rom kernel,37+// separably: columns first into a fixed-point intermediate, then rows.38+func resample(src *image.NRGBA, width, height int) *image.NRGBA {39+ bounds := src.Bounds()40+ columns := resampleAxis(bounds.Dx(), width)41+ rows := resampleAxis(bounds.Dy(), height)42+43+ out := image.NewNRGBA(image.Rect(0, 0, width, height))44+ // Each output row needs only a window of horizontally-resampled source45+ // rows, and output rows walk downward, so the intermediates live in a46+ // small cache that trails the walk instead of a whole-image buffer.47+ cache := make(map[int][]int32)48+ for y := range height {49+ taps := rows[y]50+ for r := range taps.weights {51+ row := taps.first + r52+ if _, held := cache[row]; !held {53+ cache[row] = resampleRow(src, row, columns, width)54+ }55+ }56+ for column := range width {57+ var acc [4]int6458+ for r, weight := range taps.weights {59+ held := cache[taps.first+r]60+ for channel := range 4 {61+ acc[channel] += int64(weight) * int64(held[column*4+channel])62+ }63+ }64+ at := out.PixOffset(column, y)65+ for channel := range 4 {66+ out.Pix[at+channel] = clamp8(acc[channel])67+ }68+ }69+ for row := range cache {70+ if row < taps.first {71+ delete(cache, row)72+ }73+ }74+ }75+ return out76+}77+78+// resampleRow runs the horizontal pass over one source row, answering each79+// output column's channels in 15-bit fixed point.80+func resampleRow(src *image.NRGBA, row int, columns []resampleTaps, width int) []int32 {81+ bounds := src.Bounds()82+ out := make([]int32, width*4)83+ base := src.PixOffset(bounds.Min.X, bounds.Min.Y+row)84+ line := src.Pix[base : base+bounds.Dx()*4]85+ for column := range width {86+ taps := columns[column]87+ var acc [4]int3288+ for t, weight := range taps.weights {89+ at := (taps.first + t) * 490+ for channel := range 4 {91+ acc[channel] += weight * int32(line[at+channel])92+ }93+ }94+ copy(out[column*4:], acc[:])95+ }96+ return out97+}98+99+// clamp8 rounds a doubly-scaled accumulator back to a byte, half away from100+// zero, clamped to the channel's range.101+func clamp8(value int64) uint8 {102+ value = (value + 1<<(2*resampleShift-1)) >> (2 * resampleShift)103+ if value < 0 {104+ return 0105+ }106+ if value > 255 {107+ return 255108+ }109+ return uint8(value)110+}111+112+// resampleAxis freezes the tap weights for one axis. Taps are clamped inside113+// the source, folding an edge overhang onto the edge sample, which extends the114+// picture rather than darkening its border.115+func resampleAxis(src, dst int) []resampleTaps {116+ scale := float64(src) / float64(dst)117+ if scale < 1 {118+ scale = 1119+ }120+ support := float64(2 * scale)121+ out := make([]resampleTaps, dst)122+ for i := range dst {123+ // The output pixel's centre on the source axis.124+ center := float64(float64(2*i+1)*float64(src)) / float64(2*dst)125+ first := int(math.Ceil(float64(center - support - 0.5)))126+ last := int(math.Floor(float64(center + support - 0.5)))127+ raw := make([]float64, 0, last-first+1)128+ sum := 0.0129+ for j := first; j <= last; j++ {130+ t := float64(float64(float64(j)+0.5-center) / scale)131+ weight := catmullRom(t)132+ raw = append(raw, weight)133+ sum = float64(sum + weight)134+ }135+ frozen := freezeWeights(raw, sum)136+ // Clamp the window inside the source, folding out-of-range taps onto137+ // the edge sample they would have read past.138+ taps := resampleTaps{first: first, weights: frozen}139+ out[i] = clampTaps(taps, src)140+ }141+ return out142+}143+144+// catmullRom is the kernel, spelled so every floating-point operation rounds on145+// its own: an explicit float64 conversion is a rounding point the compiler must146+// honour, which keeps a fused multiply-add from computing a differently-rounded147+// weight on one architecture than on another.148+func catmullRom(t float64) float64 {149+ if t < 0 {150+ t = -t151+ }152+ switch {153+ case t < 1:154+ // 1.5t³ − 2.5t² + 1155+ return float64(float64(float64(float64(1.5*t)-2.5)*float64(t*t)) + 1)156+ case t < 2:157+ // −0.5t³ + 2.5t² − 4t + 2158+ return float64(float64(float64(float64(float64(-0.5*t)+2.5)*t)-4)*t + 2)159+ default:160+ return 0161+ }162+}163+164+// freezeWeights quantizes one window of weights to fixed point summing exactly165+// to one, handing the rounding residual to the heaviest tap.166+func freezeWeights(raw []float64, sum float64) []int32 {167+ frozen := make([]int32, len(raw))168+ total := int32(0)169+ heaviest := 0170+ for i, weight := range raw {171+ frozen[i] = int32(math.RoundToEven(float64(weight/sum) * (1 << resampleShift)))172+ total += frozen[i]173+ if frozen[i] > frozen[heaviest] {174+ heaviest = i175+ }176+ }177+ frozen[heaviest] += 1<<resampleShift - total178+ return frozen179+}180+181+// clampTaps folds taps that fall outside the source onto its first and last182+// samples.183+func clampTaps(taps resampleTaps, src int) resampleTaps {184+ out := resampleTaps{first: taps.first}185+ if out.first < 0 {186+ out.first = 0187+ }188+ last := min(taps.first+len(taps.weights)-1, src-1)189+ if out.first > last {190+ out.first, last = 0, 0191+ }192+ out.weights = make([]int32, last-out.first+1)193+ for i, weight := range taps.weights {194+ at := min(max(taps.first+i, out.first), last)195+ out.weights[at-out.first] += weight196+ }197+ return out198+}199+200+// toNRGBA lands a decoded image in the one layout the resampler reads. The201+// conversion is the standard library's own integer arithmetic, so a JPEG202+// decodes to the same bytes on every machine.203+func toNRGBA(src image.Image) *image.NRGBA {204+ if held, ok := src.(*image.NRGBA); ok {205+ return held206+ }207+ out := image.NewNRGBA(image.Rect(0, 0, src.Bounds().Dx(), src.Bounds().Dy()))208+ draw.Draw(out, out.Bounds(), src, src.Bounds().Min, draw.Src)209+ return out210+}@@ -153,7 +153,12 @@ type Capture struct {
153153 Kind string154154 SourceID int64155155 SourceURL string156- Body []byte156+ // CapturedAt pins the moment a capture answers to, for a source whose157+ // upstream is itself pinned: a fixed publication captured under a recorded158+ // time reproduces the same archive on any machine. Empty takes the clock's159+ // first sight, which is what a live origin gets.160+ CapturedAt string161+ Body []byte157162 }158163 159164 // WriteCapture archives one capture and reports whether anything moved.@@ -178,10 +183,14 @@ func (a *Archive) WriteCapture(worldDir string, c Capture) (hash string, fresh b
178183 return hash, false, nil179184 }180185 }186+ capturedAt := c.CapturedAt187+ if capturedAt == "" {188+ capturedAt = time.Now().UTC().Format(time.RFC3339Nano)189+ }181190 index = append(index, snapshot{182191 // First-seen, and never refreshed. A volume's creation time is derived183192 // from this, so a re-verified capture must not move it.184- CapturedAt: time.Now().UTC().Format(time.RFC3339Nano),193+ CapturedAt: capturedAt,185194 ContentHash: hash,186195 Kind: c.Kind,187196 SourceID: c.SourceID,@@ -0,0 +1,79 @@
1+package doc2+3+import (4+ "math"5+ "strconv"6+)7+8+// The whole-sphere equirectangular window: how a planet's global mosaic sits in9+// the world square.10+//11+// A global equirectangular image is twice as wide as it is tall, so it fills12+// the top half of the square: x runs the full width from longitude -180 to 180,13+// y runs the top half from latitude 90 down to -90. Like the synthetic window14+// above, this is the corpus's own convention rather than any one source's --15+// NASA Trek's mosaics and the Blue Marble base map both flatten this way, and16+// two sources describing one planet this way land on the same pixels, which is17+// what lets their readings meet. The world declares the mapping through the18+// registered conventions (atlas.geometry.equirect.px/.deg), spelled from these19+// same constants, so the raster and the declaration cannot disagree.20+const (21+ // EquirectPx is the raster window the projection fills, as22+ // atlas.geometry.equirect.px spells it: x, y, w, h in world pixels.23+ EquirectPx = "0,0,8192,4096"24+ // EquirectDeg is the ground that window pictures, as25+ // atlas.geometry.equirect.deg spells it: west, north, east, south.26+ EquirectDeg = "-180,90,180,-90"27+)28+29+// EquirectCenter is where a reader opens on a whole-sphere world: the centre of30+// the planet's picture, halfway across the world square and a quarter down it,31+// which is where a 2:1 image's middle sits in a square.32+func EquirectCenter() Position {33+ return SyntheticPosition(SyntheticWorldSize/2, SyntheticWorldSize/4)34+}35+36+// EquirectLevelExtent reports the last tile column and row of one square level37+// of a whole-sphere pyramid: the full width of the world square, and the top38+// half of its height. A crawler captures exactly these tiles and the deriver39+// expects exactly these bounds, because both ask here.40+func EquirectLevelExtent(zoom int) (maxX, maxY int) {41+ if zoom == 0 {42+ return 0, 043+ }44+ return 1<<zoom - 1, 1<<(zoom-1) - 145+}46+47+// EquirectFrame declares a whole-sphere pyramid for the deriver. Every level48+// down to zero is declared: a level left unsaid would be measured against the49+// corpus's shared square, which this world does not sit in, and the half-height50+// windows are what tell frame discovery where the planet actually is. Every51+// mosaic of a body shares that window whatever its depth, which is what lets52+// siblings ride one world as its lenses.53+func EquirectFrame(maxZoom int, format string) *Frame {54+ if format == "" {55+ format = "jpg"56+ }57+ frame := &Frame{58+ MaxZoom: maxZoom,59+ Format: format,60+ Windows: make(map[string]TileWindow, maxZoom+1),61+ }62+ for zoom := 0; zoom <= maxZoom; zoom++ {63+ maxX, maxY := EquirectLevelExtent(zoom)64+ frame.Windows[strconv.Itoa(zoom)] = TileWindow{MaxX: maxX, MaxY: maxY}65+ }66+ return frame67+}68+69+// EquirectWorldPixel lands planetary coordinates on the picture. The window70+// spans longitude -180..180 west to east and latitude 90..-90 top to bottom; a71+// longitude in an east-positive 0..360 spelling wraps into that half-open72+// window first. X crosses the full world square, y only its top half, because73+// that is where a 2:1 image sits in a square.74+func EquirectWorldPixel(longitude, latitude float64) (x, y float64) {75+ wrapped := math.Mod(longitude+180, 360) - 18076+ x = (wrapped + 180) / 360 * SyntheticWorldSize77+ y = (90 - latitude) / 180 * (SyntheticWorldSize / 2)78+ return x, y79+}@@ -0,0 +1,47 @@
1+package nasabluemarble2+3+// The archived capture, as the crawler wrote it. A Blue Marble capture is one4+// pinned publication: NASA Earth Observatory's global, cloud-free, true-color5+// base map, taken whole as a single image and cut into the reference level the6+// deriver folds down. The capture body records the product's identity and the7+// derivation policy the crawler applied, so a reader of the archive can say8+// exactly what these tiles are and how they came to be -- and so a policy9+// change reads as a new capture rather than as the same one wearing new bytes.10+//11+// These are the only declarations in the tree permitted to carry this12+// publisher's vocabulary. Fields are declared in the order they marshal.13+14+type capture struct {15+ Source string `json:"source"`16+ Body string `json:"body"`17+ // Product names the publication as NASA titles it, and Credit is the line18+ // NASA Earth Observatory asks republishers to carry.19+ Product string `json:"product"`20+ Credit string `json:"credit"`21+ // AssetSHA256 is the digest of the source image the tiles were derived22+ // from -- the capture's whole claim to identity. The asset's URL rides the23+ // snapshot index as provenance, never this body.24+ AssetSHA256 string `json:"assetSha256"`25+ // Width and Height are the source image's pixels.26+ Width int `json:"width"`27+ Height int `json:"height"`28+ Map mosaic `json:"map"`29+ Derive derivation `json:"derive"`30+}31+32+// mosaic is the pyramid as captured: the deepest square-world level the source33+// was cut to, and the encoding its tiles wear. LayerTitle names the picture the34+// way a person knows it, which is what a lens picker shows.35+type mosaic struct {36+ MaxZoom int `json:"maxZoom"`37+ Extension string `json:"extension"`38+ LayerTitle string `json:"layerTitle"`39+}40+41+// derivation is the policy the crawler cut the reference level under. It is42+// provenance a person reads and identity the capture hash carries; nothing43+// downstream depends on it.44+type derivation struct {45+ Resampler string `json:"resampler"`46+ JPEGQuality int `json:"jpegQuality"`47+}@@ -0,0 +1,206 @@
1+// Package nasabluemarble reads a captured NASA Blue Marble base map and2+// translates it into the Atlas interchange document.3+//4+// The capture is one pinned publication: Blue Marble Next Generation, NASA5+// Earth Observatory's global, cloud-free, true-color composite of the whole6+// planet. Where the NASA Trek reader marries a mosaic to a gazetteer, this7+// reader carries a picture and nothing else -- the volume it emits holds one8+// world, one lens, no collections and no features. It exists so that Earth can9+// travel as an ordinary raster-only volume: the included build every library10+// starts with, and the ground later readings enrich.11+//12+// # The coordinate design13+//14+// The image is the shared whole-sphere equirectangular window (doc.EquirectPx):15+// a 2:1 global image filling the top half of the world square, -180..180 west16+// to east and 90..-90 top to bottom. The world declares the flattening as17+// registered conventions, so a reader can run any position backward through it18+// and stand on the planet, and the cell systems recognise the ground through19+// the same declarations every spherical world makes.20+//21+// # What is refused22+//23+// A capture of another source's kind or naming another source; a capture of a24+// body that is not the Earth; one declaring no pyramid, no digest, or an image25+// with no size. The reader states its preconditions and refuses what does not26+// meet them rather than guessing.27+package nasabluemarble28+29+import (30+ "encoding/json"31+ "errors"32+ "fmt"33+ "log/slog"34+35+ "github.com/FelineStateMachine/atlas/format/semconv"36+ "github.com/FelineStateMachine/atlas/internal/generate/archive"37+ "github.com/FelineStateMachine/atlas/internal/generate/doc"38+ "github.com/FelineStateMachine/atlas/internal/logging"39+)40+41+// CaptureKind is what the archive calls a Blue Marble capture, and SourceName42+// is what a well-formed capture says it is. Both are exported because the43+// crawler writes them and the reader answers only for them, and the two have to44+// agree.45+const (46+ CaptureKind = "blue-marble-map"47+ SourceName = "nasa-blue-marble"48+)49+50+// Body is the one body this source pictures. The volume and its world both51+// answer to it, which is the point: another source's capture of the same planet52+// registers the same slug and the two builds answer for one library entry.53+const Body = "earth"54+55+// TileSet is the path the base map's tiles are captured under. It is exported56+// because the crawler writes tiles at it and the reader names it, and the two57+// have to agree.58+const TileSet = Body + "/blue-marble"59+60+// radiusKM is the Earth's mean radius as the IAU nominal value spells it,61+// declared on the world so a reader that measures ground has the number the62+// picture was flattened from.63+const radiusKM = "6371.0088"64+65+// Source reads NASA Blue Marble captures.66+type Source struct{}67+68+// New builds the source. It holds no state.69+func New() Source { return Source{} }70+71+// Describe is the source's account of itself.72+func (Source) Describe() doc.Provenance {73+ return doc.Provenance{74+ Name: SourceName,75+ Label: "NASA Earth Observatory",76+ License: "Public domain. NASA imagery carries no copyright; NASA's media guidelines " +77+ "ask that NASA Earth Observatory be credited and that no NASA endorsement be implied.",78+ Attribution: "Blue Marble: Next Generation, a global true-color base map by " +79+ "NASA Earth Observatory, from NASA Goddard Space Flight Center imagery.",80+ // Nothing in the publication numbers a base map, so every identity here81+ // is derived from a stable name.82+ IDSpace: doc.IDSpaceDerived,83+ }84+}85+86+// Translate reads one archived volume.87+func (s Source) Translate(a *archive.Archive, v archive.VolumeRef, log *slog.Logger) (doc.Document, error) {88+ log = log.With(logging.Source(SourceName))89+ worlds, err := a.Worlds(v)90+ if err != nil {91+ return doc.Document{}, err92+ }93+ out := doc.Document{94+ Doc: doc.Doc,95+ Version: doc.Version,96+ Volume: doc.Volume{Slug: Body, Title: doc.Title(Body)},97+ Source: s.Describe(),98+ }99+ for _, ref := range worlds {100+ world, err := s.translateWorld(a, ref, log)101+ if err != nil {102+ if errors.Is(err, archive.ErrNotReady) {103+ log.Debug("world skipped", logging.Path(archive.TrimRoot(a.Root(), ref.Dir())),104+ "reason", err.Error())105+ continue106+ }107+ return doc.Document{}, err108+ }109+ out.Worlds = append(out.Worlds, world)110+ }111+ if len(out.Worlds) == 0 {112+ return doc.Document{}, fmt.Errorf("%w: volume %s has no readable world", archive.ErrNotReady, v.Title)113+ }114+ log.Info("volume translated", logging.Volume(out.Volume.Slug), "worlds", len(out.Worlds))115+ return out, nil116+}117+118+func (s Source) translateWorld(a *archive.Archive, ref archive.WorldRef, log *slog.Logger) (doc.World, error) {119+ archived, err := a.Newest(ref)120+ if err != nil {121+ return doc.World{}, err122+ }123+ if archived.Kind != CaptureKind {124+ return doc.World{}, fmt.Errorf(125+ "capture %s is of kind %q; the Blue Marble reader answers only for %q",126+ archived.ContentHash, archived.Kind, CaptureKind)127+ }128+ body, err := a.Body(ref, archived)129+ if err != nil {130+ return doc.World{}, err131+ }132+ var raw capture133+ if err := json.Unmarshal(body, &raw); err != nil {134+ return doc.World{}, fmt.Errorf("decode capture %s: %w", archived.ContentHash, err)135+ }136+ if err := check(&raw); err != nil {137+ return doc.World{}, fmt.Errorf("capture %s: %w", archived.ContentHash, err)138+ }139+140+ ids := doc.NewIDSpace()141+ worldID, err := ids.Claim("bluemarble:map:" + raw.Body)142+ if err != nil {143+ return doc.World{}, err144+ }145+146+ world := doc.World{147+ ID: worldID,148+ Slug: raw.Body,149+ Title: doc.Title(raw.Body),150+ // A reader opens on the middle of the planet's picture.151+ Center: doc.EquirectCenter(),152+ Capture: doc.Capture{153+ Kind: archived.Kind,154+ ID: archived.SourceID,155+ Locator: archived.SourceURL,156+ ContentHash: archived.ContentHash,157+ CapturedAt: archived.CapturedAt,158+ },159+ // The world says what it pictures: the Earth, flattened by the160+ // equirectangular projection into the top half of the world square.161+ // The declarations are the shared whole-sphere ones, plus the body and162+ // its radius, so the cell systems and any measuring reader stand on163+ // the same ground the raster pictures.164+ Attrs: map[string]string{165+ semconv.KeyGeometrySurface: semconv.SurfaceSphere,166+ semconv.KeyGeometryProjection: semconv.ProjectionEquirect,167+ semconv.KeyGeometryEquirectPx: doc.EquirectPx,168+ semconv.KeyGeometryEquirectDeg: doc.EquirectDeg,169+ semconv.KeyGeometryBody: raw.Body,170+ semconv.KeyGeometryRadiusKM: radiusKM,171+ },172+ Lenses: []doc.Lens{{173+ Name: named(raw.Map.LayerTitle, "blue-marble"),174+ TileSet: TileSet,175+ Frame: doc.EquirectFrame(raw.Map.MaxZoom, raw.Map.Extension),176+ }},177+ // No collections: the volume is its picture. Features are later178+ // sources' work, folded in by the enrich lane.179+ }180+ log.Debug("world translated", logging.World(world.Slug), "capture", archived.ContentHash)181+ return world, nil182+}183+184+// check states what a capture has to be before anything is read out of it.185+func check(raw *capture) error {186+ switch {187+ case raw.Source != SourceName:188+ return fmt.Errorf("capture says its source is %q, not %q", raw.Source, SourceName)189+ case raw.Body != Body:190+ return fmt.Errorf("capture pictures %q; the Blue Marble reader answers only for %q", raw.Body, Body)191+ case raw.AssetSHA256 == "":192+ return errors.New("capture names no source digest")193+ case raw.Width <= 0 || raw.Height <= 0:194+ return errors.New("capture names no source image size")195+ case raw.Map.MaxZoom < 1:196+ return errors.New("capture declares no pyramid")197+ }198+ return nil199+}200+201+func named(given, slug string) string {202+ if given != "" {203+ return given204+ }205+ return doc.Title(slug)206+}@@ -0,0 +1,178 @@
1+package nasabluemarble2+3+import (4+ "encoding/json"5+ "log/slog"6+ "os"7+ "path/filepath"8+ "strconv"9+ "strings"10+ "testing"11+12+ "github.com/FelineStateMachine/atlas/format/semconv"13+ "github.com/FelineStateMachine/atlas/internal/generate/archive"14+ "github.com/FelineStateMachine/atlas/internal/generate/doc"15+)16+17+func wellFormed() capture {18+ return capture{19+ Source: SourceName,20+ Body: Body,21+ Product: "Blue Marble: Next Generation, July 2004",22+ Credit: "NASA Earth Observatory",23+ AssetSHA256: strings.Repeat("d", 64),24+ Width: 21600,25+ Height: 10800,26+ Map: mosaic{MaxZoom: 5, Extension: "jpg", LayerTitle: "Blue Marble"},27+ Derive: derivation{Resampler: "catmull-rom-fixed15", JPEGQuality: 90},28+ }29+}30+31+// writeArchive lays out the smallest archive holding one volume of one world,32+// carrying the given capture.33+func writeArchive(t *testing.T, kind string, raw capture) *archive.Archive {34+ t.Helper()35+ root := t.TempDir()36+ body, err := json.Marshal(raw)37+ if err != nil {38+ t.Fatal(err)39+ }40+ worldDir := filepath.Join(root, "games", "g-1", "maps", "world-1")41+ mustWrite(t, filepath.Join(root, "archive.json"),42+ `{"games":[{"directory":"games/g-1","id":1,"title":"Earth","source":"nasa-blue-marble"}]}`)43+ mustWrite(t, filepath.Join(root, "games", "g-1", "game.json"),44+ `{"id":1,"title":"Earth","maps":[{"directory":"games/g-1/maps/world-1","id":1,"slug":"earth","title":"Earth"}]}`)45+ mustWrite(t, filepath.Join(worldDir, "snapshots", "index.json"),46+ `[{"capturedAt":"2026-08-03T14:30:39Z","contentHash":"h","kind":"`+kind+`","sourceId":1,"sourceUrl":"/x"}]`)47+ mustWrite(t, filepath.Join(worldDir, "snapshots", "map", "h.json"), string(body))48+ store, err := archive.Open(root)49+ if err != nil {50+ t.Fatal(err)51+ }52+ return store53+}54+55+func mustWrite(t *testing.T, path, body string) {56+ t.Helper()57+ if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {58+ t.Fatal(err)59+ }60+ if err := os.WriteFile(path, []byte(body), 0o644); err != nil {61+ t.Fatal(err)62+ }63+}64+65+// TestTranslateReadsACaptureWhole walks one hand-written capture into the66+// interchange document: a raster-only Earth -- one world, one lens, the shared67+// whole-sphere declarations, and nothing else.68+func TestTranslateReadsACaptureWhole(t *testing.T) {69+ store := writeArchive(t, CaptureKind, wellFormed())70+ document, err := New().Translate(store, store.Volumes()[0], slog.New(slog.DiscardHandler))71+ if err != nil {72+ t.Fatal(err)73+ }74+ if document.Volume.Slug != "earth" || document.Volume.Title != "Earth" {75+ t.Errorf("volume %s/%s; the planet is the volume", document.Volume.Slug, document.Volume.Title)76+ }77+ if document.Source.Name != SourceName || document.Source.Label != "NASA Earth Observatory" {78+ t.Errorf("provenance %s/%s; the ledger and the credit both read from here",79+ document.Source.Name, document.Source.Label)80+ }81+ if document.Source.IDSpace != doc.IDSpaceDerived {82+ t.Errorf("id space %q; a base map numbers nothing", document.Source.IDSpace)83+ }84+ if len(document.Icons) != 0 {85+ t.Errorf("the document ships %d pieces of artwork; a base map has none", len(document.Icons))86+ }87+ if len(document.Worlds) != 1 {88+ t.Fatalf("%d worlds, want the one Earth", len(document.Worlds))89+ }90+ world := document.Worlds[0]91+ if world.Slug != "earth" || world.Title != "Earth" {92+ t.Errorf("world %s/%s", world.Slug, world.Title)93+ }94+ if world.Center != doc.EquirectCenter() {95+ t.Errorf("the world opens at %v, not the picture's centre", world.Center)96+ }97+ if world.Capture.Kind != CaptureKind || world.Capture.CapturedAt != "2026-08-03T14:30:39Z" {98+ t.Errorf("capture provenance did not travel: %+v", world.Capture)99+ }100+101+ want := map[string]string{102+ semconv.KeyGeometrySurface: semconv.SurfaceSphere,103+ semconv.KeyGeometryProjection: semconv.ProjectionEquirect,104+ semconv.KeyGeometryEquirectPx: doc.EquirectPx,105+ semconv.KeyGeometryEquirectDeg: doc.EquirectDeg,106+ semconv.KeyGeometryBody: "earth",107+ semconv.KeyGeometryRadiusKM: "6371.0088",108+ }109+ if len(world.Attrs) != len(want) {110+ t.Errorf("the world carries %d attributes %v, want %d", len(world.Attrs), world.Attrs, len(want))111+ }112+ for key, value := range want {113+ if world.Attrs[key] != value {114+ t.Errorf("world says %s=%q, want %q", key, world.Attrs[key], value)115+ }116+ }117+118+ if len(world.Lenses) != 1 {119+ t.Fatalf("%d lenses, want the one base map", len(world.Lenses))120+ }121+ lens := world.Lenses[0]122+ if lens.Name != "Blue Marble" || lens.TileSet != TileSet {123+ t.Errorf("lens %s/%s", lens.Name, lens.TileSet)124+ }125+ if lens.Frame == nil || lens.Frame.MaxZoom != 5 || lens.Frame.Format != "jpg" {126+ t.Fatalf("frame %+v", lens.Frame)127+ }128+ for zoom := 0; zoom <= 5; zoom++ {129+ maxX, maxY := doc.EquirectLevelExtent(zoom)130+ window, held := lens.Frame.Windows[strconv.Itoa(zoom)]131+ if !held || window.MaxX != maxX || window.MaxY != maxY {132+ t.Errorf("zoom %d window %+v, want the whole-sphere extent %d,%d", zoom, window, maxX, maxY)133+ }134+ }135+136+ if len(world.Collections) != 0 {137+ t.Errorf("the world carries %d collections; a raster-only volume carries none", len(world.Collections))138+ }139+140+ // The same archived bytes give the same document: identities included.141+ again, err := New().Translate(store, store.Volumes()[0], slog.New(slog.DiscardHandler))142+ if err != nil {143+ t.Fatal(err)144+ }145+ if again.Worlds[0].ID != world.ID {146+ t.Error("two translations of one capture minted different identities")147+ }148+}149+150+// TestTranslateRefusesWhatItCannotStandFor states the reader's preconditions:151+// another source's bytes, another body, a capture with no pyramid, no digest or152+// no size are refusals, not guesses.153+func TestTranslateRefusesWhatItCannotStandFor(t *testing.T) {154+ tests := []struct {155+ name string156+ kind string157+ spoil func(*capture)158+ says string159+ }{160+ {"another source's kind", "trek-map", func(*capture) {}, "answers only for"},161+ {"another source's name", CaptureKind, func(c *capture) { c.Source = "nasa-trek" }, "says its source is"},162+ {"another body", CaptureKind, func(c *capture) { c.Body = "mars" }, "answers only for"},163+ {"no pyramid", CaptureKind, func(c *capture) { c.Map.MaxZoom = 0 }, "declares no pyramid"},164+ {"no digest", CaptureKind, func(c *capture) { c.AssetSHA256 = "" }, "names no source digest"},165+ {"no size", CaptureKind, func(c *capture) { c.Width = 0 }, "names no source image size"},166+ }167+ for _, tt := range tests {168+ t.Run(tt.name, func(t *testing.T) {169+ raw := wellFormed()170+ tt.spoil(&raw)171+ store := writeArchive(t, tt.kind, raw)172+ _, err := New().Translate(store, store.Volumes()[0], slog.New(slog.DiscardHandler))173+ if err == nil || !strings.Contains(err.Error(), tt.says) {174+ t.Fatalf("the capture was read anyway: %v", err)175+ }176+ })177+ }178+}@@ -60,13 +60,10 @@ const (
6060 source = "nasa-trek"6161 )6262 63-// The declared flattening, spelled once for the world's attributes and once for64-// the projection below: the mosaic fills the top half of the world square and65-// pictures the whole ground, -180..180 west to east and 90..-90 top to bottom.66-const (67- equirectPx = "0,0,8192,4096"68- equirectDeg = "-180,90,180,-90"69-)63+// The declared flattening is the corpus's shared whole-sphere window64+// (doc.EquirectPx and friends): the mosaic fills the top half of the world65+// square and pictures the whole ground, -180..180 west to east and 90..-90 top66+// to bottom.7067 7168 // Source reads NASA Trek captures.7269 type Source struct{}@@ -163,10 +160,8 @@ func (s Source) translateWorld(a *archive.Archive, ref archive.WorldRef, log *sl
163160 ID: worldID,164161 Slug: raw.MapSlug,165162 Title: named(raw.MapTitle, raw.MapSlug),166- // A reader opens on the middle of the planet's picture: halfway across167- // the world square, a quarter down it, which is where a 2:1 image's168- // centre sits in a square.169- Center: doc.SyntheticPosition(doc.SyntheticWorldSize/2, doc.SyntheticWorldSize/4),163+ // A reader opens on the middle of the planet's picture.164+ Center: doc.EquirectCenter(),170165 Capture: doc.Capture{171166 Kind: archived.Kind,172167 ID: archived.SourceID,@@ -182,8 +177,8 @@ func (s Source) translateWorld(a *archive.Archive, ref archive.WorldRef, log *sl
182177 Attrs: map[string]string{183178 semconv.KeyGeometrySurface: semconv.SurfaceSphere,184179 semconv.KeyGeometryProjection: semconv.ProjectionEquirect,185- semconv.KeyGeometryEquirectPx: equirectPx,186- semconv.KeyGeometryEquirectDeg: equirectDeg,180+ semconv.KeyGeometryEquirectPx: doc.EquirectPx,181+ semconv.KeyGeometryEquirectDeg: doc.EquirectDeg,187182 semconv.KeyGeometryBody: raw.Body,188183 },189184 }@@ -245,13 +240,9 @@ func TileSetPath(body, layer string) string { return body + "/EQ/" + layer }
245240 // LevelExtent reports the last tile column and row of one square level: the full246241 // width of the world square, and the top half of its height. The crawler asks247242 // for exactly these tiles and the deriver expects exactly these bounds, because248-// it is the same call.249-func LevelExtent(zoom int) (maxX, maxY int) {250- if zoom == 0 {251- return 0, 0252- }253- return 1<<zoom - 1, 1<<(zoom-1) - 1254-}243+// it is the same call -- the shared whole-sphere one every equirectangular244+// source measures with.245+func LevelExtent(zoom int) (maxX, maxY int) { return doc.EquirectLevelExtent(zoom) }255246 256247 // collectionsOf arranges the Gazetteer's flat feature list the way a legend257248 // reads: one collection per feature type, all under one heading. The type's@@ -338,38 +329,17 @@ func checkFeature(entry feature) error {
338329 return nil339330 }340331 341-// worldPixel lands a feature's planetary coordinates on the picture. The mosaic342-// spans longitude -180..180 west to east and latitude 90..-90 top to bottom; the343-// Gazetteer speaks east-positive 0..360, so a longitude wraps into the mosaic's344-// half-open window first. X crosses the full world square, y only its top half,345-// because that is where a 2:1 image sits in a square.332+// worldPixel lands a feature's planetary coordinates on the picture, through333+// the shared whole-sphere window. The Gazetteer speaks east-positive 0..360,334+// which the shared mapping wraps into the mosaic's half-open window first.346335 func worldPixel(longitude, latitude float64) (x, y float64) {347- wrapped := math.Mod(longitude+180, 360) - 180348- x = (wrapped + 180) / 360 * doc.SyntheticWorldSize349- y = (90 - latitude) / 180 * (doc.SyntheticWorldSize / 2)350- return x, y336+ return doc.EquirectWorldPixel(longitude, latitude)351337 }352338 353-// frameOf declares a mosaic's pyramid for the deriver. Every level down to zero354-// is declared: a level left unsaid would be measured against the corpus's shared355-// square, which this world does not sit in, and the half-height windows are what356-// tell frame discovery where the planet actually is. Every mosaic of a body357-// shares that window whatever its depth, which is what lets siblings ride one358-// world as its lenses.339+// frameOf declares a mosaic's pyramid for the deriver: the shared whole-sphere340+// frame, every level down to zero.359341 func frameOf(maxZoom int, format string) *doc.Frame {360- if format == "" {361- format = "jpg"362- }363- frame := &doc.Frame{364- MaxZoom: maxZoom,365- Format: format,366- Windows: make(map[string]doc.TileWindow, maxZoom+1),367- }368- for zoom := 0; zoom <= maxZoom; zoom++ {369- maxX, maxY := LevelExtent(zoom)370- frame.Windows[strconv.Itoa(zoom)] = doc.TileWindow{MaxX: maxX, MaxY: maxY}371- }372- return frame342+ return doc.EquirectFrame(maxZoom, format)373343 }374344 375345 func named(given, slug string) string {@@ -216,8 +216,8 @@ func TestMarsWorldSpeaksTheDeclaredFlattening(t *testing.T) {
216216 want := map[string]string{217217 semconv.KeyGeometrySurface: semconv.SurfaceSphere,218218 semconv.KeyGeometryProjection: semconv.ProjectionEquirect,219- semconv.KeyGeometryEquirectPx: equirectPx,220- semconv.KeyGeometryEquirectDeg: equirectDeg,219+ semconv.KeyGeometryEquirectPx: doc.EquirectPx,220+ semconv.KeyGeometryEquirectDeg: doc.EquirectDeg,221221 semconv.KeyGeometryBody: mars.Game.Slug,222222 }223223 if len(mars.Attrs) != len(want) {@@ -4,6 +4,7 @@ import (
44 "github.com/FelineStateMachine/atlas/internal/generate/sources/arcgishub"55 "github.com/FelineStateMachine/atlas/internal/generate/sources/ign"66 "github.com/FelineStateMachine/atlas/internal/generate/sources/mapgenie"7+ "github.com/FelineStateMachine/atlas/internal/generate/sources/nasabluemarble"78 "github.com/FelineStateMachine/atlas/internal/generate/sources/nasatrek"89 "github.com/FelineStateMachine/atlas/internal/generate/sources/piggyback"910 )@@ -24,6 +25,7 @@ func All() []Source {
2425 ign.New(),2526 piggyback.New(),2627 nasatrek.New(),28+ nasabluemarble.New(),2729 arcgishub.New(),2830 }2931 }