Planet JEPA

A neural codec for satellite history: predict the next embedding, decode shape and texture

72 seconds, captioned. Everything it shows is live on this page below.

Video codecs store keyframes and predict the frames between. We do the same for the Earth's surface — except the frames are objects, not pixels. Every field, pond, and burn scar is a token whose latent is a curve basis (30 Fourier coefficients that decode back to its exact boundary) plus its normalized multispectral state and a continuous-time clock. The model never predicts pixels: it predicts the next latent, JEPA-style, and the decoder recovers boundary and material at any timestamp, any speed, any output resolution. Scrub it:

THE CODEC, DECODED AT t
NEAREST REAL PASS

Ticks on the timeline are keyframes — real satellite passes, stored exactly. Everything between them is the model's imputation, stamped INFERRED. Boundary from the Fourier coefficients; interior color, stipple, and ripples from the imputed spectral channels — the texture is decoded physics, not decoration. The tokens come from the archive-fine-tuned segmenter now in production (105 tracks shown here, full-scene); this two-site, three-year film is under a megabyte.

2The codec, in one row

shape30 EFD coefficients
exact decode, any res
sizelog-area,
normalized
spectra7 physics fills
(r g b, NDVI, wetness…)
clocksin/cos at 3–48 mo
irregular time, native
→
transformerkeyframe + residual
output

Two design choices do the work. Missing data is a symbol, not a zero: a learned CLOUD token replaces the latent where a pass flew but saw nothing, and a QUERY token requests any timestamp in between — so irregular sampling and cloud gaps are the training task, not a preprocessing nuisance. And the output is keyframe-residual: the model predicts the change from the nearest observed latent, so visible structure anchors every decode, exactly like motion-compensated video.

Pretraining: ~34k two-window field tracks (FTW) teach the two priors real archives are too sparse to teach — boundaries copy, spectra follow the season. Fine-tune on real tracked histories.

3Prediction is the same operation as playback

Ask for a timestamp after the last keyframe and the QUERY token becomes a forecast. Below, every tracked Kansas field painted by NDVI: observed is the sensor, predicted is the model's rolling expectation from history alone (208 predictions, MAE vs 0.201 persistence), error is the disagreement. The last slider stop is a true forecast of a pass that hadn't happened yet.

Every prediction is kept beside what actually happened (INFERRED next to EXACT), so forecasts are auditable the day the next pass arrives. Numbers on cloud-gap imputation: spectral fill error 1.6–2.5× better than persistence (0.04–0.06 vs 0.093 MAE); shape ties it — boundaries barely move in three years, which is the point of the next section.

4Forty years, zero-shot

We built the codec a world to live in: every usable Landsat pass since 1985 over 32 places in 12 domains, plus monthly passes inside seven documented fires and floods — 1,290 scenes, tracked into multi-decade object histories with fused consensus boundaries (open dataset cp500/hyperkd-rs, with every script in harness/). Then the pilot codec, unchanged, ran over tracks it never saw: hide every third year behind a CLOUD token, reconstruct it. Ground truth above, decode below; observed re-rendered from their own coefficients, imputed predicted from the rest.

Honest scoreboard: on the 41-year farmland track the codec beats persistence, 0.93 vs 0.86 shape IoU, and ties the Aral Sea's slow shrink — the regimes its training taught. On event dynamics it has never seen (fire, flood, mining growth) persistence still wins. That is not a footnote, it is the roadmap: the corpus holds exactly those events, and the retrain is queued.

5What the corpus already shipped

The same consensus targets fine-tuned the segmenter itself: SAM 2.1's decoder went from 0.365 to 0.565 held-out IoU, re-quantized to 4.8 MB int8, and lifted the live tokenizer from 28/34 to 31/34 labelled fields — it is what runs when you press tokenize in the instrument (checkpoint). And because the codec's latent carries shape and material, naming is a lookup: a two-layer head on 22 deterministic token features hits 70% top-1 over 24 concepts (38.6% majority baseline), so a caption is assembled from EXACT facts — size, dates, stability, containment — with only the concept name INFERRED, score attached. Labels never touch geometry.