Two kinds of array question

There are two different things you can ask of a 3-d array with three index vectors. The orthogonal question asks for every combination: a hyperslab, the outer product of the indices. The pointwise question pairs the indices up: the first i with the first j with the first k, giving one value per point. A chunked store cares a lot about which one you mean, because they touch completely different sets of chunks.

R and numpy chose opposite defaults:

Zarr grew two named accessors, oindex and vindex, to say what S and R have said in base syntax for decades. xarray meets the same split as orthogonal versus vectorized isel. I checked the table against a real store: on the Zarr array used later in this post, z.vindex[[29, 14], [49, 24], [69, 34]] and R’s x[cbind(c(70, 35), c(50, 25), c(30, 15))] both return 104999 50714, and z.oindex returns the same 8 values as R’s x[i, j, k]. (The R indices are reversed and 1-based; more on why below.)

Matrix indexing is one of the least appreciated parts of R. It long predates lazy evaluation in most of the places people now care about it, and as it turns out, it is exactly the form that reaches a lazy array as a single, plannable request.

What R does with it

ALTREP (since R 3.5) lets a package supply a vector whose contents come from somewhere other than a block of memory. The package registers methods: Length, Elt for one element, Extract_subset for a set of indices, Dataptr for a pointer to all the data. R’s own code calls them instead of reading memory. To R, the result is just a vector.

So what happens when that vector has a dim attribute and you index it? Reading src/main/subset.c (R 4.3 and current trunk have the same structure), there are three paths:

Matrix indexing is converted to linear indices before it reaches ALTREP, but the whole set arrives in one call. That is enough: the object can recover each index’s chunk, find the unique chunks, and fetch them together.

The rectangular form is the surprise. ArraySubset normalizes each subscript with int_arraySubscript() and holds the result in a local array, subs[k]: one integer vector per dimension. That is the hyperslab, exactly. But it never offers subs to ALTREP. It loops over the result in column-major order and calls REAL_ELT() once per element.

altarr: the experiment

altarr is a small R package with a C core. altarr(dim, chunk, fetch) returns a lazy array: a double vector, a dim attribute, no class. Values arrive chunk by chunk from a fetch function you supply, so the store behind it can be object storage, Zarr, GDAL, or a generator in a test.

The whole contract with the store is one function:

fetch(chunks)
# chunks:  integer matrix, one row per chunk, one column per dimension,
#          0-based chunk coordinates
# returns: list of numeric vectors, one per row, column-major,
#          clipped at the array edge

A call to fetch is one round trip, however many chunks it asks for. The ALTREP methods do the rest:

• Elt finds the element’s chunk, fetches it alone if it is not cached, and returns one value.
• Extract_subset maps every index to its chunk, fetches all the uncached chunks in one call, then assembles.
• Dataptr materializes the whole array, and refuses above getOption("altarr.max_materialize", 1e6) values.

Here is a lazy 1440 x 720 x 365 array (378 million values, never allocated) from a generator with 20 ms of latency per call:

library(altarr)
d  <- c(1440L, 720L, 365L)
cs <- c(60L, 60L, 10L)
remote <- function(chunks) {
  Sys.sleep(0.02)                     # one round trip
  lapply(seq_len(nrow(chunks)), function(r) {
    st <- chunks[r, ] * cs + 1L
    en <- pmin(st + cs - 1L, d)
    as.vector(outer(outer(st[1]:en[1], 1e4 * (st[2]:en[2]), "+"),
                    1e8 * (st[3]:en[3]), "+"))
  })
}
x <- altarr(d, cs, remote)
class(x)           # "array"
is.object(x)       # FALSE
altarr_stats(x)    # counts Elt calls, fetch calls, chunks

altarr_stats() counts what R actually asked for, which is how the numbers below were measured.

Measured

On the 378-million-value array, the planned paths read the same chunks as the element-at-a-time paths in a single round trip, and are 20 to 40 times faster.

The latency stands in for an object-store GET, with a batch of chunks assumed to be fetched concurrently. A linear slice, x[1:1e5], also goes through Extract_subset: 48 chunks in 1 call. Point extraction through x[cbind(...)] is already first-class in unpatched R; the rectangular form is correct and lazy, but serial.

A real Zarr store

A generator proves the mechanics; a real store proves the contract is enough. zarr-python wrote a Zarr v2 array of shape (30, 50, 70), chunks (7, 16, 20), zlib-compressed, C order. The chunk shape is ragged at every edge, and one chunk is entirely fill value, so the store simply omits its file.

altarr_zarr_v2() reads .zarray and implements fetch in about 85 lines of plain R: build the chunk key, read the file (or return fill_value if it is missing), memDecompress(), readBin(), and clip Zarr’s padded edge chunks.

z <- altarr_zarr_v2("zdemo.zarr")
dim(z)                                        # 70 50 30
z[cbind(c(70, 35), c(50, 25), c(30, 15))]     # 104999 50714, one fetch call
altarr_extract(z, , , )                       # every value, one fetch call

Every value matches zarr-python’s array exactly, including the omitted chunk and the clipped edges.

The dimensions come out reversed, and that is deliberate. A C-order array of shape (s1, …, sn) has bytes that are already column-major for shape (sn, …, s1). So altarr gives R dim c(70, 50, 30) and decodes each chunk straight into place, with no transposition at all. R’s x[i, j, k] on that array is zarr’s z.oindex[k, j, i], shifted by one.

A real dataset, with GDAL doing the hard parts

The zarr-python store keeps the package self-contained. For real data, the classic xarray tutorial air_temperature dataset is published as Zarr over HTTP, compressed with Blosc. A Blosc decoder and an HTTP range-reader are exactly the things altarr should not grow, so GDAL does both in one command:

gdal mdim convert \
  "ZARR:\"/vsicurl/https://scottyhq.github.io/zarrdata/air_temperature.zarr/\"" \
  air_temperature.zarr --co FORMAT=ZARR_V2 --co ARRAY:COMPRESS=ZLIB

The result is a local, zlib-compressed Zarr v2 store, and the same reader opens it:

x <- altarr_zarr_v2("air_temperature.zarr/air")
dim(x)                 # 53 x 25 x 2920: lon, lat, time
                       # int16, chunks 27 x 13 x 730, 16 of them

x[20, 10, ]            # a time series at one cell, in kelvin
                       # orthogonal: Elt path, 4 chunks in 4 round trips
altarr_extract(x, 20, 10, )      # the same 4 chunks in 1 round trip

x[cbind(lon_i, lat_i, time_i)]   # a track, one time per position
                                 # pointwise: one planned batch

Those are the two questions from the start of this post, on real data. The time series sits under one cell, so it touches the 4 time chunks there: 4 round trips today, 1 with the hook. A 500-point track spread over the year touches all 16 chunks, and already arrives as a single request.

air is stored as int16 with a scale factor of 0.01. altarr_zarr_v2() applies the CF packing attributes from .zattrs (scale_factor, add_offset, _FillValue) inside fetch, so values arrive in kelvin and fill values become NA. Unpacking is a concern of the fetch, not of the ALTREP layer.

This is also the point to resist building a Zarr engine. Remote I/O, Blosc, sharding, Zarr v3 codecs and their combinations are a never-ending road, and every step along it is a detour from the question here. The fetch contract exists so the engine can be someone else’s: GDAL as a converter in this example, and the natural next step is a fetch that calls GDAL’s multidimensional API, or object_store and the Rust codec crates, directly. The ALTREP layer only needs chunk coordinates in and values out.

What comes for free

Because the object is an ordinary vector to R, a lot of behaviour arrives without any code:

• Shape is free. dim(), length() and str() read nothing.
• Copy-on-modify works. y <- x; y[1, 1, 1] <- 0 materializes y only; x stays lazy and unchanged.
• R’s own wrapper keeps the planning. Changing an attribute on a shared object (dimnames(y) <- ... after y <- x) makes R wrap it in its internal wrapper ALTREP class. The wrapper forwards Extract_subset, so matrix indexing still arrives as one planned request.
• Serialization stores the recipe, not the payload. ALTREP has a Serialized_state method, so saveRDS() of the 378-million-value array writes 1250 bytes: dim, chunk shape and fetch function. For the Zarr array it is 3.7 KB, and readRDS() in a fresh R session loads the package and returns a lazy array that reads from the store on first use.
• The plan is a table. altarr_plan(x, i, j, k) returns the chunks a subscript would touch, one row per chunk, with 0-based chunk coordinates, starts and extents. The plan is a table, and the array is what assembling it gives you.

That last point connects to an older thread. A chunk manifest (Icechunk’s, kerchunk’s, a COG’s tile index) is already a table of chunk keys to byte ranges. Chunk extents are min/max statistics for free, so filtering that table on coordinates is chunk pruning without anyone computing statistics.

The hook R could add

ExtractSubset already does this for vectors: if the object is ALTREP, it offers the indices to the class’s Extract_subset method first, and falls back to the element loop only if the method returns NULL. The same pattern would fit in ArraySubset and MatrixSubset, just after the subscripts are normalized and bounds-checked:

if (ALTREP(x)) {
    SEXP ans = ALTVEC_EXTRACT_ARRAY_SUBSET(x, s, call);  /* s: per-dim subs */
    if (ans != NULL) {
        /* attach dim and dimnames exactly as the existing code does */
    }
}
/* otherwise the element loop runs, as today */

The new method receives one normalized integer vector per dimension. Returning NULL keeps today’s behaviour, so nothing changes for any class that does not implement it.

altarr_extract(x, i, j, k) emulates this from package code: it normalizes each subscript with R’s own rules (by indexing a seq_len() of each dimension), plans the cartesian product of touched chunks, fetches once, and assembles. Its results are identical() to x[i, j, k] across negative, logical, character, NA, missing and zero-length subscripts, with and without drop.

The argument for the hook is that it is general. It adds no data model and no new object type, and every lazy array in R benefits: arrow, gdalraster, Zarr readers, anything ALTREP-backed with a shape.

Prior art, and what is different

Out-of-memory arrays in R are not new, and some of the older work was remarkably capable:

• ff gave R memory-mapped, chunked vectors and arrays long before most languages had anything comparable, with its own [ methods doing the paging. (I used ff heavily in my own thesis, chapter 6 to map 4D ocean model to diving elephant seal movement estimates).
• bigmemory offered shared and file-backed matrices for parallel work.
• DelayedArray and HDF5Array in Bioconductor provide lazy, chunk-aware arrays with block processing and delayed operations, and are the most complete answer to lazy compute on arrays in R.
• arrow and vroom use ALTREP to hand R vectors whose contents are not yet in R memory.

The difference here is the class. Every one of the array packages above is an S3 or S4 object with methods: code has to know about the class, or the object gets converted. An ALTREP array with a dim and no class needs no method dispatch at all. Any function that indexes a numeric array, written in 1998 or yesterday, reads it lazily, and matrix indexing reads it in one planned batch.

That is also the limit. Without a class there is nowhere to hang delayed arithmetic, which is where DelayedArray’s design earns its complexity. The two approaches fit together: lazy reads through base indexing underneath, lazy compute in a plan above.

Limits, and what next

This is a prototype, and the limits are real:

• Lazy reads, not lazy compute. Anything that needs the data pointer (arithmetic, printing, most of apply) materializes the whole array. altarr refuses that above a size option rather than fetching 3 GB because someone typed x + 1.
• The cache is unbounded. It holds one slot per chunk in the grid; it needs an LRU and a byte budget.
• Fetch runs on R’s main thread. Concurrency belongs inside the fetch implementation (object_store in Rust, GDAL), never touching the R API off-thread. The fetch contract is where a Rust backend plugs in, without touching the ALTREP layer.
• Coordinates are still dimnames, which must be character. ALTREP does not touch that; numeric coordinates are a separate question for base R.

Next steps:

1. Build R-devel with the ArraySubset hook and show x[i, j, k] going from 192 round trips to 1, with no package-side method.
2. A Rust fetch backend over object storage, so the demo reads a remote Zarr store rather than a local one.
3. Ask the people who know ALTREP and R’s subscript code best whether the hook is sound.

The code, tests and the measurement script are in the altarr package . The tests check every path against base R with identical().

Links

• R’s subsetting code: src/main/subset.c (GitHub mirror of the R sources)
• numpy indexing, including np.ix_: https://numpy.org/doc/stable/user/basics.indexing.html
• zarr-python: https://zarr.readthedocs.io/
• The air_temperature Zarr store: https://scottyhq.github.io/zarrdata/air_temperature.zarr/
• ff: https://cran.r-project.org/package=ff
• DelayedArray: https://bioconductor.org/packages/DelayedArray/

What Arrow actually changes

Arrow does not share memory between R and the page; it ships bytes the GPU can use without parsing them. R and the browser are separate processes, so data always crosses a boundary. What matters is how much work happens on the far side.

The classic mapview path was htmlwidgets. sf became GeoJSON text, the text was embedded in the page, and the browser parsed it into JS objects and then into Leaflet layers. Most of the cost was in that parse and in building millions of small objects. leafgl (WebGL drawing) and flatgeobuf attachments (binary, streamable) already moved away from it. GeoArrow is where that path was heading.

With GeoArrow, an Arrow IPC buffer becomes JS typed arrays with no parse. deck.gl-geoarrow binds those coordinate buffers straight into GPU attributes. The format is still serialized, but the cost is close to a memory copy, not a parse-and-build.

Two consequences follow:

• True shared memory exists in webR. When R runs as wasm in the same page, Arrow buffers can be handed across without a copy. Designing around Arrow now keeps that door open.
• The browser can fetch the data itself. deckglgeoarrow can already render remote GeoParquet and GeoArrow without reading it into R; only the styling is prepared in R (CRAN reference). R sends a recipe, the browser pulls the bytes.

One constraint shapes the core: only native GeoArrow geometry encoding renders, not WKB (README). The data plane must emit native coordinate buffers. A wk handler writing geoarrow does that, and so does GDAL’s OGR Arrow stream when asked for GeoArrow encoding, which keeps a gdalraster-to-browser path free of conversions.

Mercator is a basemap problem

The GPU does not care about Web Mercator; the tiled basemap does. deck.gl has orthographic and cartesian views, so projected metres can go straight in and render in any CRS. deckglgeoarrow is Mercator-bound only because it sits on mapgl and maplibre.

There is one numeric trap. GPUs work in float32, and projected coordinates in the millions of metres lose precision. The fix is to subtract a local origin before the data leaves R, and draw relative to it. deck.gl has a coordinate-origin mechanism for exactly this.

The hard parts of “any CRS in the browser” are elsewhere:

1. Basemaps. XYZ tiles are Mercator. Showing them in a polar or equal-area view means warping each tile on a reprojected mesh.
2. Rasters into a non-Mercator view. deck.gl-raster already reprojects COG and Zarr on the GPU with an adaptive mesh, from most source projections into the web map, and has prototype globe support. What remains is drawing into a polar or equal-area view CRS, and the polar and antimeridian cases its docs say are not yet tested.

So the gap is narrower than it looks. Source reprojection is solved upstream. The view CRS and polar robustness are the open work, and that is where Antarctic use cases can contribute rather than rebuild.

The design: a protocol, not a widget

The core package should describe a scene and move data; drawing it is someone else’s job. It has three parts:

1. Data plane. Vectors travel as nanoarrow streams with native GeoArrow geometry. Rasters are not Arrow: they travel as a grid descriptor (extent, dimension, CRS) plus a typed buffer, or simply as a URL the browser reads itself. Styling is data too, so colours computed in R become an RGBA column.
2. Scene spec. A small declarative JSON document: the view CRS, the view type (projected, orthographic, globe), layers that reference data by id, and legends. Being plain data, it is testable and can have more than one renderer.
3. Transport. Pluggable: embedded in a standalone page, served from a local httpuv server for large or streaming data, updated over a websocket (with selections coming back to R as row ids), or handed over in-process under webR.

[Diagram placeholder: architecture - producers, core, transport, renderers]

The scene spec is the seam. Producers only need to emit standard inputs, and renderers only need to read the spec, so either side can change without touching the other.

Two packages: a lean core and a friendly front door

The core should accept standard inputs rather than import spatial classes; the convenience package is where the opinions live.

The core depends on nanoarrow, geoarrow, wk and htmltools, and little else. Its contract is: give me an ArrowArrayStream with native GeoArrow geometry, a grid descriptor with a buffer, or a URL. gdalraster sits in Suggests as the best-supported producer, because GDAL can already hand over Arrow streams and read everything else. Anything that can emit those inputs works, with no class-specific code in the core.

The convenience package is the mapview role: view(x) for any spatial object, sensible colours, legends, popups, layer toggles. It carries the methods for sf and terra, and it is allowed to be heavy. Keeping it separate means the core stays small enough for other packages to depend on.

The convenience layer should shrink over time. sf already reaches geoarrow via wk. Once terra can emit a GeoArrow stream for a SpatVector, and a grid descriptor plus source path for a SpatRaster, the adapters become thin and the package is mostly about the user experience. That is a good outcome, not a failure.

The renderer is where the work is

Rasters are the easy half of a hand-built renderer; vectors are the long tail. A reprojected raster is one textured mesh per tile with a simple shader. Vectors need much more:

• Polygons: triangulation with holes (earcut).
• Lines: extrusion to screen-space width, joins, caps, antialiasing.
• Points: instanced sprites with sizing.
• Picking: an id framebuffer so a click resolves to a feature.
• Labels: glyph atlases, placement, collision. This is the biggest time sink, and it is why maplibre is large.
• Styling: attribute buffers and a data-driven style path.

The same list faces any standalone COG viewer that wants vector overlays. So the R viewer and the web viewer should share one renderer, not build two.

Curvature is one problem in two places. For rasters, mesh density absorbs the reprojection curvature. For vectors, a straight segment in the source CRS is not straight in the view CRS, so edges must be densified before projecting. It is the same sampling decision, applied to edges instead of a grid.

Most of the work can move out of the browser. Triangulate, densify and project on the R or Rust side, then ship vertex, index and segment buffers as Arrow columns. The renderer shrinks to three draw calls: indexed triangles, extruded segments, instanced points. The trade-off is that the payload is tied to one view CRS. For globe or spinning views, project the densified vertices on the client instead, in a vertex shader or with PROJ compiled to wasm.

Strategy: deck.gl first, with a seam for going large

Start on deck.gl in an orthographic view and find out how far the CRS game goes. “Hand the renderer to a capable team” is close behind, and the scene spec is what makes both possible without a rewrite.

The go-large option is realistic because the team largely exists. Development Seed’s lonboard puts deck.gl and GeoArrow behind a Python notebook widget, and it is the product that deck.gl-geoarrow and deck.gl-raster move with: lonboard upgrades the deck.gl-raster family in lockstep with deck.gl (lonboard #1272). If the scene spec is a neutral contract, an R front end can sit on the same JavaScript stack, and that renderer work becomes R’s renderer too.

The discipline that keeps the door open is simple: nothing deck.gl-specific in the scene spec. deck.gl is an implementation of the spec, not the spec.

Probe: a polar view on stock deck.gl

A polar view works on stock deck.gl today, as long as rasters arrive as pre-projected meshes rather than as COGs. The Polar view probe draws EPSG:3031 metres in an orthographic view from 885 KiB of Arrow, decoded in about 20 ms.

Three things came out of it:

• Source-space meshes make the pole easy. Each vertex of a lon/lat grid is projected once upstream. The pole row collapses to a point and the dateline is just a shared column edge.
• Topology depends on purpose. Natural Earth land closes through the pole, which is fine for a fill but draws a false seam if its rings are stroked. The outline has to come from a line dataset.
• EPSG:3031 is kind to float32. Its origin is the pole, so offsets stay under about 5000 km. Other projected CRSs still need a coordinateOrigin near the data.

Open questions and next steps

The first experiment is small: a polar-stereographic scene with a GeoArrow coastline and one COG, drawn by deck.gl in an orthographic view.

• Draft the scene spec as a JSON schema: view CRS, view type, layers, data references, legends. The probe’s spec is a starting point.
• Prototype the polar scene above and note where deck.gl pushes back.
• Try deck.gl-raster’s untiled RasterLayer with a 3031 reprojection function; scope a non-Mercator tile traversal upstream.
• Test a basemap in a non-Mercator view: warp XYZ tiles on a mesh, or skip tiles and draw vector land in the view CRS.
• Measure the float32 origin-offset approach on a projected CRS whose origin is far from the data.
• Decide where triangulation and densification live: R, Rust, or the client.
• Replace the probe’s Python prep with R: gdalraster Arrow streams, wk, and a mesh builder.

Open questions:

• Should styling be data only (RGBA columns computed in R), or also allow accessor expressions evaluated in the browser?
• What is the minimum grid descriptor for rasters: extent, dimension and CRS, or does it need overviews and nodata too?
• Would the lonboard maintainers welcome a shared, language-neutral scene spec, so R and Python front ends drive one renderer?

Versions

Everything below depends on these versions, and the results should be re-run when they change.

pkgs <- c("sf", "terra", "s2", "lwgeom", "geosphere", "geodist", "wk")
vapply(pkgs, function(p) as.character(utils::packageVersion(p)), "")

       sf     terra        s2    lwgeom geosphere   geodist        wk 
  "1.1.3"  "1.9.50"  "1.1.12"  "0.2.17"   "1.6.8"   "0.1.1"   "0.9.5" 

sf::sf_extSoftVersion()

          GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H 
      "3.12.1"       "3.13.3"        "9.9.0"         "true"         "true" 
          PROJ 
       "9.9.0" 

terra::gdal(lib = "all")

    gdal  proj   geos   TBB
1 3.13.3 9.9.0 3.12.1 FALSE

The parcel

This is the polygon from the thread, as a plain coordinate matrix in Lambert-93 (EPSG:2154, RGF93 on the GRS 1980 ellipsoid). It is about 40 m across. We keep three versions of it: projected, transformed to longitude and latitude (EPSG:4326), and with no CRS at all.

xy <- matrix(c(
  320102.67, 6845954.56,
  320111.85, 6845962.15,
  320131.82, 6845971.55,
  320134.33, 6845971.90,
  320138.50, 6845953.21,
  320141.93, 6845936.12,
  320135.33, 6845935.07,
  320118.58, 6845931.54,
  320113.45, 6845929.91,
  320105.18, 6845950.57,
  320102.67, 6845954.56
), ncol = 2, byrow = TRUE)

p   <- sf::st_sfc(sf::st_polygon(list(xy)), crs = "EPSG:2154")  ## projected
pll <- sf::st_transform(p, "EPSG:4326")                          ## longlat
p0  <- sf::st_set_crs(p, NA)                                     ## no crs
ll  <- sf::st_coordinates(pll)[, 1:2]
seg_i <- seq_len(nrow(xy) - 1)

A small helper sets sf_use_s2() for one expression and restores it afterwards.

with_s2 <- function(use, code) {
  old <- suppressMessages(sf::sf_use_s2(use))
  on.exit(suppressMessages(sf::sf_use_s2(old)))
  force(code)
}

Three reference numbers

Before calling any sf or terra function, we compute the perimeter and area directly under each of the three models, so that every later result can be labelled by which model it corresponds to.

• Planar: each edge is a straight line in the projected coordinates, and lengths and areas are Euclidean in those coordinates.
• Ellipsoid: each edge is a geodesic on the WGS84 ellipsoid, computed with Karney’s algorithms (GeographicLib), here via geosphere.
• Sphere: each edge is a great circle on a sphere of radius 6371010 m, which is the radius s2 uses.

planar_perim <- sum(sqrt(rowSums((xy[seg_i + 1, ] - xy[seg_i, ])^2)))
planar_area  <- abs(sum(xy[seg_i, 1] * xy[seg_i + 1, 2] -
                        xy[seg_i + 1, 1] * xy[seg_i, 2])) / 2

karney_perim <- sum(geosphere::distGeo(ll[seg_i, ], ll[seg_i + 1, ]))
karney_area  <- geosphere::areaPolygon(ll)

R_s2 <- 6371010
sphere_perim <- s2::s2_perimeter(s2::as_s2_geography(pll), radius = R_s2)
sphere_area  <- s2::s2_area(s2::as_s2_geography(pll), radius = R_s2)

refs_perim <- c(planar = planar_perim, ellipsoid = karney_perim, sphere = sphere_perim)
refs_area  <- c(planar = planar_area,  ellipsoid = karney_area,  sphere = sphere_area)
refs_perim

    planar  ellipsoid     sphere 
129.248935 129.285384 129.098150 

refs_area

    planar  ellipsoid     sphere 
1049.65051 1050.24260 1047.02612 

which_number <- function(x, refs) {
  hit <- names(refs)[abs(refs - x) / refs < 1e-6]
  if (length(hit) == 0) "OTHER" else paste(hit, collapse = "/")
}

The perimeter is 129.249 m planar, 129.285 m on the ellipsoid and 129.098 m on the sphere. The area is 1049.65, 1050.24 and 1047.03 square metres respectively.

aeqd_edge <- function(a, b) {
  crs <- sprintf("+proj=aeqd +lon_0=%.12f +lat_0=%.12f +ellps=WGS84", a[1], a[2])
  q <- terra::project(rbind(a, b), from = "EPSG:4326", to = crs)
  sqrt(sum((q[2, ] - q[1, ])^2))
}
c(aeqd = sum(vapply(seg_i, function(i) aeqd_edge(ll[i, ], ll[i + 1, ]), 0)),
  karney = karney_perim)

      aeqd     karney 
129.285384 129.285384 

sum(geosphere::distGeo(ll[seg_i, ], ll[seg_i + 1, ], a = R_s2, f = 0))

[1] 129.09815

k_perim <- planar_perim / karney_perim
k_area  <- planar_area / karney_area

cen <- colMeans(ll[seg_i, ])
nb  <- geosphere::destPoint(cen, b = 45, d = 10)
cq  <- terra::project(rbind(cen, nb), from = "EPSG:4326", to = "EPSG:2154")
k_point <- sqrt(sum((cq[2, ] - cq[1, ])^2)) / 10

c(k_point = k_point, k_perim = k_perim, k_perim_sq = k_perim^2, k_area = k_area)

    k_point     k_perim  k_perim_sq      k_area 
0.999718096 0.999718077 0.999436234 0.999436238 

laea <- sprintf("+proj=laea +lon_0=%.12f +lat_0=%.12f +ellps=WGS84", cen[1], cen[2])
c(laea = as.numeric(sf::st_area(sf::st_set_crs(sf::st_transform(pll, laea), laea))),
  karney = karney_area)

      laea     karney 
1050.24259 1050.24260 

How each package decides

body(sf::st_perimeter)

{
    x = st_geometry(x)
    if (sf_use_s2() && isTRUE(st_is_longlat(x))) {
        if (!requireNamespace("s2", quietly = TRUE)) 
            stop("package s2 required to calculate the perimeter of spherical geometries")
        units::set_units(s2::s2_perimeter(x, ...), "m", mode = "standard")
    }
    else {
        if (isTRUE(st_is_longlat(x))) 
            units::set_units(st_length(st_boundary(x)), "m", 
                mode = "standard")
        else {
            if (!requireNamespace("lwgeom", quietly = TRUE)) 
                stop("package lwgeom required, please install it first")
            lwgeom::st_perimeter_lwgeom(x)
        }
    }
}

tryCatch(lwgeom::st_perimeter(pll), error = function(e) conditionMessage(e))

Warning in lwgeom::st_perimeter(pll): 'lwgeom::st_perimeter' is deprecated.
Use 'sf::st_perimeter or lwgeom::st_perimeter_lwgeom' instead.
See help("Deprecated")

[1] "for perimeter of longlat geometry, cast to LINESTRING and use st_length"

vp  <- terra::vect(p)
pts <- terra::vect(xy[1:2, ], crs = "EPSG:2154")
r   <- terra::rast(terra::ext(320100, 320150, 6845925, 6845975), res = 10, crs = "EPSG:2154")
unlist(list(
  perim          = terra::perim(vp),
  expanse        = terra::expanse(vp),
  expanse_plane  = terra::expanse(vp, transform = FALSE),
  distance       = as.numeric(terra::distance(pts)),
  cellSize       = terra::global(terra::cellSize(r, unit = "m"), "mean")[1, 1],
  cellSize_plane = terra::global(terra::cellSize(r, unit = "m", transform = FALSE), "mean")[1, 1]
))

         perim        expanse  expanse_plane       distance       cellSize 
   129.2489354   1050.2425993   1049.6505127     11.9113601    100.0564081 
cellSize_plane 
   100.0000000 

## references for the distance row
c(planar = sqrt(sum((xy[2, ] - xy[1, ])^2)), karney = geosphere::distGeo(ll[1, ], ll[2, ]))

    planar     karney 
11.9113601 11.9147186 

The grid

Now every combination: four ways of asking for a perimeter, three ways of asking for an area, three versions of the polygon, and sf_use_s2() on and off. Each result is labelled by the reference number it matches.

inputs <- list(projected = p, longlat = pll, none = p0)

perim_calls <- list(
  "sf::st_perimeter"        = function(x) sf::st_perimeter(x),
  "sf::st_length(boundary)" = function(x) sf::st_length(sf::st_boundary(x)),
  "sf::st_length(cast MLS)" = function(x) sf::st_length(sf::st_cast(x, "MULTILINESTRING")),
  "terra::perim"            = function(x) terra::perim(terra::vect(x))
)
area_calls <- list(
  "sf::st_area"                     = function(x) sf::st_area(x),
  "terra::expanse"                  = function(x) terra::expanse(terra::vect(x)),
  "terra::expanse(transform=FALSE)" = function(x) terra::expanse(terra::vect(x), transform = FALSE)
)

run_grid <- function(calls, refs) {
  out <- expand.grid(call = names(calls), input = names(inputs), s2 = c(TRUE, FALSE),
                     stringsAsFactors = FALSE)
  out$value <- NA_real_
  out$msg <- ""
  for (i in seq_len(nrow(out))) {
    f <- calls[[out$call[i]]]
    x <- inputs[[out$input[i]]]
    r <- tryCatch(
      withCallingHandlers(
        with_s2(out$s2[i], as.numeric(f(x))),
        warning = function(w) {
          out$msg[i] <<- conditionMessage(w)
          invokeRestart("muffleWarning")
        }),
      error = function(e) {
        out$msg[i] <<- paste("ERROR:", conditionMessage(e))
        NA_real_
      })
    out$value[i] <- r
  }
  out$matches <- vapply(out$value, function(v) if (is.na(v)) "-" else which_number(v, refs), "")
  out
}

wide <- function(g) {
  w <- reshape(g[, c("call", "input", "s2", "matches")],
               idvar = c("call", "s2"), timevar = "input", direction = "wide")
  names(w) <- sub("^matches\\.", "", names(w))
  rownames(w) <- NULL
  w
}

perim_grid <- run_grid(perim_calls, refs_perim)

Linking to GEOS 3.12.1, GDAL 3.13.3, PROJ 9.9.0; sf_use_s2() is TRUE

wide(perim_grid)

                     call    s2 projected   longlat   none
1        sf::st_perimeter  TRUE    planar    sphere planar
2 sf::st_length(boundary)  TRUE    planar    sphere planar
3 sf::st_length(cast MLS)  TRUE    planar    sphere planar
4            terra::perim  TRUE    planar ellipsoid planar
5        sf::st_perimeter FALSE    planar ellipsoid planar
6 sf::st_length(boundary) FALSE    planar ellipsoid planar
7 sf::st_length(cast MLS) FALSE    planar ellipsoid planar
8            terra::perim FALSE    planar ellipsoid planar

area_grid <- run_grid(area_calls, refs_area)
wide(area_grid)

                             call    s2 projected   longlat   none
1                     sf::st_area  TRUE    planar    sphere planar
2                  terra::expanse  TRUE ellipsoid ellipsoid planar
3 terra::expanse(transform=FALSE)  TRUE    planar ellipsoid planar
4                     sf::st_area FALSE    planar ellipsoid planar
5                  terra::expanse FALSE ellipsoid ellipsoid planar
6 terra::expanse(transform=FALSE) FALSE    planar ellipsoid planar

The three sf routes to a perimeter now agree with each other in every case, which was not true at the time of the original thread. With projected input, or no CRS, everything in sf is planar. With longitude and latitude input, sf gives the spherical number when s2 is on and the ellipsoidal number when it is off.

terra does not read sf_use_s2(), so its rows are the same in both halves of each table. With longitude and latitude input it gives the ellipsoidal number for both perimeter and area. With projected input, perim() is planar and expanse() is ellipsoidal. With no CRS, terra computes in the plane and warns that the results may be wrong.

For longitude and latitude data, which is the common case for data arriving from files and web services, the two packages with their default settings give different answers: sf gives 129.098 m and 1047.03 square metres, terra gives 129.285 m and 1050.24 square metres.

One option, two roles

In sf, sf_use_s2() selects the model for measures and also the engine for predicates and overlays. With s2 off, longitude and latitude geometries go to GEOS, which treats the coordinates as planar. Measures, however, go to GeographicLib on the ellipsoid, where each edge is a geodesic. So within one session with s2 off, the same polygon has ellipsoidal geodesic edges for the purpose of its area and straight edges in longitude and latitude for the purpose of point-in-polygon.

This matters for wide polygons at high latitude. A geodesic between two points at the same latitude does not follow the parallel; it bends toward the pole. Here is a box 120 degrees wide between 70 and 60 degrees south, and a point just inside its northern edge.

bx <- sf::st_sfc(sf::st_polygon(list(rbind(
  c(-60, -70), c(60, -70), c(60, -60), c(-60, -60), c(-60, -70)))),
  crs = "EPSG:4326")
pt <- sf::st_sfc(sf::st_point(c(0, -62)), crs = "EPSG:4326")

with_s2(TRUE,  list(area = sf::st_area(bx),
                    intersects = sf::st_intersects(bx, pt, sparse = FALSE)[1, 1]))

$area
3008488091024 [m^2]

$intersects
[1] FALSE

with_s2(FALSE, list(area = sf::st_area(bx),
                    intersects = sf::st_intersects(bx, pt, sparse = FALSE)[1, 1]))

although coordinates are longitude/latitude, st_intersects assumes that they
are planar

$area
3.027757e+12 [m^2]

$intersects
[1] TRUE

With s2 on, the area is spherical and the point is outside the polygon, which is consistent: on the sphere the northern edge passes south of 62 degrees south at longitude 0. With s2 off, the area is ellipsoidal, which is the more accurate model, but the point is now inside the polygon, because GEOS treats the northern edge as the straight line at 60 degrees south. A user who switches s2 off to obtain ellipsoidal measures therefore also changes the answers to spatial predicates, and the measures and predicates no longer describe the same polygon.

Point distance and two spheres

For a single pair of points (the first two vertices of the parcel) there is a fourth number.

a <- ll[1, , drop = FALSE]
b <- ll[2, , drop = FALSE]
pa <- sf::st_sfc(sf::st_point(a), crs = "EPSG:4326")
pb <- sf::st_sfc(sf::st_point(b), crs = "EPSG:4326")
c(
  sf_s2     = with_s2(TRUE,  as.numeric(sf::st_distance(pa, pb))),
  sf_nos2   = with_s2(FALSE, as.numeric(sf::st_distance(pa, pb))),
  terra_geo = terra::distance(a, b, lonlat = TRUE),
  terra_hav = terra::distance(a, b, lonlat = TRUE, method = "haversine"),
  geodist   = geodist::geodist(a, b, measure = "geodesic"),
  karney    = geosphere::distGeo(a, b),
  planar    = sqrt(sum((xy[2, ] - xy[1, ])^2))
)

     sf_s2    sf_nos2  terra_geo  terra_hav    geodist     karney     planar 
11.8954624 11.9147186 11.9147186 11.9087694 11.9147186 11.9147186 11.9113601 

sf with s2 off, terra’s default method, geodist and geosphere all agree on the ellipsoidal distance. sf with s2 on gives the spherical distance. terra’s method = "haversine" gives a third value, because it uses a sphere with the equatorial radius of WGS84 rather than s2’s mean radius:

c(terra_hav      = terra::distance(a, b, lonlat = TRUE, method = "haversine"),
  hav_equatorial = geosphere::distHaversine(a, b, r = 6378137),
  hav_mean_s2    = geosphere::distHaversine(a, b, r = R_s2))

     terra_hav hav_equatorial    hav_mean_s2 
    11.9087694     11.9087694     11.8954624 

The two spheres differ in radius by a factor of 1.0011, so “spherical distance” is itself not one number.

How large is the difference, and where?

For a 40 m parcel the differences are centimetres. It is tempting to conclude that the choice of model only matters for large polygons. For the sphere against the ellipsoid, that is not the case: the relative error in area is set by latitude and is essentially the same for a box one metre across and a box one degree across.

box <- function(lon0, lat0, size) {
  m <- rbind(c(lon0, lat0), c(lon0 + size, lat0), c(lon0 + size, lat0 + size),
             c(lon0, lat0 + size), c(lon0, lat0))
  sf::st_sfc(sf::st_polygon(list(m)), crs = "EPSG:4326")
}
sweep_lat <- function(size, lats = seq(-89.5, 89.5 - size, by = 1)) {
  do.call(rbind, lapply(lats, function(lat0) {
    g <- box(147, lat0, size)
    k <- geosphere::areaPolygon(sf::st_coordinates(g)[, 1:2])
    data.frame(lat = lat0 + size / 2, size = size,
               s2 = with_s2(TRUE, as.numeric(sf::st_area(g))) / k - 1,
               terra = terra::expanse(terra::vect(g)) / k - 1)
  }))
}
sw <- rbind(sweep_lat(0.001), sweep_lat(0.1), sweep_lat(1))
range(sw$terra)

[1] 0 0

plot(sw$lat, 100 * sw$s2, type = "n", xlab = "latitude",
     ylab = "area error vs ellipsoid (%)", main = "s2 sphere vs Karney ellipsoid")
sizes <- unique(sw$size)
for (s in sizes) {
  d <- sw[sw$size == s, ]
  lines(d$lat, 100 * d$s2, lty = match(s, sizes))
}
abline(h = 0, col = "grey")
legend("topright", legend = paste(sizes, "deg box"), lty = seq_along(sizes))

The spherical area is 0.45% too large at the equator, correct near 35 degrees north and south, and 0.89% too small near the poles. At the latitudes of the Southern Ocean and Antarctica the spherical area is consistently between 0.5% and 0.9% smaller than the ellipsoidal area, regardless of the size of the polygon. terra agrees with Karney to within floating point precision throughout, since it uses the same library.

Planar error behaves differently: it depends on the projection’s scale factor over the polygon, which for a well-chosen local projection is small for small polygons and grows with distance from the projection’s centre or standard lines.

What is an edge?

All of the above assumes that everyone agrees on what the polygon is. The coordinates alone do not determine that. Between two vertices, an edge could be a geodesic on the ellipsoid, a great circle on the sphere, a straight line in some projection, or a line of constant latitude. For small polygons these coincide closely. For large ones they do not.

Here is a box 10 degrees on each side at 60 degrees south, first with only its four corners, then densified at 0.1 degree spacing along its edges in longitude and latitude so that the northern and southern edges follow the parallels.

b1  <- box(147, -60, 10)
b1d <- sf::st_set_crs(sf::st_segmentize(sf::st_set_crs(b1, NA), 0.1), "EPSG:4326")
c(
  corners_karney   = geosphere::areaPolygon(sf::st_coordinates(b1)[, 1:2]),
  densified_karney = geosphere::areaPolygon(sf::st_coordinates(b1d)[, 1:2]),
  corners_terra    = terra::expanse(terra::vect(b1)),
  densified_terra  = terra::expanse(terra::vect(b1d))
)

  corners_karney densified_karney    corners_terra  densified_terra 
    709651946744     711460602143     709651946744     711460602143 

The two versions differ in area by 0.25%, which is of the same order as the difference between the sphere and the ellipsoid at this latitude. terra and Karney agree on each version, so both treat the edges as geodesics and neither densifies implicitly.

The same question applies to terra’s transform = TRUE for projected polygons. The vertices are transformed to longitude and latitude and the edges are then treated as geodesics. An edge that is straight in the projection is not a geodesic, so for large polygons the result is the ellipsoidal area of a slightly different polygon. Densifying in the projected coordinates before measuring removes this.

Summary

For longitude and latitude data, terra’s rule can be stated in one line: it always measures on the ellipsoid. sf’s answer depends on an option that is set per session, is on by default, and also determines how predicates and overlays are computed. The consequence is that an sf user who wants ellipsoidal measures has to change the behaviour of their spatial predicates as well, and a user who leaves the default gets a spherical measure whose error varies from +0.45% to -0.89% with latitude, independent of the size of the feature.

Two changes would make each package’s rule simpler to state. In sf, a way to select the ellipsoid for measures independently of the engine used for predicates, for example an argument to the measure functions or a separate option. In terra, extending the transform argument to perim() and distance(), so that projected input is measured on the ellipsoid for every quantity whenever the CRS is known, as it already is for area.

In the meantime, the grid above can be run against any installed versions, and is a quick way to check which number a given function returns.

It was not a new projection

Athelstan Spilhaus published ocean-centred maps from 1942, using existing conformal and equal-area projections. The square version people recognise appeared in 1979 and was republished in 1983 and 1991. Mathematically it is an oblique aspect of the Adams World in a Square II projection (Adams, 1925/1929), rotated 45 degrees so the square sits on a corner. Spilhaus chose a centre and an azimuth; he did not invent the maths.

That matters for the story, because “implement Spilhaus” decomposes into two jobs: implement Adams WS2 (elliptic-function conformal maths), and then support an oblique, rotated aspect with the right parameters.

2018-2020: viral, then reverse-engineered

The map spread on social media in autumn 2018 with no usable coordinate system behind it. Bojan Savric, David Burrows and Melita Kennedy at Esri did the detective work, published it as a storymap, and identified the parameters:

Projection: Adams Square II
Azimuth: 40.17823482
Longitude of centre: 66.94970198 E
Latitude of centre: 49.56371678 S
XY plane rotation: 45

It shipped as a projected coordinate system in ArcGIS 10.8 and ArcGIS Pro 2.5 in February 2020, and the definitions became ESRI:54098 and ESRI:54099 in Esri’s openly published projection-engine database. So to be fair to that company: the parameters were public, the WKT was public, and the storymap is still the best explainer on the web. What did not happen was any code landing in the open library everyone else depends on.

2020: PROJ gets halfway there in three months

Asger Petersen opened OSGeo/PROJ#1851 in January 2020, pointing at the Esri storymap and the parameters. By April, Kristian Evers had added adams_hemi, adams_ws1, adams_ws2, guyou and peirce_q to PROJ (PR #2148, “a first step towards implementing the Spilhaus projection as requested in #1851”), and Even Rouault had mapped Esri’s Adams_Square_II WKT to +proj=adams_ws2 and written an iterative inverse so gdalwarp could use it (PR #2157). Both shipped in PROJ 7.1.0.

The maths was there, the oblique/rotated aspect was not, and the issue sat for five years wearing the pinned label, whose stated purpose in the PROJ repo is to “prevent stale-bot from closing an issue”.

2019-2024: everyone else routes around

While the base layer was stalled, the demand did not go away, and the community did what it always does: reimplemented the forward transform wherever it was standing.

• Cartopy issue #1376 (October 2019) asked for Spilhaus and asked whether the rotation could live in cartopy or had to come from PROJ. It stayed open until 2025.
• Torben Jansen published Observable notebooks for Adams World in a Square I and II, and for Spilhaus, on d3-geo. Joe Davies and others followed. This is where the “viral map” crowd already lived, so this is where most Spilhaus maps were actually being made.
• Ricardo Lemos published rtlemos/spilhaus, parallel R and Python scripts with from_lonlat_to_spilhaus_xy and its inverse, with a README that says the quiet part out loud: “Currently, the Spilhaus projection is missing in PROJ, making it unavailable for R and Python enthusiasts, among others: OSGeo/PROJ#1851.”
• Chen et al. (Scientific Data, 2023) published an oceanographic case for the projection. Their maps were made in ArcGIS Pro 2.5; their open supplement was a “quasi-Spilhaus” d3 notebook based on Jansen’s work, which they note carries “slight error”, and they record that GMT and QGIS could not do it.

Each of these gives you one thing: points forward, sometimes points inverse, in one language. None gives you a CRS. You cannot warp a raster with an Observable notebook, st_transform() a polygon layer with a Python script, or hand ESRI:54099 to a colleague and have their tools agree with yours. That is the difference between a projection formula and a projection in PROJ.

2025: three days in February

Javier Jimenez Shaw opened PR #4401 on 16 February 2025 and it was merged on 19 February. It implements +proj=spilhaus for both the sphere and the ellipsoid, using conformal latitudes with an additional correction that, in his words, “was not present in other implementations”. The defaults are Spilhaus’s parameters, so the bare string is the map:

+proj=spilhaus
## equivalent to
+proj=spilhaus +lon_0=66.94970198 +lat_0=-49.56371678 +azi=40.17823482 +rot=45

The same day, Even Rouault’s PR #4402 wired ESRI:54099 and ESRI:54098 to +proj=spilhaus, which surfaced the last puzzle: Esri’s coordinates differ from the raw Adams maths by a factor of sqrt(2). PROJ carries that as +k_0=1.4142135... on the Esri definitions, and Keith Jenkins’s reference points from ArcGIS agreed to well under a metre. Interoperability is exactly this kind of unglamorous detail, done in public, with the test points in the PR.

PROJ 9.6.0 was released on 15 March 2025 with “Add Spilhaus projection (#4401)” as one line in the release notes. The documentation page is a parameter table.

After: a very quiet arrival

What followed:

• Cartopy PR #2529 (May 2025) added a Spilhaus class, gated on PROJ 9.6 and pyproj 3.7.1. But pyproj 3.7.1 wheels bundle PROJ 9.5.1, so a pip install user got nothing until 3.7.2 in August 2025. Conda users got it sooner.
• A NetBSD user reported 56 of 77 Spilhaus regression tests failing with round-trip errors of tens to hundreds of kilometres, everything else in PROJ passing, almost certainly a libm quirk. Conformal maths at the edges of a square is a stress test for platform floating point.
• The QGIS map gallery got one tagged post: “The beautiful Spilhaus projection (‘oceans in a square’) is new in PROJ 9.6.0.”
• The five-year issue closed with the PR and, as far as I can find, no announcement anywhere that ocean people read.

Meanwhile, all of this just works, if and only if your PROJ is 9.6 or newer:

library(sf)
x <- st_transform(st_as_sf(rnaturalearth::ne_countries()), "+proj=spilhaus")

library(terra)
r <- project(rast("/vsicurl/https://example.org/global.tif"), "+proj=spilhaus")

gdalwarp -t_srs "+proj=spilhaus" global.tif spilhaus.tif

from pyproj import Transformer
t = Transformer.from_crs(4326, "+proj=spilhaus", always_xy=True)
t.transform(147.3, -42.9)

What this says about the landscape

The base layer is where the leverage is, and where the least attention goes. Five-plus years of demand produced maybe half a dozen independent reimplementations. One contribution to PROJ made the projection available for raster, vector and raw coordinates, with an inverse, on the ellipsoid, with a database identity, in every downstream stack at once. Nobody had to change a line in GDAL, sf, terra, QGIS or PostGIS.

“It’s in PROJ” is not “it’s in your hands.” The path from a merged PR to a working function on a user’s machine runs through release cadence, wheel and CRAN binary builds, conda-forge, OSGeo4W, Linux distro freezes and QGIS LTR policy. Six to eighteen months of skew is normal, and it is invisible unless you know to look:

projinfo --version

sf::sf_extSoftVersion()["PROJ"]
terra::gdal(lib = "proj")

import pyproj; pyproj.proj_version_str
from osgeo import osr; osr.GetPROJVersionMajor()

If folks tried Spilhaus in mid-2025 and got an “unknown projection” error probably concluded it was not supported, not that they were on 9.5 or earlier.

Provenance is lost by default. Search for the projection and you get the Esri storymap and Observable notebooks; the PROJ documentation page is a parameter table. The people who did the hard, general work are one line in a changelog. This is not a complaint about anyone; it is how the attention economy of open infrastructure works, and it is why I think projects like PROJ and GDAL need people whose job is partly to narrate what they have done.

The vendor question is less interesting than it looks. Esri published the parameters and the WKT, and that is what the open implementation was built from and tested against. The gap is not secrecy; it was just not done properly until folded into PROJ.

Next

I am putting together notebooks that walk the chain, PROJ version to pyproj to GDAL to a global bathymetry raster, vectors and city points on one Spilhaus map, partly as a demo and partly to make the case that following upstream is a skill worth teaching.

Spilhaus tracker: example python code

Links

• OSGeo/PROJ#1851 Spilhaus projection
• PR #4401 Add Spilhaus projection
• PR #4402 Export ESRI:54099 as PROJ string
• PR #2148 Adams, Guyou, Peirce projections
• PR #2157 Adams Square II ESRI WKT and inverse
• PROJ Spilhaus documentation
• Esri: The Spilhaus map of the oceans will be supported in the next release of ArcGIS
• Chen et al. 2023, Ocean Sciences with the Spilhaus Projection
• Cartopy issue #1376 and PR #2529
• rtlemos/spilhaus
• pyproj changelog

The climatology example, from dplyr

Earthmover’s headline query is a monthly climatology at a grid point, running against a 125 TB ERA5 Icechunk dataset:

SELECT DATE_PART('month', valid_time) as month,
       AVG(t2m) as t2m_mean,
       STDDEV(t2m) as t2m_std
FROM era5.'single/temporal'
WHERE longitude = 45.0 AND latitude = 60.0
GROUP BY month
ORDER BY month

Here is that query as an ordinary dplyr pipeline, and the SQL dbplyr 2.4.0 generates for it. This is reproducible with no server at all, using dbplyr’s simulated Postgres connection:

library(dplyr)

Attaching package: 'dplyr'

The following objects are masked from 'package:stats':

    filter, lag

The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union

library(dbplyr)

Attaching package: 'dbplyr'

The following objects are masked from 'package:dplyr':

    ident, sql, sql_escape_ident, sql_escape_string

era5 <- lazy_frame(
  valid_time = Sys.time(), longitude = 1, latitude = 1, t2m = 1,
  con = simulate_postgres(), .name = ident_q("era5.'single/temporal'")
)

era5 |>
  filter(longitude == 45, latitude == 60) |>
  group_by(month = date_part("month", valid_time)) |>
  summarise(t2m_mean = mean(t2m, na.rm = TRUE),
            t2m_std  = sd(t2m, na.rm = TRUE)) |>
  arrange(month) |>
  show_query()

<SQL>
SELECT "month", AVG("t2m") AS "t2m_mean", STDDEV_SAMP("t2m") AS "t2m_std"
FROM (
  SELECT *, date_part('month', "valid_time") AS "month"
  FROM era5.'single/temporal'
  WHERE ("longitude" = 45.0) AND ("latitude" = 60.0)
) AS "q01"
GROUP BY "month"
ORDER BY "month"

SELECT `month`, AVG(`t2m`) AS `t2m_mean`, STDDEV_SAMP(`t2m`) AS `t2m_std`
FROM (
  SELECT `q01`.*, date_part('month', `valid_time`) AS `month`
  FROM era5.'single/temporal' AS `q01`
  WHERE (`longitude` = 45.0) AND (`latitude` = 60.0)
) AS `q01`
GROUP BY `month`
ORDER BY `month`

(Simulated connections print backticks; a live RPostgres connection quotes identifiers with double quotes. On a real connection the table reference is tbl(con, I("era5.'single/temporal'")) rather than ident_q().)

That is, up to a subquery and spelling, the query from their post. Two details I like: dbplyr writes STDDEV_SAMP, which is what Postgres’s (and DataFusion’s) STDDEV is an alias for, so it is being more explicit, not less compatible; and date_part isn’t a special case - dbplyr passes unknown functions through verbatim, which happens to be exactly the escape hatch a young dialect needs.

The pattern extends naturally. A regional aggregation - the thing that actually exercises chunk-level pushdown - is just a bounding box in the WHERE clause:

era5 |>
  filter(between(longitude, 100, 160), between(latitude, -70, -40),
         valid_time >= "2020-01-01", valid_time < "2021-01-01") |>
  group_by(month = date_part("month", valid_time)) |>
  summarise(t2m_mean = mean(t2m, na.rm = TRUE), n = n()) |>
  arrange(month)

SELECT `month`, AVG(`t2m`) AS `t2m_mean`, COUNT(*) AS `n`
FROM (
  SELECT `q01`.*, date_part('month', `valid_time`) AS `month`
  FROM era5.'single/temporal' AS `q01`
  WHERE
    (`longitude` BETWEEN 100.0 AND 160.0) AND
    (`latitude` BETWEEN -70.0 AND -40.0) AND
    (`valid_time` >= '2020-01-01') AND
    (`valid_time` < '2021-01-01')
) AS `q01`
GROUP BY `month`
ORDER BY `month`

A BETWEEN on a coordinate column is a hyperslab. The R user wrote a filter; the Zax planner decides what that means in terms of I/O. That division of labour is precisely the one dbplyr assumes about every database it talks to.

Wiring it up

Two routes, both using infrastructure R already has:

• Postgres wire: RPostgres::dbConnect() to a Zax-SQL endpoint, then tbl() and ordinary dplyr verbs. dbplyr’s Postgres dialect should carry most pipelines, since Zax-SQL uses Apache DataFusion for the relational processing and DataFusion’s SQL is Postgres-flavoured.
• Arrow Flight SQL: the better wire for bulk results - Arrow record batches straight into R with no text protocol in the middle. The ADBC Flight SQL driver via adbcdrivermanager, with DBI compatibility through adbi, gives dbplyr the same interface.

Where it will misfire is dialect edges: DataFusion is not Postgres, so some translations (string and date functions, window frames) will emit SQL that Zax rejects. The remedy is the classic thin-backend move - a connection subclass with a sql_translation() method mapping to DataFusion names. That is a couple of hundred lines, the duckplyr-shaped package, and only worth writing once the friction is demonstrated rather than assumed.

Why this matters beyond a demo

A correctness oracle, both ways. xrdbi and Zax now implement the same tensor-to-relation flattening on either side of a wire. Running identical pipelines against xrdbi-local (xarray on your laptop) and Zax-remote (125 TB of Icechunk) cross-checks the fiddly semantics: broadcast order, coordinate column behaviour, aggregation over flattened dimensions. Where they agree, R users get a strong story - same dplyr code, laptop or petabyte. Where they disagree, one of us has found a bug or an underspecified corner of the model, and either outcome is useful feedback while the dialect is young.

Convergent evolution. Zax-SQL, xarray-sql, Beacon, and SedonaDB’s raster support are all landing on SQL-over-flattened-tensors from different directions (Earthmover’s post generously points at the other three). When the same interface emerges independently from a Rust query engine, a serverless Python project, a NetCDF client-server system, and a spatial SQL engine - and, from the R side, from a dbplyr backend - that is probably not a fad.

The relational layer is the contract. In work factoring xrdbi’s core into an engine-agnostic translation layer (working name arrplyr), the plan had been a JSON-ish intermediate representation - variables, coordinate windows, strides, axis reductions - handed to engines like xarray, GDAL multidim, or pizzarr. The arrival of SQL-speaking array engines simplifies one whole branch of that picture: for Zax and its cousins, no IR is needed, because dbplyr’s native SQL generation is the contract, and the flattened relational model is the interchange format. The IR earns its keep only for engines with no SQL front door. And Earthmover’s roadmap - queries as saved, versioned, shareable objects; views as unmaterialized derived datasets - is the recipe-not-payload position: a query is a description of data, and it is the description you should store and exchange, not the bytes.

Agents, incidentally. Earthmover’s argument for SQL-first is partly that LLMs write competent SQL but poor memory-aware chunked-array code. The same argument holds one level up: agents and humans alike write competent dplyr, and dbplyr removes even the SQL from view. The grammar R users have applied to data frames for a decade now reaches petabyte tensor stores with no new API at all.

Trying it

Zax-SQL is live on every Arraylake plan including the free Community Tier: create an organization, subscribe to a Marketplace dataset (ERA5 is there), spin up a Zax-SQL service, and connect with RPostgres exactly as you would to any Postgres. The dplyr snippets above should be a working starting point, and show_query() shows you what is crossing the wire.

I plan to write up a fuller “same table, two engines” comparison - identical pipelines against xrdbi and Zax with the results diffed. If you are interested in the R side of this (or the arrplyr factoring), issues and discussion at xrdbi are welcome.

Links

• Earthmover, “Our compute roadmap: introducing Zax SQL”: https://www.earthmover.io/blog/compute-roadmap
• Zax-SQL docs: https://docs.earthmover.io/compute/sql
• xrdbi: https://github.com/mdsumner/xrdbi
• dbplyr: https://dbplyr.tidyverse.org
• Apache DataFusion: https://datafusion.apache.org
• xarray-sql: https://xql.systems/xarray-sql/latest/
• Beacon: https://maris-development.github.io/beacon/
• SedonaDB: https://sedona.apache.org/sedonadb/latest/
• Earthmover, “Tensors vs. tables”: https://www.earthmover.io/blog/tensors-vs-tables/

Why Zarr

A Zarr array is a small metadata file plus one file per chunk. Two processes writing different chunks never touch the same bytes, so there is nothing to coordinate. GDAL’s Zarr driver also declines to write a chunk that is entirely fill value (it will even unlink an existing one), which means the store on disk is the sparse representation: chunks over open ocean and interior ice sheet do not exist.

The plan is

1. create the empty store once, in the parent, with GDAL
2. open it in update mode from each worker, with its own handle, and write whole chunks
3. if a COG is still wanted, make it with one translate() at the end

Instantiate

library(controlledburn)
library(gdalraster)
library(mirai)

ext <- ba$extent
dm  <- ba$dimension
chunk_size <- 2048L
zarr_path  <- file.path(tempdir(), "rema_approx.zarr")

ds <- create(
  format = "Zarr", dst_filename = zarr_path,
  xsize = dm[1], ysize = dm[2], nbands = 1, dataType = "Int32",
  options = c(paste0("BLOCKSIZE=", chunk_size, ",", chunk_size),
              "COMPRESS=ZSTD", "FORMAT=ZARR_V2"),
  return_obj = TRUE)
ds$setGeoTransform(vaster::extent_dim_to_gt(ext, dm))
ds$setProjection(raster_info$projection$Wkt)
ds$setNoDataValue(band = 1, 0)   # fill_value; all-zero chunks are never written
ds$close()

The 4x4 “super-tile” from the last post is gone. A chunk write is one file write, so the chunk is the natural unit of parallel work; 2048 keeps each dense allocation at 16 MB for Int32. (If 512-pixel chunks are wanted for cloud reads, set BLOCKSIZE=512,512 and keep iterating over 2048 windows; GDAL splits each write into 16 chunk files that are still disjoint per worker.)

Populate

The tile index and the pre-filter to rows that carry data are unchanged. The worker opens the store in update mode, writes a batch of chunks, and closes. It calls nothing that dirties metadata, so .zarray and .zmetadata are not rewritten on close.

write_tiles <- function(tile_batch, zarr_path) {
  ds <- methods::new(gdalraster::GDALRaster, zarr_path, read_only = FALSE)
  on.exit(ds$close(), add = TRUE)
  n <- 0L
  for (i in seq_len(nrow(tile_batch))) {
    t <- tile_batch[i, ]
    sub <- controlledburn::crop_burn(ba, c(t$xmin, t$xmax, t$ymin, t$ymax))
    if (nrow(sub$runs) == 0 && nrow(sub$edges) == 0) next
    mat <- controlledburn::materialize_chunk(sub, fun = "id")
    mat[is.na(mat)] <- 0L
    ds$write(band = 1, xoff = t$offset_x, yoff = t$offset_y,
             xsize = t$ncol, ysize = t$nrow, rasterData = as.integer(t(mat)))
    n <- n + 1L
  }
  n
}
environment(write_tiles) <- globalenv()

daemons(24)
everywhere(library(controlledburn), ba = ba)   # ships ba once per daemon

batches <- split(tiles, cut(seq_len(nrow(tiles)), 24 * 4L, labels = FALSE))
system.time({
  res <- mirai_map(batches, write_tiles, .args = list(zarr_path = zarr_path))[.progress]
})
daemons(0)

Two mirai details that cost me a round trip. Objects passed through ... to everywhere() are assigned into each daemon’s global environment, which is exactly the “send the 27 MB once” behaviour wanted here; a plain <- inside the expression is evaluated in a clean environment and lost (use <<- if you assign there). And reparenting the worker function to globalenv() stops it dragging the Quarto chunk environment along with every task.

Result

On a 32-cpu, 64 GB machine with 24 daemons:

   user  system elapsed
  ...     ...    28.0

library(terra)
rast(zarr_path)
#> class       : SpatRaster
#> size        : 182568, 170318, 1  (nrow, ncol, nlyr)
#> resolution  : 32, 32  (x, y)
#> extent      : -2700096, 2750080, -2500096, 3342080  (xmin, xmax, ymin, ymax)
#> source      : rema_approx.zarr

f0 <- fs::dir_ls(zarr_path, recurse = TRUE)
length(f0)
#> [1] 722
sum(fs::file_size(f0))
#> 9.48M

The grid could hold ceiling(170318 / 2048) x ceiling(182568 / 2048) = 84 x 90 = 7560 chunks. About 718 of them exist. That is 9.5% of the continent’s bounding box with any rock in it at the 2048-pixel scale, a number obtained by listing a directory, and the whole store is 9.5 MB - alongside the 27 MB burn object, the 5.6 MB DEFLATE COG, and the 613 MB uncompressed AADC product from last time, it is another point on the same continuum.

Coda: the COG is now cheap

If the COG is still the deliverable, the expensive part - materializing every tile - has already happened in parallel. What remains is one sequential translate, I/O-bound, with the COG driver’s own threads handling compression and overviews:

set_config_option("GDAL_NUM_THREADS", "ALL_CPUS")
translate(zarr_path, file.path(tempdir(), "rema_approx.tif"),
          cl_arg = c("-of", "COG", "-co", "BLOCKSIZE=512", "-co", "SPARSE_OK=YES",
                     "-co", "COMPRESS=DEFLATE", "-co", "OVERVIEW_RESAMPLING=NEAREST",
                     "-co", "NUM_THREADS=ALL_CPUS"))

Things I have not done yet, and want to:

• Time this step, so there is an honest end-to-end “how long to get a COG” number against the original serial loop.
• Check whether reading a sparse Zarr into the COG driver preserves sparsity as well as writing blocks directly did (SPARSE_OK=YES should skip the absent chunks, which come back as nodata).
• The Zarr driver has no classic-API overviews; for a Zarr-native product the pyramid would be additional arrays (the multiscales convention in GDAL 3.13 for Zarr V3), which is a separate post.

Appendix: populating without GDAL in the workers

The other question I set out with was whether GDAL could instantiate the store and something more generic could fill it. It can. A Zarr V2 chunk is: the full chunk shape (edge chunks padded with fill_value), C order, the declared dtype (<i4 here), passed through the declared compressor, saved as row.col in the array directory, 0-based. If the store is created with COMPRESS=ZLIB, base R already has the encoder, because memCompress(type = "gzip") emits a zlib stream, which is what the numcodecs zlib codec expects.

write_chunk_raw <- function(mat, row_chunk, col_chunk, array_dir, chunk_size) {
  full <- matrix(0L, chunk_size, chunk_size)
  full[seq_len(nrow(mat)), seq_len(ncol(mat))] <- mat
  bytes <- writeBin(as.integer(t(full)), raw(), size = 4L, endian = "little")
  writeBin(memCompress(bytes, type = "gzip"),
           file.path(array_dir, sprintf("%d.%d", row_chunk, col_chunk)))
}

This has no shared writes of any kind, not even the metadata files, and it is the mirror image of a Zarr reader, which is where I am headed with zaro. Open questions to explore:

• ZSTD or Blosc from R without pulling in a heavy dependency, since ZLIB is the slow codec.
• Whether the worker should write chunks at all, or emit (chunk key, bytes) pairs so the sink can be a local directory, an object store, or something that builds a kerchunk/Icechunk reference set instead.
• Doing the same for the coverage-mode burn (Float32, <f4), where the chunks are no longer mostly a single value and the compressor choice matters more.

Setup

Three cloud-hosted datasets, read directly via GDAL virtual filesystem paths.

# REMA 32m DEM mosaic — we read only the grid spec, no pixel values
dsn <- paste0(
  "/vsicurl/https://raw.githubusercontent.com/mdsumner/rema-ovr/",
  "main/rema-vrt/32m_dem_tiles.vrt")

# Antarctic rock outcrop polygons (v7.3, Burton-Johnson et al.)
shp <- paste0(
  "/vsizip/{/vsicurl/https://github.com/AustralianAntarcticDivision/",
  "rema.proc/raw/refs/heads/master/01_rock_classification/",
  "Medium_resolution_vector_polygons_of_Antarctic_rock_outcrop_-_",
  "VERSION_7.3.zip}/Medium resolution vector polygons of Antarctic ",
  "rock outcrop - VERSION 7.3/",
  "add_rock_outcrop_medium_res_polygon_v7.3.gpkg")

# REMA tile index — for choosing a view window
rema_tiles <- paste0(
  "/vsizip/{/vsicurl/https://data.pgc.umn.edu/elev/dem/setsm/REMA/",
  "indexes/REMA_Mosaic_Index_latest_shp.zip}/REMA_Mosaic_Index_v2_shp")

Read

The raster is never opened — vapour_raster_info() reads only the metadata (extent, dimensions, CRS). The polygons are read as raw WKB bytes.

library(terra)

terra 1.9.44

tiles <- terra::vect(rema_tiles)

Warning: [vect] Reading layer: REMA_Mosaic_Index_v2_2m
Other layers: REMA_Mosaic_Index_v2_10m, REMA_Mosaic_Index_v2_32m

raster_info <- vapour::vapour_raster_info(dsn)
rock_info <- vapour::vapour_layer_info(shp)

Warning: Unhandled projection method Polar_Stereographic (GDAL error 6)

Warning: unable export SRS to XML

rock <- wk::wkb(vapour::vapour_read_geometry(shp),
                 crs = rock_info$projection$Wkt)

cat(sprintf("Grid: %s × %s pixels (%.0f billion cells)\n",
    format(raster_info$dimension[1], big.mark = ","),
    format(raster_info$dimension[2], big.mark = ","),
    prod(as.numeric(raster_info$dimension)) / 1e9))

Grid: 170,318 × 182,568 pixels (31 billion cells)

cat(sprintf("Polygons: %s geometries\n",
    format(length(rock), big.mark = ",")))

Polygons: 25,954 geometries

Rasterize the continent

One call rasterizes every polygon onto the full continental grid. The output is a sparse table of run-length encoded interior cells — no dense matrix is allocated, no pixel values are touched.

library(controlledburn)

system.time({
  ba <- burn(rock, extent = raster_info$extent,
             dimension = raster_info$dimension, mode = "approx")
})

   user  system elapsed 
  0.294   0.056   0.350 

ba

<controlledburn> 170318 x 182568 grid, 25954 geometries
  runs:   1704432 (70650819 interior cells)
  edges:  0 polygon boundary cells
  sparsity: 99.8% empty

cat(sprintf("\nObject size: %.0f MB\n", as.numeric(lobstr::obj_size(ba)) / 1e6))

Object size: 27 MB

Coverage mode computes exact boundary fractions — slower, but the total burned area equals the true polygon area to floating-point precision:

system.time({
  be <- burn(rock, extent = raster_info$extent,
             dimension = raster_info$dimension, mode = "coverage")
})

   user  system elapsed 
  4.484   0.287   4.771 

be

<controlledburn> 170318 x 182568 grid, 25954 geometries
  runs:   1662668 (66525907 interior cells)
  edges:  6548354 polygon boundary cells
  sparsity: 99.8% empty

Look at a tile

The sparse output covers the whole continent but we can extract any sub-window instantly with crop_burn(). Here: a 50 km REMA tile of the Windmill Islands, near Casey station.

ll <- cbind(lon = 110.3, lat = -66.3)
xy <- reproj::reproj_xy(ll, crs(tiles), source = "EPSG:4326")
idx <- which(terra::is.related(tiles, vect(xy), "intersects"))
tile_ex <- as.vector(ext(tiles[idx, ]))

ma <- materialize_chunk(crop_burn(ba, tile_ex))
cat(sprintf("Tile: %d × %d pixels\n", ncol(ma), nrow(ma)))

Tile: 1570 × 1570 pixels

# Rock polygons that overlap this tile
rock_idx <- unlist(
  geos::geos_strtree_query(
    geos::geos_strtree(geos::as_geos_geometry(rock)),
    tiles[idx, ]))

par(bg = "#1a1a2e", mar = c(3, 3, 2, 1))
plot(tiles[idx, ], col = NA, border = NA,
     main = "Windmill Islands rock outcrop — 32m",
     col.main = "white", col.axis = "grey60")
ximage::ximage(ma, tile_ex, add = TRUE,
               col = c("#1a1a2e", "#e8d5b7"))

plot(rock[rock_idx], add = TRUE, border = "#e05780", lwd = 0.6)

mc <- materialize_chunk(crop_burn(be, tile_ex))

par(bg = "#1a1a2e", mar = c(3, 3, 2, 1))
plot(tiles[idx, ], col = NA, border = NA,
     main = "Windmill Islands rock outcrop — 32m, exact coverage",
     col.main = "white", col.axis = "grey60")
ximage::ximage(mc, tile_ex, add = TRUE,
               col = hcl.colors(64, "Lajolla", rev = TRUE))
plot(rock[rock_idx], add = TRUE, border = "#e05780", lwd = 0.6)

At 32m resolution the two modes are nearly identical. The difference is sub-pixel boundary treatment:

n_diff <- sum(ma != mc)
cat(sprintf("Cells that differ: %s out of %s (%.3f%%)\n",
    format(n_diff, big.mark = ","),
    format(length(ma), big.mark = ","),
    100 * n_diff / length(ma)))

Cells that differ: 7,360 out of 2,464,900 (0.299%)

Summary of the lazy rasterization process

controlledburn burned 25,954 polygons onto a 31-billion-cell grid in under a second, producing a sparse run-length table small enough to hold in memory. Any tile can be extracted and materialized on demand. The raster was never read — only its grid specification. The polygons were never validated or reprojected — just raw WKB bytes from a cloud-hosted GeoPackage.

This is rasterization as an O(perimeter) operation: the cost scales with the complexity of the polygon boundaries, not the number of pixels in the grid. The sparse output is a reusable intermediate — downstream processing (zonal statistics, masking, tiling) operates on the same table without re-burning.

Creating a raster file

Now we have a very dense version of our rock layer as a latent raster, we can materialize arbitrary windows of it and write them to an actual raster file. The strategy: create a SPARSE_OK Cloud-Optimized GeoTIFF (COG), iterate over spatial tiles, crop the sparse burn result to each tile, materialize the dense matrix, and write it via gdalraster. Tiles with no burned pixels are skipped entirely — SPARSE_OK means unwritten tiles cost nothing on disk.

library(controlledburn)
library(gdalraster)

outfile <- file.path(tempdir(), "rema_approx.tif")
block_size <- 512L

ext <- ba$extent
dm  <- ba$dimension
gdalraster::set_config_option("GDAL_NUM_THREADS", "ALL_CPUS")

ds <- create(
  format       = "COG",
  dst_filename = outfile,
  xsize = dm[1], ysize = dm[2],
  nbands   = 1,
  dataType = "Int32",
  options  = c(paste0("BLOCKSIZE=", block_size),
               "SPARSE_OK=YES",
               "COMPRESS=DEFLATE",
               "OVERVIEW_RESAMPLING=NEAREST",
               "NUM_THREADS=ALL_CPUS"),
  return_obj = TRUE
)

ds$setGeoTransform(vaster::extent_dim_to_gt(ext, dm))
ds$setNoDataValue(band = 1, 0)

superblock <- block_size * 4
gg    <- grout::grout(dm, ext, blocksize = superblock)
tiles <- grout::tile_index(gg)

# Keep only tile-rows that intersect at least one data row
data_rows <- unique(c(ba$runs$row, ba$edges$row,
                      ba$lines$row, ba$points$row))
tiles <- tiles[tiles$tile_row %in%
                 unique(ceiling(data_rows / superblock)), ]

tiles_written <- 0L
for (i in seq_len(nrow(tiles))) {
  sub <- crop_burn(ba, c(tiles$xmin[i], tiles$xmax[i],
                         tiles$ymin[i], tiles$ymax[i]))

  if (nrow(sub$runs) == 0 && nrow(sub$edges) == 0) next

  mat <- materialize_chunk(sub, fun = "id")
  mat[is.na(mat)] <- 0

  ds$write(band = 1,
           xoff = tiles$offset_x[i], yoff = tiles$offset_y[i],
           xsize = tiles$ncol[i], ysize = tiles$nrow[i],
           rasterData = as.integer(t(mat)))

  tiles_written <- tiles_written + 1L
}

ds$flushCache()
ds$close()

## ~445 seconds elapsed
## tiles written: 3,482 of 7,511
## file size: 26,782 KB
## output: 170318 × 182568 Int32, nodata=0

Extracting point values

Not every downstream operation needs a dense raster. extract_burn() queries the sparse tables directly at arbitrary point locations, with no materialisation step at all. It matches interior runs by row equality and column interval containment, and edges/lines/points by exact (row, col) match. Cost is O(points + hits), independent of grid dimension.

# Some coordinates in the burn's CRS (Antarctic Polar Stereographic)
pts <- structure(c(2412867, 2426358, 2424194, 2437557, 2437175, 2441121, 
2443539, 2443284, -911354, -909445, -915172, -913263, -913136, 
-912245, -914281, -920645), dim = c(8L, 2L))

plot(pts, asp = 1)
ximage::ximage(vapour::gdal_raster_nara(sds::wms_arcgis_mapserver_ESRI.WorldImagery_tms(), target_ext = tile_ex, target_crs = raster_info$projection, target_dim = c(512, 512)),add = TRUE)
points(pts, col = "green", pch = "+", cex = 2)
plot(rock[rock_idx], add = TRUE, border = "#e05780", lwd = 1.6)

e_approx <- extract_burn(ba, pts, fun = "id")

e_cover <- extract_burn(be, pts, fun = "sum")
cbind(approx = e_approx, cover = e_cover)

     approx     cover
[1,]  19038 1.0000000
[2,]  19024 1.0000000
[3,]     NA 0.0000000
[4,]  19048 1.0000000
[5,]  19048 1.0000000
[6,]  19051 0.5053515
[7,]  19059 1.0000000
[8,]     NA 0.0000000

The function returns NA for background cells (no geometry touched that pixel) and the geometry ID for cells inside a polygon. With fun = "sum", coverage-mode burns return exact fractional coverage at each point.

This pattern generalizes. Because the sparse representation is a reusable intermediate, other raster operations — zonal statistics, reclassification, distance transforms — can be built on top of the same sparse tables without re-burning or allocating a full dense grid. The burn is computed once; everything else is derived.

terra for example can tell us those exact same polygon identities, but it can’t easily tell us what coverage that polygon has in the related pixel (that’s a different question, we’d need to rasterize first …), we have different opportunities with this alternative representation.

terra::extract(terra::vect(geos::as_geos_geometry(rock)), pts)

     id.y  id.x
[1,]    1 19038
[2,]    2 19024
[3,]    3   NaN
[4,]    4 19048
[5,]    5 19048
[6,]    6 19051
[7,]    7 19059
[8,]    8   NaN

Packing, compression, and sparsity

A useful comparison: the Australian Antarctic Data Centre publishes Ice-free Antarctica (Toth & Terauds 2023) — a union of rock outcrop layers from imagery spanning 1960–2020, encoded as a single-band GeoTIFF at 100 m resolution (52765 × 48132 pixels). Values are 0 (ice/ocean), 1, 2, or 3 (rock, by source layer), stored at 2 bits per sample (NBITS=2) in 128 × 128 tiles. The file is 613 MB.

That 2-bit packing is already a 4× saving over a byte raster — but it is a fixed per-pixel economy. Every tile is materialized: ocean, ice sheet, and rock alike each cost their full 4096 bytes. The encoding is indifferent to the fact that the vast majority of pixels say the same thing: zero. Recompressing the same tiles with DEFLATE collapses the file to about 5.6 MB — a ~100× reduction with zero information loss, because constant regions become long runs of identical bytes, and long runs are the best possible input to any general-purpose codec.

Our controlledburn COG encodes different (but related) data at 32 m resolution — a grid 3x denser in each dimension for 26 MB on disk. The sparse burn object in memory is 27 MB. These numbers are not directly comparable (different polygons, different resolution, different pixel semantics), but the relationship between them is the point: the information content of “where is the rock?” is small relative to the grid that contains it, regardless of how you choose to encode it.

The three approaches — bit-packing, codec compression, and run-length sparse representation — are related lenses on the same underlying reality. Bit-packing reduces the cost per pixel but remains grid-dense. Compression recovers structure after the fact, collapsing constant regions that the format didn’t know to avoid. Sparse rasterization exploits the same structure before pixels exist: polygon interiors are runs by construction, so the compact representation is produced directly rather than recovered by a codec.

None of these is categorically superior. The AADC file is a finished product, designed for broad interoperability and direct random-access reads — the choice not to compress may well reflect downstream tooling constraints or the simplicity of a uniform tile stride. controlledburn’s sparse format is an intermediate: it is meant to sit between the vector source and whatever raster product or analysis comes next. The value is in seeing them as points on the same continuum — rasterization, compression, and sparse encoding are all ways of negotiating between the geometry of the data and the geometry of the grid.

Prior art: Dewey’s posts

Most of what follows was worked out first, and better, by Dewey Dunnington across a series of posts that double as the s2 package’s missing narrative documentation. If this guide is the map, these are the survey expeditions:

• wk version 0.6.0 (Dec 2021) is where the geodesic attribute enters WKT/WKB – added specifically so that wk could serve as an interchange format for s2, where vertices are joined by great circles rather than planar segments. If you have ever wondered why plot(wk::as_wkb(some_s2_thing)) refuses to cooperate, the reasoning starts here.

• Profiling point-in-polygon joins in R (Mar 2022) benchmarks geos against s2 for a 9-million-point join, is honest about where s2’s point handling was slow at the time, and foreshadows exposing the cell machinery: approximate a polygon with a covering of cells and test membership as cell arithmetic. That idea is the core of the polygon recipe below.

• s2 version 1.1.0 (Jul 2022) is the densest single source: tessellate_tol_m for importing planar-edged data honestly (adding vertices so GIS-rule edges survive the trip to the sphere), the cell polygon export problem (geodesic by definition, planar by expectation), the s2_union_agg() speedup, and the refactor of the internals into a standalone s2geography library – the lineage that later surfaces in DuckDB and Sedona.

• Partitioning strategies for bigger-than-memory spatial data and Wrangling and joining 130M points with DuckDB + the open source spatial stack (both Dec 2024) take the cell idea to its logical end: s2 cells as partition keys and join keys for larger-than-memory data, via the DuckDB Geography extension. This is the “cells are just a column” worldview that motivates keeping cell IDs in parquet in the first place.

• The GeoPython 2025 slides on DuckDB Geography give the compact cross-language overview of the Geography and S2 Cell/Center/Union types.

The through-line in all of these: s2’s cell types are not an implementation detail, they are the product – a spatial index you can store, ship, and group by. This post is about using them directly from R without the database in between.

The three type families

Everything in s2 is one of these:

1. s2_cell – a vctrs vector of doubles whose bits are reinterpreted as uint64 S2 cell IDs. This is the discrete global grid side of s2: a hierarchical quadtree on the six faces of a cube projected to the sphere, levels 0 (face) to 30 (leaf, roughly centimetre scale). Every function named s2_cell_* operates here. Cells are cheap: they sort, hash, group, and round-trip through character and bit64::integer64 losslessly. This is the type you get from BigQuery, from parquet exports, from anything that indexes points spatially.

2. s2_geography – external pointers to C++ S2 geography objects (points, polylines, polygons on the sphere). Every function with a bare s2_ prefix (s2_intersects(), s2_buffer_cells(), s2_x(), …) operates here, or coerces its input here via as_s2_geography(). The coercion is why half-wrong calls half-work: s2_x("POINT (147 -42)") succeeds because character is assumed to be WKT.

3. wk vctrs – plain coordinate vectors from the wk package, carrying a crs attribute. s2_lnglat(lng, lat) is just wk::xy() with a longlat crs; s2_point(x, y, z) is wk::xyz() with a special “unit sphere” crs. These are the plottable, data-frame-able, “normal” things. They are the exits from s2 back into everything else.

The mnemonic: s2_cell_* functions take cells, as_* functions change family, everything else takes geography.

Worth internalizing: s2_lnglat() and s2_point() are constructors from coordinates, not converters. s2_lnglat(res$cell[1]) doesn’t error helpfully, it just constructs nonsense (a point at longitude whatever-your-cell-double-is). The converters are all spelled as_: as_s2_cell(), as_s2_lnglat(), as_s2_point(), as_s2_geography(), as_s2_cell_union().

The conversion table

Notes on the sharp edges:

• as_s2_cell() on coordinates or point geographies gives the level-30 leaf cell; use s2_cell_parent(cell, level) to coarsen. Parent of a parent is fine; you can’t go below your current level except via s2_cell_child().
• as_s2_cell() on integer64 reinterprets bits directly – so a cell-ID column read from parquet as integer64 converts with no string round-trip. If arrow hands you character tokens, as_s2_cell() handles that too.
• s2_cell_to_lnglat() is the centre point in one hop. You do not need s2_cell_center() (which makes geography pointers) unless you want to feed the result into geography predicates.

The geodesic flag, or: why did my plot fail

wk::as_wkb() on an s2 geography tags the result geodesic = TRUE, because that’s the truth: S2 edges are great-circle arcs, not straight lines in longitude/latitude space. wk then refuses to plot it, because plotting geodesic edges as straight segments would be a lie and wk won’t densify them for you.

For points the edge interpretation is vacuous, so the refusal is technically principled and practically infuriating. The escape hatch:

plot(wk::wk_set_geodesic(wk::as_wkb(geog), FALSE))

For points this is always fine. For lines and polygons it is fine when your features are small (a level-10+ cell polygon spans a few km; nothing visible bends at that scale) and a lie when they are large (a level-0 face “polygon” drawn with straight lonlat edges is wildly wrong). The honest fix for large features is densification, which is section 5.

Recipe 1: raster grids, points, and polygons to s2 groupings

The pattern for all of these is: get to lonlat coordinates or geography, snap to leaf cells, coarsen to a working level, then treat the cell column as an ordinary grouping variable.

library(s2)

## from bare vectors -- one line to cells at your working level
cells <- s2_cell_parent(as_s2_cell(s2_lnglat(lon, lat)), level = 8)

## then it's just dplyr
d |>
  mutate(cell = s2_cell_parent(as_s2_cell(s2_lnglat(lon, lat)), 8)) |>
  summarize(sst = mean(sst), n = n(), .by = cell) |>
  mutate(centre = s2_cell_to_lnglat(cell))

xy <- vaster::xy_from_cell(dimension, extent, cells = seq_len(prod(dimension)))
s2cell <- s2_cell_parent(as_s2_cell(s2_lnglat(xy[, 1], xy[, 2])), level = 7)

## aggregate a value vector (e.g. one time slice of SST) onto the s2 grid
agg <- tapply(values, s2cell, mean, na.rm = TRUE)
centres <- s2_cell_to_lnglat(as_s2_cell(names(agg)))

g <- as_s2_geography(wkt_or_wkb_or_sf_geometry)

## interior/exterior covering at bounded resolution
cov <- s2_covering_cell_ids(g, min_level = 4, max_level = 10, max_cells = 64)
cov_int <- s2_covering_cell_ids(g, max_cells = 64, interior = TRUE)

## which obs cells fall in the polygon covering? no geometry predicates,
## just uint64 range containment
inside <- s2_cell_union_contains(cov, as_s2_cell_union(obs_cells))

Recipe 2: s2 to rgl (plotting on the actual sphere)

The bridge is as_s2_point(): it converts longlat wk_xy to unit-sphere xyz – which is exactly what rgl wants, no trig required.

library(rgl)

## cell centres as points on the sphere
xyz <- as.matrix(as.data.frame(as_s2_point(s2_cell_to_lnglat(cells))))
points3d(xyz, col = palr::sst_pal(64)[scales::rescale(vals, c(1, 64)) |> round()])
spheres3d(0, 0, 0, radius = 0.999, col = "grey20")  ## backdrop globe

Cells as quads is barely more work, because s2_cell_vertex(x, k) gives vertex k (0..3) as a point geography, and quads want exactly that:

cell_quads3d <- function(cells, ...) {
  v <- lapply(0:3, function(k) {
    as.matrix(as.data.frame(as_s2_point(as_s2_lnglat(s2_cell_vertex(cells, k)))))
  })
  ## interleave to vertex-per-row quad order: v0 v1 v2 v3 per cell
  m <- matrix(t(do.call(cbind, v)), ncol = 3L, byrow = TRUE)
  quads3d(m, ...)
}
cell_quads3d(cells, col = "firebrick", alpha = 0.8)

Caveats, both about coarse cells:

• A quad’s edges are drawn as chords through the sphere, so coarse cells (level < ~4) visibly cut the ball and their “flat” faces sit below the unit sphere. The fix is subdivision: replace each coarse cell with its children at a uniform deeper level (s2_cell_child() recursively, or just build your grouping at a deeper level and aggregate the values coarsely instead).
• For genuinely smooth cell patches you want the quadmesh treatment – densify each cell face in (face, i, j) space and push vertices to the sphere – which is an anglr/quadmesh shaped job and a natural follow-up post.

The general s2-geography-to-rgl path for arbitrary lines/polygons is: densify edges first (next section), then wk::wk_vertices(), as_s2_point(), and index into segments3d()/polygon3d(). Densify-first matters for the same chord-vs-arc reason.

Recipe 3: geodesic densification, bigcurve, and wkpool

Here’s the conceptual unification. wk’s geodesic flag marks “these edges are great circles, do not interpolate them linearly in lonlat”. bigcurve’s whole job is adaptive line densification under a coordinate transformation – insert vertices until the straight segments approximate the true curve to tolerance. A geodesic edge is precisely this problem where the transformation is sphere to plate carree: the true curve is the great circle, the drawing space is lonlat.

So the missing verb has an obvious signature:

## proposed: consume geodesic-flagged wkb, emit densified planar wkb
flatten_geodesic <- function(x, tolerance_m = 10000) {
  stopifnot(isTRUE(wk::wk_is_geodesic(x)))
  ## densify each edge along the great circle to tolerance, then
  wk::wk_set_geodesic(densified, FALSE)
}

with the inverse-ish operation being what bigcurve already does for projections (densify in the projected space). Composed, you get the honest pipeline for “draw this s2 polygon in LAEA”:

s2_geography --[flatten_geodesic, tol]--> planar lonlat wkb
             --[bigcurve densify under proj, tol]--> projected wkb

Three candidate engines for the great-circle interpolation itself:

1. s2 itself: s2_interpolate_normalized(line, t) walks a polyline by fraction – correct but per-feature-scalar-ish and awkward for edge-wise adaptive refinement.
2. geographiclib: direct/inverse problems on the ellipsoid – and this is the point where you decide whether “geodesic” means sphere (s2’s world) or ellipsoid (GeographicLib’s). For rendering, sphere is fine and consistent with the flag’s provenance; for measurement, s2 already defers radius choices to you.
3. a small C++ slerp in bigcurve’s core: unit-vector slerp between edge endpoints is ~5 lines, exactly matches S2’s edge definition, and fits bigcurve’s adaptive bisection (recurse while the chord midpoint deviates from the slerp midpoint by > tol). This is the one I’d do: the s2 edge model is slerp, so bigcurve gains geodesic support without a dependency, and wkpool provides the vertex pooling for shared edges (think: densifying a lattice of cell polygons without duplicating every shared boundary).

A cell-union boundary is a mesh of shared arcs, and densifying per-polygon quadruples the work and the vertex count. Pool first, densify unique edges once, reconstruct. That’s the silicate instinct applied to the sphere, and it would make “draw 10k cell polygons in any projection, correctly” cheap.

I think: flatten_geodesic() (or wk_geodesic_densify()) belongs API-wise with wk-adjacent tooling, implementation-wise in bigcurve’s C++ core, with wkpool underneath for shared-edge economy. And its existence retroactively makes wk’s plot refusal feel fair – the refusal is an unfilled socket, and this is the plug.

Appendix: the four verbs

A core of wrappers is tiny:

## cell column (character/integer64/cell) -> lonlat centres
cell_lonlat <- function(x) s2::s2_cell_to_lnglat(s2::as_s2_cell(x))

## lon, lat -> cells at level
lonlat_cell <- function(x, y, level = 30L) {
  s2::s2_cell_parent(s2::as_s2_cell(s2::s2_lnglat(x, y)), level)
}

## cell column -> plottable planar polygons (small-cell assumption)
cell_poly <- function(x) {
  wk::wk_set_geodesic(wk::as_wkb(s2::s2_cell_polygon(s2::as_s2_cell(x))), FALSE)
}

## anything s2 -> unit sphere xyz matrix for rgl
sphere_xyz <- function(x) {
  as.matrix(as.data.frame(s2::as_s2_point(s2::as_s2_lnglat(wk::as_xy(x)))))
}

I’m not sure these need a package so much as a place to be seen.

What is Icechunk?

Zarr is a chunked array format designed for cloud object storage. Icechunk is a Rust library that sits on top of Zarr and adds git-like snapshots combined with ACID transactions - (which means reliable and guaranteed read and write).

(From their docs) this provides:

• Safety: Recover from corrupted or accidentally deleted data using version history
• Consistency: Ensure readers always see complete, valid snapshots - never partial writes
• Reproducibility: Reference any version of your data permanently via commits or tags

Typical usage involves the Python packages xarray and icechunk, but Rust Icechunk can be used directly, in combination with zarrs_icechunk, and now in an upcoming GDAL release 3.14 via the read-only Icechunk driver.

Please be aware that Icechunk in GDAL is very new, and not yet in an officially supported release but if you are interested to get involved it can be used today. (GDAL 3.14 is milestoned for sometime in November 2026).

What works in GDAL and R today

GDAL 3.14 ships a /vsiicechunk/ virtual filesystem and Icechunk driver, and will be available in gdalcubes, gdalraster, sf, terra, and vapour in various forms once that support lands in R packages.

The following examples can be run in this Linux docker container, for example:

docker pull ghcr.io/hypertidy/gdal-r-full:dev
docker run --rm -ti  ghcr.io/hypertidy/gdal-r-full:dev

(If that doesn’t work, or you want help with docker/GDAL/R just contact me directly)

Here is a single cli call that gives a GDAL multidimensional summary of an Icechunk store.

gdal mdim info \
  /vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk \
  --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-west-2

Driver: Icechunk

Dimensions:
  Name (path)  Size      Type      Direction
  -----------  ----  ------------  ---------
  /init_time   3264
  /latitude     721  HORIZONTAL_Y
  /lead_time     61
  /longitude   1440  HORIZONTAL_X

Coordinates (indexing variables):
  Name (path)   Dimension    Type              Unit
  -----------  -----------  -------  ------------------------
  /init_time   (init_time)  Int64    seconds since 1970-01-01
  /latitude    (latitude)   Float64  degree_north
  /lead_time   (lead_time)  Float64  seconds
  /longitude   (longitude)  Float64  degree_east

Data variables:
                  Name (path)                   Type              Unit                    Shape             Chunk size
  -------------------------------------------  -------  ------------------------  ---------------------  -----------------

 (/init_time):
  /expected_forecast_length                    Float64  seconds                   [3264]                 [23360]
  /ingested_forecast_length                    Float64  seconds                   [3264]                 [23360]

 (/init_time, /lead_time):
  /valid_time                                  Int64    seconds since 1970-01-01  [3264, 61]             [23360, 61]

 (/init_time, /lead_time, /latitude, /longitude):
  /dew_point_temperature_2m                    Float32  degree_Celsius            [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /downward_long_wave_radiation_flux_surface   Float32  W m-2                     [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /downward_short_wave_radiation_flux_surface  Float32  W m-2                     [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /geopotential_height_500hpa                  Float32  m                         [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /geopotential_height_850hpa                  Float32  m                         [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /geopotential_height_925hpa                  Float32  m                         [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /precipitation_surface                       Float32  kg m-2 s-1                [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /pressure_reduced_to_mean_sea_level          Float32  Pa                        [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /pressure_surface                            Float32  Pa                        [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /temperature_2m                              Float32  degree_Celsius            [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /temperature_850hpa                          Float32  degree_Celsius            [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /temperature_925hpa                          Float32  degree_Celsius            [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /total_cloud_cover_atmosphere                Float32  percent                   [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /wind_u_100m                                 Float32  m s-1                     [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /wind_u_10m                                  Float32  m s-1                     [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /wind_v_100m                                 Float32  m s-1                     [3264, 61, 721, 1440]  [1, 61, 241, 240]
  /wind_v_10m                                  Float32  m s-1                     [3264, 61, 721, 1440]  [1, 61, 241, 240]

Scalar arrays:
  Name (path)   Type   Unit
  ------------  -----  ----
  /spatial_ref  Int64

Attributes:
         Name           Type                                      Value
  -------------------  ------  ----------------------------------------------------------------------------
  attribution          String  "ECMWF AIFS Single forecast data processed by dynamical.org from ECMWF Open
                               Data."
  dataset_id           String  "ecmwf-aifs-single-forecast"
  dataset_version      String  "0.1.0"
  description          String  "Weather forecasts from the ECMWF Artificial Intelligence Forecasting System
                               (AIFS) Single model."
  forecast_domain      String  "Forecast lead time 0-360 hours (0-15 days) ahead"
  forecast_resolution  String  "6 hourly"
  license              String  "CC-BY-4.0"
  name                 String  "ECMWF AIFS Single forecast"
  spatial_domain       String  "Global"
  spatial_resolution   String  "0.25 degrees (~20km)"
  time_domain          String  "Forecasts initialized 2024-04-01 00:00:00 UTC to Present"
  time_resolution      String  "Forecasts initialized every 6 hours"

Arrays:

  - /init_time:
      Dimensions:    (/init_time)
      Shape:         [3264]
      Chunk size:    [23360]
      Type:          Int64
      Unit:          seconds since 1970-01-01
      Nodata value:  0

      Attributes:
                 Name            Type                       Value
        ----------------------  ------  ---------------------------------------------
        calendar                String  "proleptic_gregorian"
        long_name               String  "Forecast initialization time"
        standard_name           String  "forecast_reference_time"
        statistics_approximate  String  {"min":"2024-04-01T00:00:00","max":"Present"}

      Structural metadata:
        COMPRESSOR  { "name": "blosc", "configuration": { "typesize": 8, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } }

  - /latitude:
      Dimensions:    (/latitude)
      Shape:         [721]
      Chunk size:    [721]
      Type:          Float64
      Unit:          degree_north
      Nodata value:  "NaN"

      Attributes:
                 Name            Type          Value
        ----------------------  ------  --------------------
        _FillValue              String  "AAAAAAAA+H8="
        axis                    String  "Y"
        long_name               String  "Latitude"
        statistics_approximate  String  {"min":-90,"max":90}

      Structural metadata:
        COMPRESSOR  { "name": "blosc", "configuration": { "typesize": 8, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } }

  - /lead_time:
      Dimensions:    (/lead_time)
      Shape:         [61]
      Chunk size:    [61]
      Type:          Float64
      Unit:          seconds
      Nodata value:  "NaN"

      Attributes:
                 Name            Type                         Value
        ----------------------  ------  --------------------------------------------------
        _FillValue              String  "AAAAAAAA+H8="
        dtype                   String  "timedelta64[us]"
        long_name               String  "Forecast lead time"
        standard_name           String  "forecast_period"
        statistics_approximate  String  {"min":"0 days 00:00:00","max":"15 days 00:00:00"}

      Structural metadata:
        COMPRESSOR  { "name": "blosc", "configuration": { "typesize": 8, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } }

  - /longitude:
      Dimensions:    (/longitude)
      Shape:         [1440]
      Chunk size:    [1440]
      Type:          Float64
      Unit:          degree_east
      Nodata value:  "NaN"

      Attributes:
                 Name            Type             Value
        ----------------------  ------  -------------------------
        _FillValue              String  "AAAAAAAA+H8="
        axis                    String  "X"
        long_name               String  "Longitude"
        statistics_approximate  String  {"min":-180,"max":179.75}

      Structural metadata:
        COMPRESSOR  { "name": "blosc", "configuration": { "typesize": 8, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } }

  - /dew_point_temperature_2m:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          degree_Celsius
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "2 metre dewpoint temperature"
        short_name     String  "2d"
        standard_name  String  "dew_point_temperature"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /downward_long_wave_radiation_flux_surface:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          W m-2
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Surface downward long-wave radiation flux"
        short_name     String  "sdlwrf"
        standard_name  String  "surface_downwelling_longwave_flux_in_air"
        step_type      String  "avg"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /downward_short_wave_radiation_flux_surface:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          W m-2
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Surface downward short-wave radiation flux"
        short_name     String  "sdswrf"
        standard_name  String  "surface_downwelling_shortwave_flux_in_air"
        step_type      String  "avg"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /expected_forecast_length:
      Dimensions:    (/init_time)
      Shape:         [3264]
      Chunk size:    [23360]
      Type:          Float64
      Unit:          seconds
      Nodata value:  "NaN"

      Attributes:
                 Name            Type                         Value
        ----------------------  ------  --------------------------------------------------
        _FillValue              String  "AAAAAAAA+H8="
        dtype                   String  "timedelta64[us]"
        long_name               String  "Expected forecast length"
        statistics_approximate  String  {"min":"0 days 00:00:00","max":"15 days 00:00:00"}

      Structural metadata:
        COMPRESSOR  { "name": "blosc", "configuration": { "typesize": 8, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } }

  - /geopotential_height_500hpa:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          m
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Geopotential height"
        short_name     String  "gh"
        standard_name  String  "geopotential_height"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /geopotential_height_850hpa:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          m
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Geopotential height"
        short_name     String  "gh"
        standard_name  String  "geopotential_height"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /geopotential_height_925hpa:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          m
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Geopotential height"
        short_name     String  "gh"
        standard_name  String  "geopotential_height"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /ingested_forecast_length:
      Dimensions:    (/init_time)
      Shape:         [3264]
      Chunk size:    [23360]
      Type:          Float64
      Unit:          seconds
      Nodata value:  "NaN"

      Attributes:
                 Name            Type                         Value
        ----------------------  ------  --------------------------------------------------
        _FillValue              String  "AAAAAAAA+H8="
        dtype                   String  "timedelta64[us]"
        long_name               String  "Ingested forecast length"
        statistics_approximate  String  {"min":"0 days 00:00:00","max":"15 days 00:00:00"}

      Structural metadata:
        COMPRESSOR  { "name": "blosc", "configuration": { "typesize": 8, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } }

  - /precipitation_surface:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          kg m-2 s-1
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                       Value
        -------------  ------  ------------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        comment        String  "Average precipitation rate since the previous forecast step. Units equivalent
                               to mm/s."
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Precipitation rate"
        short_name     String  "prate"
        standard_name  String  "precipitation_flux"
        step_type      String  "avg"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /pressure_reduced_to_mean_sea_level:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          Pa
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Pressure reduced to MSL"
        short_name     String  "prmsl"
        standard_name  String  "air_pressure_at_mean_sea_level"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /pressure_surface:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          Pa
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Surface pressure"
        short_name     String  "sp"
        standard_name  String  "surface_air_pressure"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /spatial_ref:
      Dimensions:    ()
      Shape:         []
      Type:          Int64
      Nodata value:  0

      Attributes:
                   Name               Type                                        Value
        ---------------------------  -------  -----------------------------------------------------------------------------
        comment                      String   "This coordinate reference system matches the source data which follows WMO
                                              conventions of assuming the earth is a perfect sphere with a radius of 6,371,
                                              229m. It is similar to EPSG:4326, but EPSG:4326 uses a more accurate
                                              representation of the earth's shape."
        crs_wkt                      String   "GEOGCS[\"unknown\",DATUM[\"unknown\",SPHEROID[\"unknown\",6371229,0]],
                                              PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",
                                              0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AXIS[\"Longitude\",EAST],
                                              AXIS[\"Latitude\",NORTH]]"
        geographic_crs_name          String   "unknown"
        grid_mapping_name            String   "latitude_longitude"
        horizontal_datum_name        String   "unknown"
        inverse_flattening           Float64  0
        longitude_of_prime_meridian  Float64  0
        prime_meridian_name          String   "Greenwich"
        reference_ellipsoid_name     String   "unknown"
        semi_major_axis              Float64  6371229
        semi_minor_axis              Float64  6371229
        spatial_ref                  String   "GEOGCS[\"unknown\",DATUM[\"unknown\",SPHEROID[\"unknown\",6371229,0]],
                                              PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",
                                              0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AXIS[\"Longitude\",EAST],
                                              AXIS[\"Latitude\",NORTH]]"

      Structural metadata:
        COMPRESSOR  { "name": "zstd", "configuration": { "level": 0, "checksum": false } }

  - /temperature_2m:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          degree_Celsius
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "2 metre temperature"
        short_name     String  "2t"
        standard_name  String  "air_temperature"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /temperature_850hpa:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          degree_Celsius
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Temperature"
        short_name     String  "t"
        standard_name  String  "air_temperature"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /temperature_925hpa:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          degree_Celsius
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Temperature"
        short_name     String  "t"
        standard_name  String  "air_temperature"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /total_cloud_cover_atmosphere:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          percent
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "Total cloud cover"
        short_name     String  "tcc"
        standard_name  String  "cloud_area_fraction"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /valid_time:
      Dimensions:    (/init_time, /lead_time)
      Shape:         [3264, 61]
      Chunk size:    [23360, 61]
      Type:          Int64
      Unit:          seconds since 1970-01-01
      Nodata value:  0

      Attributes:
                 Name            Type                       Value
        ----------------------  ------  ---------------------------------------------
        calendar                String  "proleptic_gregorian"
        long_name               String  "Valid time"
        standard_name           String  "time"
        statistics_approximate  String  {"min":"2024-04-01T00:00:00","max":"Present"}

      Structural metadata:
        COMPRESSOR  { "name": "blosc", "configuration": { "typesize": 8, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } }

  - /wind_u_100m:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          m s-1
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "100 metre U wind component"
        short_name     String  "100u"
        standard_name  String  "eastward_wind"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /wind_u_10m:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          m s-1
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "10 metre U wind component"
        short_name     String  "10u"
        standard_name  String  "eastward_wind"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /wind_v_100m:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          m s-1
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "100 metre V wind component"
        short_name     String  "100v"
        standard_name  String  "northward_wind"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

  - /wind_v_10m:
      Dimensions:    (/init_time, /lead_time, /latitude, /longitude)
      Shape:         [3264, 61, 721, 1440]
      Chunk size:    [1, 61, 241, 240]
      Type:          Float32
      Unit:          m s-1
      Nodata value:  "NaN"

      Attributes:
            Name        Type                                     Value
        -------------  ------  --------------------------------------------------------------------------
        _FillValue     String  "AAAAAAAA+H8="
        coordinates    String  "expected_forecast_length ingested_forecast_length spatial_ref valid_time"
        long_name      String  "10 metre V wind component"
        short_name     String  "10v"
        standard_name  String  "northward_wind"
        step_type      String  "instant"

      Structural metadata:
        COMPRESSOR  { "name": "sharding_indexed", "configuration": { "chunk_shape": [ 1, 61, 241, 240 ], "codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "blosc", "configuration": { "typesize": 4, "cname": "zstd", "clevel": 3, "shuffle": "shuffle", "blocksize": 0 } } ], "index_codecs": [ { "name": "bytes", "configuration": { "endian": "little" } }, { "name": "crc32c" } ], "index_location": "end" } }

It’s a lot of output! For comparison let’s see what xarray will show as equivalent summary.

<xarray.Dataset> Size: 14TB
Dimensions:                                     (init_time: 3264,
                                                 latitude: 721, lead_time: 61,
                                                 longitude: 1440)
Coordinates:
  * init_time                                   (init_time) datetime64[ns] 26kB ...
  * latitude                                    (latitude) float64 6kB 90.0 ....
  * lead_time                                   (lead_time) float64 488B 0.0 ...
  * longitude                                   (longitude) float64 12kB -180...
Data variables: (12/21)
    dew_point_temperature_2m                    (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
    downward_long_wave_radiation_flux_surface   (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
    downward_short_wave_radiation_flux_surface  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
    expected_forecast_length                    (init_time) float64 26kB dask.array<chunksize=(3264,), meta=np.ndarray>
    geopotential_height_500hpa                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
    geopotential_height_850hpa                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
    ...                                          ...
    total_cloud_cover_atmosphere                (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
    valid_time                                  (init_time, lead_time) float32 796kB dask.array<chunksize=(3264, 61), meta=np.ndarray>
    wind_u_100m                                 (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
    wind_u_10m                                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
    wind_v_100m                                 (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
    wind_v_10m                                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3264, 61, 721, 1440), meta=np.ndarray>
Attributes:
    attribution:          ECMWF AIFS Single forecast data processed by dynami...
    dataset_id:           ecmwf-aifs-single-forecast
    dataset_version:      0.1.0
    description:          Weather forecasts from the ECMWF Artificial Intelli...
    forecast_domain:      Forecast lead time 0-360 hours (0-15 days) ahead
    forecast_resolution:  6 hourly
    license:              CC-BY-4.0
    name:                 ECMWF AIFS Single forecast
    spatial_domain:       Global
    spatial_resolution:   0.25 degrees (~20km)
    time_domain:          Forecasts initialized 2024-04-01 00:00:00 UTC to Pr...
    time_resolution:      Forecasts initialized every 6 hours

That’s a lot denser, but the important thing is that GDAL has access to the same rich underlying Zarr data and note that xarray has clipped the full list of variables in the summary.

There are a few things to unpack in this cli call.

gdal mdim info --format json \
  /vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk \
  --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-west-2

gdalraster::set_config_option("AWS_NO_SIGN_REQUEST", "YES")
gdalraster::set_config_option("AWS_REGION", "us-west-2")
json <- gdalraster::mdim_info("/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk", cout = FALSE)
cat(substr(json, 1, 1000))
# {
#   "type": "group",
#   "driver": "Icechunk",
#   "name": "/",
#   "attributes": {
#     "attribution": "ECMWF AIFS Single forecast data processed by dynamical.org from ECMWF Open Data.",
#     "dataset_id": "ecmwf-aifs-single-forecast",
#     "dataset_version": "0.1.0",
#     "description": "Weather forecasts from the ECMWF Artificial Intelligence Forecasting System (AIFS) Single model.",
#     "forecast_domain": "Forecast lead time 0-360 hours (0-15 days) ahead",
#     "forecast_resolution": "6 hourly",
#     "license": "CC-BY-4.0",
#     "name": "ECMWF AIFS Single forecast",
#     "spatial_domain": "Global",
#     "spatial_resolution": "0.25 degrees (~20km)",
#     "time_domain": "Forecasts initialized 2024-04-01 00:00:00 UTC to Present",
#     "time_resolution": "Forecasts initialized every 6 hours"
#   },
#   "dimensions": [
#     {
#       "name": "init_time",
#       "full_name": "/init_time",
#       "size": 3264,
#       "indexing_variable": "/init_time"
#     },
#     {
#       "name": "latitude",
#       "full_name": "/latit...  

terra::setGDALconfig("AWS_NO_SIGN_REQUEST", "YES")
terra::setGDALconfig("AWS_REGION", "us-west-2")
terra::rast("/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk")
# class       : SpatRaster
# size        : 721, 1440, 3384768  (nrow, ncol, nlyr)
# resolution  : 0.25, 0.25  (x, y)
# extent      : -180.125, 179.875, -90.125, 90.125  (xmin, xmax, ymin, ymax)
# coord. ref. : lon/lat WGS 84 (CRS84) (OGC:CRS84)
# sources     : v0.1.0.icechunk
# varnames    : dew_point_temperature_2m (2 metre dewpoint temperature)
#               downward_long_wave_radiation_flux_surface (Surface downward long-wave radiation flux)
#               downward_short_wave_radiation_flux_surface (Surface downward short-wave radiation flux)
#               geopotential_height_500hpa (Geopotential height)
#               geopotential_height_850hpa (Geopotential height)
#               ...
# names       :    dew_p~600_1,    dew_p~600_2,    dew_p~600_3,    dew_p~600_4,    dew_p~600_5,    dew_p~600_6, ...
# unit        : degree_Celsius, degree_Celsius, degree_Celsius, degree_Celsius, degree_Celsius, degree_Celsius, ...
# depth       : 1.71193e+09 to 1.78241e+09 (: 3264 steps)
# time        : 0
# Warning message:
# [rast] skipped array (different geometry): /valid_time

Sys.setenv("AWS_NO_SIGN_REQUEST" = "YES")
Sys.setenv("AWS_REGION" = "us-west-2")
s <- sf::gdal_read_mdim("/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk", proxy = TRUE)
str(s)
List of 4
 $ array_list:List of 17
  ..$ dew_point_temperature_2m                  : NULL
  ..$ downward_long_wave_radiation_flux_surface : NULL
  ..$ downward_short_wave_radiation_flux_surface: NULL
...

List of 4
 $ array_list:List of 17
  ..$ dew_point_temperature_2m                  : NULL
  ..$ downward_long_wave_radiation_flux_surface : NULL
  ..$ downward_short_wave_radiation_flux_surface: NULL
  ..$ geopotential_height_500hpa                : NULL
  ..$ geopotential_height_850hpa                : NULL
  ..$ geopotential_height_925hpa                : NULL
  ..$ precipitation_surface                     : NULL
  ..$ pressure_reduced_to_mean_sea_level        : NULL
  ..$ pressure_surface                          : NULL
  ..$ temperature_2m                            : NULL
  ..$ temperature_850hpa                        : NULL
  ..$ temperature_925hpa                        : NULL
  ..$ total_cloud_cover_atmosphere              : NULL
  ..$ wind_u_100m                               : NULL
  ..$ wind_u_10m                                : NULL
  ..$ wind_v_100m                               : NULL
  ..$ wind_v_10m                                : NULL
 $ dimensions:List of 4
  ..$ init_time:List of 5
  .. ..$ from     : int 1
  .. ..$ to       : int 3264
  .. ..$ values   :List of 1
  .. .. ..$ : num [1:3264(1d)] 1.71e+09 1.71e+09 1.71e+09 1.71e+09 1.71e+09 ...
  .. .. .. ..- attr(*, "units")= chr "seconds since 1970-01-01"
  .. .. .. ..- attr(*, "d_names")= chr "init_time"
  .. .. .. ..- attr(*, "attributes")= Named chr [1:4] "proleptic_gregorian" "Forecast initialization time" "forecast_reference_time" "{ \"min\": \"2024-04-01T00:00:00\", \"max\": \"Present\" }"
  .. .. .. .. ..- attr(*, "names")= chr [1:4] "calendar" "long_name" "standard_name" "statistics_approximate"
  .. ..$ type     : chr ""
  .. ..$ direction: chr ""
  ..$ lead_time:List of 5
  .. ..$ from     : int 1
  .. ..$ to       : int 61
  .. ..$ values   :List of 1
  .. .. ..$ : num [1:61(1d)] 0 21600 43200 64800 86400 ...
  .. .. .. ..- attr(*, "units")= chr "seconds"
  .. .. .. ..- attr(*, "d_names")= chr "lead_time"
  .. .. .. ..- attr(*, "attributes")= Named chr [1:5] "AAAAAAAA+H8=" "timedelta64[us]" "Forecast lead time" "forecast_period" ...
  .. .. .. .. ..- attr(*, "names")= chr [1:5] "_FillValue" "dtype" "long_name" "standard_name" ...
  .. ..$ type     : chr ""
  .. ..$ direction: chr ""
  ..$ latitude :List of 5
  .. ..$ from     : int 1
  .. ..$ to       : int 721
  .. ..$ values   :List of 1
  .. .. ..$ : num [1:721(1d)] 90 89.8 89.5 89.2 89 ...
  .. .. .. ..- attr(*, "units")= chr "degree_north"
  .. .. .. ..- attr(*, "d_names")= chr "latitude"
  .. .. .. ..- attr(*, "attributes")= Named chr [1:4] "AAAAAAAA+H8=" "Y" "Latitude" "{ \"min\": -90.0, \"max\": 90.0 }"
  .. .. .. .. ..- attr(*, "names")= chr [1:4] "_FillValue" "axis" "long_name" "statistics_approximate"
  .. ..$ type     : chr "HORIZONTAL_Y"
  .. ..$ direction: chr ""
  ..$ longitude:List of 5
  .. ..$ from     : int 1
  .. ..$ to       : int 1440
  .. ..$ values   :List of 1
  .. .. ..$ : num [1:1440(1d)] -180 -180 -180 -179 -179 ...
  .. .. .. ..- attr(*, "units")= chr "degree_east"
  .. .. .. ..- attr(*, "d_names")= chr "longitude"
  .. .. .. ..- attr(*, "attributes")= Named chr [1:4] "AAAAAAAA+H8=" "X" "Longitude" "{ \"min\": -180.0, \"max\": 179.75 }"
  .. .. .. .. ..- attr(*, "names")= chr [1:4] "_FillValue" "axis" "long_name" "statistics_approximate"
  .. ..$ type     : chr "HORIZONTAL_X"
  .. ..$ direction: chr ""
 $ srs       : chr NA
 $ geometry  : list()

Sys.setenv("AWS_NO_SIGN_REQUEST" = "YES")
Sys.setenv("AWS_REGION" = "us-west-2")
stars::read_mdim("/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk", proxy = TRUE)

# stars_proxy object with 17 attributes in 17 file(s):
# $dew_point_temperature_2m
# [1] "[...]/v0.1.0.icechunk"
# 
# $downward_long_wave_radiation_flux_surface
# [1] "[...]/v0.1.0.icechunk"
# 
# ...
# 
# $temperature_925hpa
# [1] "[...]/v0.1.0.icechunk"
# 
# $total_cloud_cover_atmosphere
# [1] "[...]/v0.1.0.icechunk"
# 
# $wind_u_100m
# [1] "[...]/v0.1.0.icechunk"
# 
# $wind_u_10m
# [1] "[...]/v0.1.0.icechunk"
# 
# $wind_v_100m
# [1] "[...]/v0.1.0.icechunk"
# 
# $wind_v_10m
# [1] "[...]/v0.1.0.icechunk"
# 
# dimension(s):
#           from   to         offset     delta  refsys x/y
# longitude    1 1440         -180.1      0.25      NA [x]
# latitude     1  721          90.12     -0.25      NA [y]
# lead_time    1   61          0 [s] 21600 [s] udunits    
# init_time    1 3264 2024-04-01 UTC   6 hours POSIXct    

Multidimensional?

Please hang in here for a few sections, we are diverting down a GDAL and object storage rabbit hole, but it’s not long and then we’ll get back to Icechunk in the naming a thing section.

We can treat this store as raster data in the normal 2D way, but note that there are caveats and real performance implications depending on the way data is laid out in a given store.

gdalinfo /vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-west-2
Driver: Icechunk/Icechunk
Files: /vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk
Size is 512, 512
Subdatasets:
  SUBDATASET_1_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/dew_point_temperature_2m
  SUBDATASET_1_DESC=[3264x61x721x1440] /dew_point_temperature_2m (Float32)
  SUBDATASET_2_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/downward_long_wave_radiation_flux_surface
  SUBDATASET_2_DESC=[3264x61x721x1440] /downward_long_wave_radiation_flux_surface (Float32)
  SUBDATASET_3_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/downward_short_wave_radiation_flux_surface
  SUBDATASET_3_DESC=[3264x61x721x1440] /downward_short_wave_radiation_flux_surface (Float32)
  SUBDATASET_4_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/geopotential_height_500hpa
  SUBDATASET_4_DESC=[3264x61x721x1440] /geopotential_height_500hpa (Float32)
  SUBDATASET_5_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/geopotential_height_850hpa
  SUBDATASET_5_DESC=[3264x61x721x1440] /geopotential_height_850hpa (Float32)
  SUBDATASET_6_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/geopotential_height_925hpa
  SUBDATASET_6_DESC=[3264x61x721x1440] /geopotential_height_925hpa (Float32)
  SUBDATASET_7_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/precipitation_surface
  SUBDATASET_7_DESC=[3264x61x721x1440] /precipitation_surface (Float32)
  SUBDATASET_8_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/pressure_reduced_to_mean_sea_level
  SUBDATASET_8_DESC=[3264x61x721x1440] /pressure_reduced_to_mean_sea_level (Float32)
  SUBDATASET_9_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/pressure_surface
  SUBDATASET_9_DESC=[3264x61x721x1440] /pressure_surface (Float32)
  SUBDATASET_10_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/temperature_2m
  SUBDATASET_10_DESC=[3264x61x721x1440] /temperature_2m (Float32)
  SUBDATASET_11_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/temperature_850hpa
  SUBDATASET_11_DESC=[3264x61x721x1440] /temperature_850hpa (Float32)
  SUBDATASET_12_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/temperature_925hpa
  SUBDATASET_12_DESC=[3264x61x721x1440] /temperature_925hpa (Float32)
  SUBDATASET_13_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/total_cloud_cover_atmosphere
  SUBDATASET_13_DESC=[3264x61x721x1440] /total_cloud_cover_atmosphere (Float32)
  SUBDATASET_14_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/valid_time
  SUBDATASET_14_DESC=[3264x61] /valid_time (Int64)
  SUBDATASET_15_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/wind_u_100m
  SUBDATASET_15_DESC=[3264x61x721x1440] /wind_u_100m (Float32)
  SUBDATASET_16_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/wind_u_10m
  SUBDATASET_16_DESC=[3264x61x721x1440] /wind_u_10m (Float32)
  SUBDATASET_17_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/wind_v_100m
  SUBDATASET_17_DESC=[3264x61x721x1440] /wind_v_100m (Float32)
  SUBDATASET_18_NAME=ZARR:"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}":/wind_v_10m
  SUBDATASET_18_DESC=[3264x61x721x1440] /wind_v_10m (Float32)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)

Those strings like "ZARR:\"/vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk}\":/dew_point_temperature_2m" (note the embedded escaped quotes) can be used by the more tradition GDAL classic raster model.

We’ll come back to the /vsiicechunk and {/vsis3} parts of this string a bit later.

Setting config options and environment variables for GDAL

GDAL config options can be set through a package wrapper - gdalraster::set_config_option("AWS_NO_SIGN_REQUEST", "YES") or terra::setGDALconfig("AWS_NO_SIGN_REQUEST", "YES") - or as process environment variables (Sys.setenv(AWS_NO_SIGN_REQUEST = "YES"), or KEY=value in the shell). Both routes are global and sticky: once set they can affect every other step, spooky-action-at-a-distance that we want to avoid.

That’s why setting config inline - --config KEY=VALUE on the gdal mdim info call is a clean option: it lives and dies in one line, a bit like with in Python or with() in R. The R equivalent for env vars and options is withr::with_envvar() / withr::with_options(), true scoped context managers.

With online storage it can be bewildering what exact configuration you need and what you have, but it does come down to a fairly comprehensible set of ideas. This post is not a guide to online storage access configuration, or a guide to GDAL configuration options but there’s a finite set of concepts in this jungle so don’t be put off by the complexity.

Auth - keyed vs anonymous. If a dataset is not public, you will need something like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. If it is public, you will need something like AWS_NO_SIGN_REQUEST=YES (or --no-sign-request).

Addressing. Everything is a key inside a bucket. You reference an object as s3://bucket/key (Amazon), gs://bucket/key (Google Storage), abfss://… (Azure storage) - and that’s the whole model. (Azure bends it slightly: a storage account holds containers holding blobs, and the scheme carries the account, abfss://container@account.dfs.core.windows.net/key.)

Endpoint, region, hosting-style. That object is hosted on an endpoint - s3:// is shorthand for s3.<region>.amazonaws.com, gs:// for storage.googleapis.com. A custom endpoint means pointing at your own host - AWS_S3_ENDPOINT=your.endpoint.net - so that s3 in s3://bucket/key resolves to your host and not Amazon’s. Virtual hosting just means the bucket goes before the endpoint in the final url https://bucket.s3.<region>.amazonaws.com/key vs https://my.custom.endpoint.net/bucket/key. It’s confusing, until you know what it does and when you might need it. For non-AWS S3 implementations (MinIO, Ceph) this usually goes together with path-style addressing, AWS_VIRTUAL_HOSTING=NO, since most of them don’t do virtual-host style. Again on a custom S3 AWS_REGION is sometimes needed but probably shouldn’t matter - set it to ““.

/vsizip//vsis3/bucket/archive.zip/inside.tif

/vsizip/{/vsicurl/https://dap.ceda.ac.uk/bodc/gebco/global/gebco_2026/ice_surface_elevation/geotiff/gebco_2026_geotiff.zip?download=1}

ERROR 4: ... not recognized as being in a supported file format.
The archive contains several files:
  …}/gebco_2026_n0.0_s-90.0_w0.0_e90.0_geotiff.tif
  …}/gebco_2026_n0.0_s-90.0_w-180.0_e-90.0_geotiff.tif
  …   (six more files)
  …}/GEBCO_Grid_documentation.pdf
  …}/GEBCO_Grid_terms_of_use.pdf

ZARR:"/vsiicechunk/{/vsis3/.../v0.1.0.icechunk}":/dew_point_temperature_2m

gdal vsi list   /vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk \
  --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-west-2
repo
chunks
manifests
overwritten
snapshots
transactions

gdal vsi list   /vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk/chunks \
 --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-west-2
0003DSF0M0N9B5MP3R9G
0010VXY96ABV0RP45T30
0015TP2QVGT990JE9PM0
...

gdal vsi list /vsiicechunk/{/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk} \
 --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-west-2
zarr.json
dew_point_temperature_2m
downward_long_wave_radiation_flux_surface
downward_short_wave_radiation_flux_surface
expected_forecast_length
geopotential_height_500hpa
geopotential_height_850hpa
geopotential_height_925hpa
ingested_forecast_length
init_time
latitude
lead_time
longitude
precipitation_surface
pressure_reduced_to_mean_sea_level
pressure_surface
spatial_ref
temperature_2m
temperature_850hpa
temperature_925hpa
total_cloud_cover_atmosphere
valid_time
wind_u_100m
wind_u_10m
wind_v_100m
wind_v_10m

What about python?

Above we saw how GDAL and some R packages interact with the Icechunk store, how do we do that in Python?

This time we don’t have a single description of the store, we have to use infrastructure to encode the details.

import icechunk
import xarray 

storage = icechunk.s3_storage(
    bucket="dynamical-ecmwf-aifs-single",
    prefix="ecmwf-aifs-single-forecast/v0.1.0.icechunk",
    region="us-west-2",
    anonymous=True,           # the AWS_NO_SIGN_REQUEST=YES equivalent
)
repo = icechunk.Repository.open(storage)
session = repo.readonly_session("main")
ds = xarray.open_zarr(session.store, consolidated=False)
ds
# <xarray.Dataset> Size: 14TB
# Dimensions:                                     (init_time: 3265,
#                                                  lead_time: 61, latitude: 721,
#                                                  longitude: 1440)
# Coordinates:
#   * init_time                                   (init_time) datetime64[ns] 26kB ...
#     expected_forecast_length                    (init_time) timedelta64[us] 26kB dask.array<chunksize=(3265,), meta=np.ndarray>
#     ingested_forecast_length                    (init_time) timedelta64[us] 26kB dask.array<chunksize=(3265,), meta=np.ndarray>
#   * lead_time                                   (lead_time) timedelta64[us] 488B ...
#     valid_time                                  (init_time, lead_time) datetime64[ns] 2MB dask.array<chunksize=(3265, 61), meta=np.ndarray>
#   * latitude                                    (latitude) float64 6kB 90.0 ....
#   * longitude                                   (longitude) float64 12kB -180...
#     spatial_ref                                 int64 8B ...
# Data variables: (12/17)
#     dew_point_temperature_2m                    (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     downward_long_wave_radiation_flux_surface   (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     geopotential_height_500hpa                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     geopotential_height_850hpa                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     precipitation_surface                       (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     downward_short_wave_radiation_flux_surface  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     ...                                          ...
#     wind_u_100m                                 (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     temperature_925hpa                          (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     total_cloud_cover_atmosphere                (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     wind_u_10m                                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     wind_v_100m                                 (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
#     wind_v_10m                                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(1, 61, 241, 240), meta=np.ndarray>
# Attributes:
#     dataset_id:           ecmwf-aifs-single-forecast
#     dataset_version:      0.1.0
#     name:                 ECMWF AIFS Single forecast
#     description:          Weather forecasts from the ECMWF Artificial Intelli...
#     attribution:          ECMWF AIFS Single forecast data processed by dynami...
#     license:              CC-BY-4.0
#     spatial_domain:       Global
#     spatial_resolution:   0.25 degrees (~20km)
#     time_domain:          Forecasts initialized 2024-04-01 00:00:00 UTC to Pr...
#     time_resolution:      Forecasts initialized every 6 hours
#     forecast_domain:      Forecast lead time 0-360 hours (0-15 days) ahead
#     forecast_resolution:  6 hourly

We looked above at this xarray representation but up there we didn’t use the same infrastructure or show the code. The one above was made with

import os
import xarray

os.environ["AWS_NO_SIGN_REQUEST"] = "YES"
os.environ["AWS_REGION"] = "us-west-2"

ds = xarray.open_dataset("/vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk", engine = "gdalxarray", chunks = {})
# <xarray.Dataset> Size: 14TB
# Dimensions:                                     (init_time: 3265,
#                                                  latitude: 721, lead_time: 61,
#                                                  longitude: 1440)
# Coordinates:
#   * init_time                                   (init_time) datetime64[ns] 26kB ...
#     expected_forecast_length                    (init_time) float64 26kB dask.array<chunksize=(3265,), meta=np.ndarray>
#     ingested_forecast_length                    (init_time) float64 26kB dask.array<chunksize=(3265,), meta=np.ndarray>
#   * latitude                                    (latitude) float64 6kB 90.0 ....
#   * lead_time                                   (lead_time) float64 488B 0.0 ...
#     valid_time                                  (init_time, lead_time) float32 797kB dask.array<chunksize=(3265, 61), meta=np.ndarray>
#   * longitude                                   (longitude) float64 12kB -180...
#     spatial_ref                                 float32 4B ...
# Data variables: (12/17)
#     dew_point_temperature_2m                    (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     downward_long_wave_radiation_flux_surface   (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     downward_short_wave_radiation_flux_surface  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     geopotential_height_500hpa                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     geopotential_height_850hpa                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     geopotential_height_925hpa                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     ...                                          ...
#     temperature_925hpa                          (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     total_cloud_cover_atmosphere                (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     wind_u_100m                                 (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     wind_u_10m                                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     wind_v_100m                                 (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
#     wind_v_10m                                  (init_time, lead_time, latitude, longitude) float32 827GB dask.array<chunksize=(3265, 61, 721, 1440), meta=np.ndarray>
# Attributes:
#     attribution:          ECMWF AIFS Single forecast data processed by dynami...
#     dataset_id:           ecmwf-aifs-single-forecast
#     dataset_version:      0.1.0
#     description:          Weather forecasts from the ECMWF Artificial Intelli...
#     forecast_domain:      Forecast lead time 0-360 hours (0-15 days) ahead
#     forecast_resolution:  6 hourly
#     license:              CC-BY-4.0
#     name:                 ECMWF AIFS Single forecast
#     spatial_domain:       Global
#     spatial_resolution:   0.25 degrees (~20km)
#     time_domain:          Forecasts initialized 2024-04-01 00:00:00 UTC to Pr...
#     time_resolution:      Forecasts initialized every 6 hours

That seems very similar, and hopefully for the in-dev GDAL xarray backend gdalxarray it’s an equivalent connection (I’m not going to validate any further here in this post).

What this illustrates is while the configuration set up can be confusing, these are the very same generic store-auth settings cast different ways. Is it desirable to consolidate these? Is one necessarily better or more capable? Icechunk-Python auth and addressing through typed constructor arguments, GDAL through the VSI path plus config.

Other R Zarr readers

Beyond GDAL, there is growing native R support for Zarr:

• pizzarr (CRAN) - pure R Zarr V2 reader; r-universe builds include an optional high-performance Rust backend via the zarrs crate
• Rarr (Bioconducator) - A simple native R reader for Zarr Arrays, Rarr is actively developed and has good support for a growing number of codecs
• zarr (CRAN) - a native implementation in R with support for all required features of Zarr version 3.
• CopernicusMarine - Subset and download marine data from EU Copernicus Marine Service Information, has an in-built Zarr engine designed around the service
• zaro (r-universe) - Low-level Zarr interface (V2 and V3) built on Arrow filesystem and codec infrastructure
• RNetCDF, ncdf4 - with the right build set up NetCDF has Zarr support, and this for example has been proved for using with tidync

Work is ongoing on multidim API coverage in the R ecosystem, I particularly want tight api bindings to the GDAL API with gdalraster but that will be a while still in the making.

The GDAL multidim API (used by /vsiicechunk/) is distinct from the classic 2D raster driver - terra now auto-detects which to use for the multidim drivers which are VRT/HDF5/NetCDF/Zarr/GRIB. If terra gives unexpected results, check whether it’s related to recent changes in mdim support.

For some time stars via sf has support for accessing Zarr via the classic or multidimensional raster model of GDAL.

For Icechunk specifically see icechunk.io and see the GDAL icechunk driver docs at gdal.org icechunk and ZARR driver at gdal.org zarr.

Structured grids

“Structured” just means that every element in every dimension has the same number of other-dimension counterparts. It’s rectangular: in 1D that’s a set of intervals, in 2D a mosaic of axis-aligned rectangular cells, in 3D a mosaic of axis-aligned rectangular cuboids, and upwards. We can replace intervals, rectangles, cuboids, tesseracts, … with another kind of element: a point. In 1D it’s just a sequence of points, in 2D and above it’s the Cartesian product of sequences of indices. When we talk about structured vs unstructured we are not talking about the spacing of coordinates in any sense - always just index positions. The index structure is, in more formal terms, a cubical complex or regular CW complex; the Earth system modelling community calls this “logically rectangular,” a term we return to below in “Logically rectangular” and the Earth system model vocabulary.

The ladder, defined by storage

The cleanest single organizing principle comes from scientific visualization (the VTK data model, the KTH/Weinkauf taxonomy): grid types differ in how implicitly or explicitly positions must be stated.

Regular (uniform)

Positions are fully implicit. The grid is completely specified by:

• an origin
• a resolution
• dimensions

(Here if you need please pause for a pedantic ¹ footnote).

Every cell centre follows from index arithmetic alone. No coordinate arrays are stored - or if they are, they are redundant.

Code

# synthetic example
ex <- c(0, 18, -10, 5)  ## extent xmin, xmax, ymin, ymax 
# - same information as bbox xmin,ymin,xmax,ymax
#dimension, or shape (ncols,nrows)
dm <- c(8, 5)

library(terra)

terra 1.9.44

Code

r <- rast(ext(ex), ncols = dm[1], nrows = dm[2])
plot(as.polygons(r), asp = 1, lwd = 2)
points(xyFromCell(r, 1:ncell(r)))

There are two clear ways to think about a regular grid, little area rectangles or a set of centre points - each contains exactly the same information, and each is completely definable by six numbers - ncols, nrows, xmin, ymin, xmax, ymax. Those numbers might be rearrange in order or expressed as offset scale (a transform), but that doesn’t change anything materially.
The netCDF convention for a regular grid stores 1D coordinate variables, a value for every step in the dimension. This is the degenerate rectilinear case: the 1D vectors happen to be uniformly spaced, so the six-number specification is sufficient. Many files store them anyway, and many readers treat the uniform case no differently from the variable-spacing case.

What are the coordinates? Well they are the positions of the points above, the centres of a cell. It’s trivial to observe that we only need 8 points for every x coordinate, 5 for every y coordinate. But, in turn we don’t need the coordinates at all because they were generated from 6 numbers. Those six are extremely general, every regular grid ever is a variation on those values: positive ncols, nrows >= 1; any numbers at all for xmin, xmax, ymin, ymax so long at xmax > xmin; ymax > ymin.

The coordinates stored in the 1D vectors are derived values - computed from those 6 numbers by the formula xi=xmin+(i+0.5)⋅Δxx_i = x_{min} + (i + 0.5) x xi=xmin+(i+0.5)⋅Δx (or the equivalent for edges). So storing them is not just redundant, it is a lossy redundancy - the stored floats have accumulated rounding error relative to the exact values the formula would produce. The 6-number specification is not only more compact, it is more precise.

In the wild: NSIDC sea ice concentration products, ERA5 reanalysis on a regular lat/lon grid, most remote sensing imagery.

---

# synthetic example
# x <- c(0, 1, 3, 6, 10)   # non-uniform
# y <- c(0, 2, 3, 7)
# show grid lines: horizontal and vertical only, variable spacing
# contrast with regular above

# synthetic example: apply a smooth warp to a regular grid
# e.g. rotate + shear, or a polar stereographic → lonlat forward projection
# show that rows are no longer horizontal, columns no longer vertical
# this is the ESMF example: a grid warped to put the pole over land

# detecting curvilinear in vapour
# TODO: show vapour::gdal_raster_data on a curvilinear source
# show that coordinate arrays have dim > 1 in both axes

# library(vapour)
# src <- "some_ocean_model.nc"
# ...

# synthetic Delaunay triangulation over a region
# contrast with the i,j indexing of curvilinear above

The spectrum as a table

Type	Connectivity	Coordinate storage	CF signal
Regular	implicit	origin + step (4 numbers)	1D coord vars, uniform spacing
Rectilinear	implicit	two 1D vectors	1D coord vars, variable spacing
Curvilinear	implicit	two 2D arrays	2D auxiliary coord vars
Unstructured	explicit table	position pairs + connectivity	UGRID convention

The progression is strict containment: every regular grid is rectilinear; every rectilinear grid could be stored as curvilinear (wastefully); a curvilinear grid is a structured grid (logically rectangular in the sense of Balaji 2006).

“Logically rectangular” and the Earth system model vocabulary

The Earth system modelling community (ESMF, GFDL, NEMO) uses a slightly different framing that is worth knowing. Balaji (2006) defines a grid as logically rectangular if its coordinate space maps one-to-one to an integer index space - i.e., cells can be addressed as . This covers regular, rectilinear, and curvilinear in our ladder above. Unstructured grids are not logically rectangular.

ESMF then distinguishes uniform, rectilinear, and curvilinear as subtypes of logically rectangular - exactly the ladder above minus unstructured. This is consistent usage.

The key sentence from Balaji: “The coordinate space may continue to be physically curvilinear; yet, in index space, grid cells will be rectilinear boxes.” Index space is always regular. Physical space may be warped. Curvilinear is a statement about physical space, not index space.

“Curvilinear” is not a CRS

Here is the specific conflation this post exists to name.

A regular Polar Stereographic grid - say, the NSIDC 25 km Southern Ocean grid on EPSG:3412 - is regular. It has uniform spacing in projected coordinates. It can be fully specified by four numbers and a CRS string. We are not saying anything at all about the area or distance on the Earth represented by those cells. This is true in Stereographic (it varies), it’s true in Mercator, in longitude/latitude, in Lambert Azimuthal Equal Area, in Equidistant Conic: it doesn’t matter - the real-world “correctness” of area, distance, or angle is irrelevant to the regularity of the grid in its native CRS.

When a Polar Stereographic grid’s cell centres are forward-projected to geographic coordinates and stored as lat(y, x) / lon(y, x) two-dimensional arrays in a NetCDF file, the storage format looks curvilinear. The CF signal is present: 2D auxiliary coordinate variables. But the underlying geometry is not curvilinear - it is regular, with two arrays of floating-point noise standing in for four exact integers.

This is the entropy problem described in the companion post. The confusion arises because:

1. CF conventions use the shape of coordinate arrays as the proxy for grid type, and a reprojected regular grid has 2D arrays.
2. Many tools that read CF files label anything with 2D coordinate variables as “curvilinear” without checking whether those arrays have rectilinear or regular structure.
3. “Geographic coordinates” and “curvilinear coordinates” have been conflated in enough documentation that people assume any lon/lat grid is curvilinear.

The correct test is geometric, not syntactic: can the positions be expressed as the outer product of two 1D sequences (rectilinear), or as origin + step (regular)? If yes, the grid is not curvilinear regardless of how its coordinates are stored.

The detection methods in the companion post are the practical implementation of this test.

Genuinely curvilinear examples

To make the distinction concrete: these are cases where the 2D coordinate arrays are not hiding a regular grid.

ORCA bipolar ocean grid. The NEMO ocean model uses a grid that is regular in the Southern Ocean and tropics but deforms near the North Pole to place the numerical singularity over land (Greenland and Siberia). The resulting nav_lon / nav_lat arrays cannot be factored into 1D sequences by any projection. This is genuinely curvilinear.

# TODO: plot nav_lon/nav_lat from an ORCA file, zoom to Arctic
# show the non-rectangular cells near the bipolar region

MODIS/VIIRS L2 swath. Sensor scan geometry produces curved swaths. Adjacent scan lines are not parallel, and the scan angle distortion means cells grow toward swath edges. No single projected CRS straightens this.

# TODO: plot boundary polygon coloured by scan-line index
# show the bow-tie artifact at scan edges

WRF/rotated-pole regional climate model. The model domain is regular in a rotated coordinate system (pole relocated to the equator of a rotated sphere), which becomes curvilinear in standard geographic coordinates. Detecting this is the Mercator analogue for rotated-pole: project to the rotated CRS, check for regularity there.

The degenerate rectilinear case

One last terminological note. NetCDF files frequently store 1D coordinate vectors for regular grids - a lon vector of length and a lat vector of length , both uniformly spaced. Strictly speaking this is rectilinear storage (the format allows variable spacing), but the data is regular (the spacing happens to be uniform).

This is the degenerate rectilinear case: a rectilinear format carrying regular content. It matters because:

• Tools that test only for uniform spacing on 1D arrays correctly identify the grid as regular.
• Tools that check only “does it have 1D coordinate variables?” treat it as rectilinear (variable spacing allowed) and may not exploit the regularity.
• The four-number specification is recoverable from the 1D vectors trivially.

For the netCDF case it is mostly a storage efficiency question. The interesting case is when the 1D vectors are non-uniformly spaced - genuinely rectilinear - which occurs in pressure-level data, stretched ocean grids (e.g. MOM with enhanced tropical resolution), and Mercator-latitude ocean products.

References and further reading

• Balaji, V. (2006). A standard description of grids used in earth system models. GFDL/NOAA. The foundational taxonomy for the Earth system modelling community; introduces the logically rectangular / mosaic framework. GFDL grid standards

• ESMF Reference Manual §12.1 - the most consistently used set of definitions in practice for geoscience regridding tools. Distinguishes uniform / rectilinear / curvilinear as subtypes of logically rectangular. earthsystemmodeling.org

• CF Conventions §5 - the syntactic signal: 1D dimension coordinates vs 2D auxiliary coordinates. Does not define grid type geometrically but the encoding conventions follow the taxonomy here. cfconventions.org

• Weinkauf, T. (KTH, 2015). Grids and Interpolation - the clearest single slide deck on the implicit/semi-implicit/explicit storage framing. Not geoscience-specific; comes from scientific visualisation.

• Wikipedia: Regular grid - covers the structured grid sub-hierarchy with reasonable precision.

• Detecting hidden grids in curvilinear coordinates (this site, 2026-02-07) - the practical companion: how to test whether a 2D lon/lat array is actually a regular projected grid with lost metadata.

• Coordinates broken in NetCDF (this site, 2025-09-04) - an earlier treatment of the related problem of CF files with missing or misleading coordinate metadata.

The ladder is manifold-agnostic — and that is the point

The regular → rectilinear → curvilinear ladder characterises grids by their index structure and coordinate storage alone. The manifold — sphere, plane, torus, ellipsoid — is not part of the definition. You can have a regular grid on a sphere (HEALPix patches, cubed-sphere faces) or a regular grid on a plane; regularity is a property of the indexing, not the embedding. This is not a limitation of the ladder; it is the source of its generality and simplicity.

The longlat case is a clean illustration of what happens when you ignore the manifold. A global regular longlat grid is perfectly regular by the ladder’s definition: uniform spacing, Cartesian product index, four numbers and a CRS to specify it completely. And that’s fine — until your application needs to know that the top row is a pole and the left and right edges are the same meridian. At that point the ladder gives you no help. The wrapping logic, the polar singularity, the fact that cells covering Fiji and cells covering its antipode are not indexed as neighbours — none of that is encoded in the grid structure. It is external knowledge that the application must supply.

The ladder doesn’t fail at that boundary. It never claimed to address it.

What “wrapping logic” reveals

A regular longlat grid at 1° resolution has 360 × 180 = 64,800 cells. The top row contains 360 cells, all of which map to the South Pole — a single point. One cell (or at most three, for a T-shaped pole cap) would do the same geometric work. The other 357 are redundant, and the grid structure has no way to say so. Similarly, cell and cell are geographic neighbours — they share an edge at the antimeridian — but in index space they are as far apart as two cells can be. The Cartesian product index is silent about this.

This is not a pathology. It is simply what you get when you discretise a sphere with a tool designed for a flat rectangle. The grid is doing its job correctly by the ladder’s definition. The mismatch is between the ladder’s scope and the application’s needs.

When the manifold does matter to an application — global topology, correct adjacency, equal-area cells, no polar redundancy — you are outside the ladder’s scope. That is precisely the problem that DGGS are designed to solve.

DGGS: discretising the manifold directly

Discrete global grid systems (H3, S2, HEALPix, ISEA) approach the problem from the other direction. The manifold comes first; the discretisation is designed for the sphere, so adjacency, area, and topology are correct by construction. H3 hexagons near Fiji are indexed as neighbours of other hexagons near Fiji. There is no polar row of redundant cells because the addressing scheme is built around the sphere’s actual geometry.

The price is the Cartesian product index structure. DGGS cells don’t factor into rows and columns in any natural global sense (HEALPix has local patches with regular structure; H3 has a hierarchical addressing scheme with local regularity; but neither is globally Cartesian-product-indexable). This is why DGGS feel like unstructured grids to tools that expect — and in the ladder’s terms, they are. The structured/unstructured distinction still applies internally to a DGGS, but the global topology is not rectangular.

So DGGS sit outside the ladder not because they are more complex than unstructured grids, but because the ladder’s premise — index structure independent of the manifold — is exactly what DGGS reject. They are a different answer to a different question.

The ladder is the right tool when the manifold can be treated as external context. DGGS are the right tool when the manifold must be built into the addressing scheme. These are complementary, not competing.

Workarounds that live on the ladder

Most real-world geospatial work stays on the ladder but handles the manifold through external convention rather than anything the grid structure provides. This is worth naming:

• Tripolar ocean grids (NEMO ORCA, MOM6 tripolar): curvilinear grids that move the numerical pole singularity over land. The grid is still logically rectangular — still on the ladder — but the topology (two northern poles, one southern pole) is encoded in the coordinate arrays and the model’s halo-exchange logic, not in the index structure.
• NSIDC polar grids: regular projected grids (EPSG:3412, EPSG:3413) that implicitly wrap in the azimuthal sense. The grid structure doesn’t know this; applications that care about the antimeridian handle it separately.
• vaster and vapour: deliberately silent about topology. They operate on the ladder — grid arithmetic, coordinate lookup, raster I/O — and leave manifold questions to the caller. This is the right scope decision; encoding sphere topology into a raster grid library would be the wrong level of abstraction.

The “subtle relationships and workflows and workarounds” in curvilinear ocean models are mostly the accumulated cost of living on the ladder with a spherical manifold. The ladder is the floor, not the ceiling.

The SeaWiFS L3bin: the connector

The sinusoidal L3 grid and the L3bin addressing scheme together make the ladder boundary more concrete than any abstract example.

The sinusoidal regular grid is the rectangular-index version of global equal-area ocean colour compositing. It is a regular grid in sinusoidal projection: uniform spacing in projected coordinates, four numbers and a CRS, perfectly on the ladder. But sinusoidal projection preserves area by compressing x-extent toward the poles: high-latitude rows are physically narrow. When you impose a uniform column count across all rows, most high-latitude cells fall outside the Earth’s disk and are fill. The grid is valid; it is just mostly empty at the poles, and the emptiness is structural, baked in by the choice to force a sphere into a rectangle.

The SeaWiFS/MODIS/VIIRS L3bin resolves this directly. Each latitude row contains exactly as many bins as needed to tile that latitude band — no more, no fewer. Row widths vary. The result is a ragged array: not a matrix, not a Cartesian product, not on the ladder. By the ladder’s definition it is unstructured — each bin has an explicit address (the bin number) rather than a index pair.

But it is the simplest possible unstructured case. The bins are still organised by latitude band. The addressing scheme (bin number → row → column-within-row) is well-defined and compact. The geometry is explicit but not complicated. It is unstructured by necessity, not by complexity — the minimum departure from the ladder required to take the sphere seriously.

The contrast is the hook:

• The sinusoidal regular grid: technically on the ladder, equal-area, but with a wasteful rectangular fiction imposed on a spherical reality. The manifold is ignored; fill cells paper over the mismatch.
• The L3bin: off the ladder, but honest. The ragged structure is the sphere’s geometry expressed directly in the data structure. You traded the Cartesian product index for an accurate account of what you actually have.

This is the L3bin’s value as a connector. It is not an exotic case — it is used in virtually every ocean colour dataset from SeaWiFS through PACE. And it sits exactly at the boundary the ladder defines, making the boundary legible.

A post that works through the L3bin addressing scheme in R — reading bin numbers, recovering lat/lon, comparing to the sinusoidal regular grid counterpart — would make the abstract ladder concrete in a way that the regular/rectilinear/curvilinear examples cannot. That is post #2 (or #3, after the curvilinear detection post).

The affine/homogeneous coordinates aside

(Parked from a detour in the main post draft — too interesting to discard, too long to include.)

The GDAL geotransform is a 2×3 affine matrix (equivalently, a 3×3 homogeneous matrix with the bottom row fixed at [0, 0, 1]): the full projective/affine machinery for mapping pixel index → Cartesian position in a flat 2D plane. This is a genuinely 2D spatial problem, so the homogeneous coordinates are natural.

The rectilinear case exposes how much of that is a 1.5D trick. A rectilinear grid is two independent 1D problems — “where is column on the x-axis” and “where is row on the y-axis” — combined. There is no cross-term, no rotation, no shear. The “transform” collapses to two separate monotone sequences. Nobody would think to write a 3×3 homogeneous matrix for a 1D lookup table, because a lookup table is obviously just a lookup table.

The geotransform works for the regular case because regular grids are the degenerate rectilinear case — the lookup tables happen to be arithmetic sequences, so they compress back to origin + step, which is what the six geotransform numbers encode. The homogeneous machinery is present, but with most of its degrees of freedom set to zero.

The shadow quality: the geotransform is the full 2D affine apparatus applied to what is, in the regular case, a product of two 1D arithmetic sequences. When you move to rectilinear coordinates — genuinely variable spacing, two lookup tables — no one reaches for a matrix from a higher-dimensional space. The 1D lookup table is self-evidently sufficient. The geotransform convention makes the regular case look more 2D than it is. (Rotated or sheared regular grids are the exception that proves the rule — see the footnote on the Regular section above.)

The gproj package, which implements regularity detection, is at github.com/mdsumner/gproj. The vaster package provides the regular grid arithmetic used throughout the hypertidy ecosystem.

Fifteen minutes with xarray

A key moment for me was while learning Python and xarray, I was impressed that xarray could open these files — a huge list of them (16,337 days of OISST at time of writing) as local files or remote URLs. xarray checks them all for consistency, detects that what is really there is four variables in longitude, latitude, and time, and that time is an implicit concatenation from a single time step in each of those thousands of files. It’s remarkable: one object in memory in Python, ready to lazy-read any part of the four 16337×720×1440 arrays. But that took about 15 minutes to run, let alone the software and computing infrastructure setup I had been through. And then: how do you share that progress with colleagues?

Everybody knows you aren’t supposed to serialize in-memory objects to binary, we all have scary experiences with .Rdata and .rds, and Python has the same with its pickle. (I thought: “hmmm, what’s the xarray equivalent of a VRT?” Spoiler: there isn’t one — and although VirtualiZarr does serve that role, it’s missing an intermediate state where we only record what files were opened and what structure was learned.)

What the VRT already does

Back to our 23 files — a sample of the 16,337+ in the full OISST set as of writing.

Open /tmp/sst_stack.vrt in a text editor (browseURL("/tmp/sst_stack.vrt") works too). It’s a few kilobytes of XML that fully describes a virtual 23-layer SST dataset — source files, spatial extent, band mapping. It’s a tiny file: you could email it, put it in a repository, or patch the source paths to /vsicurl/https://....nc URLs so it works without any local files at all. Anyone with GDAL can open it with rast("sst_stack.vrt") and get exactly what you built, without copying a byte of the underlying data.

That’s a recipe: we’ve described our dataset as a small chunk of text that GDAL can open very quickly.

VRT (Virtual Format) is a plain XML file that describes a virtual raster dataset. It can mosaic, warp, reproject, reband, and combine source files without touching the underlying data. For the 2D raster world it is exactly the recipe format that’s missing from the terra/stars in-memory story. The 2D VRT is one of GDAL’s most underappreciated features — already powering a lot of geospatial infrastructure, especially where band-stacking or tile-mosaicking is sufficient as the solution.

So what about the full n-D case: multiple variables, time coordinates, arbitrary dimensions?

What Zarr figured out

The Python/xarray community has been working on exactly this problem, approaching it from the chunk level rather than the file level.

Zarr is a chunked array format. An array is split into regular chunks; each chunk is compressed independently and stored as a separate object — a file, an S3 object, a range of bytes in a larger store. The chunks themselves are opaque: raw numeric arrays in compressed bytes, not self-describing. What makes Zarr work is the metadata sidecar: a small JSON file that records the shape, dtype, chunk layout, compression codec, and dimension coordinates — everything xarray needs to provide a rich, self-describing interface over the data. This is really good for cloud access: fetch only the chunks you need, in parallel, without a format-specific library. It’s a bit like taking a pile of NetCDF files and turning them inside out — all the metadata (tiny in comparison to the data) is placed upfront in JSON, and all the compressed opaque chunks are stored as individually addressable objects.

The deeper insight comes when you realize that HDF5, netCDF4, GeoTIFF, GRIB, and others already store their data in chunks. The chunks are buried inside files that require format-specific libraries to navigate, but the byte ranges are all there. We just need an index.

kerchunk and its successor VirtualiZarr do exactly that: open your existing files, find where each chunk lives (file path, byte offset, byte length), and write that index as a JSON or Parquet reference store. The bulk data never moves. What you get is a Zarr-shaped recipe that points back into your original files. It’s worth noting that this idea is not new — NASA and Unidata’s DMR++ format did the same thing for OPeNDAP, indexing chunk byte-ranges in HDF5/netCDF files as a sidecar that an OPeNDAP server can serve directly. Cloud-Optimized GeoTIFF is the same idea applied to TIFF: the IFD tile index is moved to the front of the file so a client can read the chunk map in one range request and fetch only what it needs thereafter. The difference is that Zarr’s client-side framing made the pattern composable and ecosystem-wide rather than tied to a specific server or format.

NOTE: this is very much pseudo code showing the overall style - see the Usage VirtualiZarr pages for more examples, and we have a real example in the appendix.

from virtualizarr import open_virtual_dataset
import xarray as xr

# index the chunks — no data copied
combined = xr.concat(
    [open_virtual_dataset(f) for f in files],
    dim="time"
)
combined.virtualize.to_kerchunk("sst_virtual.zarr", format="zarr")

# open it — chunk-aware, lazy, S3-ready, fully structured
ds = xr.open_dataset("sst_virtual.zarr/", engine="kerchunk")
# <xarray.Dataset>
# Dimensions:  (time: 23, zlev: 1, lat: 720, lon: 1440)
# Coordinates:
#   * time     (time) datetime64[ns] 2025-01-01 ... 2025-12-19
#   * lat      (lat) float32 -90.0 ... 89.75
#   * lon      (lon) float32 0.0 ... 359.75
# Data variables: anom, err, ice, sst

The store is kilobytes of metadata, the actual data stays where it is, and the n-D structure is fully encoded. Compare this to the terra output above:

• sst, anom, err, ice as named variables, not interleaved bands
• time as a real coordinate with date values, not layer indices r_1..r_23
• lat, lon, zlev as labelled dimensions with coordinate values
• the whole thing serialized as a directory anyone with Zarr support can open

This is what stars::read_mdim() is reaching toward in R — the representation is structurally similar. The difference is that the xarray object came from a small Zarr store, not from reconstructing structure at runtime over 23 original netCDF files. That difference is the recipe.

GDAL is the bridge

GDAL already spans both worlds.

GDAL has a multidimensional API (the “mdim” layer) that represents data as named arrays with labelled dimensions — the same model as xarray, not the band-stacked model of classic GDAL rasters. GDAL’s Zarr driver can open VirtualiZarr/kerchunk reference stores through this API. A store created in Python is readable via GDAL — which means it’s reachable from R today.

More importantly, GDAL has gdal mdim mosaic — a tool that takes a collection of raster files and produces a multidim VRT describing them as a unified n-D array dataset. This VRT encodes the array structure, dimension names, coordinate values, and source file mapping. It is the GDAL-native recipe for an n-D dataset — and as shown in the appendix, stars::read_mdim() opens it faithfully, giving you the full multi-file dataset with the right structure, right coordinates, and a serializable description that travels independently of the original files.

Converting a GDAL multidim VRT to a Zarr reference store is a small transformation on kilobytes of XML, not a data operation. That’s the near-term path to R-native VirtualiZarr creation.

The gap: creation, not reading

R can read virtual stores — Zarr and multidim VRT and VirtualiZarr Parquet reference via stars::read_mdim(), gdalraster::mdim_translate or zaro or real Zarr with pizzarr or zarr. What R can’t yet do is create the chunk-level byte-reference stores that VirtualiZarr generates for HDF5 and netCDF.

That’s a frontier, and I’ve been exloring where the pieces could converge:

gdal mdim mosaic creates multidim VRTs from file collections today. The XML fully describes what a VirtualiZarr store needs — shape, chunks, coordinate values, source file paths and byte offsets. This is the hard structural work; emitting the Zarr reference store format from it is a straightforward conversion.

rhdf5 can now open remote HDF5 files at URLs or on S3 (dev version on GitHub). The next step is extracting chunk byte-reference indices directly - a core operation that VirtualiZarr performs in Python (with HDF5 library, GRIB libraries, TIFF libraries some in C++ and some in Rust). Step 7 in the appendix shows remote access, and a pending PR in my rhdf5 fork has already use that to harvest remote byte references from HDF5 and NetCDF.

zaro (pure-R Zarr reader via Arrow) already reads VirtualiZarr Parquet stores. The reading side is there, and needs comparison with pizzarr, and zarr and other zarr-enabled packages.

vrtstack generates multidim VRTs from 2D raster collections in R - the gdal mdim mosaic equivalent for simple cases of multiple 2D raster - without leaving R.

The path: vrtstack or gdal mdim mosaic for file collections → emit Zarr reference store → readable by zaro, xarray, or any Zarr-aware tool. In this way R can become a first-class creator of portable dataset descriptions, not just a consumer of what Python produces.

Why this matters beyond R

A Zarr reference store created from OISST netCDF files is immediately openable in Python, R, GDAL and in theory any tool that speaks Zarr.

STAC is complementary but different: STAC describes where files are and their spatiotemporal footprint. A virtual store describes what the dataset structurally is — how chunks map to array coordinates, what the dimensions mean, how to reassemble a slice. A STAC item can point to a virtual store the way it points to a COG.

The R community has genuine leverage here. GDAL is the common substrate between R and the xarray/Zarr ecosystem. R is not just catching up, there are things we can do that aren’t on any other roadmap.

What would help

If you work with time series of raster files and you’ve felt this friction:

• Talk about your use case — what files, what structure, what would a portable dataset description let you do that you can’t do today?
• Try terra::vrt() and try the vrt:// URI syntax for netCDF subdataset selection.
• Try stars::read_mdim() on a gdal mdim mosaic VRT — the full tour in the appendix shows how these fit together
• Try remotes::install_github("hypertidy/tidync@multi-input") if you have remote netCDF files and want to skip the download entirely — Step 8 in the appendix shows what this already looks like.
• Look at vrtstack and zaro on the hypertidy GitHub, vrtstack builds a multidim VRT from 2D rasters (multidim GDAL supports COG, but doesn’t support builiding these), zaro is a read-implementation for Zarr that is primarily based around Arrow.
• File issues against GDAL’s multidim API when you find gaps — R users exercising it in real workflows makes it better for everyone.
• Check out gdalraster (and my multidim branch) there’s a long way to go for multidim but gdalraster is a rich package has a super strong basis for faithfully carrying the core facilities of the multidim api exposed to R.

Appendix: a progressive tour on 23 OISST files

library(glue)

date  <- seq(as.Date("2025-01-01"), by = "16 days", length.out = 23)
base0 <- "https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr"
ym    <- format(date, "%Y%m")
ymd   <- format(date, "%Y%m%d")
urls  <- glue("{base0}/{ym}/oisst-avhrr-v02r01.{ymd}.nc")

dir   <- tempdir()
files <- file.path(dir, basename(urls))
download.file(urls, files, method = "libcurl", mode = "wb")

writeLines(glue::glue("/vsicurl/{urls}"), tf <- tempfile())
system(glue::glue("gdal mdim mosaic @{tf} oisst_2025_sample.vrt"))
## piggyback::pb_upload("oisst_2025_sample.vrt")

rast(files)
# class       : SpatRaster
# size        : 720, 1440, 92  (nrow, ncol, nlyr)
# sources     : oisst-avhrr-v02r01.20250101.nc (4 layers)
#               ... and 22 more sources
# varnames    : anom, err, ice, sst, anom, err, ice, sst, ...
# names       : anom_zlev=0, err_zlev=0, ice_zlev=0, sst_zlev=0, ...
# unit        :      Celsius,    Celsius,          %,    Celsius, ...
# time (days) : 2025-01-01 to 2025-12-19 (23 steps)

vrt(
  sprintf("vrt://%s?sd_name=sst", files),
  "/tmp/sst_stack.vrt",
  options = "-separate",
  overwrite = TRUE
)
# class       : SpatRaster
# size        : 720, 1440, 23  (nrow, ncol, nlyr)
# source      : sst_stack.vrt
# names       : r_1, r_2, r_3, ..., r_23

read_stars(files)
# stars_proxy object with 4 attributes in 92 file(s):
# $sst
#  [1] "[...]/oisst-avhrr-v02r01.20250101.nc:sst"
#  [2] "[...]/oisst-avhrr-v02r01.20250117.nc:sst"
#  ...
# $anom, $err, $ice  (similarly, 23 files each)
#
# dimension(s):
#      from   to offset delta  refsys x/y
# x       1 1440      0  0.25      NA [x]
# y       1  720     90 -0.25      NA [y]
# zlev    1    1  0 [m]    NA udunits
# time    1   23     NA    NA      NA

stars::read_mdim(files[1L])
# stars object with 4 dimensions and 4 attributes
# attribute(s):
#            Min.  1st Qu.  Median      Mean  3rd Qu.   Max.     NAs
# sst [°C]  -1.80    -1.80   -1.78  -1.70779    -1.64  -1.03   89635
# anom [°C] -0.72    -0.08   -0.02  -0.03531     0.00   0.49   89635
# err [°C]   0.30     0.30    0.30   0.30446     0.30   0.35   89635
# ice [%]    0.19     0.81    0.95   0.87747     0.98   1.00   91877
# dimension(s):
#      from   to  offset  delta          refsys x/y
# lon     1 1440       0   0.25  WGS 84 (CRS84) [x]
# lat     1  720     -90   0.25  WGS 84 (CRS84) [y]
# zlev    1    1  0 [m]      NA         udunits
# time    1    1  17532 [days since 1978-01-01 12:00:00]  NA  udunits

# writeLines(sprintf("/vsicurl/%s", urls), "filelist.txt")
gdal mdim mosaic --input @filelist.txt -o /tmp/sst_mdim.vrt

stars::read_mdim("/tmp/sst_mdim.vrt", proxy = TRUE)
stars_proxy object with 4 attributes in 4 file(s):
$sst
[1] "[...]/sst_mdim.vrt"

$anom
[1] "[...]/sst_mdim.vrt"

$err
[1] "[...]/sst_mdim.vrt"

$ice
[1] "[...]/sst_mdim.vrt"

dimension(s):
     from   to offset delta x/y
lon     1 1440      0  0.25 [x]
lat     1  720    -90  0.25 [y]
zlev    1    1      0    NA
time    1   23  17167    16

# GDAL-backed read direct from URL
stars::read_mdim(paste0("/vsicurl/", urls[1L]))
# stars object with 4 dimensions and 4 attributes
# ... same output as Step 4, no local file required

ncdf4::nc_open(urls[1])
# syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or
# SCAN_DATASET or SCAN_ERROR
# context: <!DOCTYPE^ HTML PUBLIC ...><h1>Not Found</h1>...
# Error in ncdf4::nc_open: NetCDF: file not found

ncdf4::nc_open(sprintf("%s#mode=bytes", urls[1]))
# File https://...oisst-avhrr-v02r01.20250101.nc#mode=bytes
# (NC_FORMAT_NETCDF4):
#   4 variables (excluding dimension variables):
#   sst, anom, err, ice ...

# remotes::install_github("grimbough/rhdf5")
library(rhdf5)

fapl <- H5Pcreate("H5P_FILE_ACCESS")
H5Pset_fapl_ros3(fapl)   # no credentials → anonymous mode

fid <- H5Fopen(urls[1], flags = "H5F_ACC_RDONLY", fapl = fapl)
fid
# HDF5 FILE
#         name /
#     filename
#
#   name       otype       dclass              dim
# 0 anom  H5I_DATASET    INTEGER  1440 x 720 x 1 x 1
# 1 err   H5I_DATASET    INTEGER  1440 x 720 x 1 x 1
# 2 ice   H5I_DATASET    INTEGER  1440 x 720 x 1 x 1
# 3 lat   H5I_DATASET      FLOAT  720
# 4 lon   H5I_DATASET      FLOAT  1440
# 5 sst   H5I_DATASET    INTEGER  1440 x 720 x 1 x 1
# 6 time  H5I_DATASET      FLOAT  1
# 7 zlev  H5I_DATASET      FLOAT  1

# remotes::install_github("hypertidy/tidync@multi-input")
tidync::tidync(
  sprintf("%s#mode=bytes", urls),
  fast = TRUE,
  concat_dim = list(name = "time", values = date)
)
# not a file:
# ' https://.../oisst-avhrr-v02r01.20250101.nc#mode=bytes '
#
# ... attempting remote connection
#
# Connection succeeded.
# Data Source (23): oisst-avhrr-v02r01.20250101.nc#mode=bytes,
#                   oisst-avhrr-v02r01.20250117.nc#mode=bytes ...
# Concatenated along 'time' (23/23 steps, 23 sources, fast mode)
#
# Grids (5) <dimension family> : <associated variables>
#
# [1]   D3,D2,D1,D0 : sst, anom, err, ice    **ACTIVE GRID**
# [2]   D0          : time
# [3]   D1          : zlev
# [4]   D2          : lat
# [5]   D3          : lon
#
# Dimensions 4 (all active):
#   dim  name  length     min       max  unlim coord_dim
#   D0   time      23   20089     20441   TRUE  TRUE
#   D1   zlev       1       0         0  FALSE  TRUE
#   D2   lat      720  -89.875   89.875  FALSE  TRUE
#   D3   lon     1440    0.125   359.88  FALSE  TRUE

from lxml import etree
import requests
from urllib.parse import urlparse
import xarray as xr
from obstore.store import HTTPStore
from obspec_utils.registry import ObjectStoreRegistry
from virtualizarr import open_virtual_mfdataset
from virtualizarr.parsers import HDFParser

# 1. parse the VRT to get source URLs
r = requests.get(
    "https://github.com/mdsumner/quarto-blog/releases/download/latest/oisst_2025_sample.vrt"
)
tree = etree.fromstring(r.content)
urls = sorted(set(             # sorted: set() loses order, filenames are date-stamped
    el.text.replace("/vsicurl/", "")
    for el in tree.findall(".//SourceFilename")
))

# 2. set up the object store (anonymous HTTP)
parsed = urlparse(urls[0])
bucket = f"{parsed.scheme}://{parsed.netloc}"
registry = ObjectStoreRegistry({bucket: HTTPStore.from_url(bucket)})
parser = HDFParser()

# 3. open as a single virtual dataset — no data read, no files downloaded
combined = open_virtual_mfdataset(
    urls,
    concat_dim="time",
    parser=parser,
    registry=registry,
    combine="nested",
)

# 4. serialize — a directory store describing all 23 files
combined.virtualize.to_kerchunk("sst_virtual.zarr", format="zarr")

# note the trailing slash — this is a Zarr directory store, not a single file
# fsspec needs it to resolve the .zmetadata index correctly
ds = xr.open_dataset("sst_virtual.zarr/", engine="kerchunk")
# <xarray.Dataset> Size: 763MB
# Dimensions:  (time: 23, zlev: 1, lat: 720, lon: 1440)
# Coordinates:
#   * time  (time) datetime64[ns] 2025-01-01T12:00:00 ... 2025-12-19T12:00:00
#   * zlev  (zlev) float32 0.0
#   * lat   (lat) float32 -89.88 -89.62 ... 89.62 89.88
#   * lon   (lon) float32 0.125 0.375 ... 359.6 359.9
# Data variables:
#     anom  (time, zlev, lat, lon) float64 191MB ...
#     err   (time, zlev, lat, lon) float64 191MB ...
#     ice   (time, zlev, lat, lon) float64 191MB ...
#     sst   (time, zlev, lat, lon) float64 191MB ...
# Attributes: (12/37)
#     Conventions:   CF-1.6, ACDD-1.3
#     title:         NOAA/NCEI 1/4 Degree Daily Optimum Interpolation ...
#     metadata_link: https://doi.org/10.25921/RE9P-PT57
#     sensor:        Thermometer, AVHRR

https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr

xr.open_dataset(
    "https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr/",
    engine="kerchunk"
)

from osgeo import gdal
ds = gdal.OpenEx(
    "ZARR:\"/vsicurl/https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr\"",
    gdal.OF_MULTIDIM_RASTER
)

gdal mdim info "ZARR:\"/vsicurl/https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr\""

z <- zaro("virtualizarr://https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr")
zaro_meta(z)
# [zaro] found .zmetadata (Zarr V2 consolidated)
# [zaro]   8 arrays: anom, err, ice, lat, lon, sst, time, zlev
# <zaro::ZaroMeta>
#  @ zarr_format    : int 2
#  @ node_type      : chr "group"
#  @ attributes     :List of 37
#  .. $ Conventions: chr "CF-1.6, ACDD-1.3"
#  .. $ title      : chr "NOAA/NCEI 1/4 Degree Daily Optimum Interpolation ..."
#  .. $ metadata_link: chr "https://doi.org/10.25921/RE9P-PT57"
#  ...

stars::read_mdim(
  "ZARR:\"/vsicurl/https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr\""
)
# stars object with 4 dimensions and 4 attributes
# attribute(s), summary of first 1e+05 cells:
#            Min. 1st Qu. Median       Mean 3rd Qu. Max.   NAs
# anom [°C] -1.58   -0.61  -0.43 -0.3861891   -0.20 1.45 89635
# err [°C]   0.30    0.30   0.30  0.3042393    0.30 0.35 89635
# ice [%]    0.16    0.62   0.87  0.7743456    0.96 1.00 93223
# sst [°C]  -1.80   -1.80  -1.66 -1.4256025   -1.29 1.19 89635
# dimension(s):
#      from   to                                 offset
# lon     1 1440                                      0
# lat     1  720                                    -90
# zlev    1    1                                  0 [m]
# time    1   23  17167 [days since 1978-01-01T12:00:00]
#                                    delta         refsys x/y
# lon                                 0.25 WGS 84 (CRS84) [x]
# lat                                 0.25 WGS 84 (CRS84) [y]
# zlev                                  NA        udunits
# time  16 [days since 1978-01-01T12:00:00]        udunits

r <- rast(  "ZARR:\"/vsicurl/https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr\"", md  = TRUE)
class       : SpatRaster
size        : 720, 1440, 92  (nrow, ncol, nlyr)
resolution  : 0.25, 0.25  (x, y)
extent      : 0, 360, -90, 90  (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84 (CRS84) (OGC:CRS84)
sources     : sst_virtual.zarr
varnames    : anom (Daily sea surface temperature anomalies)
              err (Estimated error standard deviation of analysed_sst)
              ice (Sea ice concentration)
              sst (Daily sea surface temperature)
names       : anom_~v=0_1, anom_~v=0_2, anom_~v=0_3, anom_~v=0_4, anom_~v=0_5, anom_~v=0_6, ...
unit        :     Celsius,     Celsius,     Celsius,     Celsius,     Celsius,     Celsius, ...
depth       : 0
time (days) : 2025-01-01 to 2025-12-19 (23 steps)

What PROJ accepts as a CRS definition

At the C level, PROJ’s proj_create() function accepts a remarkably wide range of CRS definition strings. The PROJ C API documentation lists the following as valid inputs:

• A proj-string, e.g. "+proj=utm +zone=55 +south +datum=WGS84 +type=crs"
• A WKT string (WKT2:2019 preferred, WKT1 accepted), e.g. GEOGCRS["WGS 84", ...]
• An object code, e.g. "EPSG:4326", "urn:ogc:def:crs:EPSG::4326", or "urn:ogc:def:coordinateOperation:EPSG::1671"
• An object name, e.g. "WGS 84" or "WGS 84 / UTM zone 31N" — matched by heuristics when the name is not unique
• An OGC URN for compound CRS, e.g. "urn:ogc:def:crs,crs:EPSG::2393,crs:EPSG::5717" or the abbreviated "EPSG:2393+5717"
• An OGC URN for concatenated operations, e.g. "urn:ogc:def:coordinateOperation,coordinateOperation:EPSG::3895,coordinateOperation:EPSG::1618"
• A PROJJSON string (added in PROJ 6.2) — the JSON representation of a CRS or coordinate operation, schema at https://proj.org/schemas/v0.4/projjson.schema.json
• A compound CRS from two names joined with " + ", e.g. "WGS 84 + EGM96 height" (added in PROJ 7.1)

That’s a lot of ways to say the same thing. In R, the PROJ package wraps this directly — proj_trans_create() and proj_crs_text() accept all of these forms as character strings.

There is a critical distinction for proj-strings: if the string contains +type=crs, it’s interpreted as a CRS definition. Geographic CRS with +type=crs are assumed to have longitude-latitude axis order with degree units. Without +type=crs, the proj-string is interpreted as a coordinate operation (a transformation step, not a CRS). This matters in some contexts but many packages (PROJ, sf, terra) will detect this case and add the ‘+type=crs’ because it’s common and convenient.

library(PROJ)

## this is a CRS ("+type=crs" present):
proj_crs_text("+proj=laea +type=crs", format = 0L)
## returns a proper WKT2 CRS

## this is technically a transformation step, NOT a CRS (no "+type=crs") but workarounds are often used (as here)
proj_crs_text("+proj=laea", format = 0L)

The PROJ library documentation explicitly says: “The use of proj-string to describe a CRS is discouraged. It is a legacy means of conveying CRS descriptions: use of object codes (EPSG:XXXX typically) or WKT description is recommended for better expressivity.” This is absolutely true but if you want to define your own CRS, just a simple equal area project such as “+proj=laea +lon_0=-147 +lat_0=-42 +datum=WGS4” then you can easily convert from a proj string to WKT, but write the WKT from scratch?? That’s hard but don’t let it put you off, use map projections, specified in a few characters as PROJ strings - if you need a specific datum you’ll need to get a full-blown WKT string by some means.

CRS representations you can work with in R

proj_crs_text() converts between CRS representations. The format argument selects the output:

• format = 0L → WKT2 (the default, most expressive)
• format = 1L → proj-string
• format = 2L → PROJJSON

library(PROJ)

## WKT2
cat(proj_crs_text("EPSG:4326", format = 0L))

GEOGCRS["WGS 84",
    ENSEMBLE["World Geodetic System 1984 ensemble",
        MEMBER["World Geodetic System 1984 (Transit)"],
        MEMBER["World Geodetic System 1984 (G730)"],
        MEMBER["World Geodetic System 1984 (G873)"],
        MEMBER["World Geodetic System 1984 (G1150)"],
        MEMBER["World Geodetic System 1984 (G1674)"],
        MEMBER["World Geodetic System 1984 (G1762)"],
        MEMBER["World Geodetic System 1984 (G2139)"],
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]],
        ENSEMBLEACCURACY[2.0]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    USAGE[
        SCOPE["Horizontal component of 3D system."],
        AREA["World."],
        BBOX[-90,-180,90,180]],
    ID["EPSG",4326]]

## proj-string
proj_crs_text("EPSG:4326", format = 1L)
#> "+proj=longlat +datum=WGS84 +no_defs +type=crs"

## PROJJSON
cat(proj_crs_text("EPSG:4326", format = 2L))

{
  "$schema": "https://proj.org/schemas/v0.7/projjson.schema.json",
  "type": "GeographicCRS",
  "name": "WGS 84",
  "datum_ensemble": {
    "name": "World Geodetic System 1984 ensemble",
    "members": [
      {
        "name": "World Geodetic System 1984 (Transit)",
        "id": {
          "authority": "EPSG",
          "code": 1166
        }
      },
      {
        "name": "World Geodetic System 1984 (G730)",
        "id": {
          "authority": "EPSG",
          "code": 1152
        }
      },
      {
        "name": "World Geodetic System 1984 (G873)",
        "id": {
          "authority": "EPSG",
          "code": 1153
        }
      },
      {
        "name": "World Geodetic System 1984 (G1150)",
        "id": {
          "authority": "EPSG",
          "code": 1154
        }
      },
      {
        "name": "World Geodetic System 1984 (G1674)",
        "id": {
          "authority": "EPSG",
          "code": 1155
        }
      },
      {
        "name": "World Geodetic System 1984 (G1762)",
        "id": {
          "authority": "EPSG",
          "code": 1156
        }
      },
      {
        "name": "World Geodetic System 1984 (G2139)",
        "id": {
          "authority": "EPSG",
          "code": 1309
        }
      },
      {
        "name": "World Geodetic System 1984 (G2296)",
        "id": {
          "authority": "EPSG",
          "code": 1383
        }
      }
    ],
    "ellipsoid": {
      "name": "WGS 84",
      "semi_major_axis": 6378137,
      "inverse_flattening": 298.257223563
    },
    "accuracy": "2.0",
    "id": {
      "authority": "EPSG",
      "code": 6326
    }
  },
  "coordinate_system": {
    "subtype": "ellipsoidal",
    "axis": [
      {
        "name": "Geodetic latitude",
        "abbreviation": "Lat",
        "direction": "north",
        "unit": "degree"
      },
      {
        "name": "Geodetic longitude",
        "abbreviation": "Lon",
        "direction": "east",
        "unit": "degree"
      }
    ]
  },
  "scope": "Horizontal component of 3D system.",
  "area": "World.",
  "bbox": {
    "south_latitude": -90,
    "west_longitude": -180,
    "north_latitude": 90,
    "east_longitude": 180
  },
  "id": {
    "authority": "EPSG",
    "code": 4326
  }
}