hypertidy

Icechunk is coming - versioned cloud-native Zarr, GDAL, and R

Michael Sumner — Fri, 26 Jun 2026 00:00:00 GMT

Icechunk is a transactional storage engine for Zarr and provides git-style versioning for large scientific datasets. This post is about Icechunk from a GDAL perspective and some impact that will have on the R community and its interactions with Zarr.

What is Icechunk?

Zarr is a chunked array format designed for cloud object storage. Icechunk is a Rust library that sits on top of Zarr and adds git-like snapshots combined with ACID transactions - (which means reliable and guaranteed read and write).

(From their docs) this provides:

Safety: Recover from corrupted or accidentally deleted data using version history
Consistency: Ensure readers always see complete, valid snapshots - never partial writes
Reproducibility: Reference any version of your data permanently via commits or tags

Typical usage involves the Python packages xarray and icechunk, but Rust Icechunk can be used directly, in combination with zarrs_icechunk, and now in an upcoming GDAL release 3.14 via the read-only Icechunk driver.

Please be aware that Icechunk in GDAL is very new, and not yet in an officially supported release but if you are interested to get involved it can be used today. (GDAL 3.14 is milestoned for sometime in November 2026).

What works in GDAL and R today

GDAL 3.14 ships a /vsiicechunk/ virtual filesystem and Icechunk driver, and will be available in gdalcubes, gdalraster, sf, terra, and vapour in various forms once that support lands in R packages.

The following examples can be run in this Linux docker container, for example:

docker pull ghcr.io/hypertidy/gdal-r-full:dev
docker run --rm -ti  ghcr.io/hypertidy/gdal-r-full:dev

(If that doesn’t work, or you want help with docker/GDAL/R just contact me directly)

Here is a single cli call that gives a GDAL multidimensional summary of an Icechunk store.

gdal mdim info \
  /vsis3/dynamical-ecmwf-aifs-single/ecmwf-aifs-single-forecast/v0.1.0.icechunk \
  --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-west-2

Click here for the full mdim info output

R, GDAL, Zarr, and the Missing Recipe Format for Raster Datasets

Michael Sumner — Wed, 27 May 2026 00:00:00 GMT

The virtual-Zarr format that lets xarray describe 16,000 files in a tiny store doesn’t exist in R yet - but the pieces are in place, this post tries to show where they fit together .

A common scenario in geospatial data workflows is multiple raster files. Let’s consider 23 snapshots of OISST sea surface temperature spread across 2025. One NetCDF file per date, each raster is the same grid, four variables each. Let’s see what R does with that. Here we assume files is a vector of paths to the NetCDF files.

(Fully reproducible setup, downloading directly from NCEI is in the appendix.)

str(basename(files))
# chr [1:23] "oisst-avhrr-v02r01.20250101.nc" "oisst-avhrr-v02r01.20250117.nc" "oisst-avhrr-v02r01.20250202.nc" ...
library(terra)
rast(files)
# class       : SpatRaster
# size        : 720, 1440, 92  (nrow, ncol, nlyr)
# resolution  : 0.25, 0.25  (x, y)
# extent      : 0, 360, -90, 90
# coord. ref. : lon/lat WGS 84 (CRS84)
# sources     : oisst-avhrr-v02r01.20250101.nc (4 layers)
#               oisst-avhrr-v02r01.20250117.nc (4 layers)
#               ... and 21 more sources
# varnames    : anom, err, ice, sst, anom, err, ...
# names       : anom_zlev=0, err_zlev=0, ice_zlev=0, sst_zlev=0, ...
# time (days) : 2025-01-01 to 2025-12-19 (23 steps)

The result is pretty good. terra::rast() is no slouch: it reads all subdatasets across all files, concatenates band-wise, preserves the time axis, carries units, and records depth metadata. For a lot of workflows this is exactly what is needed, and it’s lazy — no pixel values are read until they are needed.

But look at the names: anom_zlev=0, err_zlev=0, ice_zlev=0, sst_zlev=0 — repeated 23 times, 92 layers total. Four variables and 23 timesteps are there, flattened into band order. The n-D structure — this is a (variable × time × lat × lon) array — is implicit in the interleaving, not encoded anywhere. There’s an implicit contract about how this was unrolled, there are conventions for band ordering, workarounds for the _zlev=0 naming. Nothing is wrong, but the band-stack model forces all of that n-D structure into a single axis, and recovering it requires a lot of information that is not usual in the multi-band model.

We can sidestep the variable interleaving with the vrt:// URI syntax:

vrt(
  sprintf("vrt://%s?sd_name=sst", files),
  "/tmp/sst_stack.vrt",
  options = "-separate",
  overwrite = TRUE
)
# class       : SpatRaster
# size        : 720, 1440, 23  (nrow, ncol, nlyr)
# source      : sst_stack.vrt
# names       : r_1, r_2, r_3, ..., r_23

One variable, 23 layers. The vrt:// GDAL protocol is worth knowing: it lets us name a subdataset in a URI query string, vectorizing cleanly with sprintf and using the underlying raster-stack capability of GDAL VRT (-separate). But there’s a trade-off: no time axis, layer names are r_1 through r_23 rather than dates. We fixed the variable interleaving problem and lost the coordinate metadata.

rast(files, "sst") also selects a single subdataset without the vrt:// syntax, and produces the same 23-layer result — but it goes through the ncdf4 backend rather than GDAL, so it doesn’t produce a VRT file and doesn’t carry the same interoperability guarantees. The two approaches look similar but are doing different things underneath, and that distinction matters for what comes next.

stars approaches this differently. In default proxy (lazy) mode it uses GDAL classic raster and keeps variables as separate attributes, file-per-timestep structure explicit, doing this by harvesting metadata from the underlying files as carried by GDAL. stars::read_mdim() goes further — it uses the GDAL multidimensional API and gives named attributes, real dimension coordinates with udunits, proper n-D structure. The representation is right, but it too has limits; details are in the appendix.

None of these approaches is the best — each embeds real expertise and genuine design choices. But with excellent tools like terra and stars, you still can’t hand someone a file that says “this is a year of global SST.” You can hand them code, or you can hand them gigabytes. The code only works if they have the same files in the same place. The gigabytes require a format decision and a copy operation.

There’s no serializable recipe that is light, language-agnostic, and handles a broad variety of cases.

Fifteen minutes with xarray

A key moment for me was while learning Python and xarray, I was impressed that xarray could open these files — a huge list of them (16,337 days of OISST at time of writing) as local files or remote URLs. xarray checks them all for consistency, detects that what is really there is four variables in longitude, latitude, and time, and that time is an implicit concatenation from a single time step in each of those thousands of files. It’s remarkable: one object in memory in Python, ready to lazy-read any part of the four 16337×720×1440 arrays. But that took about 15 minutes to run, let alone the software and computing infrastructure setup I had been through. And then: how do you share that progress with colleagues?

Everybody knows you aren’t supposed to serialize in-memory objects to binary, we all have scary experiences with .Rdata and .rds, and Python has the same with its pickle. (I thought: “hmmm, what’s the xarray equivalent of a VRT?” Spoiler: there isn’t one — and although VirtualiZarr does serve that role, it’s missing an intermediate state where we only record what files were opened and what structure was learned.)

What the VRT already does

Back to our 23 files — a sample of the 16,337+ in the full OISST set as of writing.

Open /tmp/sst_stack.vrt in a text editor (browseURL("/tmp/sst_stack.vrt") works too). It’s a few kilobytes of XML that fully describes a virtual 23-layer SST dataset — source files, spatial extent, band mapping. It’s a tiny file: you could email it, put it in a repository, or patch the source paths to /vsicurl/https://....nc URLs so it works without any local files at all. Anyone with GDAL can open it with rast("sst_stack.vrt") and get exactly what you built, without copying a byte of the underlying data.

That’s a recipe: we’ve described our dataset as a small chunk of text that GDAL can open very quickly.

VRT (Virtual Format) is a plain XML file that describes a virtual raster dataset. It can mosaic, warp, reproject, reband, and combine source files without touching the underlying data. For the 2D raster world it is exactly the recipe format that’s missing from the terra/stars in-memory story. The 2D VRT is one of GDAL’s most underappreciated features — already powering a lot of geospatial infrastructure, especially where band-stacking or tile-mosaicking is sufficient as the solution.

So what about the full n-D case: multiple variables, time coordinates, arbitrary dimensions?

What Zarr figured out

The Python/xarray community has been working on exactly this problem, approaching it from the chunk level rather than the file level.

Zarr is a chunked array format. An array is split into regular chunks; each chunk is compressed independently and stored as a separate object — a file, an S3 object, a range of bytes in a larger store. The chunks themselves are opaque: raw numeric arrays in compressed bytes, not self-describing. What makes Zarr work is the metadata sidecar: a small JSON file that records the shape, dtype, chunk layout, compression codec, and dimension coordinates — everything xarray needs to provide a rich, self-describing interface over the data. This is really good for cloud access: fetch only the chunks you need, in parallel, without a format-specific library. It’s a bit like taking a pile of NetCDF files and turning them inside out — all the metadata (tiny in comparison to the data) is placed upfront in JSON, and all the compressed opaque chunks are stored as individually addressable objects.

The deeper insight comes when you realize that HDF5, netCDF4, GeoTIFF, GRIB, and others already store their data in chunks. The chunks are buried inside files that require format-specific libraries to navigate, but the byte ranges are all there. We just need an index.

kerchunk and its successor VirtualiZarr do exactly that: open your existing files, find where each chunk lives (file path, byte offset, byte length), and write that index as a JSON or Parquet reference store. The bulk data never moves. What you get is a Zarr-shaped recipe that points back into your original files. It’s worth noting that this idea is not new — NASA and Unidata’s DMR++ format did the same thing for OPeNDAP, indexing chunk byte-ranges in HDF5/netCDF files as a sidecar that an OPeNDAP server can serve directly. Cloud-Optimized GeoTIFF is the same idea applied to TIFF: the IFD tile index is moved to the front of the file so a client can read the chunk map in one range request and fetch only what it needs thereafter. The difference is that Zarr’s client-side framing made the pattern composable and ecosystem-wide rather than tied to a specific server or format.

NOTE: this is very much pseudo code showing the overall style - see the Usage VirtualiZarr pages for more examples, and we have a real example in the appendix.

from virtualizarr import open_virtual_dataset
import xarray as xr

# index the chunks — no data copied
combined = xr.concat(
    [open_virtual_dataset(f) for f in files],
    dim="time"
)
combined.virtualize.to_kerchunk("sst_virtual.zarr", format="zarr")

# open it — chunk-aware, lazy, S3-ready, fully structured
ds = xr.open_dataset("sst_virtual.zarr/", engine="kerchunk")
# 
# Dimensions:  (time: 23, zlev: 1, lat: 720, lon: 1440)
# Coordinates:
#   * time     (time) datetime64[ns] 2025-01-01 ... 2025-12-19
#   * lat      (lat) float32 -90.0 ... 89.75
#   * lon      (lon) float32 0.0 ... 359.75
# Data variables: anom, err, ice, sst

The store is kilobytes of metadata, the actual data stays where it is, and the n-D structure is fully encoded. Compare this to the terra output above:

sst, anom, err, ice as named variables, not interleaved bands
time as a real coordinate with date values, not layer indices r_1..r_23
lat, lon, zlev as labelled dimensions with coordinate values
the whole thing serialized as a directory anyone with Zarr support can open

This is what stars::read_mdim() is reaching toward in R — the representation is structurally similar. The difference is that the xarray object came from a small Zarr store, not from reconstructing structure at runtime over 23 original netCDF files. That difference is the recipe.

GDAL is the bridge

GDAL already spans both worlds.

GDAL has a multidimensional API (the “mdim” layer) that represents data as named arrays with labelled dimensions — the same model as xarray, not the band-stacked model of classic GDAL rasters. GDAL’s Zarr driver can open VirtualiZarr/kerchunk reference stores through this API. A store created in Python is readable via GDAL — which means it’s reachable from R today.

More importantly, GDAL has gdal mdim mosaic — a tool that takes a collection of raster files and produces a multidim VRT describing them as a unified n-D array dataset. This VRT encodes the array structure, dimension names, coordinate values, and source file mapping. It is the GDAL-native recipe for an n-D dataset — and as shown in the appendix, stars::read_mdim() opens it faithfully, giving you the full multi-file dataset with the right structure, right coordinates, and a serializable description that travels independently of the original files.

Converting a GDAL multidim VRT to a Zarr reference store is a small transformation on kilobytes of XML, not a data operation. That’s the near-term path to R-native VirtualiZarr creation.

The gap: creation, not reading

R can read virtual stores — Zarr and multidim VRT and VirtualiZarr Parquet reference via stars::read_mdim(), gdalraster::mdim_translate or zaro or real Zarr with pizzarr or zarr. What R can’t yet do is create the chunk-level byte-reference stores that VirtualiZarr generates for HDF5 and netCDF.

That’s a frontier, and I’ve been exloring where the pieces could converge:

gdal mdim mosaic creates multidim VRTs from file collections today. The XML fully describes what a VirtualiZarr store needs — shape, chunks, coordinate values, source file paths and byte offsets. This is the hard structural work; emitting the Zarr reference store format from it is a straightforward conversion.

rhdf5 can now open remote HDF5 files at URLs or on S3 (dev version on GitHub). The next step is extracting chunk byte-reference indices directly - a core operation that VirtualiZarr performs in Python (with HDF5 library, GRIB libraries, TIFF libraries some in C++ and some in Rust). Step 7 in the appendix shows remote access, and a pending PR in my rhdf5 fork has already use that to harvest remote byte references from HDF5 and NetCDF.

zaro (pure-R Zarr reader via Arrow) already reads VirtualiZarr Parquet stores. The reading side is there, and needs comparison with pizzarr, and zarr and other zarr-enabled packages.

vrtstack generates multidim VRTs from 2D raster collections in R - the gdal mdim mosaic equivalent for simple cases of multiple 2D raster - without leaving R.

The path: vrtstack or gdal mdim mosaic for file collections → emit Zarr reference store → readable by zaro, xarray, or any Zarr-aware tool. In this way R can become a first-class creator of portable dataset descriptions, not just a consumer of what Python produces.

Why this matters beyond R

A Zarr reference store created from OISST netCDF files is immediately openable in Python, R, GDAL and in theory any tool that speaks Zarr.

STAC is complementary but different: STAC describes where files are and their spatiotemporal footprint. A virtual store describes what the dataset structurally is — how chunks map to array coordinates, what the dimensions mean, how to reassemble a slice. A STAC item can point to a virtual store the way it points to a COG.

The R community has genuine leverage here. GDAL is the common substrate between R and the xarray/Zarr ecosystem. R is not just catching up, there are things we can do that aren’t on any other roadmap.

What would help

If you work with time series of raster files and you’ve felt this friction:

Talk about your use case — what files, what structure, what would a portable dataset description let you do that you can’t do today?
Try terra::vrt() and try the vrt:// URI syntax for netCDF subdataset selection.
Try stars::read_mdim() on a gdal mdim mosaic VRT — the full tour in the appendix shows how these fit together
Try remotes::install_github("hypertidy/tidync@multi-input") if you have remote netCDF files and want to skip the download entirely — Step 8 in the appendix shows what this already looks like.
Look at vrtstack and zaro on the hypertidy GitHub, vrtstack builds a multidim VRT from 2D rasters (multidim GDAL supports COG, but doesn’t support builiding these), zaro is a read-implementation for Zarr that is primarily based around Arrow.
File issues against GDAL’s multidim API when you find gaps — R users exercising it in real workflows makes it better for everyone.
Check out gdalraster (and my multidim branch) there’s a long way to go for multidim but gdalraster is a rich package has a super strong basis for faithfully carrying the core facilities of the multidim api exposed to R.

Appendix: a progressive tour on 23 OISST files

Setup: download 23 OISST files from NCEI

All code below uses only this setup, and we use a multidim VRT representation later to act as an intermediate form.

library(glue)

date  <- seq(as.Date("2025-01-01"), by = "16 days", length.out = 23)
base0 <- "https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr"
ym    <- format(date, "%Y%m")
ymd   <- format(date, "%Y%m%d")
urls  <- glue("{base0}/{ym}/oisst-avhrr-v02r01.{ymd}.nc")

dir   <- tempdir()
files <- file.path(dir, basename(urls))
download.file(urls, files, method = "libcurl", mode = "wb")

Setup part 2: create a GDAL multidimensional VRT

writeLines(glue::glue("/vsicurl/{urls}"), tf <- tempfile())
system(glue::glue("gdal mdim mosaic @{tf} oisst_2025_sample.vrt"))
## piggyback::pb_upload("oisst_2025_sample.vrt")

The URL structure itself is the catalogue: a simple path template over date components, no database required. 23 files, ~3.5MB each, ~80MB total.

Step 1 — terra: `rast()`, band-stacked, all variables, time preserved

rast(files)
# class       : SpatRaster
# size        : 720, 1440, 92  (nrow, ncol, nlyr)
# sources     : oisst-avhrr-v02r01.20250101.nc (4 layers)
#               ... and 22 more sources
# varnames    : anom, err, ice, sst, anom, err, ice, sst, ...
# names       : anom_zlev=0, err_zlev=0, ice_zlev=0, sst_zlev=0, ...
# unit        :      Celsius,    Celsius,          %,    Celsius, ...
# time (days) : 2025-01-01 to 2025-12-19 (23 steps)

terra uses the ncdf4 R package for netCDF reads — a design inherited from the raster package - rather than routing through GDAL. This is why its behaviour differs from stars proxy mode. The time axis survives via ncdf4’s own metadata handling. The n-D structure is flattened into 92 bands; recovering it means knowing the 4-variable interleaving convention.

Bundling all bands from all variables that share the same dimensions is a choice made by terra, GDAL has a similar capacity but it is not used here (and is in fact unusable by these files for other reasons).

Step 2 — terra: `vrt()` with `vrt://`, single variable, time lost

vrt(
  sprintf("vrt://%s?sd_name=sst", files),
  "/tmp/sst_stack.vrt",
  options = "-separate",
  overwrite = TRUE
)
# class       : SpatRaster
# size        : 720, 1440, 23  (nrow, ncol, nlyr)
# source      : sst_stack.vrt
# names       : r_1, r_2, r_3, ..., r_23

vrt:// URI syntax selects a subdataset inline; ?sd_name=sst is the query param.. The result is a real file — a 2D recipe for this one variable. But trade-offs: time coordinate is gone, layer names are positional. The 2D VRT has no slot for dimension coordinates, it can only carry raw metadata on a dataset or on a band.

Step 3 — stars: `read_stars()` proxy, GDAL classic raster, variables separated

read_stars(files)
# stars_proxy object with 4 attributes in 92 file(s):
# $sst
#  [1] "[...]/oisst-avhrr-v02r01.20250101.nc:sst"
#  [2] "[...]/oisst-avhrr-v02r01.20250117.nc:sst"
#  ...
# $anom, $err, $ice  (similarly, 23 files each)
#
# dimension(s):
#      from   to offset delta  refsys x/y
# x       1 1440      0  0.25      NA [x]
# y       1  720     90 -0.25      NA [y]
# zlev    1    1  0 [m]    NA udunits
# time    1   23     NA    NA      NA

stars proxy mode uses GDAL classic raster and separates variables as attributes rather than interleaving as bands — a meaningful structural difference from terra. Time is present as a dimension but without coordinate values, it’s fully lazy at this point as made clear by the lists of files in subdataset syntax.

Step 4 — stars: `read_mdim()` on a single file, right model, wrong scope

stars::read_mdim(files[1L])
# stars object with 4 dimensions and 4 attributes
# attribute(s):
#            Min.  1st Qu.  Median      Mean  3rd Qu.   Max.     NAs
# sst [°C]  -1.80    -1.80   -1.78  -1.70779    -1.64  -1.03   89635
# anom [°C] -0.72    -0.08   -0.02  -0.03531     0.00   0.49   89635
# err [°C]   0.30     0.30    0.30   0.30446     0.30   0.35   89635
# ice [%]    0.19     0.81    0.95   0.87747     0.98   1.00   91877
# dimension(s):
#      from   to  offset  delta          refsys x/y
# lon     1 1440       0   0.25  WGS 84 (CRS84) [x]
# lat     1  720     -90   0.25  WGS 84 (CRS84) [y]
# zlev    1    1  0 [m]      NA         udunits
# time    1    1  17532 [days since 1978-01-01 12:00:00]  NA  udunits

read_mdim() goes through GDAL’s multidimensional API: named attributes, labelled dimensions, udunits-encoded time coordinate, proper spatial reference. This is a rich representation of the dataset.

But pass stars all 23 files and it silently opens only the first, one timestep, and no error. stars has other ways of concatenating files along dimensions but it’s only applied to GDAL classic raster context.

This is a pivot point.

Step 5 — `gdal mdim mosaic` + `read_mdim()`, the full tour

gdal mdim mosaic is the missing piece. It takes a collection of files and produces a multidim VRT that describes them as a unified n-D array — dimension coordinates and all:

# writeLines(sprintf("/vsicurl/%s", urls), "filelist.txt")
gdal mdim mosaic --input @filelist.txt -o /tmp/sst_mdim.vrt

Or from R via vrtstack or a direct system() call. Now hand that VRT to read_mdim():

stars::read_mdim("/tmp/sst_mdim.vrt", proxy = TRUE)
stars_proxy object with 4 attributes in 4 file(s):
$sst
[1] "[...]/sst_mdim.vrt"

$anom
[1] "[...]/sst_mdim.vrt"

$err
[1] "[...]/sst_mdim.vrt"

$ice
[1] "[...]/sst_mdim.vrt"

dimension(s):
     from   to offset delta x/y
lon     1 1440      0  0.25 [x]
lat     1  720    -90  0.25 [y]
zlev    1    1      0    NA
time    1   23  17167    16

23 time steps in /tmp/sst_mdim.vrt which is a tiny file of XML.

This is what gdal mdim mosaic unlocks: read_mdim() and friends need VRT to make the multi-file case work.

Step 6 — skip the download: URL access, who can and can’t

The files we downloaded are publicly accessible over plain HTTPS. Why download at all? Let’s see who can open them directly.

GDAL — just works. GDAL’s /vsicurl/ virtual filesystem handles HTTP range requests transparently, and both terra::rast() (via GDAL for the spatial read) and stars::read_mdim() can open the URLs directly once you prefix them appropriately. No download needed.

# GDAL-backed read direct from URL
stars::read_mdim(paste0("/vsicurl/", urls[1L]))
# stars object with 4 dimensions and 4 attributes
# ... same output as Step 4, no local file required

ncdf4 — the picture is more complicated. ncdf4 routes through the netCDF-C library, which expects either a local file path or an OPeNDAP/Thredds dodsC URL. A plain HTTPS URL fails, and the error is instructive:

ncdf4::nc_open(urls[1])
# syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or
# SCAN_DATASET or SCAN_ERROR
# context: Not Found
...
# Error in ncdf4::nc_open: NetCDF: file not found

ncdf4 tried to parse the 404 HTML response as netCDF syntax. But there’s a workaround — the #mode=bytes suffix instructs the HDF5 layer to use HTTP byte-range requests rather than treating the URL as a file path:

ncdf4::nc_open(sprintf("%s#mode=bytes", urls[1]))
# File https://...oisst-avhrr-v02r01.20250101.nc#mode=bytes
# (NC_FORMAT_NETCDF4):
#   4 variables (excluding dimension variables):
#   sst, anom, err, ice ...

It works — but you have to know the magic suffix. And because terra’s netCDF reading is backed by ncdf4, it inherits this same limitation: plain HTTPS URLs don’t work without #mode=bytes, and OPeNDAP dodsC URLs are the usual workaround for terra users working with remote data.

This is the library-identity problem: the same URL provides three different outcomes depending on which library is doing the opening. GDAL’s cloud-native HTTP access is a first-class citizen but the ’/vsi*’ prefix is clearly not popular in some communities; NetCDF’s is an afterthought requiring a suffix that hardly anyone seems to know about. The recipe format inherits these differences — a VRT that references HTTPS URLs is GDAL-native and just works; a workflow built on NetCDF needs extra plumbing.

Step 7 — rhdf5-dev: opening the door to chunk indexing (dev, not yet on CRAN)

The development version of rhdf5 on GitHub can open remote HDF5 files over HTTPS and S3 using the ROS3 virtual file driver — no download, no #mode=bytes workaround, just direct anonymous access:

# remotes::install_github("grimbough/rhdf5")
library(rhdf5)

fapl <- H5Pcreate("H5P_FILE_ACCESS")
H5Pset_fapl_ros3(fapl)   # no credentials → anonymous mode

fid <- H5Fopen(urls[1], flags = "H5F_ACC_RDONLY", fapl = fapl)
fid
# HDF5 FILE
#         name /
#     filename
#
#   name       otype       dclass              dim
# 0 anom  H5I_DATASET    INTEGER  1440 x 720 x 1 x 1
# 1 err   H5I_DATASET    INTEGER  1440 x 720 x 1 x 1
# 2 ice   H5I_DATASET    INTEGER  1440 x 720 x 1 x 1
# 3 lat   H5I_DATASET      FLOAT  720
# 4 lon   H5I_DATASET      FLOAT  1440
# 5 sst   H5I_DATASET    INTEGER  1440 x 720 x 1 x 1
# 6 time  H5I_DATASET      FLOAT  1
# 7 zlev  H5I_DATASET      FLOAT  1

What you’re seeing here is the raw HDF5 structure of the file — variables as datasets, dimensions as separate float arrays, all visible without reading a single data value. This is exactly the level at which VirtualiZarr operates: it walks this structure, reads the chunk B-tree for each dataset, and records the byte offset and length of every chunk.

Once rhdf5 can expose those chunk byte-references — which is the next step in its development — R will have a native path to VirtualiZarr store creation without Python, without shelling out, and working directly on files at HTTPS or S3 URLs. The H5Pset_fapl_ros3() call with no credentials is the anonymous S3/HTTPS access pattern: it works for any publicly accessible endpoint that supports range requests, including NCEI, NASA Earthdata public buckets, and most open data archives.

The reading side (zaro) is already there. The indexing side (rhdf5-dev) is almost there. The assembly side (gdal mdim mosaic → Zarr reference store) is a small conversion on top of infrastructure that already exists. The pieces are in place.

Step 8 — tidync dev: multi-file remote concat in one call (unmerged dev branch)

tidync recently got an update in a dev branch to accept multiple files — long on the wishlist but out of reach in the early years of the package. I had also forgotten that remote access to NetCDF is possible via its own library by appending #mode=bytes to the URL — this means we don’t always need an OPeNDAP-enabled server for remote read. (GDAL uses its own techniques for remote access via /vsicurl/.)

# remotes::install_github("hypertidy/tidync@multi-input")
tidync::tidync(
  sprintf("%s#mode=bytes", urls),
  fast = TRUE,
  concat_dim = list(name = "time", values = date)
)
# not a file:
# ' https://.../oisst-avhrr-v02r01.20250101.nc#mode=bytes '
#
# ... attempting remote connection
#
# Connection succeeded.
# Data Source (23): oisst-avhrr-v02r01.20250101.nc#mode=bytes,
#                   oisst-avhrr-v02r01.20250117.nc#mode=bytes ...
# Concatenated along 'time' (23/23 steps, 23 sources, fast mode)
#
# Grids (5)  : 
#
# [1]   D3,D2,D1,D0 : sst, anom, err, ice    **ACTIVE GRID**
# [2]   D0          : time
# [3]   D1          : zlev
# [4]   D2          : lat
# [5]   D3          : lon
#
# Dimensions 4 (all active):
#   dim  name  length     min       max  unlim coord_dim
#   D0   time      23   20089     20441   TRUE  TRUE
#   D1   zlev       1       0         0  FALSE  TRUE
#   D2   lat      720  -89.875   89.875  FALSE  TRUE
#   D3   lon     1440    0.125   359.88  FALSE  TRUE

Twenty-three remote netCDF files concatenated into one lazy n-D dataset in a single call — #mode=bytes vectorized over the URL vector, a synthetic time axis injected via concat_dim, all four variables on the active grid, unlimited time dimension, proper coordinates throughout. We have tested this on the full set of 16,000+ URLs and it opens in less than a minute using the fast and concat_dim mechanism to declare the values on the concatenating dimension.

The “not a file / attempting remote connection / Connection succeeded” in the output is tidync doing its own detective work: it knows this isn’t a local path, tries the remote connection, and succeeds. The #mode=bytes suffix — the ncdf4 workaround from Step 6 — unlocks the whole thing.

This is an R equivalent of the xarray open: the same structure, the same laziness, the same multi-file concatenation. The recipe here is the code rather than a store on disk — and the tidync object it produces is fully navigable with hyper_filter() and hyper_array() without touching the data. The serializable recipe (the Zarr store, the multidim VRT) is still the goal for sharing and interoperability, but as a working representation for analysis, this is already there.

In the last section of the appendix we virtualize to the “normal” xarray/Zarr model, but then open that again with different tools. The same store, opened by four different tools across two languages - there’s a recipe, but it’s not a nailed down ecosystem yet.

The xarray counterpart — for comparison

First create the VirtualiZarr version. We use the GDAL mdim VRT hosted above because it’s convenient — and that’s where this effort will start: GDAL already has all the metadata and array concatenating logic, and we can pair that with byte reference extraction improvements to GDAL’s GetRawBlockInfo().

from lxml import etree
import requests
from urllib.parse import urlparse
import xarray as xr
from obstore.store import HTTPStore
from obspec_utils.registry import ObjectStoreRegistry
from virtualizarr import open_virtual_mfdataset
from virtualizarr.parsers import HDFParser

# 1. parse the VRT to get source URLs
r = requests.get(
    "https://github.com/mdsumner/quarto-blog/releases/download/latest/oisst_2025_sample.vrt"
)
tree = etree.fromstring(r.content)
urls = sorted(set(             # sorted: set() loses order, filenames are date-stamped
    el.text.replace("/vsicurl/", "")
    for el in tree.findall(".//SourceFilename")
))

# 2. set up the object store (anonymous HTTP)
parsed = urlparse(urls[0])
bucket = f"{parsed.scheme}://{parsed.netloc}"
registry = ObjectStoreRegistry({bucket: HTTPStore.from_url(bucket)})
parser = HDFParser()

# 3. open as a single virtual dataset — no data read, no files downloaded
combined = open_virtual_mfdataset(
    urls,
    concat_dim="time",
    parser=parser,
    registry=registry,
    combine="nested",
)

# 4. serialize — a directory store describing all 23 files
combined.virtualize.to_kerchunk("sst_virtual.zarr", format="zarr")

Now open with xarray:

# note the trailing slash — this is a Zarr directory store, not a single file
# fsspec needs it to resolve the .zmetadata index correctly
ds = xr.open_dataset("sst_virtual.zarr/", engine="kerchunk")
#  Size: 763MB
# Dimensions:  (time: 23, zlev: 1, lat: 720, lon: 1440)
# Coordinates:
#   * time  (time) datetime64[ns] 2025-01-01T12:00:00 ... 2025-12-19T12:00:00
#   * zlev  (zlev) float32 0.0
#   * lat   (lat) float32 -89.88 -89.62 ... 89.62 89.88
#   * lon   (lon) float32 0.125 0.375 ... 359.6 359.9
# Data variables:
#     anom  (time, zlev, lat, lon) float64 191MB ...
#     err   (time, zlev, lat, lon) float64 191MB ...
#     ice   (time, zlev, lat, lon) float64 191MB ...
#     sst   (time, zlev, lat, lon) float64 191MB ...
# Attributes: (12/37)
#     Conventions:   CF-1.6, ACDD-1.3
#     title:         NOAA/NCEI 1/4 Degree Daily Optimum Interpolation ...
#     metadata_link: https://doi.org/10.25921/RE9P-PT57
#     sensor:        Thermometer, AVHRR

That virtual store is available here:

https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr

Open with xarray directly from the URL (note the trailing slash):

xr.open_dataset(
    "https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr/",
    engine="kerchunk"
)

Open with GDAL:

from osgeo import gdal
ds = gdal.OpenEx(
    "ZARR:\"/vsicurl/https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr\"",
    gdal.OF_MULTIDIM_RASTER
)

gdal mdim info "ZARR:\"/vsicurl/https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr\""

Open with zaro:

z <- zaro("virtualizarr://https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr")
zaro_meta(z)
# [zaro] found .zmetadata (Zarr V2 consolidated)
# [zaro]   8 arrays: anom, err, ice, lat, lon, sst, time, zlev
# 
#  @ zarr_format    : int 2
#  @ node_type      : chr "group"
#  @ attributes     :List of 37
#  .. $ Conventions: chr "CF-1.6, ACDD-1.3"
#  .. $ title      : chr "NOAA/NCEI 1/4 Degree Daily Optimum Interpolation ..."
#  .. $ metadata_link: chr "https://doi.org/10.25921/RE9P-PT57"
#  ...

Open with stars:

stars::read_mdim(
  "ZARR:\"/vsicurl/https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr\""
)
# stars object with 4 dimensions and 4 attributes
# attribute(s), summary of first 1e+05 cells:
#            Min. 1st Qu. Median       Mean 3rd Qu. Max.   NAs
# anom [°C] -1.58   -0.61  -0.43 -0.3861891   -0.20 1.45 89635
# err [°C]   0.30    0.30   0.30  0.3042393    0.30 0.35 89635
# ice [%]    0.16    0.62   0.87  0.7743456    0.96 1.00 93223
# sst [°C]  -1.80   -1.80  -1.66 -1.4256025   -1.29 1.19 89635
# dimension(s):
#      from   to                                 offset
# lon     1 1440                                      0
# lat     1  720                                    -90
# zlev    1    1                                  0 [m]
# time    1   23  17167 [days since 1978-01-01T12:00:00]
#                                    delta         refsys x/y
# lon                                 0.25 WGS 84 (CRS84) [x]
# lat                                 0.25 WGS 84 (CRS84) [y]
# zlev                                  NA        udunits
# time  16 [days since 1978-01-01T12:00:00]        udunits

Oh, and turns out terra::rast() does support multidim (not our VRT file sadly):

r <- rast(  "ZARR:\"/vsicurl/https://raw.githubusercontent.com/mdsumner/quarto-blog/refs/heads/main/data/sst_virtual.zarr\"", md  = TRUE)
class       : SpatRaster
size        : 720, 1440, 92  (nrow, ncol, nlyr)
resolution  : 0.25, 0.25  (x, y)
extent      : 0, 360, -90, 90  (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84 (CRS84) (OGC:CRS84)
sources     : sst_virtual.zarr
varnames    : anom (Daily sea surface temperature anomalies)
              err (Estimated error standard deviation of analysed_sst)
              ice (Sea ice concentration)
              sst (Daily sea surface temperature)
names       : anom_~v=0_1, anom_~v=0_2, anom_~v=0_3, anom_~v=0_4, anom_~v=0_5, anom_~v=0_6, ...
unit        :     Celsius,     Celsius,     Celsius,     Celsius,     Celsius,     Celsius, ...
depth       : 0
time (days) : 2025-01-01 to 2025-12-19 (23 steps)

There’s a lot here.

Michael Sumner is a research software engineer at the Australian Antarctic Division. Packages mentioned — tidync (rOpenSci reviewed), vapour, vrtstack, zaro — are part of the hypertidy ecosystem.

Check it out: geos

Michael Sumner — Tue, 24 Mar 2026 00:00:00 GMT

The geos package provides a direct R interface to the GEOS library. It operates on a geos_geometry vector type and speaks wk natively — any wk-handleable geometry can be passed in, and the results can flow back out to wk, sf, or anything else.

This is the geometry engine for the wk ecosystem. Where wk handles representation and PROJ handles transformation, geos handles computation: buffers, intersections, unions, predicates, distances, triangulations.

Geometry vectors

geos stores geometries in an external pointer vector class:

library(geos)

g <- as_geos_geometry(c(
  "POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))",
  "POINT (5 5)",
  "LINESTRING (0 0, 10 10)"
))
g
#> 
#> [1] 
#> [2] 
#> [3]

The input can be WKT strings, WKB, wk vectors, or sf objects — anything with a wk_handle() method gets converted automatically:

## from wk::xy()
as_geos_geometry(wk::xy(1:5, 1:5))

## from sf
library(sf)
nc <- st_read(system.file("shape/nc.shp", package = "sf"), quiet = TRUE)
nc_geos <- as_geos_geometry(nc)

Operations

geos provides vectorized access to the GEOS C API. Operations are functions, not methods — they take geometry vectors and return geometry vectors or values:

## buffer
geos_buffer(g[1], distance = 2)

## intersection
geos_intersection(g[1], geos_buffer(g[2], 3))

## union
geos_unary_union(g)

## simplify
geos_simplify(g[3], tolerance = 1)

## centroid
geos_centroid(g[1])

Binary predicates are also vectorized:

geos_intersects(g[1], g[2])
#> [1] TRUE

geos_contains(g[1], g[2])
#> [1] TRUE

geos_disjoint(g[1], g[3])
#> [1] FALSE

Spatial indexing

geos includes an STRtree implementation for fast spatial queries:

## build a spatial index
pts <- as_geos_geometry(wk::xy(runif(10000), runif(10000)))
tree <- geos_strtree(pts)

## query the index
query_geom <- as_geos_geometry("POLYGON ((0.4 0.4, 0.6 0.4, 0.6 0.6, 0.4 0.6, 0.4 0.4))")
geos_strtree_query(tree, query_geom)

The newer geos_basic_strtree() uses wk::wk_envelope() for index construction, which is extremely efficient for wk::xy() input — the envelope is just the point itself, computed without any geometry parsing overhead.

Construction from coordinates

You don’t need to go through WKT to build geometries. geos provides direct constructors:

## points from x, y
geos_make_point(1:3, 1:3)

## linestring from coordinates
geos_make_linestring(c(0, 1, 2, 3), c(0, 1, 0, 1))

## polygon from coordinates (ring is closed automatically)
geos_make_polygon(c(0, 10, 10, 0), c(0, 0, 10, 10))

## collections
geos_make_collection(
  c("POINT (0 0)", "POINT (1 1)"),
  type_id = "multipoint"
)

These constructors use feature_id and ring_id arguments for building multi-part and multi-ring geometries from flat coordinate vectors, similar to how data often arrives from databases or CSV files.

wk compatibility

geos implements wk_handle() for its geometry type, so all wk operations work:

## extract coordinates
wk::wk_coords(nc_geos)

## compute bounding boxes
wk::wk_envelope(nc_geos)

## convert to wk types
wk::as_wkt(nc_geos)
wk::as_wkb(nc_geos)

## convert to sf
sf::st_as_sfc(nc_geos)

This means geos integrates into any workflow that uses wk as the interchange format. Read vector data with gdalraster (WKB output), convert to geos for geometric operations, transform with PROJ, write back with gdalraster — all through wk.

geos vs sf for geometric operations

sf uses GEOS internally for its geometry operations (st_buffer(), st_intersection(), etc.). The difference is:

sf wraps GEOS through its own C++ layer, with results returned as sfc geometry columns inside sf data frames. The operations are tightly integrated with the sf data model.
geos wraps GEOS directly, with results as standalone geometry vectors. No data frame, no attributes — just geometry in, geometry out.

If you’re working within sf, use sf’s functions. If you’re working outside sf — with wk vectors, plain data frames, or formats that don’t map to sf’s model — geos gives you the same geometric operations without the sf world view.

Workflows can deliberately separate I/O (gdalraster/vapour), geometry (wk/geos), transformation (PROJ/reproj), and even grid logic (vaster), geos is the geometric computation layer.

Measurement and properties

geos_area("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))")
#> [1] 100

geos_length("LINESTRING (0 0, 10 0, 10 10)")
#> [1] 20

geos_is_valid("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))")
#> [1] TRUE

geos_is_empty("POINT EMPTY")
#> [1] TRUE

geos_num_coordinates("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))")
#> [1] 5

Install from CRAN:

install.packages("geos")

Links:

Check it out series

This is part of a series of posts for foundational spatial support in R. These posts include:

Check it out: reproj

Michael Sumner — Tue, 24 Mar 2026 00:00:00 GMT

The reproj package provides a single generic function, reproj(), for transforming coordinates between map projections. It works with matrices, data frames, and anything else that a package author cares to add a method for.

Coordinate transformation is a basic operation on numbers, it shouldn’t require a specific spatial data format.

The generic

library(reproj)

## matrix in, matrix out
reproj(cbind(147, -42), target = "+proj=laea +datum=WGS84", source = "OGC:CRS84")
#>          [,1]     [,2] [,3]
#> [1,] 5969744 -9803200    0

The source and target arguments accept anything that the PROJ library understands: EPSG codes, proj-strings, WKT2, authority:code strings. The output is always a matrix — 2, 3, or 4 columns depending on input and the four argument.

Convenience helpers reproj_xy() and reproj_xyz() force 2- or 3-column output:

reproj_xy(cbind(147, -42), target = "+proj=laea +datum=WGS84", source = "OGC:CRS84")
#>          [,1]     [,2]
#> [1,] 5969744 -9803200

The generic method model

reproj() is an S3 generic. This means any package can import it and add a method for its own spatial type. The method extracts the source CRS from the object and the user only needs to supply the target:

## hypothetical method for a spatial type that carries its own CRS
reproj.my_spatial_thing <- function(x, target, ...) {
  coords <- extract_coords(x)
  source <- get_crs(x)
  result <- reproj(coords, target = target, source = source, ...)
  rebuild_my_thing(x, result, crs = target)
}

This pattern is already used by the silicate and quadmesh packages — their reproj() methods know where the coordinates live and what the source CRS is, so the user just says reproj(my_mesh, target_crs).

The value of this design is that the generic lives in a small, dependency-light package. A downstream package can @importFrom reproj reproj without pulling in GDAL or any heavy spatial stack. The transformation itself is done by the PROJ library via the PROJ R package (preferred) or the proj4 package as fallback.

For example we could write a reproj.sf and a .sfc:

reproj.sfc <- function(x, target, ..., source = NULL) {
  ## handle this format with wk
  handled <- wk::as_wkb(x)
  crs <- wk::wk_crs(handled)$wkt
  ## stop if not a valid crs, we need source specified
  if (is.null(source)) {
    stopifnot(!is.na(crs))
  } else {
    crs <- source
 }
 trans <- PROJ::proj_trans_create(crs, target)
 sf::st_as_sfc(wk::wk_transform(handled, trans))
}

reproj.sf <- function(x, target, ..., source = NULL) {
 sf::st_set_geometry(x, reproj.sfc(sf::st_geometry(x), target, source = source))
}

That’s not very useful since sf::st_transform already exists but demonstrates the utility for any other object, and you can force input of a source if your object does not support that already. We could decompose our object to coordinates and transform those more directly, but anything handled by wk is already supported by this simpler mechanism.

CRS preservation

A key principle: when reproj transforms an object that has a method, the output carries the new CRS. The silicate method rebuilds the mesh with the target CRS attached. The quadmesh method does the same. The idea is that format-specific methods preserve the full object structure while updating both coordinates and CRS metadata.

For plain matrices and data frames, there’s no CRS to carry, and the target and source must be specified.

Under the hood: PROJ package

reproj delegates the actual math to the PROJ R package, which wraps the PROJ C library. The PROJ package provides:

PROJ::proj_trans() — transform coordinates, accepting wk types
PROJ::proj_trans_create() — create a reusable transformation object (a wk_trans)
PROJ::proj_crs_text() — convert between CRS representations (WKT2, proj-string, PROJJSON)

reproj vs sf::st_transform vs terra::project

sf and terra both have projection functions, but they’re tied to their respective data models. sf::st_transform() works on sf objects. terra::project() works on SpatVector and SpatRaster.

reproj works on anything. It’s deliberately format-agnostic: give it a matrix of coordinates, a source CRS, and a target CRS. This makes it useful for packages that don’t depend on sf or terra, and for workflows where spatial data isn’t in a standard spatial format — track data as data frames, coordinates pulled from a netCDF, vertices from an rgl mesh, cell centres computed from a grid specification.

Example: transforming track data

library(reproj)

## some GPS fixes as a plain data frame
tracks <- data.frame(
  lon = c(147.3, 147.4, 147.5),
  lat = c(-42.9, -42.8, -42.7),
  time = as.POSIXct(c("2024-01-01 08:00", "2024-01-01 09:00", "2024-01-01 10:00"))
)

## transform to a local projection
xy <- reproj_xy(
  cbind(tracks$lon, tracks$lat),
  target = "+proj=utm +zone=55 +south +datum=WGS84",
  source = "EPSG:4326"
)

tracks$x <- xy[, 1]
tracks$y <- xy[, 2]

Install from CRAN:

install.packages("reproj")

Links:

Check it out series

This is part of a series of posts for foundational spatial support in R. These posts include:

Check it out: wk

Michael Sumner — Tue, 24 Mar 2026 00:00:00 GMT

The wk package is a foundation for geometry handling in R. It’s a lightweight, dependency-free geometry column with CRS support, vctrs integration, and a handler-based streaming API. Any data frame with a wk column works as well as the column itself with supported functions.

What wk provides

wk defines several vector types that carry geometry and coordinate reference system information as first-class attributes:

wk::wkt() — well-known text, human-readable
wk::wkb() — well-known binary, compact and fast
wk::xy(), wk::xyz(), wk::xym(), wk::xyzm() — coordinate vectors (points)
wk::rct() — rectangles (bounding boxes)
wk::crc() — circles

All of these carry a crs attribute and behave properly as vctrs types. They can live in data frame columns, get subset, combined, and compared — with CRS compatibility checked automatically.

library(wk)
pts <- wk::xy(c(147, 148, 149), c(-42, -43, -44), crs = "EPSG:4326")
pts
#> 
#> [1] (147 -42) (148 -43) (149 -44)

You can put these directly in a data frame:

d <- data.frame(name = c("Hobart", "South", "Further"), geometry = pts)
d
#>     name     geometry
#> 1 Hobart  (147 -42)
#> 2  South  (148 -43)
#> 3 Further (149 -44)

Subset, combine, and merge all work correctly. The CRS propagates through operations and mismatches are caught:

## combining vectors with different CRS will error
pts_other <- wk::xy(1, 2, crs = "EPSG:3857")
try(c(pts, pts_other))
#> Error: CRS objects are not equal

CRS handling

The CRS system in wk is deliberately general. At the C level, the PROJ library proj_create() function accepts a remarkably wide range of CRS definition strings. The PROJ C API documentation lists many valid types of inputs from ‘auth:code’, PROJJSON, WKT variants and PROJ4 strings. The CRS support in wk is similarly open to inputs, the key functions in wk are:

wk_crs() — get the CRS from any wk-compatible object
wk_set_crs() — set or replace the CRS
wk_crs_equal() — compare CRS values
wk_crs_proj_definition() — extract a PROJ-compatible string (used by PROJ and other packages)

The sentinel value wk_crs_inherit() signals “inherit from context” and is used for empty or zero-length geometry vectors. This means you can safely combine an empty geometry with a CRS-bearing one without triggering a mismatch.

Interchange

wk acts as a common interchange format. Other formats that are supported are ‘raw binary’, ‘raw text’, geos package vectors, sf data frames and geometry vectors. The gdalraster package can emit vector geometry in wk format. Conversions to and from other formats are built in:

## sf -> wk
library(sf)
nc <- st_read(system.file("shape/nc.shp", package = "sf"), quiet = TRUE)
wkb <- wk::as_wkb(nc)
wk_crs(wkb)
#> [1] "NAD27"

## wk -> sf
sf::st_as_sfc(wkb)

## terra -> wk
geos::as_geos_geometry(terra::vect(<src>))

The geos package accepts wk types directly. The PROJ package transforms them. The gdalraster package (via ogr_layer_create() and friends) can write them. This is an interchange layer that allows packages to stay independent of formats, just as the underlying libraries GDAL, PROJ, GEOS already allow.

The handler API

The real power of wk is its streaming handler API. Rather than materializing geometry in memory, wk can stream coordinates through a chain of handlers — read, transform, and write without ever building an intermediate object.

## count coordinates without materializing geometry
wk_count(wkb)

## extract coordinates as a data frame
wk_coords(wkb)

## compute bounding boxes per feature
wk_envelope(wkb)

## transform coordinates (see the PROJ post) https://www.hypertidy.org/posts/2026-03-24_check-out-proj-wk-transform/
wk_transform(pts, PROJ::proj_trans_create("EPSG:4326", "EPSG:3857"))

The wk_handle() generic is what makes this extensible. Any package can implement a wk_handle() method for its geometry type and immediately plug into the wk ecosystem — filters, transforms, writers all just work.

Geometric operations via wk

wk itself includes several useful operations that don’t require GEOS:

wk_vertices() — extract all vertices from a geometry
wk_coords() / wk_coords<-() — get/set coordinates
wk_flatten() — flatten collections to simple geometries
wk_collection() — collect simple geometries into multi/collection types
wk_linestring() / wk_polygon() — construct from coordinates
wk_set_z() / wk_drop_z() — add or remove Z dimension
wk_orient() / wk_clockwise() / wk_counterclockwise() — rewind polygon rings

For geometric predicates, buffering, intersection and the like try geos (see the geos post in this series).

Install from CRAN:

install.packages("wk")

Links:

Check it out series

This is part of a series of posts for foundational spatial support in R. These posts include:

Check it out: proj_trans_create and wk_transform

Michael Sumner — Tue, 24 Mar 2026 00:00:00 GMT

The PROJ R package provides proj_trans_create(), which creates a reusable coordinate transformation object that plugs directly into wk’s wk_transform(). This is the mechanism that connects the PROJ C library to any geometry type that speaks wk.

This post goes into some detail on what CRS representations are accepted, how R’s CRS objects get resolved to strings that PROJ understands, and how wk’s wk_crs_proj_definition() generic bridges the gap.

What PROJ accepts as a CRS definition

At the C level, PROJ’s proj_create() function accepts a remarkably wide range of CRS definition strings. The PROJ C API documentation lists the following as valid inputs:

A proj-string, e.g. "+proj=utm +zone=55 +south +datum=WGS84 +type=crs"
A WKT string (WKT2:2019 preferred, WKT1 accepted), e.g. GEOGCRS["WGS 84", ...]
An object code, e.g. "EPSG:4326", "urn:ogc:def:crs:EPSG::4326", or "urn:ogc:def:coordinateOperation:EPSG::1671"
An object name, e.g. "WGS 84" or "WGS 84 / UTM zone 31N" — matched by heuristics when the name is not unique
An OGC URN for compound CRS, e.g. "urn:ogc:def:crs,crs:EPSG::2393,crs:EPSG::5717" or the abbreviated "EPSG:2393+5717"
An OGC URN for concatenated operations, e.g. "urn:ogc:def:coordinateOperation,coordinateOperation:EPSG::3895,coordinateOperation:EPSG::1618"
A PROJJSON string (added in PROJ 6.2) — the JSON representation of a CRS or coordinate operation, schema at https://proj.org/schemas/v0.4/projjson.schema.json
A compound CRS from two names joined with " + ", e.g. "WGS 84 + EGM96 height" (added in PROJ 7.1)

That’s a lot of ways to say the same thing. In R, the PROJ package wraps this directly — proj_trans_create() and proj_crs_text() accept all of these forms as character strings.

There is a critical distinction for proj-strings: if the string contains +type=crs, it’s interpreted as a CRS definition. Geographic CRS with +type=crs are assumed to have longitude-latitude axis order with degree units. Without +type=crs, the proj-string is interpreted as a coordinate operation (a transformation step, not a CRS). This matters in some contexts but many packages (PROJ, sf, terra) will detect this case and add the ‘+type=crs’ because it’s common and convenient.

library(PROJ)

## this is a CRS ("+type=crs" present):
proj_crs_text("+proj=laea +type=crs", format = 0L)
## returns a proper WKT2 CRS

## this is technically a transformation step, NOT a CRS (no "+type=crs") but workarounds are often used (as here)
proj_crs_text("+proj=laea", format = 0L)

The PROJ library documentation explicitly says: “The use of proj-string to describe a CRS is discouraged. It is a legacy means of conveying CRS descriptions: use of object codes (EPSG:XXXX typically) or WKT description is recommended for better expressivity.” This is absolutely true but if you want to define your own CRS, just a simple equal area project such as “+proj=laea +lon_0=-147 +lat_0=-42 +datum=WGS4” then you can easily convert from a proj string to WKT, but write the WKT from scratch?? That’s hard but don’t let it put you off, use map projections, specified in a few characters as PROJ strings - if you need a specific datum you’ll need to get a full-blown WKT string by some means.

CRS representations you can work with in R

proj_crs_text() converts between CRS representations. The format argument selects the output:

format = 0L → WKT2 (the default, most expressive)
format = 1L → proj-string
format = 2L → PROJJSON

library(PROJ)

## WKT2
cat(proj_crs_text("EPSG:4326", format = 0L))

GEOGCRS["WGS 84",
    ENSEMBLE["World Geodetic System 1984 ensemble",
        MEMBER["World Geodetic System 1984 (Transit)"],
        MEMBER["World Geodetic System 1984 (G730)"],
        MEMBER["World Geodetic System 1984 (G873)"],
        MEMBER["World Geodetic System 1984 (G1150)"],
        MEMBER["World Geodetic System 1984 (G1674)"],
        MEMBER["World Geodetic System 1984 (G1762)"],
        MEMBER["World Geodetic System 1984 (G2139)"],
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]],
        ENSEMBLEACCURACY[2.0]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    USAGE[
        SCOPE["Horizontal component of 3D system."],
        AREA["World."],
        BBOX[-90,-180,90,180]],
    ID["EPSG",4326]]

## proj-string
proj_crs_text("EPSG:4326", format = 1L)
#> "+proj=longlat +datum=WGS84 +no_defs +type=crs"

## PROJJSON
cat(proj_crs_text("EPSG:4326", format = 2L))

{
  "$schema": "https://proj.org/schemas/v0.7/projjson.schema.json",
  "type": "GeographicCRS",
  "name": "WGS 84",
  "datum_ensemble": {
    "name": "World Geodetic System 1984 ensemble",
    "members": [
      {
        "name": "World Geodetic System 1984 (Transit)",
        "id": {
          "authority": "EPSG",
          "code": 1166
        }
      },
      {
        "name": "World Geodetic System 1984 (G730)",
        "id": {
          "authority": "EPSG",
          "code": 1152
        }
      },
      {
        "name": "World Geodetic System 1984 (G873)",
        "id": {
          "authority": "EPSG",
          "code": 1153
        }
      },
      {
        "name": "World Geodetic System 1984 (G1150)",
        "id": {
          "authority": "EPSG",
          "code": 1154
        }
      },
      {
        "name": "World Geodetic System 1984 (G1674)",
        "id": {
          "authority": "EPSG",
          "code": 1155
        }
      },
      {
        "name": "World Geodetic System 1984 (G1762)",
        "id": {
          "authority": "EPSG",
          "code": 1156
        }
      },
      {
        "name": "World Geodetic System 1984 (G2139)",
        "id": {
          "authority": "EPSG",
          "code": 1309
        }
      },
      {
        "name": "World Geodetic System 1984 (G2296)",
        "id": {
          "authority": "EPSG",
          "code": 1383
        }
      }
    ],
    "ellipsoid": {
      "name": "WGS 84",
      "semi_major_axis": 6378137,
      "inverse_flattening": 298.257223563
    },
    "accuracy": "2.0",
    "id": {
      "authority": "EPSG",
      "code": 6326
    }
  },
  "coordinate_system": {
    "subtype": "ellipsoidal",
    "axis": [
      {
        "name": "Geodetic latitude",
        "abbreviation": "Lat",
        "direction": "north",
        "unit": "degree"
      },
      {
        "name": "Geodetic longitude",
        "abbreviation": "Lon",
        "direction": "east",
        "unit": "degree"
      }
    ]
  },
  "scope": "Horizontal component of 3D system.",
  "area": "World.",
  "bbox": {
    "south_latitude": -90,
    "west_longitude": -180,
    "north_latitude": 90,
    "east_longitude": 180
  },
  "id": {
    "authority": "EPSG",
    "code": 4326
  }
}

PROJJSON is the most machine-friendly format — it’s the full CRS definition in a structure you can parse, manipulate, and round-trip without information loss. WKT2 is the canonical text representation and equally complete. Proj-strings are the most compact, but lose information (datum ensemble members, axis order details, usage scope). Every conversion to proj-string is potentially lossy.

How wk bridges CRS objects to PROJ strings

In the wk package, a CRS can be any R object. The crs attribute on a wk::wkt(), wk::wkb(), or wk::xy() vector is deliberately untyped — it could be a string like "EPSG:4326", an integer like 4326, an sf crs object, or something else entirely. wk doesn’t interpret the CRS, it just propagates it.

The bridge to PROJ is wk_crs_proj_definition(). This is an S3 generic that takes a CRS object and returns a character string that the PROJ library can understand. This is the function that PROJ::proj_trans() and PROJ::proj_trans_create() call internally to resolve CRS arguments.

The built-in methods handle the common cases:

character → returned as-is (assumed to be an authority:code string, WKT, PROJJSON, or proj-string already)
integer or double → formatted as "EPSG:" (e.g. 4326 → "EPSG:4326")


NULL → NA_character_ (no CRS)
wk_crs_inherit() → NA_character_ (inherit from context)


library(wk)

wk_crs_proj_definition("EPSG:4326")
#> [1] "EPSG:4326"

wk_crs_proj_definition(4326)
#> [1] "EPSG:4326"

wk_crs_proj_definition(4326L)
#> [1] "EPSG:4326"
For sf’s crs class, sf registers its own method so that wk_crs_proj_definition() returns the correct string for the installed version of PROJ. Other packages can do the same — this is the extension point for custom CRS types.
The verbose argument controls how much detail is returned. With verbose = FALSE (the default), it returns the most compact form — typically the authority:code string. With verbose = TRUE, it returns PROJJSON or WKT2 for maximum expressivity.

wk_crs_projjson()
There’s also wk_crs_projjson(), which returns the PROJJSON representation of a CRS. For shortcut-style CRS values like "EPSG:4326" or 4326, wk includes cached lookup tables — wk_proj_crs_view and wk_proj_crs_json — containing pre-rendered PROJJSON for all CRS definitions in the PROJ database (based on PROJ 9.3.0 as of wk 0.9.x). This means wk_crs_projjson("EPSG:4326") works without PROJ being installed — the lookup is pure R against shipped data.
## get PROJJSON for a known EPSG code, no PROJ library needed
pj <- wk_crs_projjson("EPSG:4326")
cat(pj)
## full PROJJSON output
This is the mechanism that allows wk-based packages to inspect CRS properties without linking to the PROJ C library.



The transformation pipeline
With all that context, here’s how proj_trans_create() works:

It calls wk_crs_proj_definition() on the source_crs and target_crs arguments to get PROJ-compatible strings
It passes those strings to the PROJ C library’s proj_create_crs_to_crs() function, which finds the best available transformation pipeline between the two CRS
It wraps the result in a wk_trans object — an external pointer with an wk_trans_inverse() method

library(PROJ)
library(wk)

## create a transformation object
trans <- proj_trans_create("EPSG:4326", "EPSG:3857")
<proj_trans at 0x55800593fd60>
type: Concatenated Operation
id: pipeline
description: axis order change (2D) + Popular Visualisation Pseudo-Mercator
definition: proj=pipeline step proj=unitconvert xy_in=deg xy_out=rad step proj=webmerc lat_0=0 lon_0=0 x_0=0 y_0=0 ellps=WGS84
area_of_use:
  name: World
  bounds: [-180 -90 180 90]
source_crs:
  type: Geographic 2D CRS
  id: EPSG:4326
  name: WGS 84
  area_of_use:
    name: World.
    bounds: [-180 -90 180 90]
target_crs:
  type: Projected CRS
  id: EPSG:3857
  name: WGS 84 / Pseudo-Mercator
  area_of_use:
    name: World between 85.06°S and 85.06°N.
    bounds: [-180 -85.06 180 85.06]
The source and target can be any form PROJ accepts — EPSG codes, WKT2, PROJJSON, proj-strings with +type=crs, object names:
## all of these create equivalent transformations:
proj_trans_create("EPSG:4326", "EPSG:3857")
proj_trans_create(4326, 3857)
proj_trans_create("WGS 84", "WGS 84 / Pseudo-Mercator")
proj_trans_create(
  "+proj=longlat +datum=WGS84 +type=crs",
  "+proj=merc +a=6378137 +b=6378137 +lat_ts=0 +lon_0=0 +x_0=0 +y_0=0 +k=1 +units=m +type=crs"
)


Applying the transformation with wk_transform
wk::wk_transform() takes any wk-handleable geometry and applies the transformation through wk’s streaming handler API. This means the transformation works on any geometry type — points, lines, polygons, collections — without materializing intermediate representations:
## transform point coordinates
pts <- wk::xy(c(147, 148, 149), c(-42, -43, -44), crs = "EPSG:4326")
wk_transform(pts, trans)
#> 
#> [1] (16362893 -5160980) (16474013 -5311971) (16585132 -5465442)

## transform well-known text geometries
poly <- wk::wkt(
  "POLYGON ((147 -42, 148 -42, 148 -43, 147 -43, 147 -42))",
  crs = "EPSG:4326"
)
wk_transform(poly, trans)
#> 
#> [1] POLYGON ((16362893 -5160980, ...))

## transform well-known binary
wkb_geom <- wk::as_wkb(poly)
wk_transform(wkb_geom, trans)

## transform sf geometries (sf implements wk_handle)
## wk_transform(sf_sfc_object, trans) — just works
The CRS of the output is updated automatically. The output carries the target CRS from the transformation.
The transformation object is reusable — create it once, apply it to many geometry vectors. This avoids paying the PROJ pipeline setup cost for each call.


Inverse transformations
wk_trans_inverse() flips a transformation:
inv <- wk_trans_inverse(trans)

## round-trip
pts_proj <- wk_transform(pts, trans)
wk_transform(pts_proj, inv)
#> 
#> [1] (147 -42) (148 -43) (149 -44)


The convenience wrapper: PROJ::proj_trans()
If you don’t need a reusable transformation object, PROJ::proj_trans() combines the create and transform steps. It accepts matrices, data frames, wk geometry vectors, and sf sfc columns:
## matrix form — requires explicit source
PROJ::proj_trans(cbind(147, -42), "+proj=laea +type=crs", "EPSG:4326")
#>          x        y
#> [1,] 5969744 -9803200

## wk type with CRS — source is inferred from the data
PROJ::proj_trans(wk::xy(147, -42, crs = "EPSG:4326"), "+proj=laea +type=crs")
#> 
#> [1] (5969744 -9803200)

## full WKT geometries work too
PROJ::proj_trans(
  wk::wkt("POLYGON ((1 1, 0 1, 0 0, 1 0, 1 1))", crs = "EPSG:4326"),
  3112
)
#> 
#> [1] POLYGON ((-1.351177e+07 -7779443, ...))
When the input has a CRS (via wk_crs()), the source_crs argument can be omitted — proj_trans() reads it from the data using wk_crs_proj_definition().


The +type=crs distinction
This is worth emphasizing because it catches people. In PROJ 6+, a bare proj-string like "+proj=laea" is a coordinate operation, not a CRS. The library draws a hard line between “a description of a coordinate reference system” and “a description of a mathematical operation to apply to coordinates.”
To specify a CRS using a proj-string, you must include +type=crs:
## CRS — works as source/target for transformations
proj_crs_text("+proj=laea +type=crs", format = 0L)
## returns proper WKT2

## coordinate operation — not a CRS
proj_crs_text("+proj=laea", format = 0L)
## NOT a CRS definition
Object codes ("EPSG:4326"), WKT2, and PROJJSON are unambiguous — they always describe a CRS. Proj-strings need the +type=crs disambiguation. The recommendation from the PROJ documentation is to prefer authority:code strings or WKT2 for CRS definitions. But, use proj strings because they are easy and expressive, when you need a formal WKT2 for a particular operation define it carefully.


How this connects to reproj
The reproj package (see the reproj post in this series) uses PROJ::proj_trans() under the hood.
The relationship is:

reproj — high-level generic, format-agnostic, delegates to PROJ
PROJ — mid-level wrapper around the PROJ C library, returns wk types, uses wk_crs_proj_definition() for CRS resolution
wk — the geometry representation, streaming protocol, and CRS bridge generic
PROJ C library (libproj) — the actual math, proj_create() does the parsing

If you want maximum control and reusable transformation objects, use proj_trans_create() + wk_transform() directly. If you want a quick one-liner for a matrix of coordinates, use reproj(). They’re different interfaces to the same underlying capability.


Further reading

PROJ C API: Transformation setup — the canonical reference for what proj_create() accepts
PROJJSON schema
wk wk_crs_proj_definition() docs
PROJ R package proj_trans() docs
wk wk_transform() docs

Install from CRAN:
install.packages("PROJ")
install.packages("wk")


Check it out series
This is part of a series of posts for foundational spatial support in R. These posts include:

wk
reproj
PROJ/wk transform (this one!)
gdalraster
geos



Check it out: gdalraster
Michael Sumner — Tue, 24 Mar 2026 00:00:00 GMT
 






The gdalraster package provides a comprehensive interface to the GDAL library for R. It handles raster I/O, vector I/O, the virtual filesystem, coordinate transformation, and a large collection of GDAL utilities — all through a direct, low-level API that doesn’t impose a high-level spatial data model.

Raster I/O
The core raster interface is GDALRaster, an Rcpp module class that wraps a GDAL dataset handle:
library(gdalraster)

f <- system.file("extdata/storm_lake.lcp", package = "gdalraster")
ds <- new(GDALRaster, f, TRUE)

ds$info()
ds$dim()       # c(ncol, nrow, nbands)
ds$res()       # c(xres, yres)
ds$bbox()      # c(xmin, ymin, xmax, ymax)
ds$getProjection()

## read a band
data <- ds$read(band = 1,
                xoff = 0, yoff = 0,
                xsize = ds$getRasterXSize(),
                ysize = ds$getRasterYSize(),
                out_xsize = ds$getRasterXSize(),
                out_ysize = ds$getRasterYSize())

ds$close()
The read() method maps directly to GDAL’s RasterIO — you specify the source window and the output size. This gives full control over decimation, oversampling, and partial reads. No intermediate abstraction, no automatic resampling decisions you didn’t ask for.


Vector I/O
gdalraster provides vector read/write through OGR:
## list layers in a data source
dsn <- system.file("extdata/ynp_fires_1984_2022.gpkg", package = "gdalraster")
ogr_ds_layer_names(dsn)

## feature count and extent
ogr_ds_layer_count(dsn)
ogr_layer_field_names(dsn, "mtbs_perims")

## read features as a data frame with WKB geometry
lyr <- new(GDALVector, dsn, "mtbs_perims", read_only = TRUE)
feat <- lyr$fetch(-1)  # all features
lyr$close()
The geometry comes back as WKB — directly compatible with wk. You can pass it to wk::as_wkt(), geos::as_geos_geometry(), or sf::st_as_sfc() as needed. gdalraster reads the data; what you do with the geometry is up to you and whatever packages you choose.


Virtual filesystem
GDAL’s /vsicurl/, /vsizip/, /vsimem/ and other virtual filesystem handlers are all available:
## read a remote file without downloading
vsi_read_dir("/vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/")

## open a GeoPackage over HTTP
dsn <- "/vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AUS.gpkg"
ogr_ds_layer_names(dsn)

## read inside a remote zip
vsi_read_dir("/vsizip//vsicurl/https://github.com/OSGeo/gdal/raw/refs/heads/master/autotest/gcore/data/byte.tif.zip")
#[1] "byte.tif"
This is essential for cloud-native workflows. The same dataset string that works locally works over HTTP, S3, or Azure — GDAL handles the transport.


The warper
warp() provides access to GDAL’s warper API for raster reprojection and resampling:
## warp to a new projection
warp(
  src_filename = "input.tif",
  dst_filename = "output.tif",
  t_srs = "EPSG:3577",
  tr = c(100, 100),           # target resolution
  r = "bilinear"              # resampling method
)
This is the same warper that gdalwarp uses on the command line. For building VRTs, mosaics, and complex raster pipelines, gdalraster provides buildVRT(), translate(), and rasterFromRaster().


VRT construction
One of the most powerful features is programmatic VRT (Virtual Raster) construction:
## create a VRT from multiple source files
buildVRT(
  vrt_filename = "mosaic.vrt",
  input_rasters = c("tile1.tif", "tile2.tif", "tile3.tif")
)

## add sources to a VRT programmatically
ds <- new(GDALRaster, "output.vrt", FALSE)
ds$addSimpleSource(band = 1, src_filename = "source.tif", ...)
ds$close()
VRTs are GDAL’s answer to lazy raster operations — the XML file describes the operation but nothing is computed until you read pixels.


Configuration and utilities
gdalraster exposes GDAL configuration options:
## set GDAL config for performance tuning
set_config_option("GDAL_CACHEMAX", "512")
set_config_option("GDAL_HTTP_MULTIRANGE", "YES")
set_config_option("GDAL_NUM_THREADS", "ALL_CPUS")

## check what's set
get_config_option("GDAL_CACHEMAX")

## GDAL/PROJ/GEOS version info
gdal_version()
Plus a full set of geometry utilities (g_buffer(), g_intersection(), etc.) that use GEOS through GDAL’s OGR layer, and coordinate transformation via transform_xy().
Install from CRAN:
install.packages("gdalraster")
Links:

gdalraster on CRAN
gdalraster documentation
GitHub



Check it out series
This is part of a series of posts for foundational spatial support in R. These posts include:

wk
reproj
PROJ/wk transform
gdalraster (this one!)
geos





 


What’s on geodata.ucdavis.edu and how to access it
Michael Sumner — Tue, 17 Mar 2026 00:00:00 GMT
 






The R package geodata provides functions for downloading climate, elevation, soil, crop, species occurrence, and administrative boundary data. All of its raster and vector data is hosted on a single Apache web server at geodata.ucdavis.edu.
Every file on that server is directly accessible via HTTP. This means we can enumerate, inspect, and in some cases read them using GDAL’s virtual filesystem — without downloading anything first.
This post documents the full site structure, the file formats served, and which files can be streamed via /vsicurl/ (they all can, but without efficient tiling it’s probably best to make local copies for large raster).

Helper functions
The code snippets below use a small set of helper functions that wrap gdalraster::vsi_read_dir() and related GDAL calls. Source them with:

source("https://github.com/mdsumner/quarto-blog/releases/download/latest/geodata-ucdavis-tools.R")

This provides: vsi_list(), geodata_files(), gdalinfo(), check_block_size(), and streamability_report().


Site structure
The server has three top-level paths that contain spatial data:







Path
Contents




/climate/
WorldClim 2.1 and 1.4, CMIP5 legacy data


/cmip6/
Downscaled CMIP6 projections


/geodata/
Soil, landcover, crops, elevation, travel time, OSM, buildings


/gadm/
GADM administrative boundaries (versions 2.8, 3.6, 4.0, 4.1)



The root URL itself and some intermediate directories serve an index.html that says “Hello World!” rather than an Apache directory listing. The data directories below these do have standard Apache autoindex enabled, which is what makes vsi_read_dir() work.


Enumerating files
GDAL’s /vsicurl/ handler can parse Apache directory listings. The gdalraster package exposes this via vsi_read_dir():

library(gdalraster)

GDAL 3.13.0dev-a3b7b01d3e-dirty (released 2026-02-23), GEOS 3.12.1, PROJ 9.7.0

## flat directory — returns filenames
vsi_read_dir("/vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/")

 [1] "agg"                           "bdod_0-5cm_mean_30s.tif"      
 [3] "bdod_15-30cm_mean_30s.tif"     "bdod_30-60cm_mean_30s.tif"    
 [5] "bdod_5-15cm_mean_30s.tif"      "cec_0-5cm_mean_30s.tif"       
 [7] "cec_15-30cm_mean_30s.tif"      "cec_30-60cm_mean_30s.tif"     
 [9] "cec_5-15cm_mean_30s.tif"       "cfvo_0-5cm_mean_30s.tif"      
[11] "cfvo_15-30cm_mean_30s.tif"     "cfvo_30-60cm_mean_30s.tif"    
[13] "cfvo_5-15cm_mean_30s.tif"      "clay_0-5cm_mean_30s.tif"      
[15] "clay_15-30cm_mean_30s.tif"     "clay_30-60cm_mean_30s.tif"    
[17] "clay_5-15cm_mean_30s.tif"      "files.txt"                    
[19] "fill"                          "nitrogen_0-5cm_mean_30s.tif"  
[21] "nitrogen_15-30cm_mean_30s.tif" "nitrogen_30-60cm_mean_30s.tif"
[23] "nitrogen_5-15cm_mean_30s.tif"  "ocd_0-5cm_mean_30s.tif"       
[25] "ocd_15-30cm_mean_30s.tif"      "ocd_30-60cm_mean_30s.tif"     
[27] "ocd_5-15cm_mean_30s.tif"       "phh2o_0-5cm_Q0.05_30s.tif"    
[29] "phh2o_0-5cm_Q0.95_30s.tif"     "phh2o_0-5cm_mean_30s.tif"     
[31] "phh2o_15-30cm_mean_30s.tif"    "phh2o_30-60cm_mean_30s.tif"   
[33] "phh2o_5-15cm_Q0.05_30s.tif"    "phh2o_5-15cm_Q0.95_30s.tif"   
[35] "phh2o_5-15cm_mean_30s.tif"     "raw"                          
[37] "sand_0-5cm_mean_30s.tif"       "sand_15-30cm_mean_30s.tif"    
[39] "sand_30-60cm_mean_30s.tif"     "sand_5-15cm_mean_30s.tif"     
[41] "silt_0-5cm_mean_30s.tif"       "silt_15-30cm_mean_30s.tif"    
[43] "silt_30-60cm_mean_30s.tif"     "silt_5-15cm_mean_30s.tif"     
[45] "soc_0-5cm_mean_30s.tif"        "soc_15-30cm_mean_30s.tif"     
[47] "soc_30-60cm_mean_30s.tif"      "soc_5-15cm_mean_30s.tif"      


For structured enumeration with URLs and GDAL dataset names:

soilgrids <- geodata_files("geodata/soil/soilgrids")
head(soilgrids)

                                                                           url
1   https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_0-5cm_mean_30s.tif
2 https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_15-30cm_mean_30s.tif
3 https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_30-60cm_mean_30s.tif
4  https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_5-15cm_mean_30s.tif
5    https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_0-5cm_mean_30s.tif
6  https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_15-30cm_mean_30s.tif
                       file ext
1   bdod_0-5cm_mean_30s.tif tif
2 bdod_15-30cm_mean_30s.tif tif
3 bdod_30-60cm_mean_30s.tif tif
4  bdod_5-15cm_mean_30s.tif tif
5    cec_0-5cm_mean_30s.tif tif
6  cec_15-30cm_mean_30s.tif tif
                                                                                gdaldsn
1   /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_0-5cm_mean_30s.tif
2 /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_15-30cm_mean_30s.tif
3 /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_30-60cm_mean_30s.tif
4  /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_5-15cm_mean_30s.tif
5    /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_0-5cm_mean_30s.tif
6  /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_15-30cm_mean_30s.tif


This returns a data.frame with columns url, file, ext, and gdaldsn. The gdaldsn column contains the /vsicurl/ path for files GDAL can read, and NA for files it cannot (.rds, .txt, etc.).


What’s there: a complete inventory

SoilGrids (/geodata/soil/soilgrids/)
Global soil properties at 30 arc-second resolution. 48 TIF files covering 8 properties (bdod, cec, cfvo, clay, nitrogen, ocd, phh2o, sand, silt, soc) at 4 depth intervals, plus pH quantiles. Files range from 83 MB to 546 MB.

soilgrids <- geodata_files("geodata/soil/soilgrids", pattern = "\\.tif$")
nrow(soilgrids)

[1] 44


Format: Single-band GeoTIFF, LZW compressed, Block=43200x1 (row-striped). No overviews. No internal tiling.

check_block_size(soilgrids$gdaldsn[1])

xblock yblock 
 43200      1 
attr(,"full_width")
[1] 43200
attr(,"is_row_striped")
[1] TRUE
attr(,"is_tiled")
[1] FALSE


Streamability: Row-striped layout means sub-region reads via /vsicurl/ require decompressing entire rows spanning the full 43200 pixel width. Reading a small bounding box will transfer most of the file. These files are best downloaded locally.
The geodata::soil_world_vsi() function constructs /vsicurl/ paths to these same files — the name suggests streaming is the intended use, but the row-striped layout makes it impractical for anything less than a large regional or global read.


WorldClim 2.1 tiles (/climate/worldclim/2_1/tiles/tile/)
30 arc-second resolution climate data, pre-tiled into 72 spatial tiles. 8 variables (bio, elev, prec, srad, tavg, tmax, tmin, wind). Bio tiles have 19 bands; monthly variables have 12 bands; elevation has 1. File sizes range from ~5 MB (ocean/ice tiles) to ~550 MB (land-dense tiles).

wc_tiles <- geodata_files("climate/worldclim/2_1/tiles/tile", pattern = "\\.tif$")
nrow(wc_tiles)  ## expect 576 = 72 tiles × 8 variables

[1] 576


Format: Multi-band GeoTIFF. Block layout should be checked:

check_block_size(wc_tiles$gdaldsn[1])

xblock yblock 
  3600      1 
attr(,"full_width")
[1] 3600
attr(,"is_row_striped")
[1] TRUE
attr(,"is_tiled")
[1] FALSE




WorldClim 2.1 global (/climate/worldclim/2_1/base/)
Global grids at 10m, 5m, 2.5m, and 30s resolution. All stored as .zip files containing one or more TIF files per variable.








Resolution
Variables
Example size




10 minute
bio, elev, prec, srad, tavg, tmax, tmin, vapr, wind
1.3 MB – 48 MB


5 minute
same
4.6 MB – 171 MB


2.5 minute
same
17 MB – 628 MB


30 second
same
323 MB – 9.7 GB



GDAL can read inside these zips via /vsizip//vsicurl/:

## example: 10-minute bioclim, first file inside the zip
dsn <- "/vsizip//vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/base/wc2.1_10m_bio.zip"
## list the contents of the zip
vsi_read_dir(dsn)

 [1] "wc2.1_10m_bio_1.tif"  "wc2.1_10m_bio_10.tif" "wc2.1_10m_bio_11.tif"
 [4] "wc2.1_10m_bio_12.tif" "wc2.1_10m_bio_13.tif" "wc2.1_10m_bio_14.tif"
 [7] "wc2.1_10m_bio_15.tif" "wc2.1_10m_bio_16.tif" "wc2.1_10m_bio_17.tif"
[10] "wc2.1_10m_bio_18.tif" "wc2.1_10m_bio_19.tif" "wc2.1_10m_bio_2.tif" 
[13] "wc2.1_10m_bio_3.tif"  "wc2.1_10m_bio_4.tif"  "wc2.1_10m_bio_5.tif" 
[16] "wc2.1_10m_bio_6.tif"  "wc2.1_10m_bio_7.tif"  "wc2.1_10m_bio_8.tif" 
[19] "wc2.1_10m_bio_9.tif" 


The internal TIF filenames follow the pattern wc2.1_{res}_{var}_{number}.tif.


WorldClim 2.1 country extracts (/climate/worldclim/2_1/tiles/iso/)
30 arc-second data clipped to individual countries, identified by ISO 3166-1 alpha-3 codes.

iso <- geodata_files("climate/worldclim/2_1/tiles/iso")
head(iso)

                                                                                 url
1  https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_bio.tif
2 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_elev.tif
3 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_prec.tif
4 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_srad.tif
5 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_tavg.tif
6 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_tmax.tif
                    file ext
1  ABW_wc2.1_30s_bio.tif tif
2 ABW_wc2.1_30s_elev.tif tif
3 ABW_wc2.1_30s_prec.tif tif
4 ABW_wc2.1_30s_srad.tif tif
5 ABW_wc2.1_30s_tavg.tif tif
6 ABW_wc2.1_30s_tmax.tif tif
                                                                                      gdaldsn
1  /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_bio.tif
2 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_elev.tif
3 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_prec.tif
4 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_srad.tif
5 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_tavg.tif
6 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_tmax.tif




WorldClim 1.4 (/climate/worldclim/1_4/)
Legacy WorldClim version in BIL (Band Interleaved by Line) format inside zip files. Three subdirectories: cur/ (current), fut/ (future projections), pst/ (past / LGM / mid-Holocene).
BIL format is GDAL-readable but, like the row-striped TIFs, does not support efficient random access.


CMIP6 projections (/cmip6/)
Downscaled and bias-corrected CMIP6 future climate data. Organised as:
/cmip6/{resolution}/{model}/{ssp}/
Resolutions: 30s, 2.5m, 5m, 10m. 23 GCMs. 4 SSPs (ssp126, ssp245, ssp370, ssp585). 4 time periods (2021-2040, 2041-2060, 2061-2080, 2081-2100). Variables: bioc, prec, tmax, tmin.
At 30s resolution these are bare TIF files, some exceeding 20 GB.

## list models at 2.5 minute resolution
vsi_read_dir("/vsicurl/https://geodata.ucdavis.edu/cmip6/2.5m/")

 [1] "ACCESS-CM2"       "ACCESS-ESM1-5"    "AWI-CM-1-1-MR"    "BCC-CSM2-MR"     
 [5] "CMCC-ESM2"        "CNRM-CM6-1"       "CNRM-CM6-1-HR"    "CNRM-ESM2-1"     
 [9] "CanESM5"          "CanESM5-CanOE"    "EC-Earth3-Veg"    "EC-Earth3-Veg-LR"
[13] "FIO-ESM-2-0"      "GFDL-ESM4"        "GISS-E2-1-G"      "GISS-E2-1-H"     
[17] "HadGEM3-GC31-LL"  "INM-CM4-8"        "INM-CM5-0"        "IPSL-CM6A-LR"    
[21] "MIROC-ES2L"       "MIROC6"           "MPI-ESM1-2-HR"    "MPI-ESM1-2-LR"   
[25] "MRI-ESM2-0"       "UKESM1-0-LL"     


There is also a /cmip6/tiles/ directory with tiled versions.


CMIP5 projections (/climate/cmip5/)
Legacy CMIP5 data at 30s, 2.5m, 5m, 10m resolutions. Separate directories for lgm/ (Last Glacial Maximum) and mid/ (mid-Holocene).


Elevation
TODO: locate and document. The geodata package provides elevation_3s(), elevation_30s(), and elevation_global() — the URLs for these need to be traced from the source code.


Landcover (/geodata/landcover/)
Two subdirectories:

/geodata/landcover/30s/ — 30 arc-second aggregated landcover fractions
/geodata/landcover/3deg_cogs/ — ESA WorldCover 10m data as 3-degree tiles, labelled as COGs in the directory name


lc_cogs <- geodata_files("geodata/landcover/3deg_cogs", pattern = "\\.tif$")
nrow(lc_cogs)

[1] 2651


The 3deg_cogs files follow the naming convention ESA_WorldCover_10m_2020_v100_{lat}{lon}_Map.tif. If these are genuine COGs (internally tiled with overviews), they would be efficiently streamable via /vsicurl/.

## check whether these are really COGs
check_block_size(lc_cogs$gdaldsn[1])

xblock yblock 
  1024   1024 
attr(,"full_width")
[1] 36000
attr(,"is_row_striped")
[1] FALSE
attr(,"is_tiled")
[1] TRUE




Crops (/geodata/crops/)
Subdirectories for SPAM and Monfreda crop data, plus cropland density and crop calendars. File formats and structure need to be enumerated.


GADM administrative boundaries (/gadm/gadm4.1/)
Per-country boundaries in multiple formats:









Subdirectory
Format
GDAL-readable
Notes




gpkg/
GeoPackage
Yes
SQLite-based, supports HTTP range requests


shp/
Shapefile (zipped)
Yes, via /vsizip/
Multiple component files


json/
GeoJSON
Yes



pck/
RDS (R serialization)
No
Used by geodata::gadm()


kmz/
KMZ (zipped KML)
Yes, via /vsizip/




Whole-dataset archives:

gadm_410-gpkg.zip — 1.4 GB, all countries in one GeoPackage
gadm_410-gdb.zip — 913 MB, FileGDB format
gadm_410-levels.zip — 2.5 GB, separated by admin level
gadm_410-raw.gpkg — 2.6 GB, bare GeoPackage on HTTP

The gadm_410-raw.gpkg is directly streamable:

## open GADM as a single GeoPackage over HTTP
dsn <- "/vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gadm_410-raw.gpkg"
## list layers
gdalraster::ogr_ds_layer_names(dsn)

[1] "gadm41_raw"


Per-country GeoPackage files are also streamable:

dsn <- "/vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AUS.gpkg"
gdalraster::ogr_ds_layer_names(dsn)

[1] "ADM_ADM_0" "ADM_ADM_1" "ADM_ADM_2"




Other datasets








Path
Contents
geodata function




/geodata/travel/
Travel time rasters
travel_time()


/geodata/osm/
OpenStreetMap extracts (highways, places)
osm()


/geodata/buildings/
Building footprint data
—



These need further enumeration.


Datasets not on this server
Some geodata functions fetch data from external APIs rather than geodata.ucdavis.edu:

sp_occurrence() — queries the GBIF API
bio_oracle() — queries the Bio-Oracle server at bio-oracle.org




Block layout and streamability
The central question for remote access is whether a file’s internal layout supports efficient HTTP range requests. GDAL’s /vsicurl/ handler issues byte-range requests to read specific portions of a file. This works well when the file is internally tiled (e.g., COG format with Block=256x256) but poorly when the file is row-striped (Block=width×1).
For a row-striped file, reading a small spatial window requires decompressing one or more full rows, each spanning the entire raster width. For a 43200-pixel-wide global grid, every row is a complete scan line from -180° to +180°. A single HTTP range request per row, times the number of rows in the window, with full LZW decompression of each row. For a small bounding box this is extremely wasteful.

What we know








Collection
Block layout
Streamable?




SoilGrids TIFs
43200×1 (row-striped)
No — download


WorldClim 2.1 tiles
TODO: check
Probably row-striped


WorldClim 2.1 global zips
N/A (zip)
Via /vsizip/, but still row-striped inside


CMIP6 30s TIFs
TODO: check
Unknown


Landcover 3deg COGs
TODO: check
Likely yes (name says COGs)


Landcover 30s
TODO: check
Unknown


GADM GeoPackage
N/A (vector)
Yes — SQLite random access works over HTTP


GADM raw GeoPackage
N/A (vector)
Yes





How to check

## for any TIF on the server
bs <- check_block_size("/vsicurl/https://geodata.ucdavis.edu/path/to/file.tif")
bs
## if is_row_striped is TRUE: download, don't stream
## if is_tiled is TRUE: streaming is efficient




Generating file tables
For any collection, generate a complete table of URLs and GDAL dataset names:

## SoilGrids: all TIFs
sg <- geodata_files("geodata/soil/soilgrids", pattern = "\\.tif$")
head(sg[, c("url", "gdaldsn")])

                                                                           url
1   https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_0-5cm_mean_30s.tif
2 https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_15-30cm_mean_30s.tif
3 https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_30-60cm_mean_30s.tif
4  https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_5-15cm_mean_30s.tif
5    https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_0-5cm_mean_30s.tif
6  https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_15-30cm_mean_30s.tif
                                                                                gdaldsn
1   /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_0-5cm_mean_30s.tif
2 /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_15-30cm_mean_30s.tif
3 /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_30-60cm_mean_30s.tif
4  /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_5-15cm_mean_30s.tif
5    /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_0-5cm_mean_30s.tif
6  /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_15-30cm_mean_30s.tif



## WorldClim tiles: all TIFs
wc <- geodata_files("climate/worldclim/2_1/tiles/tile", pattern = "\\.tif$")
head(wc[, c("url", "gdaldsn")])

                                                                                      url
1  https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_bio.tif
2 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_elev.tif
3 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_prec.tif
4 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_srad.tif
5 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_tavg.tif
6 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_tmax.tif
                                                                                           gdaldsn
1  /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_bio.tif
2 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_elev.tif
3 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_prec.tif
4 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_srad.tif
5 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_tavg.tif
6 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_tmax.tif



## GADM 4.1 GeoPackages
gadm <- geodata_files("gadm/gadm4.1/gpkg", pattern = "\\.gpkg$")
head(gadm[, c("url", "gdaldsn")])

                                                            url
1 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ABW.gpkg
2 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AFG.gpkg
3 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AGO.gpkg
4 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AIA.gpkg
5 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ALA.gpkg
6 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ALB.gpkg
                                                                 gdaldsn
1 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ABW.gpkg
2 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AFG.gpkg
3 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AGO.gpkg
4 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AIA.gpkg
5 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ALA.gpkg
6 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ALB.gpkg



## CMIP6 models at 2.5 minute resolution
## first list the model directories, then enumerate within each
models <- vsi_read_dir("/vsicurl/https://geodata.ucdavis.edu/cmip6/2.5m/")
models

 [1] "ACCESS-CM2"       "ACCESS-ESM1-5"    "AWI-CM-1-1-MR"    "BCC-CSM2-MR"     
 [5] "CMCC-ESM2"        "CNRM-CM6-1"       "CNRM-CM6-1-HR"    "CNRM-ESM2-1"     
 [9] "CanESM5"          "CanESM5-CanOE"    "EC-Earth3-Veg"    "EC-Earth3-Veg-LR"
[13] "FIO-ESM-2-0"      "GFDL-ESM4"        "GISS-E2-1-G"      "GISS-E2-1-H"     
[17] "HadGEM3-GC31-LL"  "INM-CM4-8"        "INM-CM5-0"        "IPSL-CM6A-LR"    
[21] "MIROC-ES2L"       "MIROC6"           "MPI-ESM1-2-HR"    "MPI-ESM1-2-LR"   
[25] "MRI-ESM2-0"       "UKESM1-0-LL"     




Still to document
Several areas need further investigation:

Elevation data paths (elevation_3s, elevation_30s, elevation_global)
Exact contents of /geodata/crops/, /geodata/travel/, /geodata/osm/, /geodata/buildings/
Block layout of WorldClim tiles, CMIP6 30s TIFs, and landcover COGs
Contents of /geodata/landcover/30s/
Contents of /geodata/soil/soilgrids/agg/, fill/, and raw/ subdirectories
The /cmip6/tiles/ tiled variant structure





 


Map Reprojection and Map Edges
Michael Sumner — Fri, 20 Feb 2026 00:00:00 GMT
 







Map Reprojection
A recent issue on the tmap package reports a white patch appearing at the top of a map rendered in an Albers Equal Area projection. The basemap tiles don’t fill the whole plot area. It’s tempting to call this a tmap bug, but the underlying problem is more general. It affects any tool that needs to compute the geographic footprint of a projected extent: tile-fetching packages, GDAL utilities, raster warping workflows, and vector reprojection. An almost identical issue was reported against GDAL’s gdal_translate and fixed in GDAL 3.11 by densifying the projwin boundary instead of transforming only corner points.
The core question is: when you have a rectangular region in a map, what area does it cover in another map projection? If you only transform the four corners, you get a quadrilateral. But the actual footprint is a curved shape, because straight lines in the projected CRS trace curves through geographic space. For conic projections like Albers, the top edge of the rectangle sweeps a wider arc in longitude than the corners alone suggest. The difference between the four-corner bounding box and the true footprint is the missing area in the map.

A picture of the problem
The fix is to densify the boundary of the projected rectangle before transforming it. Add intermediate vertices along each edge, then reproject the resulting polygon, and compute the bounding box of that. The more vertices you add, the better the approximation.
library(terra)
vdsn <- "/vsizip//vsicurl/https://github.com/user-attachments/files/20692539/AEA_polygon.zip"
v <- vect(vdsn)

## The extent as a polygon in projected space
box_proj <- as.polygons(ext(v))
crs(box_proj) <- crs(v)

## Transform with sparse corners only
box_ll_sparse <- project(box_proj, "EPSG:4326")

## Densify first (add vertices every 10 km), then transform
box_dense <- densify(box_proj, interval = 10000)
box_ll_dense <- project(box_dense, "EPSG:4326")

## Compare the extents
ext(box_ll_sparse)
ext(box_ll_dense)

## Plot them together
plot(box_ll_dense, border = "blue", lwd = 2, main = "Sparse corners vs. densified boundary")
plot(box_ll_sparse, border = "red", lwd = 2, add = TRUE)
legend("bottomright", legend = c("Densified", "Sparse (4 corners)"),
       col = c("blue", "red"), lwd = 2)
The gap between the red quadrilateral and the blue curve is exactly the area that goes missing when software transforms only the corner points.


How this relates to map tiles
Web map tile services (TMS, WMTS, XYZ) serve imagery in tiles at fixed zoom levels, typically in Web Mercator (EPSG:3857). Packages like maptiles, ceramic, and rosm do an excellent job of fetching, compositing, and caching these tiles for use in R. Their workflow is roughly: determine the geographic extent needed, request the right tiles at the right zoom level, stitch them, and return a raster in the desired CRS.
The extent determination step is where the densification problem bites. If the user’s map is in a conic projection and the package computes the longlat extent by transforming only the viewport corners, the requested tiles won’t cover the full area.
There is an alternative approach that sidesteps tile-fetching entirely, by treating the tile server as a GDAL data source and using gdalwarp to produce an image directly on a target grid. GDAL’s warper handles extent computation internally via GDALSuggestedWarpOutput(), which densifies the source boundary as part of its calculation. This means the “white patch” problem doesn’t occur.


Using GDAL warp with a tile server
A TMS or WMTS endpoint can be opened by GDAL as a raster data source, given the right connection string. The esri() helper function below constructs a WMTS connection string from an ArcGIS REST service URL:
## Helper functions:
## esri() builds a WMTS connection string from an ArcGIS REST Services URL
## fit_dims() computes aspect-ratio-preserving pixel dimensions
## (fit_dims is available in the hypertidy/vaster package)

esri <- function(url) {
  sprintf("WMTS:%s/WMTS/1.0.0/WMTSCapabilities.xml", url)
}

## fit_dims is also available from vaster::fit_dims()
fit_dims <- function(size = 1024L, wh = c(size, size)) {
  wh <- rep(wh, length.out = 2L)
  w <- wh[1L]; h <- wh[2L]
  as.integer(ceiling(size * c(w, h) / max(w, h)))
}
With these in hand, the workflow is:
library(terra)

dsn <- esri(url = "https://services.arcgisonline.com/arcgis/rest/services/Ocean/World_Ocean_Base/MapServer")
vdsn <- "/vsizip//vsicurl/https://github.com/user-attachments/files/20692539/AEA_polygon.zip"

v <- vect(vdsn)

## Compute pixel dimensions that preserve the aspect ratio of the extent
dm <- fit_dims(1024, diff(as.vector(ext(v)))[c(1L, 3L)])

source <- rast(dsn)

## Build the target grid: same CRS and extent as the input vector data,
## with the computed dimensions, and matching number of layers (RGB)
target <- rast(ext(v), ncols = dm[1L], nrows = dm[2L],
               crs = crs(v), nlyr = nlyr(source))

## Warp the tile server imagery onto the target grid
## by_util = TRUE calls gdalwarp rather than terra's internal resampler
im <- project(rast(dsn), target, by_util = TRUE)

plotRGB(im)
plot(v, add = TRUE)
The project() call with by_util = TRUE invokes gdalwarp under the hood, which reads tiles from the server as needed to fill the target grid. GDAL computes the correct source extent internally, including the densification step.
A few details worth noting about this code:

The nlyr = nlyr(source) argument on the target raster is necessary so that all bands (RGB) are included. Without it, the target would default to a single band.
The by_util = TRUE flag is essential. Without it, project() uses terra’s internal logic driving a lower level part of the GDAL api, which does not automatically target the appropriate resolution level (it will take forever to process pixels that are not needed).
The dsn variable is a WMTS connection string that GDAL opens as a raster source. The same approach works with TMS (XYZ) endpoints, though the connection string format is different (it requires a small XML file or an inline XML specification; a good reference is the GDAL TMS driver documentation).



Reprojecting the target grid definition
If the desired output CRS is different from the input vector CRS, the key insight is to reproject the target grid definition, not the image data. Build the target grid in the source CRS first (where the extent is known), then reproject the empty grid to the desired output CRS. GDAL’s warp step will then fill it correctly.
## Build the target in the vector's native CRS
target0 <- rast(ext(v), ncols = dm[1L], nrows = dm[2L],
                crs = crs(v), nlyr = nlyr(source))

## Reproject the empty grid definition to Web Mercator
target <- project(target0, "EPSG:3857")

## Warp the source imagery onto the reprojected target
im <- project(rast(dsn), target, by_util = TRUE)
plotRGB(im)
plot(project(v, "EPSG:3857"), add = TRUE)
Reprojecting an empty raster grid with project() (or equivalently, gdalwarp on a small raster) computes the appropriate output extent and resolution by densifying the source boundary before transforming. This is a lightweight way to answer the question “what does this extent look like in another CRS?”


Extent reprojection as a general operation
The task of computing a bounding box in one CRS from a bounding box in another CRS appears in many contexts. Several tools address it:

reproj::reproj_extent() in the reproj package does this directly. It densifies the input extent boundary, transforms the points, and returns the bounding box. The documentation describes it as a simple version of what GDAL’s GDALSuggestedWarpOutput() does.
terra::project() applied to a SpatExtent or an empty SpatRaster achieves the same result, using GDAL internally.
In sf and stars, sf::st_transform() on an sfc_POLYGON representing the bounding box will reproject the boundary (sf applies segmentization as part of its transformation pipeline when working with s2). st_bbox() of the result gives the target extent.
In gdalraster, gdalraster::warp() with appropriate -te and -te_srs arguments lets you specify the target extent in a different CRS from the output. The -te_srs option tells gdalwarp to interpret the -te coordinates in the given CRS and transform them (with densification) to the output CRS.
Similarly, gdal_translate has -projwin_srs, which specifies the CRS of the -projwin coordinates. As of GDAL 3.11, the projwin boundary is fully densified before transformation. In earlier versions, only the two corner points were transformed, leading to the same class of error described here.

All of these ultimately rely on the same principle: you cannot determine the geographic footprint of a projected rectangle from its corners alone. The boundary must be sampled densely enough to capture the curvature of the transformation.


The vector side: densifying features
The same issue applies to vector geometries. A polygon or line with sparse vertices in one CRS will have incorrect geometry after reprojection if the segments are long relative to the curvature of the transformation. This is not about datum shifts or projection accuracy at individual points — point reprojection is exact. It is about the fact that a straight line segment in one CRS is not, in general, a straight line in another.
Consider a coastline polygon with vertices every 100 km, stored in an Albers Equal Area CRS. Those long straight edges are reasonable approximations of the coastline in that projection. But when reprojected to longitude/latitude, each straight edge should trace a curve through geographic space. Without densification, the reprojected polygon connects the transformed endpoints with straight lines in the target CRS, which follow different paths than the original segments.
sf::st_segmentize() and terra::densify() add intermediate vertices along existing edges. The critical point is that this must happen before reprojection, in the source CRS, so that the added vertices capture the geometry as it was defined. After reprojection, the denser vertex set produces a polygon that faithfully represents the original shape in the new CRS.
Neither sf nor terra applies densification automatically during reprojection. It is the caller’s responsibility to decide whether it is needed and to choose an appropriate segment length. For small-area, high-resolution data the effect is negligible, but for continental extents or coarse geometries it can be significant.


Summary
The white patch in the tmap basemap, the missing data in gdal_translate -projwin_srs before GDAL 3.11, and the distortion of coarse polygons after reprojection are all the same problem. A rectangular extent in a projected CRS is not a rectangle in geographic coordinates. The boundary curves, and capturing that curvature requires sampling it with enough points.
This applies whenever software needs to answer the question “what geographic area does this projected region cover?” The answer is to densify before transforming — whether the input is a viewport boundary for fetching tiles, a bounding box for subsetting, or the edges of a polygon for display.
The GDAL warp approach to tile servers avoids the problem for raster imagery by letting GDAL handle extent computation internally. For vector data, st_segmentize() or densify() should be part of any reprojection workflow involving coarse geometries.
Code and helper functions used in this post are available at this gist.

Possible future additions: equivalent workflows using sf::gdal_utils(), gdalraster::warp(), and vapour::vapour_warp_raster() to show the same operations without terra.





 


Detecting hidden grids in curvilinear coordinates
Michael Sumner — Sat, 07 Feb 2026 00:00:00 GMT
 







The problem
A lot of gridded data arrives with two-dimensional longitude and latitude arrays: one value per pixel, stored explicitly. This is the standard representation for curvilinear grids in NetCDF and HDF — satellite swaths, model output on rotated grids, polar data. The coordinates describe the geographic location of every cell, and in many cases they genuinely need to: the grid may be irregular, oblique, or follow a sensor’s scan geometry.
But sometimes those coordinates are hiding something. The data was originally produced on a regular projected grid — polar stereographic, Lambert conformal conic, Mercator — and at some point in the processing chain the projection was discarded and replaced with lon/lat arrays computed from it. The regularity is gone. The CRS is gone. What was four numbers and a coordinate reference system is now two arrays of floating-point values, one per pixel, carrying no more information than the originals but consuming orders of magnitude more storage and, critically, obscuring the actual structure of the data.
This is an entropy problem. A 316×332 grid in EPSG:3412 at 25 km resolution is fully specified by a CRS string and four extent values. Storing lon/lat arrays for every pixel replaces those with 209,824 coordinate values — a 52,000:1 expansion — and the extra values are noisier than the originals because they’ve accumulated floating-point error from the forward projection.
It would be useful to be able to detect this automatically.


Detecting regularity
The in-development R package gproj includes tools for characterising coordinate geometry and selecting appropriate projections. Among these is a regularity detection pipeline that can identify “cryptic curvilinear” grids — data that presents as irregular lon/lat arrays but is in fact a regular projected grid with lost metadata.
The approach is layered.

Step 1: Is it rectilinear in lon/lat?
The simplest hidden structure is a rectilinear grid: longitude varies only along one axis, latitude only along the other. This is tested by checking the standard deviation of each coordinate along each matrix dimension.
detect_rectilinear() classifies these into subtypes. If both marginal sequences have uniform spacing, it’s a regular lon/lat grid. If longitude is uniform but latitude spacing follows the Mercator y-formula — y = ln(tan(π/4 + φ/2)) — then it’s a Mercator grid that was “devolved” to lon/lat.
Running this on an AVISO altimetry product with 1080×915 lon/lat arrays:
Rectilinear:        TRUE
Type:               mercator_devolved
Lon regular:        TRUE (0.3333° spacing)
Mercator test:      TRUE (max deviation: 1e-12)
Mercator resolution: 37106 m
The deviation from perfect Mercator spacing is at machine epsilon — the coordinates were computed analytically. The 1,976,400 stored values can be replaced by a CRS and an extent.


Step 2: Is it regular in a projected CRS?
When the grid isn’t rectilinear in lon/lat, it may be regular in some projected coordinate system. The test:

Characterise the data geometry — centre, extent, hemisphere — to select candidate projections.
Project the lon/lat arrays into each candidate CRS.
Check whether the projected coordinates form a regular grid: constant spacing along rows and columns, with rows horizontal in projected space.

The params() function in gproj computes a geometric characterisation from any set of lon/lat coordinates:

  centre:     -0.0000, -88.1448
  extent:     [0.17, 359.84] x [-89.84, -39.36]
  span:       359.67° lon x 50.47° lat
  hemisphere: polar_south
  aspect:     wide
  axis:       azimuth=180.0°, tilt=90.0°, var_ratio=0.51/0.46/0.02
This is a CryoSat-2 sea ice product distributed as a NetCDF file with 316×332 lon/lat arrays. The params() output immediately tells us it’s circumpolar Antarctic data with no dominant axis orientation (variance split evenly between two components — characteristic of polar data rather than an elongated swath).
Projecting the coordinates into EPSG:3412 (NSIDC Sea Ice Polar Stereographic South) and testing regularity:
Max row-y SD:      13.42 m
Median dx:         25000 m
Max dx deviation:  74.8 m
Median dy:         -25000 m
Max dy deviation:  71.0 m
Row-level y standard deviation of 13 metres on a 25 km grid means rows are horizontal to within 0.05% of a cell width. The x and y spacing deviations of ~70 m (0.3%) are the accumulated floating-point noise from the trig functions in the lon/lat → projection → lon/lat → projection roundtrip.
These numbers are unambiguous. A genuinely curvilinear grid would show deviations of 10–100% of the cell size. The gap between 0.3% and “not regular” is enormous.


Step 3: Recover the grid specification
Once regularity is confirmed, the resolution is snapped to the nearest “round” value (25000 rather than 24999.7) and the extent is recovered from the cell centres plus half-cell margins:
Recovered extent:  -3950030  3950000  -3949995  4349967
Known NSIDC extent: -3950000  3950000  -3950000  4350000
Difference:         -30       0        5        -33
Within 33 metres of the exact grid specification. The extent values snap to exact multiples of the resolution, completing the reconstruction:
CRS:          EPSG:3412
Resolution:   25000 × 25000 m
Dimensions:   316 × 332
Extent:       -3950000  3950000  -3950000  4350000
The 209,824 coordinate values have been replaced by four numbers and a CRS string. The reconstructed grid is not an approximation — it is the original grid, recovered (near enough to) exactly.



The Mercator case
A subtler variant of this problem occurs with global ocean products. A Mercator projection produces a regular grid in eastings and northings, but when projected to lon/lat the latitude spacing becomes non-uniform: cells are compressed toward the poles following the inverse Mercator formula. The longitude spacing remains uniform.
This is easy to miss because the grid looks rectilinear — it has separable 1D coordinate axes — and many tools will happily treat it as a lon/lat grid with variable spacing. But it’s not: it’s a Mercator grid with specific metric properties that are lost when treated as plain lon/lat.
The detection is straightforward: convert the latitude values to Mercator y-coordinates and check whether the spacing becomes uniform. For the AVISO product tested above, the deviation from perfect Mercator spacing is 10⁻¹² — indistinguishable from the analytical formula.


Swath geometry and projection selection
Not all lon/lat arrays hide regular grids. Satellite swath data from sensors like VIIRS or MODIS is genuinely curvilinear — the coordinates follow the sensor’s scan geometry, and there is no single projected CRS that makes them regular. But the coordinates still have structure that can be exploited.
The params() function characterises this structure using PCA on the 3D Cartesian coordinates of the data points. Converting lon/lat to points on the unit sphere before PCA avoids the distortions that would arise from operating directly in longitude/latitude space (particularly near the poles or antimeridian).
For a VIIRS swath boundary:

  centre:     -113.56, -20.81
  extent:     [-180, 180] x [-84.44, 57.66]
  hemisphere: polar_south
  axis:       azimuth=12.4°, tilt=77.6°, var_ratio=0.85/0.14/0.02
The variance ratio of 0.85 in the first component indicates a strongly elongated dataset. The azimuth of 12.4° gives the bearing of the dominant axis — essentially the satellite ground track orientation.
This feeds directly into projection selection. The proj() function uses the geometric characterisation to generate appropriate CRS parameters:

Polar data with no dominant axis → stereographic or Lambert azimuthal equal-area, centred on the pole
Elongated data with a dominant axis → oblique Mercator aligned to the data’s axis
Mid-latitude wide bands → conic projections with secant latitudes derived from the data extent
Small regions → stereographic centred on the data

For the VIIRS swath, this produces an oblique Mercator projection aligned to the 12.4° ground track, which straightens the swath into a narrow strip in projected coordinates (aspect ratio ~11:1).


The opportunity
The detection of hidden regular grids is not a theoretical exercise. These grids exist throughout operational data archives — sea ice products, ocean altimetry, atmospheric reanalyses, satellite imagery. Every file that stores redundant lon/lat arrays for a regular projected grid is:

Wasting storage (often 2× the data volume for the coordinate arrays alone)
Losing precision (floating-point coordinates are noisier than the exact grid specification)
Obscuring structure (downstream tools treat the data as irregular, triggering unnecessary resampling)
Creating friction (users must discover the projection through external documentation or domain expertise)

The regularity test demonstrated here is simple — a few lines of linear algebra and some tolerance checks — and could be applied routinely during data ingest or quality assessment. The tolerance gap between “regular with floating-point noise” (0.3% deviation) and “genuinely irregular” is wide enough that false positives are unlikely.
For data producers, the message is: if your data is on a regular projected grid, store the grid specification, not the coordinates. A CRS string and four extent values are complete, exact, and universally understood.
For data consumers and tool developers, the message is: when you encounter 2D lon/lat arrays, test for regularity before treating them as curvilinear. The answer may save you considerable complexity.
The gproj package is in early development at github.com/mdsumner/gproj.
A previous iteration of this discussion was published here hypertidy.org: Coordinates broken in NetCDF.




 


Raster logic without pixels
Michael Sumner — Sun, 21 Dec 2025 00:00:00 GMT
 







Raster logic is in our software
Every geospatial package that works with grids has raster logic inside. The old {raster} package in R established powerful abstraction functions and now {terra} includes an improved suite of those, the {stars} package has its own internally, GDAL obviously uses these abstractions deeply, and xarray and many other Python packages also provide this. There is similar logic built into R’s matrix and array functions, its visualization image() function, and the newer rasterImage().
Is there a problem? This logic is almost always coupled to a data structure. You can’t use raster’s cellFromXY() without a RasterLayer. You can’t use terra’s xyFromCell() or cells() without a SpatRaster. Many packages have embedded this reinvented wheel, and to more or less degree lock the wheel inside a bigger machine.
The logic itself is beautifully simple: given a grid with certain dimensions (ncol, nrow) and a spatial extent (xmin, xmax, ymin, ymax, or identically ‘bbox’ xmin,ymin,xmax,ymax), you can compute everything else. Cell indices, row/column positions, coordinate centres and corners, cropping and snapping—all of it flows from six numbers and some basic arithmetic.
That’s what {vaster} extracts: the logic alone, without any data. (This logic of course is not especially 2D and extends into n-dimensional concepts as well illustrated by xarray and others, but please let’s park that from our discussion here.)


R already knows this (sort of)
R’s base graphics already embody two distinct models for placing gridded data in space.

The image() model: rectilinear coordinates

m <- volcano[1:10, 1:15]
x <- seq(0, 1, length.out = nrow(m) + 1)
y <- seq(0, 2, length.out = ncol(m) + 1)

image(x, y, m, col = terrain.colors(12))








The image() function takes explicit coordinate vectors for cell boundaries. This is the rectilinear model—you define where every edge falls. It’s flexible (cells don’t need to be square, or even uniform), but it means carrying around those vectors.
Notice x has nrow(m) + 1 elements, and y has ncol(m) + 1. These are edge coordinates, not centres. The function figures out that a 10×15 matrix needs 11×16 edges. (Centres are also a valid input for image(), it quietly handles either case of ‘n’ or ‘n+1’, note this has implications for interpretation of a grid but that takes us away from the main topic here).


The rasterImage() model: bounding box placement

plot(NA, xlim = c(0, 1), ylim = c(0, 2), asp = 1, xlab = "x", ylab = "y")
rasterImage(as.raster(scales::rescale(m)), xleft = 0, ybottom = 0, xright = 1, ytop = 2, interpolate = F)








The rasterImage() function takes a different approach: you hand it an image and four numbers defining the bounding box. The function stretches or compresses the image to fit. This is the affine model—the grid is implicitly regular within the box.
These two models are part of a foundation of all raster handling. GeoTIFFs use the affine model (extent + dimension → implicit coordinates). NetCDF often uses the rectilinear model (explicit coordinate arrays). Both are valid; both are useful; both involve the same underlying logic.


{ximage}: unifying both models
There’s a sort of frustration in base with image() and rasterImage(), each has features the other lacks. image() handles numeric data with colour palettes and is geared to R’s matrix orientation, can create a plot or add to an existing one, but by default will draw into a unit square. rasterImage() handles the orientation more aligned to external raster data and graphics, but only works with unit-scaled data or pre-rendered images—no palette mapping, and no plot creation (only adding to an existing plot) .
ximage merges the features of both into one function that uses the rasterImage orientation:

library(ximage)
library(vaster)
## Plot numeric data in GIS orientation with extent
ximage(volcano, extent = c(0, 61, 0, 87), col = terrain.colors(24))

## Add contours that respect the same extent
xcontour(volcano, extent = c(0, 61, 0, 87), add = TRUE, levels = c(120, 140, 160))







## Or RGB arrays, or nativeRaster, or hex colours—all work
img <- png::readPNG(system.file("img", "Rlogo.png", package="png"), native = TRUE)
ximage(img, extent = c(100, 160, -50, -10))
maps::map(add = TRUE)  
## nonsensical map, but we can do what we want and maps why not
# they can be fun and used with non spatial data too
ximage(img, extent = c(152, 158, -44, -40), add = TRUE)
plot_extent(c(152, 158, -44, -40), add = TRUE, border = "firebrick", lwd = 2)








Once we separate the data (a matrix of values) from the placement (extent as four numbers), you can handle any input type with the same logic. The orientation confusion disappears because {ximage} adopts the rasterImage convention—the one that matches how GDAL and every other geospatial tool returns data.
This is {vaster}’s philosophy applied to plotting: the spatial meaning comes from the six numbers (dimension + extent), not from the data structure.



What vaster provides
{vaster} gives that underlying logic with no attachment to any data format or structure. All you need are dimension and extent (and extent is optional, a sensible default is [0,nx], [0,ny] rather than unit-square).

library(vaster)

dm <- c(40, 20)  # ncol, nrow
ex <- c(100, 160, -50, -10)  # xmin, xmax, ymin, ymax

# Cell centres - implicit from dimension and extent
x_centre(dm, ex)

 [1] 100.75 102.25 103.75 105.25 106.75 108.25 109.75 111.25 112.75 114.25
[11] 115.75 117.25 118.75 120.25 121.75 123.25 124.75 126.25 127.75 129.25
[21] 130.75 132.25 133.75 135.25 136.75 138.25 139.75 141.25 142.75 144.25
[31] 145.75 147.25 148.75 150.25 151.75 153.25 154.75 156.25 157.75 159.25

y_centre(dm, ex)

 [1] -49 -47 -45 -43 -41 -39 -37 -35 -33 -31 -29 -27 -25 -23 -21 -19 -17 -15 -13
[20] -11


What cell contains a given point?

# Some arbitrary coordinates
pts <- cbind(x = c(120.5, 145.2, 110.8), 
             y = c(-25.3, -42.1, -15.7))

cell_from_xy(dm, ex, pts)

[1] 294 671  88


What are the coordinates of cells 1, 100, and 800?

xy_from_cell(dm, ex, c(1, 100, 800))

       [,1] [,2]
[1,] 100.75  -11
[2,] 129.25  -15
[3,] 159.25  -49


Convert between row/column and cell index:

# Cell to row, column
cell <- 42
row_from_cell(dm, cell)

[1] 2

col_from_cell(dm, cell)

[1] 2

# Row, column to cell
cell_from_row_col(dm, row = 3, col = 15)

[1] 95


None of this requires loading imagery or touching files. It’s pure computation from first principles.
Note here that indexing is via R’s convention: 1-based, both for cell and row and col.


The snap problem
One of the most common operations in raster work is “snapping”—taking an arbitrary region and aligning it to an existing grid. You want to crop to an area of interest, but you need cell-aligned boundaries.

# A reference grid: 3-degree cells covering the globe
ref_dm <- c(30, 10)
ref_ex <- c(-180, 180, -90, 90)

# Some arbitrary region of interest (not aligned to anything)
roi <- c(15, 154.7, -44.2, -9.8)

# Snap to the reference grid
snapped <- vcrop(roi, ref_dm, extent = ref_ex)
snapped

$extent
[1]  12 156 -54   0

$dimension
[1] 12  3


The result gives you the exact dimension and extent for the crop window, aligned to the reference grid. No data needed—just the numbers.

plot_extent(ref_ex, border = "grey")
abline(v = x_corner(ref_dm, ref_ex), col = "grey90")
abline(h = y_corner(ref_dm, ref_ex), col = "grey90")

# Original ROI (arbitrary)
plot_extent(roi, border = "red", lwd = 2, add = TRUE)

# Snapped ROI (grid-aligned)
plot_extent(snapped$extent, border = "blue", lwd = 2, add = TRUE)








The red box is what we asked for; the blue box is what we get when we respect the grid alignment.


The geotransform connection
GDAL uses a six-element “geotransform” to encode the affine relationship between pixel coordinates and geographic coordinates. This is the workhorse of georeferenced raster data. {vaster} speaks this language:

# From extent and dimension to geotransform
gt <- extent_dim_to_gt(ex, dm)
gt

 xmin  xres yskew  ymax xskew  yres 
100.0   1.5   0.0 -10.0   0.0  -2.0 

# And back again
gt_dim_to_extent(gt, dm)

xmin xmin ymax ymax 
 100  160  -50  -10 


This matters when you’re working directly with GDAL (via {gdalraster}, {vapour}, or Python’s osgeo.gdal) and need to set up or interpret raster metadata without constructing heavyweight objects.


Why does this matter?

Lightweight tooling: Sometimes we just need to compute cell indices or snap extents. We shouldn’t need to load terra or GDAL for that.
Cross-package compatibility: The logic is the same whether you’re working with terra, stars, GDAL, or raw arrays. Having it in one place means consistent behaviour everywhere.
Teaching and understanding: Separating the logic from the data makes it clearer what raster operations actually do. The magic isn’t in the file format or the object class—it’s in the relationship between dimension and extent.



The ecosystem: what you can build with pure logic
Once you separate grid logic from data, interesting things become possible. Here are three packages that build on this foundation, taken (ahem) shamelessly from the hypertidy suite. (I’m not ignoring other important packages in R and Python that use these ideas, but these are chosen to focus on the core idea of what vaster is and why it exists. )

{fasterize}: fast polygon rasterization
fasterize is a high-performance replacement for raster::rasterize(). It uses the classic scanline algorithm to burn polygons onto a grid. The original fasterize required an actual RasterLayer as a template—not for its data, but for its six numbers (dimension and extent).
## fasterize needs a "raster", but really it just needs dimension + extent
library(fasterize)
r <- raster::raster(ncol = 1000, nrow = 800, 
                    xmn = 0, xmx = 100, ymn = 0, ymx = 80)
result <- fasterize(polygons_sf, r, field = "value")
The raster object is just a vessel for grid metadata, but fasterize also will create an actual numeric matrix in memory - so it has these two limitations at input and output. With raster logic available separately, we can drive the same algorithm with nothing but numbers.


{controlledburn}: don’t materialize, just rasterize
controlledburn takes the fasterize algorithm and strips away the last step: instead of filling a raster with values, it returns the indices of which cells each polygon covers. The output is run-length encoded scanline segments—start, end, row, polygon_id.
library(controlledburn)
library(vaster)

## Define a grid with just numbers
ext <- c(100, 160, -50, -10)
dm <- c(500, 400)

## Get the cell coverage index, not pixel values
idx <- burn_polygon(polygons_sf, extent = ext, dimension = dm)
For a 500,000 × 400,000 grid, materializing a raster would require terabytes. But storing the polygon coverage as run-length scanline indices? Tens of megabytes for typical distributions that real world polygons embody. The grid logic is the same—only the output changes.
This is the “cell abstraction” always implicit in raster operations. We can use these indices for extraction, for streaming aggregation, for anything that doesn’t require all pixels to exist at once.


{grout}: tiling without tiles
grout applies the same principle to tiling. Given a grid’s dimension and extent, plus a block size, it computes the complete tiling scheme: how many tiles, where each one falls, how much overlap (“dangle”) occurs when dimensions don’t divide evenly.

library(grout)

## A raster that's 87 × 61 with 8 × 8 tiles
scheme <- grout(c(87, 61), extent = c(0, 87, 0, 61), blocksize = c(8, 16))
scheme

          tiles: 4, 11 (x * y = 44)
          block: 8, 16 
         dangle: 1, 3 
tile resolution: 8, 16 
    tile extent: 0, 88, -3, 61 (xmin,xmax,ymin,ymax)
          grain: 8, 16 (8 : x, 16 : y)

## Get offset/size for each tile (for GDAL RasterIO)
dplyr::glimpse(tile_index(scheme))

Rows: 44
Columns: 11
$ tile      1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18…
$ offset_x  0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 0, 8, 16, 24, 32, 4…
$ offset_y  0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16, 16, 16, 16, 16, 16, 16, …
$ tile_col  1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8, 9,…
$ tile_row  1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2…
$ ncol      8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8…
$ nrow      16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1…
$ xmin      0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 0, 8, 16, 24, 32, 4…
$ xmax      8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 87, 8, 16, 24, 32, 40,…
$ ymin      45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 29, 29, 29, 29, 2…
$ ymax      61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 45, 45, 45, 45, 4…


No data loaded. No files opened. Just the arithmetic of how a grid subdivides. This is exactly what you need to drive tiled reading from GDAL, or to generate web map tile pyramids, or to parallelize processing across spatial chunks.


The common thread
All three packages share a design principle: the grid specification is just six numbers, and most operations don’t need anything else.
{fasterize} showed that fast rasterization doesn’t need heavy objects—just geometry and grid metadata. {controlledburn} showed you don’t even need to materialize pixels—indices are enough. {grout} showed that tiling is pure arithmetic on dimension and extent.
{vaster} is the foundation that makes this all clean. Instead of each package reinventing cellFromXY() and friends, they can share a common vocabulary for grid logic.



Python has a similar mix of history, concepts, and tools
The Python geospatial ecosystem has grappled with the same issues that motivated {vaster}.
A recent pangeo discussion compared odc.stac and stackstac—two packages for loading satellite imagery into xarray. Despite ostensibly doing the same thing, they produce different results. Why?
Pixel coordinates: edge or centre? When you say a pixel is at coordinate (100.0, 200.0), do you mean that’s its centre, or its top-left corner? odc.stac defaults to centre, stackstac defaults to edge. Both are valid conventions; GIS tools tend toward corners, while climate/ocean data tends toward centres. But if you don’t know which convention your tool uses, your data shifts by half a pixel.
Coordinate snapping: When your requested bounding box doesn’t align perfectly with the source grid, what happens? Different tools make different choices about how to snap—and those choices are usually invisible, buried in the library internals.
These are exactly the questions {vaster} makes explicit. Dimension and extent. Corner or centre. Snap in or snap out. The logic is universal; only the defaults vary.

xarray’s new RasterIndex
The xarray team recently announced flexible indexing, including a RasterIndex that computes coordinates on-the-fly from an affine transform rather than storing explicit coordinate arrays. Sound familiar?
From the post:

For 2D raster images, this function often takes the form of an Affine Transform. The rasterix library extends Xarray with a RasterIndex which computes coordinates for geospatial images such as GeoTiffs via Affine Transform.

This is the same insight: coordinates are implicit in dimension and extent. You don’t need to materialize a 7-terabyte coordinate array when six numbers suffice. The xarray team is building infrastructure for raster grid logic itself.
The Python ecosystem will align around this conceptual foundation—grid logic as well as R and other language.



The R matrix connection
R’s matrices already have most of this logic, just without the spatial semantics. A matrix has dimension (nrow, ncol). When you use image() with explicit coordinates, you’re adding extent. The [i, j] indexing is cell-from-row-column. which(m > threshold, arr.ind = TRUE) gives you row-column-from-cell.
{vaster} makes this connection explicit. It treats dimension and extent as first-class inputs, just as R treats matrices as first-class objects. The spatial meaning is in the numbers, not the container.


Example: raster logic is a simplifying principle
Enough theory! Let’s make a map. The following string defines a huge world-coverage image server.

dsn <- "WMTS:https://services.arcgisonline.com/arcgis/rest/services/World_Imagery/MapServer/WMTS/1.0.0/WMTSCapabilities.xml,layer=World_Imagery"

So, obviously we need a huge stack of Python geospatial to make a map image … or maybe we can just use generic tools and do it in R.
We’re using {gdalraster} here because it’s the thinnest wrapper over the GDAL API, but the same calls work through {terra}, {vapour}, or Python’s osgeo.gdal. The point isn’t the package — it’s that the interface is just numbers and strings. I’m still using a mix of tools and plotting … but that’s life.

library(gdalraster)
library(ximage)
ds <- new(GDALRaster, dsn)  ## same as terra::rast(dsn) or osgeo.gdal.Open(dsn) or rioxarray.open_rasterio(dsn)
ds

C++ object of class GDALRaster
 Driver : OGC Web Map Tile Service (WMTS)
 DSN    : WMTS:https://services.arcgisonline.com/arcgis/rest/services/World_Imagery/MapServer/WMTS/1.0.0/WMTSCapabilities.xml,layer=World_Imagery
 Dim    : 1073741766, 1070224430, 4
 CRS    : WGS 84 / Pseudo-Mercator (EPSG:3857)
 Res    : 0.037323, 0.037323
 Bbox   : -20037507.260427, -19971868.890929, 20037507.248165, 19971868.903191


We don’t want the whole world and certainly not that much detail, how about Mawson Station in Antarctica, Mawson is at approximately 62.87E and -67.6S.
We should define a local equal area projection, on Mawson.

prj <- "+proj=laea +lon_0=62.8742 +lat_0=-67.6033"

What is the right extent or bbox for that projection? It’s just metres around the zero-point, let’s go with a range.

b <- 125000
tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt") ## use GDAL to manage IO and cleanup, same in other tools
chk <- warp(dsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 0))

0...10...20...30...40...50...60...70...80...90...100 - done.

ximage(read_ds(new(GDALRaster, tfile), bands = 1:3), asp = 1)








What about a different data source, can we see any meaningful data in sea ice at this time of year?

icedsn <- "/vsicurl/https://data.seaice.uni-bremen.de/amsr2/asi_daygrid_swath/s3125/2025/dec/Antarctic3125/asi-AMSR2-s3125-20251220-v5.4.tif"

tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt")
chk <- warp(icedsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 0))

0...10...20...30...40...50...60...70...80...90...100 - done.

dat <- read_ds(new(GDALRaster, tfile), bands = 1)
dat[dat > 100] <- NA

ximage(dat, col = grey.colors(100), asp = 1)







summary(dat)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
   1.00   94.00   98.00   90.99  100.00  100.00  576851 


Not unexpectedly at 3.125km resolution it’s a bit sparse on information, but we didn’t have to use a different approach to get coincident data. The ice is thick here, to see Mawson at about this time from Sentinel 2 imagery, try the estinel catalog, click ‘ESA’ and zoom around for more, you’ll see that dark hole in the sea ice).
What about the bathymetry? Even at this scale we can see the disparity between bathymetric and topographic detail.

bathdsn <- "/vsicurl/https://projects.pawsey.org.au/idea-gebco-tif/GEBCO_2024.tif"
tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt")
chk <- warp(bathdsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 0))

0...10...20...30...40...50...60...70...80...90...100 - done.

ximage(dat <- read_ds(new(GDALRaster, tfile), bands = 1), col = hcl.colors(128), asp = 1)








Let’s zoom right in, and this time we’ll use terra so that adding contours is easy, read the REMA v2 2m topography and add it as contours.

b <- 3000
tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt")
chk <- warp(dsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 0))

0...10...20...30...40...50...60...70...80...90...100 - done.

ximage(read_ds(new(GDALRaster, tfile), bands = 1:3), asp = TRUE)

remadsn <- "/vsicurl/https://raw.githubusercontent.com/mdsumner/rema-ovr/main/REMA-2m_dem_ovr.vrt"
library(terra)
r <- rast(ext(-b, b, -b, b), crs = prj, ncols = 1024, nrows = 1024)
tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt")
chk <- warp(remadsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 1024))

0...10...20...30...40...50...60...70...80...90...100 - done.

rema <- read_ds(new(GDALRaster, tfile), bands = 1)
contour(setValues(r, rema), add = TRUE)








Note that throughout we’ve done no juggling of input coverage or extent or determining output resolution - we ask GDAL for “enough pixels” to see, because that’s what we’re doing and it sorts that out. Reading enough pixels for a view from an efficient data source takes very little time, with raster logic as a core guideline it also takes very little time to craft our scripts.


Thanks
Hopefully this sparks some interest in the simplicity of raster grid logic and more use of the GDAL warp api to its full potential. Please contact me if you have any interest or questions.
https://github.com/hypertidy/vaster/issues

sessionInfo()

R version 4.5.1 (2025-06-13)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.5 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_AU.UTF-8        LC_COLLATE=en_AU.UTF-8    
 [5] LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8   
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] grout_0.0.2.9009      ximage_0.0.0.9012     gdalraster_2.3.0.9007
[4] terra_1.8-80          vaster_0.5.0         

loaded via a namespace (and not attached):
 [1] bit_4.6.0          jsonlite_2.0.0     dplyr_1.1.4        compiler_4.5.1    
 [5] maps_3.4.1.1       tidyselect_1.2.1   Rcpp_1.1.0         xml2_1.5.1        
 [9] palr_0.4.0         nanoarrow_0.7.0-2  parallel_4.5.1     dichromat_2.0-0.1 
[13] scales_1.4.0       png_0.1-8          yaml_2.3.10        fastmap_1.2.0     
[17] R6_2.6.1           generics_0.1.4     yyjsonr_0.1.21     knitr_1.50        
[21] htmlwidgets_1.6.4  tibble_3.3.0       pillar_1.11.1      RColorBrewer_1.1-3
[25] rlang_1.1.6        xfun_0.53          bit64_4.6.0-1      cli_3.6.5         
[29] magrittr_2.0.4     wk_0.9.4.9000      digest_0.6.39      rstudioapi_0.15.0 
[33] rappdirs_0.3.3     lifecycle_1.0.4    vctrs_0.6.5        evaluate_1.0.5    
[37] glue_1.8.0         farver_2.1.2       codetools_0.2-19   rmarkdown_2.29    
[41] pkgconfig_2.0.3    tools_4.5.1        htmltools_0.5.8.1 






 


Coordinates broken in NetCDF
Michael Sumner — Thu, 04 Sep 2025 00:00:00 GMT
 







A NetCDF file
There’s a NetCDF file at this URL:
"https://zenodo.org/records/7327711/files/CS2WFA_25km_201007.nc?download=1"
It’s fairly straightforward:
netcdf CS2WFA_25km_201007 {
dimensions:
        y = 332 ;
        x = 316 ;
        time = 1 ;
variables:
        float time(time) ;
                time:long_name = "time" ;
                time:units = "months since 2000-01-01" ;
                time:calendar = "360_day" ;
        double lat(y, x) ;
                lat:long_name = "latitude" ;
                lat:units = "degrees_north" ;
        double lon(y, x) ;
                lon:long_name = "longitude" ;
                lon:units = "degrees_east" ;
...
        double sea_ice_concentration(time, y, x) ;
                sea_ice_concentration:least_significant_digit = 4LL ;
                sea_ice_concentration:units = "1" ;
                sea_ice_concentration:standard_name = "sea_ice_area_fraction" ;
                sea_ice_concentration:description = "Mean sea ice concentration in grid cell, from Bootstrap V3 concentration algorithm (Comiso, 2017 https://doi.org/10.5067/7Q8HCCWS4I0R)" ;

// global attributes:
                :title = "Antarctic sea ice physical properties obtained from CryoSat-2 using the CS2WFA algorithm" ;
                :institution = "NASA GSFC Cryospheric Sciences Laboratory and University of Maryland-College Park" ;
                :history = "File created on November 15, 2022, 12:09:06" ;

group: sea_ice_thickness {
  ...
group: density {
  variables:
        double ice_density(time, y, x) ;
                ice_density:least_significant_digit = 4LL ;
 ...
  } // group density
}
There’s a root group with four variables defined in time,y,x and another two groups with further variables on that same grid.
Each data array is 316x332x1 and we can tell that every lon,lat pair is stored explicitly. This fits a convention in NetCDF where time is “unlimited” and we’re only looking at a subset of an overall time series, that could well be continually generated day to day still now.
Let’s investigate the coordinates. We’ll use GDAL via a commonly used wrapper in R, its ‘vsicurl’ protocol to stream from the internet, and declare the driver explicitly (lest we get the less sophisticated HDF5 interpretation).

#dsn <- "NETCDF:/vsicurl/https://zenodo.org/records/7327711/files/CS2WFA_25km_201007.nc?download=1"
dsn <- "NETCDF:../CS2WFA_25km_201007.nc"
library(terra)

terra 1.8.62

lon <- rast(dsn, "lon")

Warning: [rast] GDAL did not find an extent. installing the ncdf4 package may
help

lat <- rast(dsn, "lat")

Warning: [rast] GDAL did not find an extent. installing the ncdf4 package may
help

plot(c(lon, lat))








We get a message from terra about “cells not equally spaced”, which means that when reading an array the geospatial context in GDAL didn’t find anything specifying a compact representation of coordinates, but lon and lat are the coordinates as data, so we can safely ignore this message. (There’s another message about ncdf4, but ignore that too it’s terra try(ing) stuff. )
We won’t pay attention to “time”, because our file only has one time step. We’re interested in these lon,lat coordinates. What does the data itself look like? Again we choose one of the 3D variables (with degenerate 3rd dimension ‘time’) by name.

ice <- rast(dsn, "sea_ice_concentration")

Warning: [rast] GDAL did not find an extent. installing the ncdf4 package may
help

plot(ice)








That should look familiar, though notice that we don’t have any spatial information it’s just a matrix in its own index coordinates, 0,316 for x and 0,332 for y.
Let’s plot the coordinates. Now we can see our data in a much more spatial-ish context.

xyz <- values(c(lon, lat, ice))
plot(xyz[,1:2], pch = ".", col = palr::d_pal(xyz[,3]), asp = 2)
maps::map("world2", add = TRUE)








We now don’t have a nice gridded dataset, it’s points in longitude latitude. What can we do to plot it in that nice polar aspect with proper spatial referencing?
Let’s have a look at this dataset from a GDAL perspective. Now we use the API package gdalraster which gives a lot more control over GDAL itself. I’m going to skip over interrogating the syntax for a particular array, like we did with terra and ‘sea_ice_concentration’ above, I construct the subdataset syntax to open.

"NETCDF:/vsicurl/https://zenodo.org/records/7327711/files/CS2WFA_25km_201007.nc?download=1"

[1] "NETCDF:/vsicurl/https://zenodo.org/records/7327711/files/CS2WFA_25km_201007.nc?download=1"

sds <- sprintf("%s:sea_ice_concentration", dsn)
library(gdalraster)

GDAL 3.12.0dev-4e2b27da8f (released 2025-08-26), GEOS 3.12.1, PROJ 9.3.1



Attaching package: 'gdalraster'


The following object is masked from 'package:terra':

    rasterize

ds <- new(GDALRaster, sds)
ds$info()

Driver: netCDF/Network Common Data Format
Files: ../CS2WFA_25km_201007.nc
Size is 316, 332
Metadata:
  NC_GLOBAL#history=File created on November 15, 2022, 12:09:06
  NC_GLOBAL#institution=NASA GSFC Cryospheric Sciences Laboratory and University of Maryland-College Park
  NC_GLOBAL#title=Antarctic sea ice physical properties obtained from CryoSat-2 using the CS2WFA algorithm
  NETCDF_DIM_EXTRA={time}
  NETCDF_DIM_time_DEF={1,5}
  NETCDF_DIM_time_VALUES=126
  sea_ice_concentration#description=Mean sea ice concentration in grid cell, from Bootstrap V3 concentration algorithm (Comiso, 2017 https://doi.org/10.5067/7Q8HCCWS4I0R)
  sea_ice_concentration#least_significant_digit=4
  sea_ice_concentration#standard_name=sea_ice_area_fraction
  sea_ice_concentration#units=1
  time#calendar=360_day
  time#long_name=time
  time#units=months since 2000-01-01
Geolocation:
  SRS=GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
  X_DATASET=NETCDF:"../CS2WFA_25km_201007.nc":lon
  X_BAND=1
  Y_DATASET=NETCDF:"../CS2WFA_25km_201007.nc":lat
  Y_BAND=1
  PIXEL_OFFSET=0
  PIXEL_STEP=1
  LINE_OFFSET=0
  LINE_STEP=1
  GEOREFERENCING_CONVENTION=PIXEL_CENTER
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  332.0)
Upper Right (  316.0,    0.0)
Lower Right (  316.0,  332.0)
Center      (  158.0,  166.0)
Band 1 Block=316x332 Type=Float64, ColorInterp=Undefined
  NoData Value=9.969209968386869e+36
  Unit Type: 1
  Metadata:
    NETCDF_VARNAME=sea_ice_concentration
    least_significant_digit=4
    units=1
    standard_name=sea_ice_area_fraction
    description=Mean sea ice concentration in grid cell, from Bootstrap V3 concentration algorithm (Comiso, 2017 https://doi.org/10.5067/7Q8HCCWS4I0R)
    NETCDF_DIM_time=126


The interesting part of that output is under “Geolocation:”. We can see that while GDAL doesn’t have spatial referencing for this array, it does seem to know that’s it’s possible because of the X_DATASET and Y_DATASET. These can be used by the GDAL warper API (warp means “reprojection” or “reshaping” for an image) in order to resolve to a new spatial gridded dataset … but also note, the same facilities used to “reproject” one regular grid to another are also the same that can be used to generate a regular grid from irregular geolocation coordinates (we have a point for every data point, but those coordinates might be less dense, or might be GCPs or RCPs, and no matter what coordinate system those latent coordinates are using, we can warp to whatever spatial grid we like). So let’s warp.
We don’t have to specifying the geolocation arrays, we saw that GDAL already knows this. We’ll write our new gridded dataset conveniently to a temporary virtual “file” so we can avoid more cleanup.

gdalraster::warp(sds,  tf <- tempfile(fileext = ".tif", tmpdir = "/vsimem"), t_srs = "EPSG:4326")

0...10...20...30...40...50...60...70...80...90...100 - done.

plot(icegrid <- rast(tf))







icegrid

class       : SpatRaster 
size        : 127, 902, 1  (nrow, ncol, nlyr)
resolution  : 0.398722, 0.3987662  (x, y)
extent      : 0.164645, 359.8119, -89.80441, -39.1611  (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84 (EPSG:4326) 
source      : filedb3943dea1d4.tif 
name        : filedb3943dea1d4 


We indeed now have a spatial grid, a raster. But perhaps we don’t like the 0,360 convention. We can set the target extent, GDAL had to figure one out from the input geolocation arrays, and it will usually do a good job but it’s really our responsibility to specify what we want for reproducibility in later workflows.

gdalraster::warp(sds,  tf <- tempfile(fileext = ".tif", tmpdir = "/vsimem"), t_srs = "EPSG:4326", 
                 cl_arg = c("-te", -180, -90, 180, -39, "-to", "GEOLOC_NORMALIZE_LONGITUDE_MINUS_180_PLUS_180=YES"))

0...10...20...30...40...50...60...70...80...90...100 - done.

plot(icegrid <- rast(tf))







icegrid

class       : SpatRaster 
size        : 194, 1371, 1  (nrow, ncol, nlyr)
resolution  : 0.2625821, 0.2628866  (x, y)
extent      : -180, 180, -90, -39  (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84 (EPSG:4326) 
source      : filedb394444fc021.tif 
name        : filedb394444fc021 


Another thing that is also our responsibility is the dimensions of the grid and the resolution, obviously this and extent (or bbox, bounding box) are all interlinked so we augment our bounding box / extent setting with a nice clean resolution.

gdalraster::warp(sds,  tf <- tempfile(fileext = ".tif", tmpdir = "/vsimem"), t_srs = "EPSG:4326", 
                 cl_arg = c("-te", -180, -90, 180, -39, "-tr", 0.25, 0.25,
                            "-to", "GEOLOC_NORMALIZE_LONGITUDE_MINUS_180_PLUS_180=YES"))

0...10...20...30...40...50...60...70...80...90...100 - done.

plot(icegrid <- rast(tf))
maps::map(add = TRUE)







icegrid

class       : SpatRaster 
size        : 204, 1440, 1  (nrow, ncol, nlyr)
resolution  : 0.25, 0.25  (x, y)
extent      : -180, 180, -90, -39  (xmin, xmax, ymin, ymax)
coord. ref. : lon/lat WGS 84 (EPSG:4326) 
source      : filedb39443fd3375.tif 
name        : filedb39443fd3375 


That is all well and nice, and we’ve quietly demonstrated some of the key powers of GDAL:

virtualization, streaming read, read a particular variable
investigate GDAL logic (coordinates are linked to data via latent “geolocation” arrays)
warping with heuristics to a CRS
warping with a grid specification, and using geolocation arrays to guide the grid-resolving process
setting grid specification with any of crs, extent(bbox), resolution, dimension, and allowing GDAL to internally wrap from 0,360 context to -180,180

But, we still don’t have that nice polar aspect we saw from the raw array above. So we change CRS. ‘EPSG:3412’ is a Polar Stereographic map projection on the south pole, commonly used for standard sea ice products. As a grid this has nice properties, that don’t require edge-wrap for a matrix model, it’s true scale at approximately where the coastline of Antarctica is, and it preserves shape (this means not all cells are exactly the same size, but they’re close enough for where the sea ice is).

gdalraster::warp(sds,  tf <- tempfile(fileext = ".tif", tmpdir = "/vsimem"), t_srs = "EPSG:3412", 
                cl_arg = c("-to", "GEOLOC_NORMALIZE_LONGITUDE_MINUS_180_PLUS_180=YES"))

0...10...20...30...40...50...60...70...80...90...100 - done.

plot(polargrid <- rast(tf))







polargrid

class       : SpatRaster 
size        : 348, 332, 1  (nrow, ncol, nlyr)
resolution  : 24999.84, 24999.84  (x, y)
extent      : -3950080, 4349868, -4337584, 4362361  (xmin, xmax, ymin, ymax)
coord. ref. : NSIDC Sea Ice Polar Stereographic South (EPSG:3412) 
source      : filedb39427e3e998.tif 
name        : filedb39427e3e998 


To make sure let’s get some spatial data to plot with this.

v <- vect("/vsizip//vsicurl/https://github.com/wmgeolab/geoBoundaries/raw/main/releaseData/CGAZ/geoBoundariesCGAZ_ADM0.zip", 
     query = "SELECT shapeGroup FROM geoBoundariesCGAZ_ADM0 WHERE shapeGroup IN ('ATA')")
v <- crop(v, ext(-180, 180, -84, 0)) ## small hack to remove the pole seam
v <- project(v, "EPSG:3412")
plot(polargrid)
plot(v, add = TRUE)








Looks good! We’ve plotted the original data in the correct polar aspect.
But, the data aren’t exactly the same, we’ve added pixels in the heuristic used by GDAL to determine the grid specification from the extent and resolution of the geolocation array longitude and latitudes.

ice

class       : SpatRaster 
size        : 332, 316, 1  (nrow, ncol, nlyr)
resolution  : 1, 1  (x, y)
extent      : 0, 316, 0, 332  (xmin, xmax, ymin, ymax)
coord. ref. :  
source      : CS2WFA_25km_201007.nc:sea_ice_concentration 
varname     : sea_ice_concentration (sea_ice_area_fraction) 
name        : sea_ice_concentration 
unit        :                     1 

polargrid

class       : SpatRaster 
size        : 348, 332, 1  (nrow, ncol, nlyr)
resolution  : 24999.84, 24999.84  (x, y)
extent      : -3950080, 4349868, -4337584, 4362361  (xmin, xmax, ymin, ymax)
coord. ref. : NSIDC Sea Ice Polar Stereographic South (EPSG:3412) 
source      : filedb39427e3e998.tif 
name        : filedb39427e3e998 


So, let’s try to use the same grid.

gdalraster::warp(sds,  tf <- tempfile(fileext = ".tif", tmpdir = "/vsimem"), t_srs = "EPSG:3412", 
                cl_arg = c("-ts", 316, 332,
                           
                  "-to", "GEOLOC_NORMALIZE_LONGITUDE_MINUS_180_PLUS_180=YES"))

0...10...20...30...40...50...60...70...80...90...100 - done.

polargrid <- rast(tf)

polargrid

class       : SpatRaster 
size        : 332, 316, 1  (nrow, ncol, nlyr)
resolution  : 26265.66, 26204.65  (x, y)
extent      : -3950080, 4349868, -4337584, 4362361  (xmin, xmax, ymin, ymax)
coord. ref. : NSIDC Sea Ice Polar Stereographic South (EPSG:3412) 
source      : filedb394ad7590c.tif 
name        : filedb394ad7590c 


We can’t do that without giving them the same extent, but we still don’t know what that is for the original data. So let’s calculate it.

xy <- project(values(c(lon, lat)), to = "EPSG:3412", from = "EPSG:4326")
range(xy[,2])

[1] -3937527  4337533




Throw all that away, ignore everything above.
You should never ever do the above for a dataset that doesn’t need it. This is a major problem in array computation worlds, there’s a lowest common denominator that will work but why not be smart about it, and keep fidelity with the original scheme for a dataset.
The problem here is that we can’t reconstruct the actual grid in the file, we have to treat it as points in geolocation arrays, and infer something about the grid. Here’s the answer:

ex <- c(xmin = -3950000,  xmax = 3950000, ymin  = -3950000, ymax = 4350000)
diff(ex)[c(1, 3)] / c(316, 332)  ## look familiar?

 xmax  ymax 
25000 25000 



set_ext <- function(x, ex) {terra::ext(x) <- ex; x}

plot(set_ext(ice, ext(ex)))
plot(v, add = T)







plot(polargrid); plot(v, add = TRUE)








Now finally, we get exactly the same answer.

gdalraster::warp(sds,  tf <- tempfile(fileext = ".tif", tmpdir = "/vsimem"), t_srs = "EPSG:3412", 
                cl_arg = c("-ts", 316, 332, "-te", ex[1], ex[3], ex[2], ex[4],
                           
                  "-to", "GEOLOC_NORMALIZE_LONGITUDE_MINUS_180_PLUS_180=YES"))

0...10...20...30...40...50...60...70...80...90...100 - done.

polargrid <- rast(tf)

polargrid

class       : SpatRaster 
size        : 332, 316, 1  (nrow, ncol, nlyr)
resolution  : 25000, 25000  (x, y)
extent      : -3950000, 3950000, -3950000, 4350000  (xmin, xmax, ymin, ymax)
coord. ref. : NSIDC Sea Ice Polar Stereographic South (EPSG:3412) 
source      : filedb3946450fee3.tif 
name        : filedb3946450fee3 

polargrid - set_ext(ice, ext(ex))

Warning: [-] CRS do not match


class       : SpatRaster 
size        : 332, 316, 1  (nrow, ncol, nlyr)
resolution  : 25000, 25000  (x, y)
extent      : -3950000, 3950000, -3950000, 4350000  (xmin, xmax, ymin, ymax)
coord. ref. : NSIDC Sea Ice Polar Stereographic South (EPSG:3412) 
source(s)   : memory
varname     : filedb3946450fee3 
name        : filedb3946450fee3 
min value   :                 0 
max value   :                 0 

# class       : SpatRaster 
# size        : 332, 316, 1  (nrow, ncol, nlyr)
# resolution  : 25000, 25000  (x, y)
# extent      : -3950000, 3950000, -3950000, 4350000  (xmin, xmax, ymin, ymax)
# coord. ref. : NSIDC Sea Ice Polar Stereographic South (EPSG:3412) 
# source(s)   : memory
# varname     : fileb328ade0cec2 
# name        : fileb328ade0cec2 
# min value   :                0 
# max value   :                0 

But how did we do that? We plucked some magic numbers out of the air. We can’t obtain those four numbers from the lon lat arrays, we can’t find them in the file, they aren’t in the metadata or the website. We just know, because NSIDC sea ice products are standard on a 3412 grid at 25km resolution.
These four numbers: -3950000, 3950000, -3950000, 4350000 are entirely cryptic in the data store itself, and require an expert to notice the problem. We can throw away 332 * 316 * 2 values from every file, replace them with these four values and we have a better and more faithful representation of the original data.
This is an entropy problem.
I refer to this kind of problem as “cryptic curvilinear”. The data present as an irregular grid with curvlinear coordinates, but they are entirely unnecessary, we can replace them with 4 values that anyone can remember and we have the following easy ways to fix, roughly in increasing difficulty and decreasing utility:

set “-a_srs”, and -a_ullr” in gdal raster convert, or via “vrt://{dsn}?a_ullr=,,,” syntax
OR set “-a_srs”, and “-a_gt” in gdal raster convert, or via “vrt://{dsn}?a_gt=,,,,,” syntax
set extent with terra in R and set the crs
open with rasterio and set geotransform and set spatial ref (same as set extent with R)
modify the coordinates in xarray, and set the crs



Oh, but sure there are bad files out there
There are so many.

GHRSST noisy, low precision degenerate rectilinear coordinates
OISST and NSIDC, have degenerate rectilinear coordinates that don’t cause problems





 


GDAL multidim and cloud-ready ZARR
Michael Sumner — Mon, 30 Jun 2025 00:00:00 GMT
 







The GDAL API
I have been working on a better understanding of the GDAL multidimensional model and to do that really needs a closer look at the GDAL API itself.
This post demonstrates loading the GDAL API via its Python bindings into R. We connect to a modern “cloud-ready” ZARR dataset, find out some details about its contents and then convert from its native multidimensional form to a more classic 2D raster model, then use that to create a map from Sentinel 2 imagery.
We don’t go very deep into any part, but just want to show a quick tour of some parts of GDAL that don’t get as much attention as deserved (IMO). I’m using a very recent version of GDAL, which might mean some code doesn’t work for you. If that’s the case please let me know and I can explore alternatives and identify when/how the newer features will be more available. There are some echoes here of an older post I made about GDAL in R: https://www.hypertidy.org/posts/2017-09-01_gdal-in-r/


ZARR and the European Space Agency (ESA)
The ESA is moving Copernicus to ZARR, launching its Earth Observation Processing Framework (EOPF) data format (Zarr). ZARR is “a community project to develop specifications and software for storage of large N-dimensional typed arrays”.
A ZARR is a dataset consisting of trees of array chunks stored (usually) in object storage and indexed by fairly simple JSON metadata that describes how those chunks align together in one potentially very large array. The idea is that all the metadata lives upfront in instantly readable JSON, and changes made to the dataset (extending it each day as new data arrives) affects only the relevant chunks and the small parts of the JSON. This is different to a long list of NetCDFs that grows every day, where the metadata is self-contained for each file and there is no overarching abstraction for the entire file set.
ZARR is usually in object storage, and loaded by datacube software such as xarray. It’s not intended to be zipped into a huge file and downloaded or read, the real power lies in the entire dataset being lazy, and understood by software that needs just one data set description (url, or S3 path, etc).


ESA sample Zarr datasets
The ESA provide a set of example ZARRs that are available in zip files:
https://eopf-public.s3.sbg.perf.cloud.ovh.net/product.html
We choose one that is described by this URL:

url <- "https://eopf-public.s3.sbg.perf.cloud.ovh.net/eoproducts/S02MSIL1C_20230629T063559_0000_A064_T3A5.zarr.zip"



GDAL urls and zip files
GDAL doesn’t force us to download data, but we need some syntax to leverage its remote capabilities in the simplest way. The Virtual File System (VSI) allows us to declare special sources like zip files /vsizip/ and urls /vsicurl/, which we can chain together. With ZARR we also need careful quoting of the description, and we declare the ZARR driver upfront.

(dsn <- sprintf('ZARR:"/vsizip//vsicurl/%s"', url))

[1] "ZARR:\"/vsizip//vsicurl/https://eopf-public.s3.sbg.perf.cloud.ovh.net/eoproducts/S02MSIL1C_20230629T063559_0000_A064_T3A5.zarr.zip\""




GDAL and multidimensional datasets
GDAL has a multidimensional mode for data sources that aren’t “imagery” in the traditional sense. (If we open one of these datasets in “classic” mode we end up with a lot of bands on a 2D raster, or potentially many bands on many subdatasets within a more general container. Zarr is a container format, much like HDF5 and NetCDF).
Multidimensional mode is avaible in the API via OpenEx() and declaring type OF_MULTIDIM_RASTER.
To actually load this python library we use {reticulate} py_require() which drives the awesome Python uv package manager.
(For some reason the pypi name of the package is “gdal”, but the actual module is obtained with “osgeo.gdal”).

reticulate::py_require("gdal")
gdal <- reticulate::import("osgeo.gdal")
gdal$UseExceptions()

sample(names(gdal), 40)  ## see that we have a huge coverage of the underlying API, 544 elements at time of writing

 [1] "GCI_NIRBand"                       "GCI_CyanBand"                     
 [3] "GFU_RedMax"                        "GRT_COMPOSITE"                    
 [5] "GRT_AGGREGATION"                   "GRIORA_Gauss"                     
 [7] "GPI_RGB"                           "GFT_Integer"                      
 [9] "GFU_AlphaMin"                      "GCI_SAR_C_Band"                   
[11] "GCI_OtherIRBand"                   "ApplyVerticalShiftGrid"           
[13] "CPLE_NoWriteAccess"                "deprecation_warn"                 
[15] "Rasterize"                         "GCI_SAR_K_Band"                   
[17] "TermProgress_nocb"                 "GMF_ALL_VALID"                    
[19] "DCAP_CREATECOPY"                   "GRA_Q1"                           
[21] "config_option"                     "DCAP_FIELD_DOMAINS"               
[23] "GDAL_GCP_Info_set"                 "wrapper_GDALVectorTranslateDestDS"
[25] "ColorTable"                        "ContourGenerate"                  
[27] "DCAP_COORDINATE_EPOCH"             "GDT_UInt16"                       
[29] "GDsCDeleteRelationship"            "GCI_YCbCr_CbBand"                 
[31] "IsLineOfSightVisible"              "BuildVRTInternalObjects"          
[33] "DEMProcessing"                     "GDALDestroyDriverManager"         
[35] "DCAP_RENAME_LAYERS"                "MultiDimInfoOptions"              
[37] "CPLES_XML_BUT_QUOTES"              "DitherRGB2PCT"                    
[39] "GDT_TypeCount"                     "GARIO_ERROR"                      


The API elements chain in the usual way that works in python with ‘object.element.thing.etc’ syntax uses R’s $ accessor.

gdal$Dimension$GetIndexingVariable


 signature: (self, *args) -> 'GDALMDArrayHS *'




Open the data
It’s not very exciting yet.

ds <- gdal$OpenEx(dsn, gdal$OF_MULTIDIM_RASTER)
ds

 >


To actually find out what’s in there we have to traverse a tree of potentially nested “groups” that organize actual datasets in a hierarchy.
Get the root group and dive in, it’s very tediuous but shows some of what is there. There might be MDArrays in a group, or there might just be more groups.

rg <- ds$GetRootGroup()
rg$GetMDArrayNames()

list()

rg$GetGroupNames()

[1] "conditions"   "measurements" "quality"     

g1 <- rg$OpenGroup("quality")
g1$GetMDArrayNames()

list()

g1$GetGroupNames()

[1] "l1c_quicklook" "mask"         

g2 <- g1$OpenGroup("l1c_quicklook")
g2$GetMDArrayNames()

list()

g2$GetGroupNames()

[1] "r10m"

g3 <- g2$OpenGroup("r10m")
g3$GetMDArrayNames()

[1] "band" "x"    "y"    "tci" 

g3$GetGroupNames()

list()


Finally we got to actual data, we recognize ‘tci’ as being the quicklook RGB of Sentinel 2.
To avoid tedium write a quick recursive function to find all the MDArray, at each level use GetFullName() which provides the cumulative path to where we are in the tree.

get_all_mdnames <- function(rootgroup) {
  groups <- rootgroup$GetGroupNames()
  groupname <- rootgroup$GetFullName()
  amd <- rootgroup$GetMDArrayNames()
  md <- sprintf("%s/%s", groupname, amd)
  md <- c(md, unlist(lapply(groups, \(.g) get_all_mdnames(rootgroup$OpenGroup(.g)))))
  md
}

get_all_mdnames(ds$GetRootGroup())

  [1] "/conditions/geometry/angle"                        
  [2] "/conditions/geometry/band"                         
  [3] "/conditions/geometry/detector"                     
  [4] "/conditions/geometry/x"                            
  [5] "/conditions/geometry/y"                            
  [6] "/conditions/geometry/mean_sun_angles"              
  [7] "/conditions/geometry/mean_viewing_incidence_angles"
  [8] "/conditions/geometry/sun_angles"                   
  [9] "/conditions/geometry/viewing_incidence_angles"     
 [10] "/conditions/mask/detector_footprint/r10m/x"        
 [11] "/conditions/mask/detector_footprint/r10m/y"        
 [12] "/conditions/mask/detector_footprint/r10m/b02"      
 [13] "/conditions/mask/detector_footprint/r10m/b03"      
 [14] "/conditions/mask/detector_footprint/r10m/b04"      
 [15] "/conditions/mask/detector_footprint/r10m/b08"      
 [16] "/conditions/mask/detector_footprint/r20m/x"        
 [17] "/conditions/mask/detector_footprint/r20m/y"        
 [18] "/conditions/mask/detector_footprint/r20m/b05"      
 [19] "/conditions/mask/detector_footprint/r20m/b06"      
 [20] "/conditions/mask/detector_footprint/r20m/b07"      
 [21] "/conditions/mask/detector_footprint/r20m/b11"      
 [22] "/conditions/mask/detector_footprint/r20m/b12"      
 [23] "/conditions/mask/detector_footprint/r20m/b8a"      
 [24] "/conditions/mask/detector_footprint/r60m/x"        
 [25] "/conditions/mask/detector_footprint/r60m/y"        
 [26] "/conditions/mask/detector_footprint/r60m/b01"      
 [27] "/conditions/mask/detector_footprint/r60m/b09"      
 [28] "/conditions/mask/detector_footprint/r60m/b10"      
 [29] "/conditions/mask/l1c_classification/r60m/x"        
 [30] "/conditions/mask/l1c_classification/r60m/y"        
 [31] "/conditions/mask/l1c_classification/r60m/b00"      
 [32] "/conditions/meteorology/cams/latitude"             
 [33] "/conditions/meteorology/cams/longitude"            
 [34] "/conditions/meteorology/cams/aod1240"              
 [35] "/conditions/meteorology/cams/aod469"               
 [36] "/conditions/meteorology/cams/aod550"               
 [37] "/conditions/meteorology/cams/aod670"               
 [38] "/conditions/meteorology/cams/aod865"               
 [39] "/conditions/meteorology/cams/bcaod550"             
 [40] "/conditions/meteorology/cams/duaod550"             
 [41] "/conditions/meteorology/cams/isobaricInhPa"        
 [42] "/conditions/meteorology/cams/number"               
 [43] "/conditions/meteorology/cams/omaod550"             
 [44] "/conditions/meteorology/cams/ssaod550"             
 [45] "/conditions/meteorology/cams/step"                 
 [46] "/conditions/meteorology/cams/suaod550"             
 [47] "/conditions/meteorology/cams/surface"              
 [48] "/conditions/meteorology/cams/time"                 
 [49] "/conditions/meteorology/cams/valid_time"           
 [50] "/conditions/meteorology/cams/z"                    
 [51] "/conditions/meteorology/ecmwf/latitude"            
 [52] "/conditions/meteorology/ecmwf/longitude"           
 [53] "/conditions/meteorology/ecmwf/isobaricInhPa"       
 [54] "/conditions/meteorology/ecmwf/msl"                 
 [55] "/conditions/meteorology/ecmwf/number"              
 [56] "/conditions/meteorology/ecmwf/r"                   
 [57] "/conditions/meteorology/ecmwf/step"                
 [58] "/conditions/meteorology/ecmwf/surface"             
 [59] "/conditions/meteorology/ecmwf/tco3"                
 [60] "/conditions/meteorology/ecmwf/tcwv"                
 [61] "/conditions/meteorology/ecmwf/time"                
 [62] "/conditions/meteorology/ecmwf/u10"                 
 [63] "/conditions/meteorology/ecmwf/v10"                 
 [64] "/conditions/meteorology/ecmwf/valid_time"          
 [65] "/measurements/reflectance/r10m/x"                  
 [66] "/measurements/reflectance/r10m/y"                  
 [67] "/measurements/reflectance/r10m/b02"                
 [68] "/measurements/reflectance/r10m/b03"                
 [69] "/measurements/reflectance/r10m/b04"                
 [70] "/measurements/reflectance/r10m/b08"                
 [71] "/measurements/reflectance/r20m/x"                  
 [72] "/measurements/reflectance/r20m/y"                  
 [73] "/measurements/reflectance/r20m/b05"                
 [74] "/measurements/reflectance/r20m/b06"                
 [75] "/measurements/reflectance/r20m/b07"                
 [76] "/measurements/reflectance/r20m/b11"                
 [77] "/measurements/reflectance/r20m/b12"                
 [78] "/measurements/reflectance/r20m/b8a"                
 [79] "/measurements/reflectance/r60m/x"                  
 [80] "/measurements/reflectance/r60m/y"                  
 [81] "/measurements/reflectance/r60m/b01"                
 [82] "/measurements/reflectance/r60m/b09"                
 [83] "/measurements/reflectance/r60m/b10"                
 [84] "/quality/l1c_quicklook/r10m/band"                  
 [85] "/quality/l1c_quicklook/r10m/x"                     
 [86] "/quality/l1c_quicklook/r10m/y"                     
 [87] "/quality/l1c_quicklook/r10m/tci"                   
 [88] "/quality/mask/r10m/x"                              
 [89] "/quality/mask/r10m/y"                              
 [90] "/quality/mask/r10m/b02"                            
 [91] "/quality/mask/r10m/b03"                            
 [92] "/quality/mask/r10m/b04"                            
 [93] "/quality/mask/r10m/b08"                            
 [94] "/quality/mask/r20m/x"                              
 [95] "/quality/mask/r20m/y"                              
 [96] "/quality/mask/r20m/b05"                            
 [97] "/quality/mask/r20m/b06"                            
 [98] "/quality/mask/r20m/b07"                            
 [99] "/quality/mask/r20m/b11"                            
[100] "/quality/mask/r20m/b12"                            
[101] "/quality/mask/r20m/b8a"                            
[102] "/quality/mask/r60m/x"                              
[103] "/quality/mask/r60m/y"                              
[104] "/quality/mask/r60m/b01"                            
[105] "/quality/mask/r60m/b09"                            
[106] "/quality/mask/r60m/b10"                            


Happily, we see our target MDArray name in there /quality/l1c_quicklook/r10m/tci.


Actual data
Finally let’s get some data out. We can obtain the MDArray now by full name and find out some properties.

reticulate::py_require("gdal")
gdal <- reticulate::import("osgeo.gdal")
gdal$UseExceptions()

dsn <- "ZARR:\"/vsizip//vsicurl/https://eopf-public.s3.sbg.perf.cloud.ovh.net/eoproducts/S02MSIL1C_20230629T063559_0000_A064_T3A5.zarr.zip\""

ds <- gdal$OpenEx(dsn, gdal$OF_MULTIDIM_RASTER)
rg <- ds$GetRootGroup()
mdname <- "/quality/l1c_quicklook/r10m/tci"
md <- rg$OpenMDArrayFromFullname(mdname)
md$GetDimensionCount()

[1] 3


Traverse the dimensions to get their sizes and names.

vapply(md$GetDimensions(), \(.d) .d$GetSize(), 0L)

[1]     3 10980 10980

vapply(md$GetDimensions(), \(.d) .d$GetName(), "")

[1] "band" "y"    "x"   


Explore some metadata attributes.

mnames <- vapply(md$GetAttributes(), \(.a) .a$GetName(), "")

(meta <- setNames(lapply(md$GetAttributes(), \(.a) unlist(.a$Read())), mnames))

$long_name
[1] "TCI: True Color Image"

$`proj:bbox`
[1]  300000 4490220  409800 4600020

$`proj:epsg`
[1] 32636

$`proj:shape`
[1] 10980 10980

$`proj:transform`
[1]      10       0  300000       0     -10 4600020       0       0       1

$`proj:wkt2`
[1] "PROJCS[\"WGS 84 / UTM zone 36N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",33],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32636\"]]"


We see that the CRS information is in there, but sadly while the geotransform of the data is maintained when we convert to classic raster the CRS does not make the journey (the geotransform is just the image bbox intermingled with its resolution in a mathematical abstraction used in matrix manipulation).
A multidim raster can be converted to classic form, either in whole or after applying a $GetView() operation that acts like numpy ‘[:]’ array subsetting.
When converting to classic raster we specify the x, then y dimensions of the dataset, which from multdim convention are in reverse order. (Note that dimension 0 is the “3rd” dimension in normal thinking, here that dimension is “band” or a dimension for each of red, green, blue in the quicklook image).

cc <- md$AsClassicDataset(2L, 1L)
tr <- unlist(cc$GetGeoTransform())
dm <- c(cc$RasterXSize, cc$RasterYSize)
cc$GetSpatialRef() ## empty, so we retrieve from the attributes
crs <- cc$GetMetadata()[["proj:wkt2"]]

Now, I used reproj_extent to convert the dataset’s bbox to one in longlat, but I won’t share that here, we’ll just use the result so our map is a little more familiar. The source data xy bbox is xmin: 300000 ymin: 4490220  xmax: 409800 ymax:4600020, and we take small section of that which is this in longitude latitude:

# xmin,xmax,ymin,ymax converted below to bbox 
ex <- c(31.689, 31.774, 40.665, 40.724)

To warp the imagery to this region we first need to set the CRS properly on the dataset, so translate to a temporary VRT and use that dataset for the next step.

dsvrt <- gdal$Translate(tempfile(fileext = ".vrt", tmpdir = "/vsimem"), cc, options = c("-a_srs", crs))
tf <- tf <- "/vsimem/result.tif"
ww <- gdal$Warp(tf, dsvrt,  dstSRS = "EPSG:4326", outputBounds = ex[c(1, 3, 2, 4)])
ww$Close()

[1] 0


Finally we can read the image, plot it, obtain some contextual data and be assured that our map is correct.

library(gdalraster)
data <- read_ds(new(GDALRaster, tf))


library(osmdata)
x <- opq(ex[c(1, 3, 2, 4)]) |> add_osm_feature(key = "highway") |> osmdata_sf()

library(sf)
plot_raster(data)
plot(x$osm_lines[0], add = TRUE, col = "hotpink")










Summary
We had a look at GDAL multidimensional API, converting a 3D dataset to classic raster, augmenting the missing CRS information on a virtual copy and then warping an image to a familiar map context.
There is ongoing work to bring the full GDAL API to R in {gdalraster}, and in-development version of gdalraster to add the GDALMultiDimRaster class and helpers: https://github.com/mdsumner/gdalraster/tree/multidimnew.
Please get in touch if any of this is of interest!


Late Bonus Section: Bluelink ocean model
This is something I want to use VRT to encapsulate as one whole dataset, here just a test on one file (there are 174 of these ocean_temp files, up to June 2024 at time of writing).
Note that we are using ‘/fileServer/’ from Thredds, not ‘/dodsC/’. GDAL uses userfaultd as a trick to read NetCDF remotely.
from osgeo import gdal
dsn = "/vsicurl/https://thredds.nci.org.au/thredds/fileServer/gb6/BRAN/BRAN2023/daily/ocean_temp_2010_01.nc"
ds = gdal.OpenEx(dsn, gdal.OF_MULTIDIM_RASTER)
rg = ds.GetRootGroup()
rg.GetMDArrayNames()
# ['xt_ocean', 'yt_ocean', 'st_ocean', 'Time', 'nv', 'st_edges_ocean', 'average_T1', 'average_T2', 'average_DT', 'Time_bnds', 'temp']

temp = rg.OpenMDArrayFromFullname("//temp")
[d.GetName() for d in temp.GetDimensions()]
#['Time', 'st_ocean', 'yt_ocean', 'xt_ocean']
[d.GetSize() for d in temp.GetDimensions()]
#[31, 51, 1500, 3600]
I don’t know why xarray can seemingly read netcdf from S3, but not from normal url? Maybe just something fsspec I need to understand?
I also noticed that if the file is local the array names at root level are like this:
['//xt_ocean', '//yt_ocean', '//st_ocean', '//Time', '//nv', '//st_edges_ocean', '//average_T1', '//average_T2', '//average_DT', '//Time_bnds', '//temp']


gdalraster::gdal_version()

[1] "GDAL 3.11.0dev-dd6009a0eb, released 2018/99/99"
[2] "3110000"                                       
[3] "20189999"                                      
[4] "3.11.0dev-dd6009a0eb"                          

devtools::session_info()

─ Session info ───────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.4.2 (2024-10-31)
 os       Ubuntu 20.04.6 LTS
 system   x86_64, linux-gnu
 ui       X11
 language (EN)
 collate  en_AU.UTF-8
 ctype    en_AU.UTF-8
 tz       Australia/Hobart
 date     2025-03-12
 pandoc   3.2 @ /usr/lib/rstudio-server/bin/quarto/bin/tools/x86_64/ (via rmarkdown)

─ Packages ───────────────────────────────────────────────────────────────────
 package     * version     date (UTC) lib source
 bit           4.5.0.1     2024-12-03 [2] CRAN (R 4.4.2)
 bit64         4.6.0-1     2025-01-16 [2] CRAN (R 4.4.2)
 cachem        1.1.0       2024-05-16 [2] CRAN (R 4.4.1)
 class         7.3-23      2025-01-01 [4] CRAN (R 4.4.2)
 classInt      0.4-11      2025-01-08 [2] CRAN (R 4.4.2)
 cli           3.6.4       2025-02-13 [2] CRAN (R 4.4.2)
 codetools     0.2-20      2024-03-31 [4] CRAN (R 4.4.1)
 curl          6.2.1       2025-02-19 [2] CRAN (R 4.4.2)
 DBI           1.2.3       2024-06-02 [2] CRAN (R 4.4.1)
 devtools      2.4.5       2022-10-11 [2] CRAN (R 4.4.1)
 digest        0.6.37      2024-08-19 [2] CRAN (R 4.4.1)
 e1071         1.7-16      2024-09-16 [2] CRAN (R 4.4.1)
 ellipsis      0.3.2       2021-04-29 [2] CRAN (R 4.4.1)
 evaluate      1.0.3       2025-01-10 [2] CRAN (R 4.4.2)
 fastmap       1.2.0       2024-05-15 [2] CRAN (R 4.4.1)
 fs            1.6.5.9000  2024-12-18 [2] Github (r-lib/fs@8d40528)
 gdalraster  * 1.12.0.9250 2025-03-10 [1] local
 generics      0.1.3       2022-07-05 [2] CRAN (R 4.4.1)
 glue          1.8.0       2024-09-30 [2] CRAN (R 4.4.1)
 here          1.0.1       2020-12-13 [2] CRAN (R 4.4.1)
 htmltools     0.5.8.1     2024-04-04 [2] CRAN (R 4.4.1)
 htmlwidgets   1.6.4       2023-12-06 [2] CRAN (R 4.4.1)
 httpuv        1.6.15      2024-03-26 [2] CRAN (R 4.4.1)
 httr2         1.1.0       2025-01-18 [2] CRAN (R 4.4.2)
 jsonlite      1.9.0       2025-02-19 [2] CRAN (R 4.4.2)
 KernSmooth    2.23-26     2025-01-01 [4] CRAN (R 4.4.2)
 knitr         1.49        2024-11-08 [2] CRAN (R 4.4.2)
 later         1.3.2       2023-12-06 [2] CRAN (R 4.4.1)
 lattice       0.22-6      2024-03-20 [4] CRAN (R 4.4.1)
 lifecycle     1.0.4       2023-11-07 [2] CRAN (R 4.4.1)
 lubridate     1.9.4       2024-12-08 [2] CRAN (R 4.4.2)
 magrittr      2.0.3       2022-03-30 [2] CRAN (R 4.4.1)
 Matrix        1.7-0       2024-04-26 [2] CRAN (R 4.4.1)
 memoise       2.0.1       2021-11-26 [2] CRAN (R 4.4.1)
 mime          0.12        2021-09-28 [2] CRAN (R 4.4.1)
 miniUI        0.1.1.1     2018-05-18 [2] CRAN (R 4.4.1)
 nanoarrow     0.6.0       2024-10-13 [2] CRAN (R 4.4.2)
 osmdata     * 0.2.5       2023-08-14 [2] CRAN (R 4.4.2)
 pkgbuild      1.4.4       2024-03-17 [2] CRAN (R 4.4.1)
 pkgload       1.4.0       2024-06-28 [2] CRAN (R 4.4.1)
 png           0.1-8       2022-11-29 [2] CRAN (R 4.4.1)
 profvis       0.4.0       2024-09-20 [2] CRAN (R 4.4.1)
 promises      1.3.0       2024-04-05 [2] CRAN (R 4.4.1)
 proxy         0.4-27      2022-06-09 [2] CRAN (R 4.4.1)
 purrr         1.0.4.9000  2025-02-19 [2] Github (tidyverse/purrr@9c8beb4)
 R6            2.6.1       2025-02-15 [2] CRAN (R 4.4.2)
 rappdirs      0.3.3       2021-01-31 [2] CRAN (R 4.4.1)
 Rcpp          1.0.14      2025-01-12 [2] CRAN (R 4.4.2)
 remotes       2.5.0       2024-03-17 [2] CRAN (R 4.4.1)
 reticulate  * 1.41.0      2025-02-24 [2] CRAN (R 4.4.2)
 rlang         1.1.5       2025-01-17 [2] CRAN (R 4.4.2)
 rmarkdown     2.29        2024-11-04 [2] CRAN (R 4.4.2)
 rprojroot     2.0.4       2023-11-05 [2] CRAN (R 4.4.1)
 rstudioapi    0.16.0      2024-03-24 [2] CRAN (R 4.4.1)
 sessioninfo   1.2.2       2021-12-06 [2] CRAN (R 4.4.1)
 sf          * 1.0-19      2024-11-05 [2] CRAN (R 4.4.2)
 shiny         1.9.1       2024-08-01 [2] CRAN (R 4.4.1)
 timechange    0.3.0       2024-01-18 [2] CRAN (R 4.4.1)
 units         0.8-5       2023-11-28 [2] CRAN (R 4.4.1)
 urlchecker    1.0.1       2021-11-30 [2] CRAN (R 4.4.1)
 usethis       3.0.0       2024-07-29 [2] CRAN (R 4.4.1)
 vctrs         0.6.5       2023-12-01 [2] CRAN (R 4.4.1)
 withr         3.0.2       2024-10-28 [2] CRAN (R 4.4.1)
 wk            0.9.4       2024-10-11 [2] CRAN (R 4.4.1)
 xfun          0.50        2025-01-07 [2] CRAN (R 4.4.2)
 xml2          1.3.6       2023-12-04 [2] CRAN (R 4.4.1)
 xtable        1.8-4       2019-04-21 [2] CRAN (R 4.4.1)
 yaml          2.3.10      2024-07-26 [2] CRAN (R 4.4.1)

 [1] /perm_storage/home/mdsumner/R/x86_64-pc-linux-gnu-library/4.4
 [2] /usr/local/lib/R/site-library
 [3] /usr/lib/R/site-library
 [4] /usr/lib/R/library

─ Python configuration ───────────────────────────────────────────────────────
 python:         /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf/bin/python3
 libpython:      /perm_storage/home/mdsumner/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/libpython3.11.so
 pythonhome:     /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf:/perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf
 virtualenv:     /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf/bin/activate_this.py
 version:        3.11.11 (main, Feb 12 2025, 14:51:05) [Clang 19.1.6 ]
 numpy:          /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf/lib/python3.11/site-packages/numpy
 numpy_version:  2.2.3
 osgeo:          /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf/lib/python3.11/site-packages/osgeo
 
 NOTE: Python version was forced by py_require()

──────────────────────────────────────────────────────────────────────────────

reticulate::py_require()

══════════════════════════ Python requirements ══════════════════════════


── Current requirements ─────────────────────────────────────────────────


 Python:   [No Python version specified. Will default to '3.11.11']
 Packages: numpy, gdal


── R package requests ───────────────────────────────────────────────────


R package  Python packages                           Python version      
reticulate numpy                                                         






 


Conservative regridding with GDAL (?)
Michael Sumner — Wed, 11 Dec 2024 00:00:00 GMT
 






Can GDAL do conservative re-gridding? For cases of regular grid to regular grid, yes I think it can.
Please note that I’m using tools I’m comfortable with, because I wrote them. I will reframe in other tools and other languages in time. For some reason I’d been blocked on understanding this issue.

Simple grid with four values
Take a grid, 2x2 with values 1,2,3,4 that sums to 10 and warp it to a new size.

## target
dm <- c(2, 2)

g1 <- matrix(c(1, 2, 3, 4), dm[2], dm[1])
e1 <- c(-1, 1, -1, 1)

library(vapour)
library(terra)

terra 1.8.29

## we need this so we can use MEM: via dsn::mem()
Sys.setenv(GDAL_MEM_ENABLE_OPEN = "YES")


## dsn::mem generates a Memory raster, 
## gdal_raster_data is the warper, here we get identity
sg <- gdal_raster_data(dsn::mem(g1, extent = e1, projection = "EPSG:4326"))

unique(sg[[1]])

[1] 1 2 3 4

sum(sg[[1]])

[1] 10

## spatialize for easy plotting
tor <- function(x) {
 
  dm <- attr(x, "dimension")[2:1]
  r <- terra::rast(terra::ext(attr(x, "extent")), ncols = dm[1], nrows = dm[2], crs = attr(x, "projection"), 
                   vals = x[[1]])
  r
}

r <- "sum"

##   (always use mem() "live" to avoid the garbage collector, and only for Float64 I'm afraid)


## now use the same extent but reduce pixel size
tg <- gdal_raster_data(dsn::mem(g1, extent = e1, projection = "EPSG:4326"), target_ext = e1,
                         target_dim = dm * 8, resample = r)

str(tg)

List of 1
 $ : num [1:256] 0.0156 0.0156 0.0156 0.0156 0.0156 ...
 - attr(*, "dimension")= num [1:2] 16 16
 - attr(*, "extent")= num [1:4] -1 1 -1 1
 - attr(*, "projection")= chr "GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AU"| __truncated__


So we have a lot more values as we resized from 2x2 to 16x16.
It still looks the same, but the overall quantity has been distributed.

plot(tor(tg), main = r)
## draw boundaries on
abline(v = vaster::x_corner(dm * 8, e1), h = vaster::y_corner(dm * 8, e1))







print(sum(tg[[1]]))

[1] 10

## our sum of 10 was distributed across 256 pixels
unique(tg[[1]])

[1] 0.015625 0.031250 0.046875 0.062500


Now a different example, an actual map projection change.
But, gawd … there’s a bug here. WIP

## now try reprojecting our unit 1x1 longlat grid to LAEA centred a few degrees east and north
crs <- "+proj=laea +lon_0=5 +lat_0=0"
## reproject the extent, we use a densified boundary to find the new extent
prex <- reproj::reproj_extent(c(-1, 1, -1, 1), crs, source = "EPSG:4326")
newg <- gdal_raster_data(dsn::mem(g1, extent = e1, projection = "EPSG:4326"), 
                         target_ext = prex, target_res = c(10000, 10000), target_crs = crs, resample = "sum", options = "-tap")
plot(tor(newg))







sum(newg[[1]])

[1] 10


And what about trying to return, of course this can only be approximate (I want to know how this compares to other tools.)

## what if we go the other way, 

newdm <- attr(newg, "dimension")
newext <- attr(newg, "extent")
x <- gdal_raster_data(dsn::mem(matrix(newg[[1]],newdm[2], byrow = TRUE),  extent = newext, projection = crs), 
                 target_dim = dm, target_ext = e1, target_crs = "EPSG:4326", resample = "sum")
plot(tor(x))







sum(x[[1]])

[1] 9.496864






 


Michael Sumner — Thu, 05 Dec 2024 00:00:00 GMT
 undefined 


Plot at native resolution, with R
Michael D. Sumner — Wed, 04 Dec 2024 00:00:00 GMT
 






Plot native, something I should have done long ago.

#' Plot raster at native resolution
#'
#' Determines the current device size and plots the raster centred on its own
#' middle to plot at native resolution. 
#'
#' @param x as SpatRaster
#' @param ... passed to terra::plot
#'
#' @return the input raster, cropped corresponding to the plot made
#' @export
#'
#' @examples
#' plot_native(terra::rast(volcano))
#' plot_native(terra::disagg(terra::rast(volcano), 64))
plot_native <- function(x, ...) {
  ex <- as.vector(terra::ext(x))
  at <- NULL
  ## take the centre
  if (is.null(at)) {
    at <- apply(matrix(ex, 2), 2, mean)
  }
  dv <- dev.size("px")
  scl <- terra::res(x)
  halfx <- dv[1]/2 * scl[1]
  halfy <- dv[2]/2 * scl[2]
  cropex <- c(at[1] - halfx, at[1] + halfx, at[2] - halfy, at[2] + halfy)
  x <- terra::crop(x, terra::ext(cropex), extend = TRUE)
  add <- FALSE
  if (terra::nlyr(x) >= 3) terra::plotRGB(x, add = add) else plot(x, ..., add = add)
  x
}

So as an example read this world imagery.

dsn <- "\"TMS\">http://services.arcgisonline.com/ArcGIS/rest/services/World_Imagery/MapServer/tile/${z}/${y}/${x}-20037508.3420037508.3420037508.34-20037508.341711topEPSG:900913256256310204,404,403"

library(terra)

terra 1.7.83

im <- project(rast(dsn), rast(ext(-1, 1, -1, 1) * 1e6, res = 1000, crs = "+proj=laea +lon_0=147 +lat_0=-42"), by_util = TRUE)
plotRGB(im)








Do we have more, or less resolution than our device can handle? This next plot shows that we have more, and the cropped raster is returned to match the device.

print(dim(im))

[1] 2000 2000    3

print(dev.size("px"))

[1] 1344  960

plot_native(im)








class       : SpatRaster 
dimensions  : 960, 1344, 3  (nrow, ncol, nlyr)
resolution  : 1000, 1000  (x, y)
extent      : -672000, 672000, -480000, 480000  (xmin, xmax, ymin, ymax)
coord. ref. : +proj=laea +lat_0=-42 +lon_0=147 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs 
source(s)   : memory
colors RGB  : 1, 2, 3 
names       : GDAL_WMS>_1, GDAL_WMS>_2, GDAL_WMS>_3 
min values  :           0,           0,           0 
max values  :         255,         237,         213 





 


GDAL and image tiles, the {ceramic} package
Michael D. Sumner — Sat, 22 Apr 2023 00:00:00 GMT
 






A new version of {ceramic} is now on CRAN, version 0.8.0.
The package exists for two purpose

to download tiles from image servers
to load raster data from image servers

NOTE: we need a Mapbox key for the Mapbox servers see, please see ceramic::get_api_key().
The original versions of ceramic didn’t really separate these tasks but the new version does.

Get raster data from online
We can read satellite imagery or elevation with a central point and a buffer (in metres).

library(ceramic)

Loading required package: terra


terra 1.7.23

pt <- cbind(147.3257, -42.8826)

(im <- cc_location(pt, buffer = c(15000, 25000)))

class       : SpatRaster 
dimensions  : 960, 576, 3  (nrow, ncol, nlyr)
resolution  : 52.08333, 52.08333  (x, y)
extent      : 16385222, 16415222, -5319119, -5269119  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
colors RGB  : 1, 2, 3 
names       : red, green, blue 
min values  :   0,     0,    0 
max values  : 255,   255,  252 

(el <- cc_elevation(pt, buffer = c(15000, 25000)))

class       : SpatRaster 
dimensions  : 960, 576, 1  (nrow, ncol, nlyr)
resolution  : 52.08333, 52.08333  (x, y)
extent      : 16385222, 16415222, -5319119, -5269119  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
name        :  lyr.1 
min value   :   -1.9 
max value   : 1264.9 

op <- par(mfcol = c(1, 2))
plot(el, legend = F); plot(im, add = TRUE); plot(el, legend = F, 
                                                 col = hcl.colors(64)[tail(seq_len(64), 45)])







par(op)

These raster objects are in terra ‘SpatRaster’ format (older ceramic used raster package).
These use the Mapbox ‘mapbox.satellite’ and ‘mapbox-terrain-rgb’ tile servers.


Get raster tiles from online
If we want the actual tiles, we can use the original get_tiles() function. (In older versions cc_location() and cc_elevation() would invoke get_tiles, but no longer).

pt <- cbind(147.3257, -42.8826)

imtiles <- get_tiles(pt, buffer = c(15000, 25000))

Preparing to download: 24 tiles at zoom = 12 from 
https://api.mapbox.com/v4/mapbox.satellite/

str(imtiles)

List of 3
 $ files : chr [1:24] "/perm_storage/home/mdsumner/.cache/.ceramic/api.mapbox.com/v4/mapbox.satellite/12/3722/2586.jpg" "/perm_storage/home/mdsumner/.cache/.ceramic/api.mapbox.com/v4/mapbox.satellite/12/3723/2586.jpg" "/perm_storage/home/mdsumner/.cache/.ceramic/api.mapbox.com/v4/mapbox.satellite/12/3724/2586.jpg" "/perm_storage/home/mdsumner/.cache/.ceramic/api.mapbox.com/v4/mapbox.satellite/12/3725/2586.jpg" ...
 $ tiles :List of 2
  ..$ tiles:'data.frame':   24 obs. of  2 variables:
  .. ..$ x: int [1:24] 3722 3723 3724 3725 3722 3723 3724 3725 3722 3723 ...
  .. ..$ y: int [1:24] 2586 2586 2586 2586 2587 2587 2587 2587 2588 2588 ...
  .. ..- attr(*, "out.attrs")=List of 2
  .. .. ..$ dim     : Named int [1:2] 4 6
  .. .. .. ..- attr(*, "names")= chr [1:2] "x" "y"
  .. .. ..$ dimnames:List of 2
  .. .. .. ..$ x: chr [1:4] "x=3722" "x=3723" "x=3724" "x=3725"
  .. .. .. ..$ y: chr [1:6] "y=2586" "y=2587" "y=2588" "y=2589" ...
  ..$ zoom : int 12
  ..- attr(*, "class")= chr "tile_grid"
 $ extent: num [1:4] 16385222 16415222 -5319119 -5269119

eltiles <- get_tiles(pt, buffer = c(15000, 25000))

Preparing to download: 24 tiles at zoom = 12 from 
https://api.mapbox.com/v4/mapbox.satellite/

head(gsub(ceramic::ceramic_cache(), "",  eltiles$files))

[1] "/api.mapbox.com/v4/mapbox.satellite/12/3722/2586.jpg"
[2] "/api.mapbox.com/v4/mapbox.satellite/12/3723/2586.jpg"
[3] "/api.mapbox.com/v4/mapbox.satellite/12/3724/2586.jpg"
[4] "/api.mapbox.com/v4/mapbox.satellite/12/3725/2586.jpg"
[5] "/api.mapbox.com/v4/mapbox.satellite/12/3722/2587.jpg"
[6] "/api.mapbox.com/v4/mapbox.satellite/12/3723/2587.jpg"


And to see what tiles we have we can materialize their footprint in wk::rct() form, this is way more efficient than having to use polygons.

zoomtiles <- ceramic::ceramic_tiles(imtiles$tiles$zoom)
## sub out the ones we just triggered
zoomtiles <- dplyr::filter(zoomtiles, fullname %in% imtiles$files)
rc <- ceramic::tiles_to_polygon(zoomtiles)
plot(rc)
plot(ext(im), add = TRUE)  ## see the image from above is not the tiles, but the buffer around a point







plot(read_tiles(pt, buffer = c(15000, 25000)))  ## but we can read thos tiles exactly

Preparing to download: 24 tiles at zoom = 12 from 
https://api.mapbox.com/v4/mapbox.satellite/

plot(rc, add = TRUE, border = "white")










Diverse query inputs
Finally, we can use a point and buffer to get imagery, or we can use a spatial object, currently supported are objects from {geos}, {wk}, {terra}, {sf}, {sp}, {raster}, and {stars}.
Here’s an example.

wkarea <- wk::rct(xmin = 5e6, ymin = -2e6, xmax = 2e6, ymax = 3e6, crs = "+proj=laea +lon_0=25 +lat_0=31")
im2 <- cc_location(wkarea)
el2 <- cc_elevation(wkarea)
op <- par(mfcol = c(1, 2))
plot(el2, legend = F); plot(im2, add = TRUE); plot(el2, col = grey.colors(128), legend = FALSE)







par(op)

(Please note that the result is still in Mercator, the query can be in any projection but we’re not matching that here, only its extent - ceramic version 0.8.0 is merely a stepping stone to some of the things we can do better with GDAL).
Previous versions of cc_location() and cc_elevation() were stuck with only ‘zoom’ and ‘max_tiles’ arguments, these make sense when you restrict exactly to the available tiles but when you just want a given area and a resolution, ‘dimension’ is more appropriate.
By default, the dimension is chosen relative to the graphics device. But, we can also specify exactly what we want. (Use zero for one of the dimensions to let the system figure out an appropriate size for the query you have).

cc_location(wkarea)

class       : SpatRaster 
dimensions  : 960, 776, 3  (nrow, ncol, nlyr)
resolution  : 7094.906, 7094.906  (x, y)
extent      : 4203326, 9708973, 105375.6, 6916485  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
colors RGB  : 1, 2, 3 
names       : red, green, blue 
min values  :   0,     0,    0 
max values  : 255,   255,  255 

cc_location(wkarea, dimension = c(400, 700))

class       : SpatRaster 
dimensions  : 700, 400, 3  (nrow, ncol, nlyr)
resolution  : 13766.3, 9730.157  (x, y)
extent      : 4203326, 9709847, 105375.6, 6916485  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
colors RGB  : 1, 2, 3 
names       : red, green, blue 
min values  :   0,     2,    0 
max values  : 255,   255,  255 

cc_location(wkarea, dimension = c(1024, 0))

class       : SpatRaster 
dimensions  : 1267, 1024, 3  (nrow, ncol, nlyr)
resolution  : 5377.463, 5377.463  (x, y)
extent      : 4203326, 9709847, 103240.4, 6916485  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
colors RGB  : 1, 2, 3 
names       : red, green, blue 
min values  :   0,     1,    0 
max values  : 255,   255,  255 




Some other improvements

it’s faster, for loading reads data from image servers directly with the GDAL warper API
we have separation of tile downloading and interaction from raster loading via GDAL
we don’t materialize tiles as polygons, we have the tile description, or the compact rct representation
new functions read_tiles(), unpack_rgb(),

See the full list of changes here: https://hypertidy.github.io/ceramic/news/index.html.
ceramic is my final CRAN package that had direct dependencies on rgdal, rgeos, or maptools - in some ways it was the most challenging for me and had loomed as a problem for some time. But, I’ve progressed my own tool kit well in the time and I learnt a lot with this update.
A future version will probably make the separation (are we tiles, or are we loading raster?) more complete, at any rate there are better ways to organize these things now. There are data sources, data readers and writers, data structures, and algorithms. This work is part of a family of tools that aims to make that orthogonality real and accessible.




 


Degenerate Rectilinear (WIP)
Michael D. Sumner — Fri, 09 Dec 2022 00:00:00 GMT
 






There are a lot of problems when it comes to array representation in data formats and software. There is a whole family of complex issues with sometimes subtle or even cryptic causes, and here we focus on one particular detail.

Degenerate Rectilinear Coordinates
I have taken to saying this phrase to describe a particular situation in array handling. I get confused responses, or normally just no response or feedback on the term itself. I think it’s extremely important though. Python communities raced up to where R was a few years back and just sped on by without any nod to what R folks had learned, I think that’s a huge failure and I take it personally.
Here’s what it is. Let’s say we have a pretty low resolution grid of the world, we break up the entire range of longitude and latitude into intervals, 360 for meridians and 180 for parallels.
(This is neat because in whole number terms, this is exactly the right number).
So, define a grid, we make it a farily typical 1-degree-per-pixel grid of the entire globe flattened unceremoniously from an

angular coordinate system “longitude and latitude” to one where we just plot those numbers in x,y on a flat plane.

## define a grid in -180,180 -90,90 (360x180)
dm <- c(360, 180)
xlim <- c(-180, 180)
ylim <- c(-90, 90)

## create a grid
dat <- matrix(c(0, 1, -1, 0), 2L, 2L)[rep(1:2, each = 180), rep(1:2, each = 90)]
## we don't get much from this because 0,1 0,1
## and the world isn't 0,360 0,180 anyway (that would have been handy!)
#image(dat)


xs <- seq(xlim[1], xlim[2], length.out = dm[1] + 1)
ys <- seq(ylim[1], ylim[2], length.out = dm[2] + 1)
image(xs, ys, dat, asp = 1)
maps::map(add = TRUE)








What happened there? Well, dat is a matrix, a 2D array in R. It has 360 rows and 180 columns (but yes, we are treating the world map as if it had 360 columns, 180 rows - but that is not this blogpost). Just don’t question it right now, but be assured we are treating the world as having 360 unique whole number longitudes, and 180 unique whole number latitudes.
We can image() that, by which I mean draw a pixel map on the screen of all the values in the matrix as if they were a field of little rectangles.

image(dat)








But, that’s very boring because we don’t have any idea where anything is in 0,1 0,1 space.
What if we use xlim and ylim, these are plot() arguments in R (and exist in image() too).

image(dat, xlim = xlim, ylim = ylim)








Ok that was a trick, our matrix is drawn as a tiny dot right at 0,0, because xlim/ylim is about the plot not the data.
No more tricks, image() has 3 arguments x,y,z for normal usage.

image(x = xs, y = ys, z = dat, asp = 1)








Why does that work? We essentially have an x coordinate and a y coordinate for every pixel edge. We have one for the left side of the leftmost pixel column, and the second one is the right side of the left most pixel, and so on. We used dm[1] + 1 and dm[2] + 1 up there exactly because we wanted xlim and ylim to be the very extremes of the data.
Cool, explained - a matrix has a coordinate array for each side, we plot it like a map with those. Done.


We are so so so not done.
I’m not here to explain how image() works in R, but that’s kind of necessary because I want people who don’t use R to also read this.


What is Rectilinear?
Rectilinear means varying in one dimension. When applied to axis coordinates it just means you have a position along the axis for each step. These can vary in how big each step is, and that is exactly when we say it’s rectilinear, because otherwise it’s regular.
Our ys above are a regular list of coordinates, each one is exactly the same distance apart (distance in the frame we are using, forget about the actual Earth).

op <- par(mfrow = c(2, 1))
plot(ys)
range(diff(ys))

[1] 1 1

plot(diff(ys))








See? They just plod along with exactly the same delta between each, in this case the step size is 1 but it could be anything.


What do you mean Degenerate?

Isn’t this a bad word, pejorative?

Well no, it means something very specific.
It means even though there is a quite a lot of data, the information is confused or confounded. The required information could be expressed very compactly, but it’s actually obscured by the details.




 


GDAL raster read/write by blocks
Michael D. Sumner — Wed, 04 May 2022 00:00:00 GMT
 







Read and write raster by blocks.
A block is another word for a tile, a tile in a small-ish raster window within a larger raster. Tiles can be very clever, such as 256x256, they make a nice way to organize large data- keeping pieces of data that are nearby spatial nearby to each other in memory. There is a pair of functions internal to vapour that will 1) read data from a raster block 2) write data to a raster block. At the moment only Float64 data is handled.


Tiles
Tiles are a way of organizing rasters, a tile may be 256x256, 512x512 (typically powers of 2 for sensible reasons), or they may be of higher dimensions 256x256x8 - this is really a private detail for a storage format, such as a file. The data values stored in a tile are a nother matter, these are like variables - we might have 1 variable (say elevation) stored in a double floating point value, or we might have 3 variables of byte values for storing an RGB image. These are independent concepts to the size of the tile, and strictly there may be no variables at all, just the abstract idea of the tiling.
Another kind of block, or tile is a the scanline of a raster. Imagine reading each row of the raster, from the top row to the bottom. Each line is its own kind of degenerate tile. We can read from a raster this way no matter what its internal tiling is, just software might be opening several tiles and reading just one line from each. So, best if we match our “tile attack” to the native structure of the data.
Here we find a large, tiled raster, obtain its internal tiling, and use that to update a copy of the same file.
Please note how we copy the source file, then write to that. There’s probably other software better suited to doing this atm, this post simply aims to air the topic a little in an R context.
Here we use only temporary files that we download, and copy as needed. In practice, you should set this up to work across different physical disks, and for better workflow we need the ability to open an empty file to write to (WIP).

raster_url <- "ftp://ftp.data.pgc.umn.edu/elev/dem/setsm/REMA/mosaic/v1.1/200m/REMA_200m_dem.tif"

readfrom_file <- tempfile(pattern = "readfrom", fileext = ".tif")
writeto_file <- tempfile(pattern = "writeto", fileext = ".tif")
## file size is 1.3Gb
curl::curl_download(raster_url, readfrom_file)

## this now makes a copy of the 1.3Gb file, so we have two of them
fs::file_copy(readfrom_file, writeto_file)

Now we want the tiling

info <- vapour::vapour_raster_info(readfrom_file)
tiling <- list(dimension = info$dimXY, tiles = info$tilesXY)
fac <- 100
fac * tiling$tiles
if (tiling$tiles[2] == 1) {
  ## let's take fac scanlines at a time
  tiling$tiles[2] <- fac
}
calc_steps <- function(dimension, tiles) {
  bounds_x <- seq(0, dimension, by = tiles)
  steps_x <-  rep(tiles, length.out = length(bounds_x)-1)
  dangle_x <- sum(steps_x) -dimension
  if (dangle_x > 0) steps_x[length(steps_x)] <- steps_x[length(steps_x)] - dangle_x
  
  list(head(bounds_x, -1), steps_x)
}
x_step <- calc_steps(tiling$dimension[1], tiling$tiles[1])
y_step <- calc_steps(tiling$dimension[2], tiling$tiles[2])
y_step
system.time({
for (i in seq_along(x_step[[1]])) {
  startx <- x_step[[1]][i]
  countx <- x_step[[2]][i]
  for (j in seq_along(y_step[[1]])) {
  starty <- y_step[[1]][j]
  county <- y_step[[2]][j]
    
  ## now read
  offset <- c(startx, starty)
  dimension <- c(countx, county)
  vals <- vapour:::vapour_read_raster_block(readfrom_file, offset = offset, dimension = dimension, band_output_type = info$datatype, band = 1)[[1L]]
  
  ## do something to the values
  if (any(na.omit(vals) > info$nodata_value)) {
    vals <- vals * -1
    vapour:::vapour_write_raster_block(writeto_file, vals, offset, dimension, band = 1, overwrite = TRUE)
  }
  }
}





 


GDAL warper with R
Michael D. Sumner — Mon, 25 Apr 2022 00:00:00 GMT
 







An unfinished post
There’s several meanings floating around when you say the “GDAL warper”. It can mean

command-line gdalwarp
the rasterio package in Python, and its WarpedVRT
the GDAL C++ API warping library

We can also mean

the sf package in R, and its gdal_utils() function
the stars package in R, and its st_warp() function
the terra package in R, and its project() function
the gdalUtils package in R
the gdalUtilities package in R

But, what I mean is the GDAL C++ API warping library.


The GDAL C++ API warping library
GDAL is complex. It deals with very many different formats, and has many tools. In terms of this post, it has very low-level development facilities, written in C++. This means you can usually go as deep as you need into a geospatial problem by writing C++ code against the library directly. Often you don’t need C++ and can use Python, and sometimes you can use R depending on what is exposed there. Sometimes you need to modify GDAL itself, and you can do anything you want then(!), which is how open source is supposed to work.
Key feature of the gdalwarp_lib.cpp GDALWarp() C++ function, versus the GDALWarpDirect(), GDALWarpIndirect() and the lower level GDALWarpMulti() and GDALWarpImage() functions.
When we use gdalwarp_lib, we get handling of multiple-zoom level sources without intervention on our own.
Most use cases I see of the warper facilities are for whole-sale conversion of large datasets, multiple input files converted to a another projection in a large output file or series of tiles.


The warper is also a generalized RasterIO
(rasterio is a famous python package for using GDAL’s raster facilities, it’s not what we mean here).
RasterIO is a C++ function in the GDAL library, its job is to take a source data set (i.e. a path to GeoTIFF) and to provide you with a window of pixels from that source. A window can be a subset, the entire raster, or a resampling (i.e. fewer pixels than native) of either the entire or a subset of the raster.
It’s really, cool and using it looks like this:
err = rasterBand->RasterIO(GF_Read, 
                           Xoffset, Yoffset, 
                           nXSize, nYSize,
                           &double_scanline[0], 
                           outXSize, outYSize, 
                           GDT_Float64,
                           0, 0, &psExtraArg);
There is a lot going on in that function call, but in short the key parts are (these are my argument-variable-names):

rasterBand->RasterIO() is the way we read a pixel values, we have a raster band and we call the member function RasterIO()
Xoffset, Yoffset is the first column and row we should consider for reading (0,0 if we start at the top left corner)
nXSize, nYSize
outXSize, outYSize is the dimension of the window we get out

That last bits, the two kinds of Size is the magic, we can ask to start at a particular row,column and read out a given number of pixels in x and y. But, not only that we can specifiy where to end in the source. If nXSize and outXSize are not equal in value then we have asked for a resampling of the source “only read every nXSize / outXSize values in the x direction. If they are equal, just read every one. Note also that we might ask for an outX/YSize that is larger than the source nXSize/nYSize - and this would give us a resampling to higher resolution. This is really where the resampling algorithm comes in. ‘Nearest neighbour’ would give us copies of pixels, ‘Bilinear’ and interpolation between source pixels for new pixels in between. Other algorithms include ‘Cubic’, ‘Lanczos’, ‘Average’, ‘Sum’.
This is the key behind the fast and lazy reads provided by the terra and sf packages in R, this was originally available also in rgdal and raster made heavy use of it.
But, what is the offset and the size? We really want to think in geographic coordinates, and this is exactly what crop() does for example. We get a discretized crop, not an exact one because we only get to read by this raster-based mechanism - we are bound to the size and alignment of the source pixels.
Under the hood, functions like crop() do the following:
offsets/scale vs. xlim,ylim
The other arguments in RasterIO.

GF_Read controls the mode we are into (read or write or update)
GDT_Float64 controls the type of data we get out (64 bit doubles here, GDAL will auto-convert if the source is different type)
0, 0, &psExtraArg are further details we won’t discuss (though, the way resampling is done is controlled in the extra args options).

What are its limitations:

only one source at a time, you can’t resample from multiple rasters at once (we are using offset/size indexing so the rasters must all be the same shape, you can read multiple bands)


What is interesting about RasterIO vs. Warping is that we would never ever use warping for pure graphics. It doesn’t make sens


Enter the WuTang Warper!
Doing discretized extent raster math is boring. With the warper we don’t need to do it.
This is because the warper is for changing projections, this is an entire re-modelling of raster data. (in the 2000s it was possible to do but still quite slow and problematic, hence you have very slowly changing standards and perceptions of what is possible/normal/appropriate in software comunities etc.).
This is a global longlat data set, but we want a local projection so we adopt one by finding its specification.


If we were using RasterIO we’d have to do this kind of math to run the index function.
But with the warper, we can use an extent (but we still need to align it, or we won’t get what the RasterIO facility would faithfully give).

Path	Contents
`/climate/`	WorldClim 2.1 and 1.4, CMIP5 legacy data
`/cmip6/`	Downscaled CMIP6 projections
`/geodata/`	Soil, landcover, crops, elevation, travel time, OSM, buildings
`/gadm/`	GADM administrative boundaries (versions 2.8, 3.6, 4.0, 4.1)

Resolution	Variables	Example size
10 minute	bio, elev, prec, srad, tavg, tmax, tmin, vapr, wind	1.3 MB – 48 MB
5 minute	same	4.6 MB – 171 MB
2.5 minute	same	17 MB – 628 MB
30 second	same	323 MB – 9.7 GB

Subdirectory	Format	GDAL-readable	Notes
`gpkg/`	GeoPackage	Yes	SQLite-based, supports HTTP range requests
`shp/`	Shapefile (zipped)	Yes, via /vsizip/	Multiple component files
`json/`	GeoJSON	Yes
`pck/`	RDS (R serialization)	No	Used by `geodata::gadm()`
`kmz/`	KMZ (zipped KML)	Yes, via /vsizip/

Path	Contents	geodata function
`/geodata/travel/`	Travel time rasters	`travel_time()`
`/geodata/osm/`	OpenStreetMap extracts (highways, places)	`osm()`
`/geodata/buildings/`	Building footprint data	—

Collection	Block layout	Streamable?
SoilGrids TIFs	43200×1 (row-striped)	No — download
WorldClim 2.1 tiles	TODO: check	Probably row-striped
WorldClim 2.1 global zips	N/A (zip)	Via /vsizip/, but still row-striped inside
CMIP6 30s TIFs	TODO: check	Unknown
Landcover 3deg COGs	TODO: check	Likely yes (name says COGs)
Landcover 30s	TODO: check	Unknown
GADM GeoPackage	N/A (vector)	Yes — SQLite random access works over HTTP
GADM raw GeoPackage	N/A (vector)	Yes

hypertidy

Icechunk is coming - versioned cloud-native Zarr, GDAL, and R

What is Icechunk?

What works in GDAL and R today

R, GDAL, Zarr, and the Missing Recipe Format for Raster Datasets

Fifteen minutes with xarray

What the VRT already does

What Zarr figured out

GDAL is the bridge

The gap: creation, not reading

Why this matters beyond R

What would help

Appendix: a progressive tour on 23 OISST files

Setup: download 23 OISST files from NCEI

Setup part 2: create a GDAL multidimensional VRT

Step 1 — terra: rast(), band-stacked, all variables, time preserved

Step 2 — terra: vrt() with vrt://, single variable, time lost

Step 3 — stars: read_stars() proxy, GDAL classic raster, variables separated

Step 4 — stars: read_mdim() on a single file, right model, wrong scope

Step 5 — gdal mdim mosaic + read_mdim(), the full tour

Step 6 — skip the download: URL access, who can and can’t

Not Found

Step 7 — rhdf5-dev: opening the door to chunk indexing (dev, not yet on CRAN)

Step 8 — tidync dev: multi-file remote concat in one call (unmerged dev branch)

The xarray counterpart — for comparison

Check it out: geos

Geometry vectors

Operations

Spatial indexing

Construction from coordinates

wk compatibility

geos vs sf for geometric operations

Measurement and properties

Check it out series

Check it out: reproj

The generic

The generic method model

CRS preservation

Under the hood: PROJ package

reproj vs sf::st_transform vs terra::project

Example: transforming track data

Check it out series

Check it out: wk

What wk provides

CRS handling

Interchange

The handler API

Geometric operations via wk

Check it out series

Check it out: proj_trans_create and wk_transform

What PROJ accepts as a CRS definition

CRS representations you can work with in R

How wk bridges CRS objects to PROJ strings

wk_crs_projjson()

The transformation pipeline

Applying the transformation with wk_transform

Inverse transformations

The convenience wrapper: PROJ::proj_trans()

The +type=crs distinction

How this connects to reproj

Further reading

Check it out series

Check it out: gdalraster

Raster I/O

Vector I/O

Virtual filesystem

The warper

VRT construction

Configuration and utilities

Check it out series

What’s on geodata.ucdavis.edu and how to access it

Helper functions

Site structure

Enumerating files

What’s there: a complete inventory

SoilGrids (/geodata/soil/soilgrids/)

WorldClim 2.1 tiles (/climate/worldclim/2_1/tiles/tile/)

WorldClim 2.1 global (/climate/worldclim/2_1/base/)

WorldClim 2.1 country extracts (/climate/worldclim/2_1/tiles/iso/)

WorldClim 1.4 (/climate/worldclim/1_4/)

Step 1 — terra: `rast()`, band-stacked, all variables, time preserved

Step 2 — terra: `vrt()` with `vrt://`, single variable, time lost

Step 3 — stars: `read_stars()` proxy, GDAL classic raster, variables separated

Step 4 — stars: `read_mdim()` on a single file, right model, wrong scope

Step 5 — `gdal mdim mosaic` + `read_mdim()`, the full tour

SoilGrids (`/geodata/soil/soilgrids/`)

WorldClim 2.1 tiles (`/climate/worldclim/2_1/tiles/tile/`)

WorldClim 2.1 global (`/climate/worldclim/2_1/base/`)

WorldClim 2.1 country extracts (`/climate/worldclim/2_1/tiles/iso/`)

WorldClim 1.4 (`/climate/worldclim/1_4/`)

CMIP6 projections (`/cmip6/`)

CMIP5 projections (`/climate/cmip5/`)

Landcover (`/geodata/landcover/`)

Crops (`/geodata/crops/`)

GADM administrative boundaries (`/gadm/gadm4.1/`)

The `image()` model: rectilinear coordinates

The `rasterImage()` model: bounding box placement