Geometry vectors

geos stores geometries in an external pointer vector class:

library(geos)

g <- as_geos_geometry(c(
  "POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))",
  "POINT (5 5)",
  "LINESTRING (0 0, 10 10)"
))
g
#> <geos_geometry[3]>
#> [1] <POLYGON [0 0...10 10]>
#> [2] <POINT [5 5]>
#> [3] <LINESTRING [0 0...10 10]>

The input can be WKT strings, WKB, wk vectors, or sf objects — anything with a wk_handle() method gets converted automatically:

## from wk::xy()
as_geos_geometry(wk::xy(1:5, 1:5))

## from sf
library(sf)
nc <- st_read(system.file("shape/nc.shp", package = "sf"), quiet = TRUE)
nc_geos <- as_geos_geometry(nc)

Operations

geos provides vectorized access to the GEOS C API. Operations are functions, not methods — they take geometry vectors and return geometry vectors or values:

## buffer
geos_buffer(g[1], distance = 2)

## intersection
geos_intersection(g[1], geos_buffer(g[2], 3))

## union
geos_unary_union(g)

## simplify
geos_simplify(g[3], tolerance = 1)

## centroid
geos_centroid(g[1])

Binary predicates are also vectorized:

geos_intersects(g[1], g[2])
#> [1] TRUE

geos_contains(g[1], g[2])
#> [1] TRUE

geos_disjoint(g[1], g[3])
#> [1] FALSE

Spatial indexing

geos includes an STRtree implementation for fast spatial queries:

## build a spatial index
pts <- as_geos_geometry(wk::xy(runif(10000), runif(10000)))
tree <- geos_strtree(pts)

## query the index
query_geom <- as_geos_geometry("POLYGON ((0.4 0.4, 0.6 0.4, 0.6 0.6, 0.4 0.6, 0.4 0.4))")
geos_strtree_query(tree, query_geom)

The newer geos_basic_strtree() uses wk::wk_envelope() for index construction, which is extremely efficient for wk::xy() input — the envelope is just the point itself, computed without any geometry parsing overhead.

Construction from coordinates

You don’t need to go through WKT to build geometries. geos provides direct constructors:

## points from x, y
geos_make_point(1:3, 1:3)

## linestring from coordinates
geos_make_linestring(c(0, 1, 2, 3), c(0, 1, 0, 1))

## polygon from coordinates (ring is closed automatically)
geos_make_polygon(c(0, 10, 10, 0), c(0, 0, 10, 10))

## collections
geos_make_collection(
  c("POINT (0 0)", "POINT (1 1)"),
  type_id = "multipoint"
)

These constructors use feature_id and ring_id arguments for building multi-part and multi-ring geometries from flat coordinate vectors, similar to how data often arrives from databases or CSV files.

wk compatibility

geos implements wk_handle() for its geometry type, so all wk operations work:

## extract coordinates
wk::wk_coords(nc_geos)

## compute bounding boxes
wk::wk_envelope(nc_geos)

## convert to wk types
wk::as_wkt(nc_geos)
wk::as_wkb(nc_geos)

## convert to sf
sf::st_as_sfc(nc_geos)

This means geos integrates into any workflow that uses wk as the interchange format. Read vector data with gdalraster (WKB output), convert to geos for geometric operations, transform with PROJ, write back with gdalraster — all through wk.

geos vs sf for geometric operations

sf uses GEOS internally for its geometry operations (st_buffer(), st_intersection(), etc.). The difference is:

• sf wraps GEOS through its own C++ layer, with results returned as sfc geometry columns inside sf data frames. The operations are tightly integrated with the sf data model.
• geos wraps GEOS directly, with results as standalone geometry vectors. No data frame, no attributes — just geometry in, geometry out.

If you’re working within sf, use sf’s functions. If you’re working outside sf — with wk vectors, plain data frames, or formats that don’t map to sf’s model — geos gives you the same geometric operations without the sf world view.

Workflows can deliberately separate I/O (gdalraster/vapour), geometry (wk/geos), transformation (PROJ/reproj), and even grid logic (vaster), geos is the geometric computation layer.

Measurement and properties

geos_area("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))")
#> [1] 100

geos_length("LINESTRING (0 0, 10 0, 10 10)")
#> [1] 20

geos_is_valid("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))")
#> [1] TRUE

geos_is_empty("POINT EMPTY")
#> [1] TRUE

geos_num_coordinates("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 0))")
#> [1] 5

Install from CRAN:

install.packages("geos")

Links:

• geos on CRAN
• geos documentation
• GitHub

Check it out series

This is part of a series of posts for foundational spatial support in R. These posts include:

• wk
• reproj
• PROJ/wk transform
• gdalraster
• geos (this one!)

What wk provides

wk defines several vector types that carry geometry and coordinate reference system information as first-class attributes:

• wk::wkt() — well-known text, human-readable
• wk::wkb() — well-known binary, compact and fast
• wk::xy(), wk::xyz(), wk::xym(), wk::xyzm() — coordinate vectors (points)
• wk::rct() — rectangles (bounding boxes)
• wk::crc() — circles

All of these carry a crs attribute and behave properly as vctrs types. They can live in data frame columns, get subset, combined, and compared — with CRS compatibility checked automatically.

library(wk)
pts <- wk::xy(c(147, 148, 149), c(-42, -43, -44), crs = "EPSG:4326")
pts
#> <wk_xy[3] with CRS=EPSG:4326>
#> [1] (147 -42) (148 -43) (149 -44)

You can put these directly in a data frame:

d <- data.frame(name = c("Hobart", "South", "Further"), geometry = pts)
d
#>     name     geometry
#> 1 Hobart  (147 -42)
#> 2  South  (148 -43)
#> 3 Further (149 -44)

Subset, combine, and merge all work correctly. The CRS propagates through operations and mismatches are caught:

## combining vectors with different CRS will error
pts_other <- wk::xy(1, 2, crs = "EPSG:3857")
try(c(pts, pts_other))
#> Error: CRS objects are not equal

CRS handling

The CRS system in wk is deliberately general. At the C level, the PROJ library proj_create() function accepts a remarkably wide range of CRS definition strings. The PROJ C API documentation lists many valid types of inputs from ‘auth:code’, PROJJSON, WKT variants and PROJ4 strings. The CRS support in wk is similarly open to inputs, the key functions in wk are:

• wk_crs() — get the CRS from any wk-compatible object
• wk_set_crs() — set or replace the CRS
• wk_crs_equal() — compare CRS values
• wk_crs_proj_definition() — extract a PROJ-compatible string (used by PROJ and other packages)

The sentinel value wk_crs_inherit() signals “inherit from context” and is used for empty or zero-length geometry vectors. This means you can safely combine an empty geometry with a CRS-bearing one without triggering a mismatch.

Interchange

wk acts as a common interchange format. Other formats that are supported are ‘raw binary’, ‘raw text’, geos package vectors, sf data frames and geometry vectors. The gdalraster package can emit vector geometry in wk format. Conversions to and from other formats are built in:

## sf -> wk
library(sf)
nc <- st_read(system.file("shape/nc.shp", package = "sf"), quiet = TRUE)
wkb <- wk::as_wkb(nc)
wk_crs(wkb)
#> [1] "NAD27"

## wk -> sf
sf::st_as_sfc(wkb)

## terra -> wk
geos::as_geos_geometry(terra::vect(<src>))

The geos package accepts wk types directly. The PROJ package transforms them. The gdalraster package (via ogr_layer_create() and friends) can write them. This is an interchange layer that allows packages to stay independent of formats, just as the underlying libraries GDAL, PROJ, GEOS already allow.

The handler API

The real power of wk is its streaming handler API. Rather than materializing geometry in memory, wk can stream coordinates through a chain of handlers — read, transform, and write without ever building an intermediate object.

## count coordinates without materializing geometry
wk_count(wkb)

## extract coordinates as a data frame
wk_coords(wkb)

## compute bounding boxes per feature
wk_envelope(wkb)

## transform coordinates (see the PROJ post) https://www.hypertidy.org/posts/2026-03-24_check-out-proj-wk-transform/
wk_transform(pts, PROJ::proj_trans_create("EPSG:4326", "EPSG:3857"))

The wk_handle() generic is what makes this extensible. Any package can implement a wk_handle() method for its geometry type and immediately plug into the wk ecosystem — filters, transforms, writers all just work.

Geometric operations via wk

wk itself includes several useful operations that don’t require GEOS:

• wk_vertices() — extract all vertices from a geometry
• wk_coords() / wk_coords<-() — get/set coordinates
• wk_flatten() — flatten collections to simple geometries
• wk_collection() — collect simple geometries into multi/collection types
• wk_linestring() / wk_polygon() — construct from coordinates
• wk_set_z() / wk_drop_z() — add or remove Z dimension
• wk_orient() / wk_clockwise() / wk_counterclockwise() — rewind polygon rings

For geometric predicates, buffering, intersection and the like try geos (see the geos post in this series).

Install from CRAN:

install.packages("wk")

Links:

• wk on CRAN
• wk documentation
• GitHub

Check it out series

This is part of a series of posts for foundational spatial support in R. These posts include:

• wk (this one!)
• reproj
• PROJ/wk transform
• gdalraster
• geos

The generic

library(reproj)

## matrix in, matrix out
reproj(cbind(147, -42), target = "+proj=laea +datum=WGS84", source = "OGC:CRS84")
#>          [,1]     [,2] [,3]
#> [1,] 5969744 -9803200    0

The source and target arguments accept anything that the PROJ library understands: EPSG codes, proj-strings, WKT2, authority:code strings. The output is always a matrix — 2, 3, or 4 columns depending on input and the four argument.

Convenience helpers reproj_xy() and reproj_xyz() force 2- or 3-column output:

reproj_xy(cbind(147, -42), target = "+proj=laea +datum=WGS84", source = "OGC:CRS84")
#>          [,1]     [,2]
#> [1,] 5969744 -9803200

The generic method model

reproj() is an S3 generic. This means any package can import it and add a method for its own spatial type. The method extracts the source CRS from the object and the user only needs to supply the target:

## hypothetical method for a spatial type that carries its own CRS
reproj.my_spatial_thing <- function(x, target, ...) {
  coords <- extract_coords(x)
  source <- get_crs(x)
  result <- reproj(coords, target = target, source = source, ...)
  rebuild_my_thing(x, result, crs = target)
}

This pattern is already used by the silicate and quadmesh packages — their reproj() methods know where the coordinates live and what the source CRS is, so the user just says reproj(my_mesh, target_crs).

The value of this design is that the generic lives in a small, dependency-light package. A downstream package can @importFrom reproj reproj without pulling in GDAL or any heavy spatial stack. The transformation itself is done by the PROJ library via the PROJ R package (preferred) or the proj4 package as fallback.

For example we could write a reproj.sf and a .sfc:

reproj.sfc <- function(x, target, ..., source = NULL) {
  ## handle this format with wk
  handled <- wk::as_wkb(x)
  crs <- wk::wk_crs(handled)$wkt
  ## stop if not a valid crs, we need source specified
  if (is.null(source)) {
    stopifnot(!is.na(crs))
  } else {
    crs <- source
 }
 trans <- PROJ::proj_trans_create(crs, target)
 sf::st_as_sfc(wk::wk_transform(handled, trans))
}

reproj.sf <- function(x, target, ..., source = NULL) {
 sf::st_set_geometry(x, reproj.sfc(sf::st_geometry(x), target, source = source))
}

That’s not very useful since sf::st_transform already exists but demonstrates the utility for any other object, and you can force input of a source if your object does not support that already. We could decompose our object to coordinates and transform those more directly, but anything handled by wk is already supported by this simpler mechanism.

CRS preservation

A key principle: when reproj transforms an object that has a method, the output carries the new CRS. The silicate method rebuilds the mesh with the target CRS attached. The quadmesh method does the same. The idea is that format-specific methods preserve the full object structure while updating both coordinates and CRS metadata.

For plain matrices and data frames, there’s no CRS to carry, and the target and source must be specified.

Under the hood: PROJ package

reproj delegates the actual math to the PROJ R package, which wraps the PROJ C library. The PROJ package provides:

• PROJ::proj_trans() — transform coordinates, accepting wk types
• PROJ::proj_trans_create() — create a reusable transformation object (a wk_trans)
• PROJ::proj_crs_text() — convert between CRS representations (WKT2, proj-string, PROJJSON)

reproj vs sf::st_transform vs terra::project

sf and terra both have projection functions, but they’re tied to their respective data models. sf::st_transform() works on sf objects. terra::project() works on SpatVector and SpatRaster.

reproj works on anything. It’s deliberately format-agnostic: give it a matrix of coordinates, a source CRS, and a target CRS. This makes it useful for packages that don’t depend on sf or terra, and for workflows where spatial data isn’t in a standard spatial format — track data as data frames, coordinates pulled from a netCDF, vertices from an rgl mesh, cell centres computed from a grid specification.

Example: transforming track data

library(reproj)

## some GPS fixes as a plain data frame
tracks <- data.frame(
  lon = c(147.3, 147.4, 147.5),
  lat = c(-42.9, -42.8, -42.7),
  time = as.POSIXct(c("2024-01-01 08:00", "2024-01-01 09:00", "2024-01-01 10:00"))
)

## transform to a local projection
xy <- reproj_xy(
  cbind(tracks$lon, tracks$lat),
  target = "+proj=utm +zone=55 +south +datum=WGS84",
  source = "EPSG:4326"
)

tracks$x <- xy[, 1]
tracks$y <- xy[, 2]

Install from CRAN:

install.packages("reproj")

Links:

• reproj on CRAN
• reproj documentation
• PROJ on CRAN
• GitHub (reproj)
• GitHub (PROJ)

Check it out series

This is part of a series of posts for foundational spatial support in R. These posts include:

• wk
• reproj (this one!)
• PROJ/wk transform
• gdalraster
• geos

Raster I/O

The core raster interface is GDALRaster, an Rcpp module class that wraps a GDAL dataset handle:

library(gdalraster)

f <- system.file("extdata/storm_lake.lcp", package = "gdalraster")
ds <- new(GDALRaster, f, TRUE)

ds$info()
ds$dim()       # c(ncol, nrow, nbands)
ds$res()       # c(xres, yres)
ds$bbox()      # c(xmin, ymin, xmax, ymax)
ds$getProjection()

## read a band
data <- ds$read(band = 1,
                xoff = 0, yoff = 0,
                xsize = ds$getRasterXSize(),
                ysize = ds$getRasterYSize(),
                out_xsize = ds$getRasterXSize(),
                out_ysize = ds$getRasterYSize())

ds$close()

The read() method maps directly to GDAL’s RasterIO — you specify the source window and the output size. This gives full control over decimation, oversampling, and partial reads. No intermediate abstraction, no automatic resampling decisions you didn’t ask for.

Vector I/O

gdalraster provides vector read/write through OGR:

## list layers in a data source
dsn <- system.file("extdata/ynp_fires_1984_2022.gpkg", package = "gdalraster")
ogr_ds_layer_names(dsn)

## feature count and extent
ogr_ds_layer_count(dsn)
ogr_layer_field_names(dsn, "mtbs_perims")

## read features as a data frame with WKB geometry
lyr <- new(GDALVector, dsn, "mtbs_perims", read_only = TRUE)
feat <- lyr$fetch(-1)  # all features
lyr$close()

The geometry comes back as WKB — directly compatible with wk. You can pass it to wk::as_wkt(), geos::as_geos_geometry(), or sf::st_as_sfc() as needed. gdalraster reads the data; what you do with the geometry is up to you and whatever packages you choose.

Virtual filesystem

GDAL’s /vsicurl/, /vsizip/, /vsimem/ and other virtual filesystem handlers are all available:

## read a remote file without downloading
vsi_read_dir("/vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/")

## open a GeoPackage over HTTP
dsn <- "/vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AUS.gpkg"
ogr_ds_layer_names(dsn)

## read inside a remote zip
vsi_read_dir("/vsizip//vsicurl/https://github.com/OSGeo/gdal/raw/refs/heads/master/autotest/gcore/data/byte.tif.zip")
#[1] "byte.tif"

This is essential for cloud-native workflows. The same dataset string that works locally works over HTTP, S3, or Azure — GDAL handles the transport.

The warper

warp() provides access to GDAL’s warper API for raster reprojection and resampling:

## warp to a new projection
warp(
  src_filename = "input.tif",
  dst_filename = "output.tif",
  t_srs = "EPSG:3577",
  tr = c(100, 100),           # target resolution
  r = "bilinear"              # resampling method
)

This is the same warper that gdalwarp uses on the command line. For building VRTs, mosaics, and complex raster pipelines, gdalraster provides buildVRT(), translate(), and rasterFromRaster().

VRT construction

One of the most powerful features is programmatic VRT (Virtual Raster) construction:

## create a VRT from multiple source files
buildVRT(
  vrt_filename = "mosaic.vrt",
  input_rasters = c("tile1.tif", "tile2.tif", "tile3.tif")
)

## add sources to a VRT programmatically
ds <- new(GDALRaster, "output.vrt", FALSE)
ds$addSimpleSource(band = 1, src_filename = "source.tif", ...)
ds$close()

VRTs are GDAL’s answer to lazy raster operations — the XML file describes the operation but nothing is computed until you read pixels.

Configuration and utilities

gdalraster exposes GDAL configuration options:

## set GDAL config for performance tuning
set_config_option("GDAL_CACHEMAX", "512")
set_config_option("GDAL_HTTP_MULTIRANGE", "YES")
set_config_option("GDAL_NUM_THREADS", "ALL_CPUS")

## check what's set
get_config_option("GDAL_CACHEMAX")

## GDAL/PROJ/GEOS version info
gdal_version()

Plus a full set of geometry utilities (g_buffer(), g_intersection(), etc.) that use GEOS through GDAL’s OGR layer, and coordinate transformation via transform_xy().

Install from CRAN:

install.packages("gdalraster")

Links:

• gdalraster on CRAN
• gdalraster documentation
• GitHub

Check it out series

This is part of a series of posts for foundational spatial support in R. These posts include:

• wk
• reproj
• PROJ/wk transform
• gdalraster (this one!)
• geos

What PROJ accepts as a CRS definition

At the C level, PROJ’s proj_create() function accepts a remarkably wide range of CRS definition strings. The PROJ C API documentation lists the following as valid inputs:

• A proj-string, e.g. "+proj=utm +zone=55 +south +datum=WGS84 +type=crs"
• A WKT string (WKT2:2019 preferred, WKT1 accepted), e.g. GEOGCRS["WGS 84", ...]
• An object code, e.g. "EPSG:4326", "urn:ogc:def:crs:EPSG::4326", or "urn:ogc:def:coordinateOperation:EPSG::1671"
• An object name, e.g. "WGS 84" or "WGS 84 / UTM zone 31N" — matched by heuristics when the name is not unique
• An OGC URN for compound CRS, e.g. "urn:ogc:def:crs,crs:EPSG::2393,crs:EPSG::5717" or the abbreviated "EPSG:2393+5717"
• An OGC URN for concatenated operations, e.g. "urn:ogc:def:coordinateOperation,coordinateOperation:EPSG::3895,coordinateOperation:EPSG::1618"
• A PROJJSON string (added in PROJ 6.2) — the JSON representation of a CRS or coordinate operation, schema at https://proj.org/schemas/v0.4/projjson.schema.json
• A compound CRS from two names joined with " + ", e.g. "WGS 84 + EGM96 height" (added in PROJ 7.1)

That’s a lot of ways to say the same thing. In R, the PROJ package wraps this directly — proj_trans_create() and proj_crs_text() accept all of these forms as character strings.

There is a critical distinction for proj-strings: if the string contains +type=crs, it’s interpreted as a CRS definition. Geographic CRS with +type=crs are assumed to have longitude-latitude axis order with degree units. Without +type=crs, the proj-string is interpreted as a coordinate operation (a transformation step, not a CRS). This matters in some contexts but many packages (PROJ, sf, terra) will detect this case and add the ‘+type=crs’ because it’s common and convenient.

library(PROJ)

## this is a CRS ("+type=crs" present):
proj_crs_text("+proj=laea +type=crs", format = 0L)
## returns a proper WKT2 CRS

## this is technically a transformation step, NOT a CRS (no "+type=crs") but workarounds are often used (as here)
proj_crs_text("+proj=laea", format = 0L)

The PROJ library documentation explicitly says: “The use of proj-string to describe a CRS is discouraged. It is a legacy means of conveying CRS descriptions: use of object codes (EPSG:XXXX typically) or WKT description is recommended for better expressivity.” This is absolutely true but if you want to define your own CRS, just a simple equal area project such as “+proj=laea +lon_0=-147 +lat_0=-42 +datum=WGS4” then you can easily convert from a proj string to WKT, but write the WKT from scratch?? That’s hard but don’t let it put you off, use map projections, specified in a few characters as PROJ strings - if you need a specific datum you’ll need to get a full-blown WKT string by some means.

CRS representations you can work with in R

proj_crs_text() converts between CRS representations. The format argument selects the output:

• format = 0L → WKT2 (the default, most expressive)
• format = 1L → proj-string
• format = 2L → PROJJSON

library(PROJ)

## WKT2
cat(proj_crs_text("EPSG:4326", format = 0L))

GEOGCRS["WGS 84",
    ENSEMBLE["World Geodetic System 1984 ensemble",
        MEMBER["World Geodetic System 1984 (Transit)"],
        MEMBER["World Geodetic System 1984 (G730)"],
        MEMBER["World Geodetic System 1984 (G873)"],
        MEMBER["World Geodetic System 1984 (G1150)"],
        MEMBER["World Geodetic System 1984 (G1674)"],
        MEMBER["World Geodetic System 1984 (G1762)"],
        MEMBER["World Geodetic System 1984 (G2139)"],
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]],
        ENSEMBLEACCURACY[2.0]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    USAGE[
        SCOPE["Horizontal component of 3D system."],
        AREA["World."],
        BBOX[-90,-180,90,180]],
    ID["EPSG",4326]]

## proj-string
proj_crs_text("EPSG:4326", format = 1L)
#> "+proj=longlat +datum=WGS84 +no_defs +type=crs"

## PROJJSON
cat(proj_crs_text("EPSG:4326", format = 2L))

{
  "$schema": "https://proj.org/schemas/v0.7/projjson.schema.json",
  "type": "GeographicCRS",
  "name": "WGS 84",
  "datum_ensemble": {
    "name": "World Geodetic System 1984 ensemble",
    "members": [
      {
        "name": "World Geodetic System 1984 (Transit)",
        "id": {
          "authority": "EPSG",
          "code": 1166
        }
      },
      {
        "name": "World Geodetic System 1984 (G730)",
        "id": {
          "authority": "EPSG",
          "code": 1152
        }
      },
      {
        "name": "World Geodetic System 1984 (G873)",
        "id": {
          "authority": "EPSG",
          "code": 1153
        }
      },
      {
        "name": "World Geodetic System 1984 (G1150)",
        "id": {
          "authority": "EPSG",
          "code": 1154
        }
      },
      {
        "name": "World Geodetic System 1984 (G1674)",
        "id": {
          "authority": "EPSG",
          "code": 1155
        }
      },
      {
        "name": "World Geodetic System 1984 (G1762)",
        "id": {
          "authority": "EPSG",
          "code": 1156
        }
      },
      {
        "name": "World Geodetic System 1984 (G2139)",
        "id": {
          "authority": "EPSG",
          "code": 1309
        }
      },
      {
        "name": "World Geodetic System 1984 (G2296)",
        "id": {
          "authority": "EPSG",
          "code": 1383
        }
      }
    ],
    "ellipsoid": {
      "name": "WGS 84",
      "semi_major_axis": 6378137,
      "inverse_flattening": 298.257223563
    },
    "accuracy": "2.0",
    "id": {
      "authority": "EPSG",
      "code": 6326
    }
  },
  "coordinate_system": {
    "subtype": "ellipsoidal",
    "axis": [
      {
        "name": "Geodetic latitude",
        "abbreviation": "Lat",
        "direction": "north",
        "unit": "degree"
      },
      {
        "name": "Geodetic longitude",
        "abbreviation": "Lon",
        "direction": "east",
        "unit": "degree"
      }
    ]
  },
  "scope": "Horizontal component of 3D system.",
  "area": "World.",
  "bbox": {
    "south_latitude": -90,
    "west_longitude": -180,
    "north_latitude": 90,
    "east_longitude": 180
  },
  "id": {
    "authority": "EPSG",
    "code": 4326
  }
}

PROJJSON is the most machine-friendly format — it’s the full CRS definition in a structure you can parse, manipulate, and round-trip without information loss. WKT2 is the canonical text representation and equally complete. Proj-strings are the most compact, but lose information (datum ensemble members, axis order details, usage scope). Every conversion to proj-string is potentially lossy.

How wk bridges CRS objects to PROJ strings

In the wk package, a CRS can be any R object. The crs attribute on a wk::wkt(), wk::wkb(), or wk::xy() vector is deliberately untyped — it could be a string like "EPSG:4326", an integer like 4326, an sf crs object, or something else entirely. wk doesn’t interpret the CRS, it just propagates it.

The bridge to PROJ is wk_crs_proj_definition(). This is an S3 generic that takes a CRS object and returns a character string that the PROJ library can understand. This is the function that PROJ::proj_trans() and PROJ::proj_trans_create() call internally to resolve CRS arguments.

The built-in methods handle the common cases:

• character → returned as-is (assumed to be an authority:code string, WKT, PROJJSON, or proj-string already)
• integer or double → formatted as "EPSG:<code>" (e.g. 4326 → "EPSG:4326")
• NULL → NA_character_ (no CRS)
• wk_crs_inherit() → NA_character_ (inherit from context)

library(wk)

wk_crs_proj_definition("EPSG:4326")
#> [1] "EPSG:4326"

wk_crs_proj_definition(4326)
#> [1] "EPSG:4326"

wk_crs_proj_definition(4326L)
#> [1] "EPSG:4326"

For sf’s crs class, sf registers its own method so that wk_crs_proj_definition() returns the correct string for the installed version of PROJ. Other packages can do the same — this is the extension point for custom CRS types.

The verbose argument controls how much detail is returned. With verbose = FALSE (the default), it returns the most compact form — typically the authority:code string. With verbose = TRUE, it returns PROJJSON or WKT2 for maximum expressivity.

## get PROJJSON for a known EPSG code, no PROJ library needed
pj <- wk_crs_projjson("EPSG:4326")
cat(pj)
## full PROJJSON output

The transformation pipeline

With all that context, here’s how proj_trans_create() works:

1. It calls wk_crs_proj_definition() on the source_crs and target_crs arguments to get PROJ-compatible strings
2. It passes those strings to the PROJ C library’s proj_create_crs_to_crs() function, which finds the best available transformation pipeline between the two CRS
3. It wraps the result in a wk_trans object — an external pointer with an wk_trans_inverse() method

library(PROJ)
library(wk)

## create a transformation object
trans <- proj_trans_create("EPSG:4326", "EPSG:3857")
<proj_trans at 0x55800593fd60>
type: Concatenated Operation
id: pipeline
description: axis order change (2D) + Popular Visualisation Pseudo-Mercator
definition: proj=pipeline step proj=unitconvert xy_in=deg xy_out=rad step proj=webmerc lat_0=0 lon_0=0 x_0=0 y_0=0 ellps=WGS84
area_of_use:
  name: World
  bounds: [-180 -90 180 90]
source_crs:
  type: Geographic 2D CRS
  id: EPSG:4326
  name: WGS 84
  area_of_use:
    name: World.
    bounds: [-180 -90 180 90]
target_crs:
  type: Projected CRS
  id: EPSG:3857
  name: WGS 84 / Pseudo-Mercator
  area_of_use:
    name: World between 85.06°S and 85.06°N.
    bounds: [-180 -85.06 180 85.06]

The source and target can be any form PROJ accepts — EPSG codes, WKT2, PROJJSON, proj-strings with +type=crs, object names:

## all of these create equivalent transformations:
proj_trans_create("EPSG:4326", "EPSG:3857")
proj_trans_create(4326, 3857)
proj_trans_create("WGS 84", "WGS 84 / Pseudo-Mercator")
proj_trans_create(
  "+proj=longlat +datum=WGS84 +type=crs",
  "+proj=merc +a=6378137 +b=6378137 +lat_ts=0 +lon_0=0 +x_0=0 +y_0=0 +k=1 +units=m +type=crs"
)

Applying the transformation with wk_transform

wk::wk_transform() takes any wk-handleable geometry and applies the transformation through wk’s streaming handler API. This means the transformation works on any geometry type — points, lines, polygons, collections — without materializing intermediate representations:

## transform point coordinates
pts <- wk::xy(c(147, 148, 149), c(-42, -43, -44), crs = "EPSG:4326")
wk_transform(pts, trans)
#> <wk_xy[3] with CRS=EPSG:3857>
#> [1] (16362893 -5160980) (16474013 -5311971) (16585132 -5465442)

## transform well-known text geometries
poly <- wk::wkt(
  "POLYGON ((147 -42, 148 -42, 148 -43, 147 -43, 147 -42))",
  crs = "EPSG:4326"
)
wk_transform(poly, trans)
#> <wk_wkt[1] with CRS=EPSG:3857>
#> [1] POLYGON ((16362893 -5160980, ...))

## transform well-known binary
wkb_geom <- wk::as_wkb(poly)
wk_transform(wkb_geom, trans)

## transform sf geometries (sf implements wk_handle)
## wk_transform(sf_sfc_object, trans) — just works

The CRS of the output is updated automatically. The output carries the target CRS from the transformation.

The transformation object is reusable — create it once, apply it to many geometry vectors. This avoids paying the PROJ pipeline setup cost for each call.

Inverse transformations

wk_trans_inverse() flips a transformation:

inv <- wk_trans_inverse(trans)

## round-trip
pts_proj <- wk_transform(pts, trans)
wk_transform(pts_proj, inv)
#> <wk_xy[3] with CRS=EPSG:4326>
#> [1] (147 -42) (148 -43) (149 -44)

The convenience wrapper: PROJ::proj_trans()

If you don’t need a reusable transformation object, PROJ::proj_trans() combines the create and transform steps. It accepts matrices, data frames, wk geometry vectors, and sf sfc columns:

## matrix form — requires explicit source
PROJ::proj_trans(cbind(147, -42), "+proj=laea +type=crs", "EPSG:4326")
#>          x        y
#> [1,] 5969744 -9803200

## wk type with CRS — source is inferred from the data
PROJ::proj_trans(wk::xy(147, -42, crs = "EPSG:4326"), "+proj=laea +type=crs")
#> <wk_xy[1] with CRS=+proj=laea +type=crs>
#> [1] (5969744 -9803200)

## full WKT geometries work too
PROJ::proj_trans(
  wk::wkt("POLYGON ((1 1, 0 1, 0 0, 1 0, 1 1))", crs = "EPSG:4326"),
  3112
)
#> <wk_wkt[1] with CRS=EPSG:3112>
#> [1] POLYGON ((-1.351177e+07 -7779443, ...))

When the input has a CRS (via wk_crs()), the source_crs argument can be omitted — proj_trans() reads it from the data using wk_crs_proj_definition().

The +type=crs distinction

This is worth emphasizing because it catches people. In PROJ 6+, a bare proj-string like "+proj=laea" is a coordinate operation, not a CRS. The library draws a hard line between “a description of a coordinate reference system” and “a description of a mathematical operation to apply to coordinates.”

To specify a CRS using a proj-string, you must include +type=crs:

## CRS — works as source/target for transformations
proj_crs_text("+proj=laea +type=crs", format = 0L)
## returns proper WKT2

## coordinate operation — not a CRS
proj_crs_text("+proj=laea", format = 0L)
## NOT a CRS definition

Object codes ("EPSG:4326"), WKT2, and PROJJSON are unambiguous — they always describe a CRS. Proj-strings need the +type=crs disambiguation. The recommendation from the PROJ documentation is to prefer authority:code strings or WKT2 for CRS definitions. But, use proj strings because they are easy and expressive, when you need a formal WKT2 for a particular operation define it carefully.

How this connects to reproj

The reproj package (see the reproj post in this series) uses PROJ::proj_trans() under the hood.

The relationship is:

• reproj — high-level generic, format-agnostic, delegates to PROJ
• PROJ — mid-level wrapper around the PROJ C library, returns wk types, uses wk_crs_proj_definition() for CRS resolution
• wk — the geometry representation, streaming protocol, and CRS bridge generic
• PROJ C library (libproj) — the actual math, proj_create() does the parsing

If you want maximum control and reusable transformation objects, use proj_trans_create() + wk_transform() directly. If you want a quick one-liner for a matrix of coordinates, use reproj(). They’re different interfaces to the same underlying capability.

Further reading

• PROJ C API: Transformation setup — the canonical reference for what proj_create() accepts
• PROJJSON schema
• wk wk_crs_proj_definition() docs
• PROJ R package proj_trans() docs
• wk wk_transform() docs

Install from CRAN:

install.packages("PROJ")
install.packages("wk")

Check it out series

This is part of a series of posts for foundational spatial support in R. These posts include:

• wk
• reproj
• PROJ/wk transform (this one!)
• gdalraster
• geos

Helper functions

The code snippets below use a small set of helper functions that wrap gdalraster::vsi_read_dir() and related GDAL calls. Source them with:

source("https://github.com/mdsumner/quarto-blog/releases/download/latest/geodata-ucdavis-tools.R")

This provides: vsi_list(), geodata_files(), gdalinfo(), check_block_size(), and streamability_report().

Site structure

The server has three top-level paths that contain spatial data:

The root URL itself and some intermediate directories serve an index.html that says “Hello World!” rather than an Apache directory listing. The data directories below these do have standard Apache autoindex enabled, which is what makes vsi_read_dir() work.

Enumerating files

GDAL’s /vsicurl/ handler can parse Apache directory listings. The gdalraster package exposes this via vsi_read_dir():

library(gdalraster)

GDAL 3.13.0dev-a3b7b01d3e-dirty (released 2026-02-23), GEOS 3.12.1, PROJ 9.7.0

## flat directory — returns filenames
vsi_read_dir("/vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/")

 [1] "agg"                           "bdod_0-5cm_mean_30s.tif"      
 [3] "bdod_15-30cm_mean_30s.tif"     "bdod_30-60cm_mean_30s.tif"    
 [5] "bdod_5-15cm_mean_30s.tif"      "cec_0-5cm_mean_30s.tif"       
 [7] "cec_15-30cm_mean_30s.tif"      "cec_30-60cm_mean_30s.tif"     
 [9] "cec_5-15cm_mean_30s.tif"       "cfvo_0-5cm_mean_30s.tif"      
[11] "cfvo_15-30cm_mean_30s.tif"     "cfvo_30-60cm_mean_30s.tif"    
[13] "cfvo_5-15cm_mean_30s.tif"      "clay_0-5cm_mean_30s.tif"      
[15] "clay_15-30cm_mean_30s.tif"     "clay_30-60cm_mean_30s.tif"    
[17] "clay_5-15cm_mean_30s.tif"      "files.txt"                    
[19] "fill"                          "nitrogen_0-5cm_mean_30s.tif"  
[21] "nitrogen_15-30cm_mean_30s.tif" "nitrogen_30-60cm_mean_30s.tif"
[23] "nitrogen_5-15cm_mean_30s.tif"  "ocd_0-5cm_mean_30s.tif"       
[25] "ocd_15-30cm_mean_30s.tif"      "ocd_30-60cm_mean_30s.tif"     
[27] "ocd_5-15cm_mean_30s.tif"       "phh2o_0-5cm_Q0.05_30s.tif"    
[29] "phh2o_0-5cm_Q0.95_30s.tif"     "phh2o_0-5cm_mean_30s.tif"     
[31] "phh2o_15-30cm_mean_30s.tif"    "phh2o_30-60cm_mean_30s.tif"   
[33] "phh2o_5-15cm_Q0.05_30s.tif"    "phh2o_5-15cm_Q0.95_30s.tif"   
[35] "phh2o_5-15cm_mean_30s.tif"     "raw"                          
[37] "sand_0-5cm_mean_30s.tif"       "sand_15-30cm_mean_30s.tif"    
[39] "sand_30-60cm_mean_30s.tif"     "sand_5-15cm_mean_30s.tif"     
[41] "silt_0-5cm_mean_30s.tif"       "silt_15-30cm_mean_30s.tif"    
[43] "silt_30-60cm_mean_30s.tif"     "silt_5-15cm_mean_30s.tif"     
[45] "soc_0-5cm_mean_30s.tif"        "soc_15-30cm_mean_30s.tif"     
[47] "soc_30-60cm_mean_30s.tif"      "soc_5-15cm_mean_30s.tif"      

For structured enumeration with URLs and GDAL dataset names:

soilgrids <- geodata_files("geodata/soil/soilgrids")
head(soilgrids)

                                                                           url
1   https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_0-5cm_mean_30s.tif
2 https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_15-30cm_mean_30s.tif
3 https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_30-60cm_mean_30s.tif
4  https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_5-15cm_mean_30s.tif
5    https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_0-5cm_mean_30s.tif
6  https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_15-30cm_mean_30s.tif
                       file ext
1   bdod_0-5cm_mean_30s.tif tif
2 bdod_15-30cm_mean_30s.tif tif
3 bdod_30-60cm_mean_30s.tif tif
4  bdod_5-15cm_mean_30s.tif tif
5    cec_0-5cm_mean_30s.tif tif
6  cec_15-30cm_mean_30s.tif tif
                                                                                gdaldsn
1   /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_0-5cm_mean_30s.tif
2 /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_15-30cm_mean_30s.tif
3 /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_30-60cm_mean_30s.tif
4  /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_5-15cm_mean_30s.tif
5    /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_0-5cm_mean_30s.tif
6  /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_15-30cm_mean_30s.tif

This returns a data.frame with columns url, file, ext, and gdaldsn. The gdaldsn column contains the /vsicurl/ path for files GDAL can read, and NA for files it cannot (.rds, .txt, etc.).

What’s there: a complete inventory

soilgrids <- geodata_files("geodata/soil/soilgrids", pattern = "\\.tif$")
nrow(soilgrids)

[1] 44

check_block_size(soilgrids$gdaldsn[1])

xblock yblock 
 43200      1 
attr(,"full_width")
[1] 43200
attr(,"is_row_striped")
[1] TRUE
attr(,"is_tiled")
[1] FALSE

wc_tiles <- geodata_files("climate/worldclim/2_1/tiles/tile", pattern = "\\.tif$")
nrow(wc_tiles)  ## expect 576 = 72 tiles × 8 variables

[1] 576

check_block_size(wc_tiles$gdaldsn[1])

xblock yblock 
  3600      1 
attr(,"full_width")
[1] 3600
attr(,"is_row_striped")
[1] TRUE
attr(,"is_tiled")
[1] FALSE

## example: 10-minute bioclim, first file inside the zip
dsn <- "/vsizip//vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/base/wc2.1_10m_bio.zip"
## list the contents of the zip
vsi_read_dir(dsn)

 [1] "wc2.1_10m_bio_1.tif"  "wc2.1_10m_bio_10.tif" "wc2.1_10m_bio_11.tif"
 [4] "wc2.1_10m_bio_12.tif" "wc2.1_10m_bio_13.tif" "wc2.1_10m_bio_14.tif"
 [7] "wc2.1_10m_bio_15.tif" "wc2.1_10m_bio_16.tif" "wc2.1_10m_bio_17.tif"
[10] "wc2.1_10m_bio_18.tif" "wc2.1_10m_bio_19.tif" "wc2.1_10m_bio_2.tif" 
[13] "wc2.1_10m_bio_3.tif"  "wc2.1_10m_bio_4.tif"  "wc2.1_10m_bio_5.tif" 
[16] "wc2.1_10m_bio_6.tif"  "wc2.1_10m_bio_7.tif"  "wc2.1_10m_bio_8.tif" 
[19] "wc2.1_10m_bio_9.tif" 

iso <- geodata_files("climate/worldclim/2_1/tiles/iso")
head(iso)

                                                                                 url
1  https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_bio.tif
2 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_elev.tif
3 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_prec.tif
4 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_srad.tif
5 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_tavg.tif
6 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_tmax.tif
                    file ext
1  ABW_wc2.1_30s_bio.tif tif
2 ABW_wc2.1_30s_elev.tif tif
3 ABW_wc2.1_30s_prec.tif tif
4 ABW_wc2.1_30s_srad.tif tif
5 ABW_wc2.1_30s_tavg.tif tif
6 ABW_wc2.1_30s_tmax.tif tif
                                                                                      gdaldsn
1  /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_bio.tif
2 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_elev.tif
3 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_prec.tif
4 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_srad.tif
5 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_tavg.tif
6 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/iso/ABW_wc2.1_30s_tmax.tif

/cmip6/{resolution}/{model}/{ssp}/

## list models at 2.5 minute resolution
vsi_read_dir("/vsicurl/https://geodata.ucdavis.edu/cmip6/2.5m/")

 [1] "ACCESS-CM2"       "ACCESS-ESM1-5"    "AWI-CM-1-1-MR"    "BCC-CSM2-MR"     
 [5] "CMCC-ESM2"        "CNRM-CM6-1"       "CNRM-CM6-1-HR"    "CNRM-ESM2-1"     
 [9] "CanESM5"          "CanESM5-CanOE"    "EC-Earth3-Veg"    "EC-Earth3-Veg-LR"
[13] "FIO-ESM-2-0"      "GFDL-ESM4"        "GISS-E2-1-G"      "GISS-E2-1-H"     
[17] "HadGEM3-GC31-LL"  "INM-CM4-8"        "INM-CM5-0"        "IPSL-CM6A-LR"    
[21] "MIROC-ES2L"       "MIROC6"           "MPI-ESM1-2-HR"    "MPI-ESM1-2-LR"   
[25] "MRI-ESM2-0"       "UKESM1-0-LL"     

lc_cogs <- geodata_files("geodata/landcover/3deg_cogs", pattern = "\\.tif$")
nrow(lc_cogs)

[1] 2651

## check whether these are really COGs
check_block_size(lc_cogs$gdaldsn[1])

xblock yblock 
  1024   1024 
attr(,"full_width")
[1] 36000
attr(,"is_row_striped")
[1] FALSE
attr(,"is_tiled")
[1] TRUE

## open GADM as a single GeoPackage over HTTP
dsn <- "/vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gadm_410-raw.gpkg"
## list layers
gdalraster::ogr_ds_layer_names(dsn)

[1] "gadm41_raw"

dsn <- "/vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AUS.gpkg"
gdalraster::ogr_ds_layer_names(dsn)

[1] "ADM_ADM_0" "ADM_ADM_1" "ADM_ADM_2"

Block layout and streamability

The central question for remote access is whether a file’s internal layout supports efficient HTTP range requests. GDAL’s /vsicurl/ handler issues byte-range requests to read specific portions of a file. This works well when the file is internally tiled (e.g., COG format with Block=256x256) but poorly when the file is row-striped (Block=width×1).

For a row-striped file, reading a small spatial window requires decompressing one or more full rows, each spanning the entire raster width. For a 43200-pixel-wide global grid, every row is a complete scan line from -180° to +180°. A single HTTP range request per row, times the number of rows in the window, with full LZW decompression of each row. For a small bounding box this is extremely wasteful.

## for any TIF on the server
bs <- check_block_size("/vsicurl/https://geodata.ucdavis.edu/path/to/file.tif")
bs
## if is_row_striped is TRUE: download, don't stream
## if is_tiled is TRUE: streaming is efficient

Generating file tables

For any collection, generate a complete table of URLs and GDAL dataset names:

## SoilGrids: all TIFs
sg <- geodata_files("geodata/soil/soilgrids", pattern = "\\.tif$")
head(sg[, c("url", "gdaldsn")])

                                                                           url
1   https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_0-5cm_mean_30s.tif
2 https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_15-30cm_mean_30s.tif
3 https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_30-60cm_mean_30s.tif
4  https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_5-15cm_mean_30s.tif
5    https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_0-5cm_mean_30s.tif
6  https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_15-30cm_mean_30s.tif
                                                                                gdaldsn
1   /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_0-5cm_mean_30s.tif
2 /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_15-30cm_mean_30s.tif
3 /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_30-60cm_mean_30s.tif
4  /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/bdod_5-15cm_mean_30s.tif
5    /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_0-5cm_mean_30s.tif
6  /vsicurl/https://geodata.ucdavis.edu/geodata/soil/soilgrids/cec_15-30cm_mean_30s.tif

## WorldClim tiles: all TIFs
wc <- geodata_files("climate/worldclim/2_1/tiles/tile", pattern = "\\.tif$")
head(wc[, c("url", "gdaldsn")])

                                                                                      url
1  https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_bio.tif
2 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_elev.tif
3 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_prec.tif
4 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_srad.tif
5 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_tavg.tif
6 https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_tmax.tif
                                                                                           gdaldsn
1  /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_bio.tif
2 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_elev.tif
3 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_prec.tif
4 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_srad.tif
5 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_tavg.tif
6 /vsicurl/https://geodata.ucdavis.edu/climate/worldclim/2_1/tiles/tile/tile_10_wc2.1_30s_tmax.tif

## GADM 4.1 GeoPackages
gadm <- geodata_files("gadm/gadm4.1/gpkg", pattern = "\\.gpkg$")
head(gadm[, c("url", "gdaldsn")])

                                                            url
1 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ABW.gpkg
2 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AFG.gpkg
3 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AGO.gpkg
4 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AIA.gpkg
5 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ALA.gpkg
6 https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ALB.gpkg
                                                                 gdaldsn
1 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ABW.gpkg
2 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AFG.gpkg
3 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AGO.gpkg
4 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_AIA.gpkg
5 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ALA.gpkg
6 /vsicurl/https://geodata.ucdavis.edu/gadm/gadm4.1/gpkg/gadm41_ALB.gpkg

## CMIP6 models at 2.5 minute resolution
## first list the model directories, then enumerate within each
models <- vsi_read_dir("/vsicurl/https://geodata.ucdavis.edu/cmip6/2.5m/")
models

 [1] "ACCESS-CM2"       "ACCESS-ESM1-5"    "AWI-CM-1-1-MR"    "BCC-CSM2-MR"     
 [5] "CMCC-ESM2"        "CNRM-CM6-1"       "CNRM-CM6-1-HR"    "CNRM-ESM2-1"     
 [9] "CanESM5"          "CanESM5-CanOE"    "EC-Earth3-Veg"    "EC-Earth3-Veg-LR"
[13] "FIO-ESM-2-0"      "GFDL-ESM4"        "GISS-E2-1-G"      "GISS-E2-1-H"     
[17] "HadGEM3-GC31-LL"  "INM-CM4-8"        "INM-CM5-0"        "IPSL-CM6A-LR"    
[21] "MIROC-ES2L"       "MIROC6"           "MPI-ESM1-2-HR"    "MPI-ESM1-2-LR"   
[25] "MRI-ESM2-0"       "UKESM1-0-LL"     

Still to document

Several areas need further investigation:

• Elevation data paths (elevation_3s, elevation_30s, elevation_global)
• Exact contents of /geodata/crops/, /geodata/travel/, /geodata/osm/, /geodata/buildings/
• Block layout of WorldClim tiles, CMIP6 30s TIFs, and landcover COGs
• Contents of /geodata/landcover/30s/
• Contents of /geodata/soil/soilgrids/agg/, fill/, and raw/ subdirectories
• The /cmip6/tiles/ tiled variant structure

A picture of the problem

The fix is to densify the boundary of the projected rectangle before transforming it. Add intermediate vertices along each edge, then reproject the resulting polygon, and compute the bounding box of that. The more vertices you add, the better the approximation.

library(terra)
vdsn <- "/vsizip//vsicurl/https://github.com/user-attachments/files/20692539/AEA_polygon.zip"
v <- vect(vdsn)

## The extent as a polygon in projected space
box_proj <- as.polygons(ext(v))
crs(box_proj) <- crs(v)

## Transform with sparse corners only
box_ll_sparse <- project(box_proj, "EPSG:4326")

## Densify first (add vertices every 10 km), then transform
box_dense <- densify(box_proj, interval = 10000)
box_ll_dense <- project(box_dense, "EPSG:4326")

## Compare the extents
ext(box_ll_sparse)
ext(box_ll_dense)

## Plot them together
plot(box_ll_dense, border = "blue", lwd = 2, main = "Sparse corners vs. densified boundary")
plot(box_ll_sparse, border = "red", lwd = 2, add = TRUE)
legend("bottomright", legend = c("Densified", "Sparse (4 corners)"),
       col = c("blue", "red"), lwd = 2)

The gap between the red quadrilateral and the blue curve is exactly the area that goes missing when software transforms only the corner points.

How this relates to map tiles

Web map tile services (TMS, WMTS, XYZ) serve imagery in tiles at fixed zoom levels, typically in Web Mercator (EPSG:3857). Packages like maptiles, ceramic, and rosm do an excellent job of fetching, compositing, and caching these tiles for use in R. Their workflow is roughly: determine the geographic extent needed, request the right tiles at the right zoom level, stitch them, and return a raster in the desired CRS.

The extent determination step is where the densification problem bites. If the user’s map is in a conic projection and the package computes the longlat extent by transforming only the viewport corners, the requested tiles won’t cover the full area.

There is an alternative approach that sidesteps tile-fetching entirely, by treating the tile server as a GDAL data source and using gdalwarp to produce an image directly on a target grid. GDAL’s warper handles extent computation internally via GDALSuggestedWarpOutput(), which densifies the source boundary as part of its calculation. This means the “white patch” problem doesn’t occur.

Using GDAL warp with a tile server

A TMS or WMTS endpoint can be opened by GDAL as a raster data source, given the right connection string. The esri() helper function below constructs a WMTS connection string from an ArcGIS REST service URL:

## Helper functions:
## esri() builds a WMTS connection string from an ArcGIS REST Services URL
## fit_dims() computes aspect-ratio-preserving pixel dimensions
## (fit_dims is available in the hypertidy/vaster package)

esri <- function(url) {
  sprintf("WMTS:%s/WMTS/1.0.0/WMTSCapabilities.xml", url)
}

## fit_dims is also available from vaster::fit_dims()
fit_dims <- function(size = 1024L, wh = c(size, size)) {
  wh <- rep(wh, length.out = 2L)
  w <- wh[1L]; h <- wh[2L]
  as.integer(ceiling(size * c(w, h) / max(w, h)))
}

With these in hand, the workflow is:

library(terra)

dsn <- esri(url = "https://services.arcgisonline.com/arcgis/rest/services/Ocean/World_Ocean_Base/MapServer")
vdsn <- "/vsizip//vsicurl/https://github.com/user-attachments/files/20692539/AEA_polygon.zip"

v <- vect(vdsn)

## Compute pixel dimensions that preserve the aspect ratio of the extent
dm <- fit_dims(1024, diff(as.vector(ext(v)))[c(1L, 3L)])

source <- rast(dsn)

## Build the target grid: same CRS and extent as the input vector data,
## with the computed dimensions, and matching number of layers (RGB)
target <- rast(ext(v), ncols = dm[1L], nrows = dm[2L],
               crs = crs(v), nlyr = nlyr(source))

## Warp the tile server imagery onto the target grid
## by_util = TRUE calls gdalwarp rather than terra's internal resampler
im <- project(rast(dsn), target, by_util = TRUE)

plotRGB(im)
plot(v, add = TRUE)

The project() call with by_util = TRUE invokes gdalwarp under the hood, which reads tiles from the server as needed to fill the target grid. GDAL computes the correct source extent internally, including the densification step.

A few details worth noting about this code:

• The nlyr = nlyr(source) argument on the target raster is necessary so that all bands (RGB) are included. Without it, the target would default to a single band.
• The by_util = TRUE flag is essential. Without it, project() uses terra’s internal logic driving a lower level part of the GDAL api, which does not automatically target the appropriate resolution level (it will take forever to process pixels that are not needed).
• The dsn variable is a WMTS connection string that GDAL opens as a raster source. The same approach works with TMS (XYZ) endpoints, though the connection string format is different (it requires a small XML file or an inline XML specification; a good reference is the GDAL TMS driver documentation).

Reprojecting the target grid definition

If the desired output CRS is different from the input vector CRS, the key insight is to reproject the target grid definition, not the image data. Build the target grid in the source CRS first (where the extent is known), then reproject the empty grid to the desired output CRS. GDAL’s warp step will then fill it correctly.

## Build the target in the vector's native CRS
target0 <- rast(ext(v), ncols = dm[1L], nrows = dm[2L],
                crs = crs(v), nlyr = nlyr(source))

## Reproject the empty grid definition to Web Mercator
target <- project(target0, "EPSG:3857")

## Warp the source imagery onto the reprojected target
im <- project(rast(dsn), target, by_util = TRUE)
plotRGB(im)
plot(project(v, "EPSG:3857"), add = TRUE)

Reprojecting an empty raster grid with project() (or equivalently, gdalwarp on a small raster) computes the appropriate output extent and resolution by densifying the source boundary before transforming. This is a lightweight way to answer the question “what does this extent look like in another CRS?”

Extent reprojection as a general operation

The task of computing a bounding box in one CRS from a bounding box in another CRS appears in many contexts. Several tools address it:

• reproj::reproj_extent() in the reproj package does this directly. It densifies the input extent boundary, transforms the points, and returns the bounding box. The documentation describes it as a simple version of what GDAL’s GDALSuggestedWarpOutput() does.

• terra::project() applied to a SpatExtent or an empty SpatRaster achieves the same result, using GDAL internally.

• In sf and stars, sf::st_transform() on an sfc_POLYGON representing the bounding box will reproject the boundary (sf applies segmentization as part of its transformation pipeline when working with s2). st_bbox() of the result gives the target extent.

• In gdalraster, gdalraster::warp() with appropriate -te and -te_srs arguments lets you specify the target extent in a different CRS from the output. The -te_srs option tells gdalwarp to interpret the -te coordinates in the given CRS and transform them (with densification) to the output CRS.

• Similarly, gdal_translate has -projwin_srs, which specifies the CRS of the -projwin coordinates. As of GDAL 3.11, the projwin boundary is fully densified before transformation. In earlier versions, only the two corner points were transformed, leading to the same class of error described here.

All of these ultimately rely on the same principle: you cannot determine the geographic footprint of a projected rectangle from its corners alone. The boundary must be sampled densely enough to capture the curvature of the transformation.

The vector side: densifying features

The same issue applies to vector geometries. A polygon or line with sparse vertices in one CRS will have incorrect geometry after reprojection if the segments are long relative to the curvature of the transformation. This is not about datum shifts or projection accuracy at individual points — point reprojection is exact. It is about the fact that a straight line segment in one CRS is not, in general, a straight line in another.

Consider a coastline polygon with vertices every 100 km, stored in an Albers Equal Area CRS. Those long straight edges are reasonable approximations of the coastline in that projection. But when reprojected to longitude/latitude, each straight edge should trace a curve through geographic space. Without densification, the reprojected polygon connects the transformed endpoints with straight lines in the target CRS, which follow different paths than the original segments.

sf::st_segmentize() and terra::densify() add intermediate vertices along existing edges. The critical point is that this must happen before reprojection, in the source CRS, so that the added vertices capture the geometry as it was defined. After reprojection, the denser vertex set produces a polygon that faithfully represents the original shape in the new CRS.

Neither sf nor terra applies densification automatically during reprojection. It is the caller’s responsibility to decide whether it is needed and to choose an appropriate segment length. For small-area, high-resolution data the effect is negligible, but for continental extents or coarse geometries it can be significant.

Summary

The white patch in the tmap basemap, the missing data in gdal_translate -projwin_srs before GDAL 3.11, and the distortion of coarse polygons after reprojection are all the same problem. A rectangular extent in a projected CRS is not a rectangle in geographic coordinates. The boundary curves, and capturing that curvature requires sampling it with enough points.

This applies whenever software needs to answer the question “what geographic area does this projected region cover?” The answer is to densify before transforming — whether the input is a viewport boundary for fetching tiles, a bounding box for subsetting, or the edges of a polygon for display.

The GDAL warp approach to tile servers avoids the problem for raster imagery by letting GDAL handle extent computation internally. For vector data, st_segmentize() or densify() should be part of any reprojection workflow involving coarse geometries.

Code and helper functions used in this post are available at this gist.

Possible future additions: equivalent workflows using sf::gdal_utils(), gdalraster::warp(), and vapour::vapour_warp_raster() to show the same operations without terra.

Detecting regularity

The in-development R package gproj includes tools for characterising coordinate geometry and selecting appropriate projections. Among these is a regularity detection pipeline that can identify “cryptic curvilinear” grids — data that presents as irregular lon/lat arrays but is in fact a regular projected grid with lost metadata.

The approach is layered.

Rectilinear:        TRUE
Type:               mercator_devolved
Lon regular:        TRUE (0.3333° spacing)
Mercator test:      TRUE (max deviation: 1e-12)
Mercator resolution: 37106 m

<params: 104912 points>
  centre:     -0.0000, -88.1448
  extent:     [0.17, 359.84] x [-89.84, -39.36]
  span:       359.67° lon x 50.47° lat
  hemisphere: polar_south
  aspect:     wide
  axis:       azimuth=180.0°, tilt=90.0°, var_ratio=0.51/0.46/0.02

Max row-y SD:      13.42 m
Median dx:         25000 m
Max dx deviation:  74.8 m
Median dy:         -25000 m
Max dy deviation:  71.0 m

Recovered extent:  -3950030  3950000  -3949995  4349967
Known NSIDC extent: -3950000  3950000  -3950000  4350000
Difference:         -30       0        5        -33

CRS:          EPSG:3412
Resolution:   25000 × 25000 m
Dimensions:   316 × 332
Extent:       -3950000  3950000  -3950000  4350000

The Mercator case

A subtler variant of this problem occurs with global ocean products. A Mercator projection produces a regular grid in eastings and northings, but when projected to lon/lat the latitude spacing becomes non-uniform: cells are compressed toward the poles following the inverse Mercator formula. The longitude spacing remains uniform.

This is easy to miss because the grid looks rectilinear — it has separable 1D coordinate axes — and many tools will happily treat it as a lon/lat grid with variable spacing. But it’s not: it’s a Mercator grid with specific metric properties that are lost when treated as plain lon/lat.

The detection is straightforward: convert the latitude values to Mercator y-coordinates and check whether the spacing becomes uniform. For the AVISO product tested above, the deviation from perfect Mercator spacing is 10⁻¹² — indistinguishable from the analytical formula.

Swath geometry and projection selection

Not all lon/lat arrays hide regular grids. Satellite swath data from sensors like VIIRS or MODIS is genuinely curvilinear — the coordinates follow the sensor’s scan geometry, and there is no single projected CRS that makes them regular. But the coordinates still have structure that can be exploited.

The params() function characterises this structure using PCA on the 3D Cartesian coordinates of the data points. Converting lon/lat to points on the unit sphere before PCA avoids the distortions that would arise from operating directly in longitude/latitude space (particularly near the poles or antimeridian).

For a VIIRS swath boundary:

<params: 32436 points>
  centre:     -113.56, -20.81
  extent:     [-180, 180] x [-84.44, 57.66]
  hemisphere: polar_south
  axis:       azimuth=12.4°, tilt=77.6°, var_ratio=0.85/0.14/0.02

The variance ratio of 0.85 in the first component indicates a strongly elongated dataset. The azimuth of 12.4° gives the bearing of the dominant axis — essentially the satellite ground track orientation.

This feeds directly into projection selection. The proj() function uses the geometric characterisation to generate appropriate CRS parameters:

• Polar data with no dominant axis → stereographic or Lambert azimuthal equal-area, centred on the pole
• Elongated data with a dominant axis → oblique Mercator aligned to the data’s axis
• Mid-latitude wide bands → conic projections with secant latitudes derived from the data extent
• Small regions → stereographic centred on the data

For the VIIRS swath, this produces an oblique Mercator projection aligned to the 12.4° ground track, which straightens the swath into a narrow strip in projected coordinates (aspect ratio ~11:1).

The opportunity

The detection of hidden regular grids is not a theoretical exercise. These grids exist throughout operational data archives — sea ice products, ocean altimetry, atmospheric reanalyses, satellite imagery. Every file that stores redundant lon/lat arrays for a regular projected grid is:

• Wasting storage (often 2× the data volume for the coordinate arrays alone)
• Losing precision (floating-point coordinates are noisier than the exact grid specification)
• Obscuring structure (downstream tools treat the data as irregular, triggering unnecessary resampling)
• Creating friction (users must discover the projection through external documentation or domain expertise)

The regularity test demonstrated here is simple — a few lines of linear algebra and some tolerance checks — and could be applied routinely during data ingest or quality assessment. The tolerance gap between “regular with floating-point noise” (0.3% deviation) and “genuinely irregular” is wide enough that false positives are unlikely.

For data producers, the message is: if your data is on a regular projected grid, store the grid specification, not the coordinates. A CRS string and four extent values are complete, exact, and universally understood.

For data consumers and tool developers, the message is: when you encounter 2D lon/lat arrays, test for regularity before treating them as curvilinear. The answer may save you considerable complexity.

The gproj package is in early development at github.com/mdsumner/gproj.

A previous iteration of this discussion was published here hypertidy.org: Coordinates broken in NetCDF.

R already knows this (sort of)

R’s base graphics already embody two distinct models for placing gridded data in space.

m <- volcano[1:10, 1:15]
x <- seq(0, 1, length.out = nrow(m) + 1)
y <- seq(0, 2, length.out = ncol(m) + 1)

image(x, y, m, col = terrain.colors(12))

plot(NA, xlim = c(0, 1), ylim = c(0, 2), asp = 1, xlab = "x", ylab = "y")
rasterImage(as.raster(scales::rescale(m)), xleft = 0, ybottom = 0, xright = 1, ytop = 2, interpolate = F)

library(ximage)
library(vaster)
## Plot numeric data in GIS orientation with extent
ximage(volcano, extent = c(0, 61, 0, 87), col = terrain.colors(24))

## Add contours that respect the same extent
xcontour(volcano, extent = c(0, 61, 0, 87), add = TRUE, levels = c(120, 140, 160))

## Or RGB arrays, or nativeRaster, or hex colours—all work
img <- png::readPNG(system.file("img", "Rlogo.png", package="png"), native = TRUE)
ximage(img, extent = c(100, 160, -50, -10))
maps::map(add = TRUE)  
## nonsensical map, but we can do what we want and maps why not
# they can be fun and used with non spatial data too
ximage(img, extent = c(152, 158, -44, -40), add = TRUE)
plot_extent(c(152, 158, -44, -40), add = TRUE, border = "firebrick", lwd = 2)

What vaster provides

{vaster} gives that underlying logic with no attachment to any data format or structure. All you need are dimension and extent (and extent is optional, a sensible default is [0,nx], [0,ny] rather than unit-square).

library(vaster)

dm <- c(40, 20)  # ncol, nrow
ex <- c(100, 160, -50, -10)  # xmin, xmax, ymin, ymax

# Cell centres - implicit from dimension and extent
x_centre(dm, ex)

 [1] 100.75 102.25 103.75 105.25 106.75 108.25 109.75 111.25 112.75 114.25
[11] 115.75 117.25 118.75 120.25 121.75 123.25 124.75 126.25 127.75 129.25
[21] 130.75 132.25 133.75 135.25 136.75 138.25 139.75 141.25 142.75 144.25
[31] 145.75 147.25 148.75 150.25 151.75 153.25 154.75 156.25 157.75 159.25

y_centre(dm, ex)

 [1] -49 -47 -45 -43 -41 -39 -37 -35 -33 -31 -29 -27 -25 -23 -21 -19 -17 -15 -13
[20] -11

What cell contains a given point?

# Some arbitrary coordinates
pts <- cbind(x = c(120.5, 145.2, 110.8), 
             y = c(-25.3, -42.1, -15.7))

cell_from_xy(dm, ex, pts)

[1] 294 671  88

What are the coordinates of cells 1, 100, and 800?

xy_from_cell(dm, ex, c(1, 100, 800))

       [,1] [,2]
[1,] 100.75  -11
[2,] 129.25  -15
[3,] 159.25  -49

Convert between row/column and cell index:

# Cell to row, column
cell <- 42
row_from_cell(dm, cell)

[1] 2

col_from_cell(dm, cell)

[1] 2

# Row, column to cell
cell_from_row_col(dm, row = 3, col = 15)

[1] 95

None of this requires loading imagery or touching files. It’s pure computation from first principles.

Note here that indexing is via R’s convention: 1-based, both for cell and row and col.

The snap problem

One of the most common operations in raster work is “snapping”—taking an arbitrary region and aligning it to an existing grid. You want to crop to an area of interest, but you need cell-aligned boundaries.

# A reference grid: 3-degree cells covering the globe
ref_dm <- c(30, 10)
ref_ex <- c(-180, 180, -90, 90)

# Some arbitrary region of interest (not aligned to anything)
roi <- c(15, 154.7, -44.2, -9.8)

# Snap to the reference grid
snapped <- vcrop(roi, ref_dm, extent = ref_ex)
snapped

$extent
[1]  12 156 -54   0

$dimension
[1] 12  3

The result gives you the exact dimension and extent for the crop window, aligned to the reference grid. No data needed—just the numbers.

plot_extent(ref_ex, border = "grey")
abline(v = x_corner(ref_dm, ref_ex), col = "grey90")
abline(h = y_corner(ref_dm, ref_ex), col = "grey90")

# Original ROI (arbitrary)
plot_extent(roi, border = "red", lwd = 2, add = TRUE)

# Snapped ROI (grid-aligned)
plot_extent(snapped$extent, border = "blue", lwd = 2, add = TRUE)

The red box is what we asked for; the blue box is what we get when we respect the grid alignment.

The geotransform connection

GDAL uses a six-element “geotransform” to encode the affine relationship between pixel coordinates and geographic coordinates. This is the workhorse of georeferenced raster data. {vaster} speaks this language:

# From extent and dimension to geotransform
gt <- extent_dim_to_gt(ex, dm)
gt

 xmin  xres yskew  ymax xskew  yres 
100.0   1.5   0.0 -10.0   0.0  -2.0 

# And back again
gt_dim_to_extent(gt, dm)

xmin xmin ymax ymax 
 100  160  -50  -10 

This matters when you’re working directly with GDAL (via {gdalraster}, {vapour}, or Python’s osgeo.gdal) and need to set up or interpret raster metadata without constructing heavyweight objects.

Why does this matter?

1. Lightweight tooling: Sometimes we just need to compute cell indices or snap extents. We shouldn’t need to load terra or GDAL for that.

2. Cross-package compatibility: The logic is the same whether you’re working with terra, stars, GDAL, or raw arrays. Having it in one place means consistent behaviour everywhere.

3. Teaching and understanding: Separating the logic from the data makes it clearer what raster operations actually do. The magic isn’t in the file format or the object class—it’s in the relationship between dimension and extent.

The ecosystem: what you can build with pure logic

Once you separate grid logic from data, interesting things become possible. Here are three packages that build on this foundation, taken (ahem) shamelessly from the hypertidy suite. (I’m not ignoring other important packages in R and Python that use these ideas, but these are chosen to focus on the core idea of what vaster is and why it exists. )

## fasterize needs a "raster", but really it just needs dimension + extent
library(fasterize)
r <- raster::raster(ncol = 1000, nrow = 800, 
                    xmn = 0, xmx = 100, ymn = 0, ymx = 80)
result <- fasterize(polygons_sf, r, field = "value")

library(controlledburn)
library(vaster)

## Define a grid with just numbers
ext <- c(100, 160, -50, -10)
dm <- c(500, 400)

## Get the cell coverage index, not pixel values
idx <- burn_polygon(polygons_sf, extent = ext, dimension = dm)

library(grout)

## A raster that's 87 × 61 with 8 × 8 tiles
scheme <- grout(c(87, 61), extent = c(0, 87, 0, 61), blocksize = c(8, 16))
scheme

          tiles: 4, 11 (x * y = 44)
          block: 8, 16 
         dangle: 1, 3 
tile resolution: 8, 16 
    tile extent: 0, 88, -3, 61 (xmin,xmax,ymin,ymax)
          grain: 8, 16 (8 : x, 16 : y)

## Get offset/size for each tile (for GDAL RasterIO)
dplyr::glimpse(tile_index(scheme))

Rows: 44
Columns: 11
$ tile     <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18…
$ offset_x <dbl> 0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 0, 8, 16, 24, 32, 4…
$ offset_y <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16, 16, 16, 16, 16, 16, 16, …
$ tile_col <dbl> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 1, 2, 3, 4, 5, 6, 7, 8, 9,…
$ tile_row <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2…
$ ncol     <dbl> 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8…
$ nrow     <dbl> 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 1…
$ xmin     <dbl> 0, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 0, 8, 16, 24, 32, 4…
$ xmax     <dbl> 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 87, 8, 16, 24, 32, 40,…
$ ymin     <dbl> 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 45, 29, 29, 29, 29, 2…
$ ymax     <dbl> 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 61, 45, 45, 45, 45, 4…

Python has a similar mix of history, concepts, and tools

The Python geospatial ecosystem has grappled with the same issues that motivated {vaster}.

A recent pangeo discussion compared odc.stac and stackstac—two packages for loading satellite imagery into xarray. Despite ostensibly doing the same thing, they produce different results. Why?

Pixel coordinates: edge or centre? When you say a pixel is at coordinate (100.0, 200.0), do you mean that’s its centre, or its top-left corner? odc.stac defaults to centre, stackstac defaults to edge. Both are valid conventions; GIS tools tend toward corners, while climate/ocean data tends toward centres. But if you don’t know which convention your tool uses, your data shifts by half a pixel.

Coordinate snapping: When your requested bounding box doesn’t align perfectly with the source grid, what happens? Different tools make different choices about how to snap—and those choices are usually invisible, buried in the library internals.

These are exactly the questions {vaster} makes explicit. Dimension and extent. Corner or centre. Snap in or snap out. The logic is universal; only the defaults vary.

The R matrix connection

R’s matrices already have most of this logic, just without the spatial semantics. A matrix has dimension (nrow, ncol). When you use image() with explicit coordinates, you’re adding extent. The [i, j] indexing is cell-from-row-column. which(m > threshold, arr.ind = TRUE) gives you row-column-from-cell.

{vaster} makes this connection explicit. It treats dimension and extent as first-class inputs, just as R treats matrices as first-class objects. The spatial meaning is in the numbers, not the container.

Example: raster logic is a simplifying principle

Enough theory! Let’s make a map. The following string defines a huge world-coverage image server.

dsn <- "WMTS:https://services.arcgisonline.com/arcgis/rest/services/World_Imagery/MapServer/WMTS/1.0.0/WMTSCapabilities.xml,layer=World_Imagery"

So, obviously we need a huge stack of Python geospatial to make a map image … or maybe we can just use generic tools and do it in R.

We’re using {gdalraster} here because it’s the thinnest wrapper over the GDAL API, but the same calls work through {terra}, {vapour}, or Python’s osgeo.gdal. The point isn’t the package — it’s that the interface is just numbers and strings. I’m still using a mix of tools and plotting … but that’s life.

library(gdalraster)
library(ximage)
ds <- new(GDALRaster, dsn)  ## same as terra::rast(dsn) or osgeo.gdal.Open(dsn) or rioxarray.open_rasterio(dsn)
ds

C++ object of class GDALRaster
 Driver : OGC Web Map Tile Service (WMTS)
 DSN    : WMTS:https://services.arcgisonline.com/arcgis/rest/services/World_Imagery/MapServer/WMTS/1.0.0/WMTSCapabilities.xml,layer=World_Imagery
 Dim    : 1073741766, 1070224430, 4
 CRS    : WGS 84 / Pseudo-Mercator (EPSG:3857)
 Res    : 0.037323, 0.037323
 Bbox   : -20037507.260427, -19971868.890929, 20037507.248165, 19971868.903191

We don’t want the whole world and certainly not that much detail, how about Mawson Station in Antarctica, Mawson is at approximately 62.87E and -67.6S.

We should define a local equal area projection, on Mawson.

prj <- "+proj=laea +lon_0=62.8742 +lat_0=-67.6033"

What is the right extent or bbox for that projection? It’s just metres around the zero-point, let’s go with a range.

b <- 125000
tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt") ## use GDAL to manage IO and cleanup, same in other tools
chk <- warp(dsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 0))

0...10...20...30...40...50...60...70...80...90...100 - done.

ximage(read_ds(new(GDALRaster, tfile), bands = 1:3), asp = 1)

What about a different data source, can we see any meaningful data in sea ice at this time of year?

icedsn <- "/vsicurl/https://data.seaice.uni-bremen.de/amsr2/asi_daygrid_swath/s3125/2025/dec/Antarctic3125/asi-AMSR2-s3125-20251220-v5.4.tif"

tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt")
chk <- warp(icedsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 0))

0...10...20...30...40...50...60...70...80...90...100 - done.

dat <- read_ds(new(GDALRaster, tfile), bands = 1)
dat[dat > 100] <- NA

ximage(dat, col = grey.colors(100), asp = 1)

summary(dat)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.    NA's 
   1.00   94.00   98.00   90.99  100.00  100.00  576851 

Not unexpectedly at 3.125km resolution it’s a bit sparse on information, but we didn’t have to use a different approach to get coincident data. The ice is thick here, to see Mawson at about this time from Sentinel 2 imagery, try the estinel catalog, click ‘ESA’ and zoom around for more, you’ll see that dark hole in the sea ice).

What about the bathymetry? Even at this scale we can see the disparity between bathymetric and topographic detail.

bathdsn <- "/vsicurl/https://projects.pawsey.org.au/idea-gebco-tif/GEBCO_2024.tif"
tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt")
chk <- warp(bathdsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 0))

0...10...20...30...40...50...60...70...80...90...100 - done.

ximage(dat <- read_ds(new(GDALRaster, tfile), bands = 1), col = hcl.colors(128), asp = 1)

Let’s zoom right in, and this time we’ll use terra so that adding contours is easy, read the REMA v2 2m topography and add it as contours.

b <- 3000
tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt")
chk <- warp(dsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 0))

0...10...20...30...40...50...60...70...80...90...100 - done.

ximage(read_ds(new(GDALRaster, tfile), bands = 1:3), asp = TRUE)

remadsn <- "/vsicurl/https://raw.githubusercontent.com/mdsumner/rema-ovr/main/REMA-2m_dem_ovr.vrt"
library(terra)
r <- rast(ext(-b, b, -b, b), crs = prj, ncols = 1024, nrows = 1024)
tfile <- tempfile(tmpdir = "/vsimem", fileext = ".vrt")
chk <- warp(remadsn, tfile, t_srs = prj, cl_arg = c("-te", -b, -b, b, b,  "-ts", 1024, 1024))

0...10...20...30...40...50...60...70...80...90...100 - done.

rema <- read_ds(new(GDALRaster, tfile), bands = 1)
contour(setValues(r, rema), add = TRUE)

Note that throughout we’ve done no juggling of input coverage or extent or determining output resolution - we ask GDAL for “enough pixels” to see, because that’s what we’re doing and it sorts that out. Reading enough pixels for a view from an efficient data source takes very little time, with raster logic as a core guideline it also takes very little time to craft our scripts.

Thanks

Hopefully this sparks some interest in the simplicity of raster grid logic and more use of the GDAL warp api to its full potential. Please contact me if you have any interest or questions.

https://github.com/hypertidy/vaster/issues

sessionInfo()

R version 4.5.1 (2025-06-13)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 22.04.5 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_AU.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_AU.UTF-8        LC_COLLATE=en_AU.UTF-8    
 [5] LC_MONETARY=en_AU.UTF-8    LC_MESSAGES=en_AU.UTF-8   
 [7] LC_PAPER=en_AU.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_AU.UTF-8 LC_IDENTIFICATION=C       

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] grout_0.0.2.9009      ximage_0.0.0.9012     gdalraster_2.3.0.9007
[4] terra_1.8-80          vaster_0.5.0         

loaded via a namespace (and not attached):
 [1] bit_4.6.0          jsonlite_2.0.0     dplyr_1.1.4        compiler_4.5.1    
 [5] maps_3.4.1.1       tidyselect_1.2.1   Rcpp_1.1.0         xml2_1.5.1        
 [9] palr_0.4.0         nanoarrow_0.7.0-2  parallel_4.5.1     dichromat_2.0-0.1 
[13] scales_1.4.0       png_0.1-8          yaml_2.3.10        fastmap_1.2.0     
[17] R6_2.6.1           generics_0.1.4     yyjsonr_0.1.21     knitr_1.50        
[21] htmlwidgets_1.6.4  tibble_3.3.0       pillar_1.11.1      RColorBrewer_1.1-3
[25] rlang_1.1.6        xfun_0.53          bit64_4.6.0-1      cli_3.6.5         
[29] magrittr_2.0.4     wk_0.9.4.9000      digest_0.6.39      rstudioapi_0.15.0 
[33] rappdirs_0.3.3     lifecycle_1.0.4    vctrs_0.6.5        evaluate_1.0.5    
[37] glue_1.8.0         farver_2.1.2       codetools_0.2-19   rmarkdown_2.29    
[41] pkgconfig_2.0.3    tools_4.5.1        htmltools_0.5.8.1 

Throw all that away, ignore everything above.

You should never ever do the above for a dataset that doesn’t need it. This is a major problem in array computation worlds, there’s a lowest common denominator that will work but why not be smart about it, and keep fidelity with the original scheme for a dataset.

The problem here is that we can’t reconstruct the actual grid in the file, we have to treat it as points in geolocation arrays, and infer something about the grid. Here’s the answer:

ex <- c(xmin = -3950000,  xmax = 3950000, ymin  = -3950000, ymax = 4350000)
diff(ex)[c(1, 3)] / c(316, 332)  ## look familiar?

 xmax  ymax 
25000 25000 

set_ext <- function(x, ex) {terra::ext(x) <- ex; x}

plot(set_ext(ice, ext(ex)))
plot(v, add = T)

plot(polargrid); plot(v, add = TRUE)

Now finally, we get exactly the same answer.

gdalraster::warp(sds,  tf <- tempfile(fileext = ".tif", tmpdir = "/vsimem"), t_srs = "EPSG:3412", 
                cl_arg = c("-ts", 316, 332, "-te", ex[1], ex[3], ex[2], ex[4],
                           
                  "-to", "GEOLOC_NORMALIZE_LONGITUDE_MINUS_180_PLUS_180=YES"))

0...10...20...30...40...50...60...70...80...90...100 - done.

polargrid <- rast(tf)

polargrid

class       : SpatRaster 
size        : 332, 316, 1  (nrow, ncol, nlyr)
resolution  : 25000, 25000  (x, y)
extent      : -3950000, 3950000, -3950000, 4350000  (xmin, xmax, ymin, ymax)
coord. ref. : NSIDC Sea Ice Polar Stereographic South (EPSG:3412) 
source      : filedb3946450fee3.tif 
name        : filedb3946450fee3 

polargrid - set_ext(ice, ext(ex))

Warning: [-] CRS do not match

class       : SpatRaster 
size        : 332, 316, 1  (nrow, ncol, nlyr)
resolution  : 25000, 25000  (x, y)
extent      : -3950000, 3950000, -3950000, 4350000  (xmin, xmax, ymin, ymax)
coord. ref. : NSIDC Sea Ice Polar Stereographic South (EPSG:3412) 
source(s)   : memory
varname     : filedb3946450fee3 
name        : filedb3946450fee3 
min value   :                 0 
max value   :                 0 

# class       : SpatRaster 
# size        : 332, 316, 1  (nrow, ncol, nlyr)
# resolution  : 25000, 25000  (x, y)
# extent      : -3950000, 3950000, -3950000, 4350000  (xmin, xmax, ymin, ymax)
# coord. ref. : NSIDC Sea Ice Polar Stereographic South (EPSG:3412) 
# source(s)   : memory
# varname     : fileb328ade0cec2 
# name        : fileb328ade0cec2 
# min value   :                0 
# max value   :                0 

But how did we do that? We plucked some magic numbers out of the air. We can’t obtain those four numbers from the lon lat arrays, we can’t find them in the file, they aren’t in the metadata or the website. We just know, because NSIDC sea ice products are standard on a 3412 grid at 25km resolution.

These four numbers: -3950000, 3950000, -3950000, 4350000 are entirely cryptic in the data store itself, and require an expert to notice the problem. We can throw away 332 * 316 * 2 values from every file, replace them with these four values and we have a better and more faithful representation of the original data.

This is an entropy problem.

I refer to this kind of problem as “cryptic curvilinear”. The data present as an irregular grid with curvlinear coordinates, but they are entirely unnecessary, we can replace them with 4 values that anyone can remember and we have the following easy ways to fix, roughly in increasing difficulty and decreasing utility:

• set “-a_srs”, and -a_ullr” in gdal raster convert, or via “vrt://{dsn}?a_ullr=,,,” syntax
• OR set “-a_srs”, and “-a_gt” in gdal raster convert, or via “vrt://{dsn}?a_gt=,,,,,” syntax
• set extent with terra in R and set the crs
• open with rasterio and set geotransform and set spatial ref (same as set extent with R)
• modify the coordinates in xarray, and set the crs

Oh, but sure there are bad files out there

There are so many.

• GHRSST noisy, low precision degenerate rectilinear coordinates
• OISST and NSIDC, have degenerate rectilinear coordinates that don’t cause problems

ZARR and the European Space Agency (ESA)

The ESA is moving Copernicus to ZARR, launching its Earth Observation Processing Framework (EOPF) data format (Zarr). ZARR is “a community project to develop specifications and software for storage of large N-dimensional typed arrays”.

A ZARR is a dataset consisting of trees of array chunks stored (usually) in object storage and indexed by fairly simple JSON metadata that describes how those chunks align together in one potentially very large array. The idea is that all the metadata lives upfront in instantly readable JSON, and changes made to the dataset (extending it each day as new data arrives) affects only the relevant chunks and the small parts of the JSON. This is different to a long list of NetCDFs that grows every day, where the metadata is self-contained for each file and there is no overarching abstraction for the entire file set.

ZARR is usually in object storage, and loaded by datacube software such as xarray. It’s not intended to be zipped into a huge file and downloaded or read, the real power lies in the entire dataset being lazy, and understood by software that needs just one data set description (url, or S3 path, etc).

ESA sample Zarr datasets

The ESA provide a set of example ZARRs that are available in zip files:

https://eopf-public.s3.sbg.perf.cloud.ovh.net/product.html

We choose one that is described by this URL:

url <- "https://eopf-public.s3.sbg.perf.cloud.ovh.net/eoproducts/S02MSIL1C_20230629T063559_0000_A064_T3A5.zarr.zip"

GDAL urls and zip files

GDAL doesn’t force us to download data, but we need some syntax to leverage its remote capabilities in the simplest way. The Virtual File System (VSI) allows us to declare special sources like zip files /vsizip/ and urls /vsicurl/, which we can chain together. With ZARR we also need careful quoting of the description, and we declare the ZARR driver upfront.

(dsn <- sprintf('ZARR:"/vsizip//vsicurl/%s"', url))

[1] "ZARR:\"/vsizip//vsicurl/https://eopf-public.s3.sbg.perf.cloud.ovh.net/eoproducts/S02MSIL1C_20230629T063559_0000_A064_T3A5.zarr.zip\""

GDAL and multidimensional datasets

GDAL has a multidimensional mode for data sources that aren’t “imagery” in the traditional sense. (If we open one of these datasets in “classic” mode we end up with a lot of bands on a 2D raster, or potentially many bands on many subdatasets within a more general container. Zarr is a container format, much like HDF5 and NetCDF).

Multidimensional mode is avaible in the API via OpenEx() and declaring type OF_MULTIDIM_RASTER.

To actually load this python library we use {reticulate} py_require() which drives the awesome Python uv package manager.

(For some reason the pypi name of the package is “gdal”, but the actual module is obtained with “osgeo.gdal”).

reticulate::py_require("gdal")
gdal <- reticulate::import("osgeo.gdal")
gdal$UseExceptions()

sample(names(gdal), 40)  ## see that we have a huge coverage of the underlying API, 544 elements at time of writing

 [1] "GCI_NIRBand"                       "GCI_CyanBand"                     
 [3] "GFU_RedMax"                        "GRT_COMPOSITE"                    
 [5] "GRT_AGGREGATION"                   "GRIORA_Gauss"                     
 [7] "GPI_RGB"                           "GFT_Integer"                      
 [9] "GFU_AlphaMin"                      "GCI_SAR_C_Band"                   
[11] "GCI_OtherIRBand"                   "ApplyVerticalShiftGrid"           
[13] "CPLE_NoWriteAccess"                "deprecation_warn"                 
[15] "Rasterize"                         "GCI_SAR_K_Band"                   
[17] "TermProgress_nocb"                 "GMF_ALL_VALID"                    
[19] "DCAP_CREATECOPY"                   "GRA_Q1"                           
[21] "config_option"                     "DCAP_FIELD_DOMAINS"               
[23] "GDAL_GCP_Info_set"                 "wrapper_GDALVectorTranslateDestDS"
[25] "ColorTable"                        "ContourGenerate"                  
[27] "DCAP_COORDINATE_EPOCH"             "GDT_UInt16"                       
[29] "GDsCDeleteRelationship"            "GCI_YCbCr_CbBand"                 
[31] "IsLineOfSightVisible"              "BuildVRTInternalObjects"          
[33] "DEMProcessing"                     "GDALDestroyDriverManager"         
[35] "DCAP_RENAME_LAYERS"                "MultiDimInfoOptions"              
[37] "CPLES_XML_BUT_QUOTES"              "DitherRGB2PCT"                    
[39] "GDT_TypeCount"                     "GARIO_ERROR"                      

The API elements chain in the usual way that works in python with ‘object.element.thing.etc’ syntax uses R’s $ accessor.

gdal$Dimension$GetIndexingVariable

<function Dimension.GetIndexingVariable at 0x7f08c9f61760>
 signature: (self, *args) -> 'GDALMDArrayHS *'

Open the data

It’s not very exciting yet.

ds <- gdal$OpenEx(dsn, gdal$OF_MULTIDIM_RASTER)
ds

<osgeo.gdal.Dataset; proxy of <Swig Object of type 'GDALDatasetShadow *' at 0x7f08ca0ebcf0> >

To actually find out what’s in there we have to traverse a tree of potentially nested “groups” that organize actual datasets in a hierarchy.

Get the root group and dive in, it’s very tediuous but shows some of what is there. There might be MDArrays in a group, or there might just be more groups.

rg <- ds$GetRootGroup()
rg$GetMDArrayNames()

list()

rg$GetGroupNames()

[1] "conditions"   "measurements" "quality"     

g1 <- rg$OpenGroup("quality")
g1$GetMDArrayNames()

list()

g1$GetGroupNames()

[1] "l1c_quicklook" "mask"         

g2 <- g1$OpenGroup("l1c_quicklook")
g2$GetMDArrayNames()

list()

g2$GetGroupNames()

[1] "r10m"

g3 <- g2$OpenGroup("r10m")
g3$GetMDArrayNames()

[1] "band" "x"    "y"    "tci" 

g3$GetGroupNames()

list()

Finally we got to actual data, we recognize ‘tci’ as being the quicklook RGB of Sentinel 2.

To avoid tedium write a quick recursive function to find all the MDArray, at each level use GetFullName() which provides the cumulative path to where we are in the tree.

get_all_mdnames <- function(rootgroup) {
  groups <- rootgroup$GetGroupNames()
  groupname <- rootgroup$GetFullName()
  amd <- rootgroup$GetMDArrayNames()
  md <- sprintf("%s/%s", groupname, amd)
  md <- c(md, unlist(lapply(groups, \(.g) get_all_mdnames(rootgroup$OpenGroup(.g)))))
  md
}

get_all_mdnames(ds$GetRootGroup())

  [1] "/conditions/geometry/angle"                        
  [2] "/conditions/geometry/band"                         
  [3] "/conditions/geometry/detector"                     
  [4] "/conditions/geometry/x"                            
  [5] "/conditions/geometry/y"                            
  [6] "/conditions/geometry/mean_sun_angles"              
  [7] "/conditions/geometry/mean_viewing_incidence_angles"
  [8] "/conditions/geometry/sun_angles"                   
  [9] "/conditions/geometry/viewing_incidence_angles"     
 [10] "/conditions/mask/detector_footprint/r10m/x"        
 [11] "/conditions/mask/detector_footprint/r10m/y"        
 [12] "/conditions/mask/detector_footprint/r10m/b02"      
 [13] "/conditions/mask/detector_footprint/r10m/b03"      
 [14] "/conditions/mask/detector_footprint/r10m/b04"      
 [15] "/conditions/mask/detector_footprint/r10m/b08"      
 [16] "/conditions/mask/detector_footprint/r20m/x"        
 [17] "/conditions/mask/detector_footprint/r20m/y"        
 [18] "/conditions/mask/detector_footprint/r20m/b05"      
 [19] "/conditions/mask/detector_footprint/r20m/b06"      
 [20] "/conditions/mask/detector_footprint/r20m/b07"      
 [21] "/conditions/mask/detector_footprint/r20m/b11"      
 [22] "/conditions/mask/detector_footprint/r20m/b12"      
 [23] "/conditions/mask/detector_footprint/r20m/b8a"      
 [24] "/conditions/mask/detector_footprint/r60m/x"        
 [25] "/conditions/mask/detector_footprint/r60m/y"        
 [26] "/conditions/mask/detector_footprint/r60m/b01"      
 [27] "/conditions/mask/detector_footprint/r60m/b09"      
 [28] "/conditions/mask/detector_footprint/r60m/b10"      
 [29] "/conditions/mask/l1c_classification/r60m/x"        
 [30] "/conditions/mask/l1c_classification/r60m/y"        
 [31] "/conditions/mask/l1c_classification/r60m/b00"      
 [32] "/conditions/meteorology/cams/latitude"             
 [33] "/conditions/meteorology/cams/longitude"            
 [34] "/conditions/meteorology/cams/aod1240"              
 [35] "/conditions/meteorology/cams/aod469"               
 [36] "/conditions/meteorology/cams/aod550"               
 [37] "/conditions/meteorology/cams/aod670"               
 [38] "/conditions/meteorology/cams/aod865"               
 [39] "/conditions/meteorology/cams/bcaod550"             
 [40] "/conditions/meteorology/cams/duaod550"             
 [41] "/conditions/meteorology/cams/isobaricInhPa"        
 [42] "/conditions/meteorology/cams/number"               
 [43] "/conditions/meteorology/cams/omaod550"             
 [44] "/conditions/meteorology/cams/ssaod550"             
 [45] "/conditions/meteorology/cams/step"                 
 [46] "/conditions/meteorology/cams/suaod550"             
 [47] "/conditions/meteorology/cams/surface"              
 [48] "/conditions/meteorology/cams/time"                 
 [49] "/conditions/meteorology/cams/valid_time"           
 [50] "/conditions/meteorology/cams/z"                    
 [51] "/conditions/meteorology/ecmwf/latitude"            
 [52] "/conditions/meteorology/ecmwf/longitude"           
 [53] "/conditions/meteorology/ecmwf/isobaricInhPa"       
 [54] "/conditions/meteorology/ecmwf/msl"                 
 [55] "/conditions/meteorology/ecmwf/number"              
 [56] "/conditions/meteorology/ecmwf/r"                   
 [57] "/conditions/meteorology/ecmwf/step"                
 [58] "/conditions/meteorology/ecmwf/surface"             
 [59] "/conditions/meteorology/ecmwf/tco3"                
 [60] "/conditions/meteorology/ecmwf/tcwv"                
 [61] "/conditions/meteorology/ecmwf/time"                
 [62] "/conditions/meteorology/ecmwf/u10"                 
 [63] "/conditions/meteorology/ecmwf/v10"                 
 [64] "/conditions/meteorology/ecmwf/valid_time"          
 [65] "/measurements/reflectance/r10m/x"                  
 [66] "/measurements/reflectance/r10m/y"                  
 [67] "/measurements/reflectance/r10m/b02"                
 [68] "/measurements/reflectance/r10m/b03"                
 [69] "/measurements/reflectance/r10m/b04"                
 [70] "/measurements/reflectance/r10m/b08"                
 [71] "/measurements/reflectance/r20m/x"                  
 [72] "/measurements/reflectance/r20m/y"                  
 [73] "/measurements/reflectance/r20m/b05"                
 [74] "/measurements/reflectance/r20m/b06"                
 [75] "/measurements/reflectance/r20m/b07"                
 [76] "/measurements/reflectance/r20m/b11"                
 [77] "/measurements/reflectance/r20m/b12"                
 [78] "/measurements/reflectance/r20m/b8a"                
 [79] "/measurements/reflectance/r60m/x"                  
 [80] "/measurements/reflectance/r60m/y"                  
 [81] "/measurements/reflectance/r60m/b01"                
 [82] "/measurements/reflectance/r60m/b09"                
 [83] "/measurements/reflectance/r60m/b10"                
 [84] "/quality/l1c_quicklook/r10m/band"                  
 [85] "/quality/l1c_quicklook/r10m/x"                     
 [86] "/quality/l1c_quicklook/r10m/y"                     
 [87] "/quality/l1c_quicklook/r10m/tci"                   
 [88] "/quality/mask/r10m/x"                              
 [89] "/quality/mask/r10m/y"                              
 [90] "/quality/mask/r10m/b02"                            
 [91] "/quality/mask/r10m/b03"                            
 [92] "/quality/mask/r10m/b04"                            
 [93] "/quality/mask/r10m/b08"                            
 [94] "/quality/mask/r20m/x"                              
 [95] "/quality/mask/r20m/y"                              
 [96] "/quality/mask/r20m/b05"                            
 [97] "/quality/mask/r20m/b06"                            
 [98] "/quality/mask/r20m/b07"                            
 [99] "/quality/mask/r20m/b11"                            
[100] "/quality/mask/r20m/b12"                            
[101] "/quality/mask/r20m/b8a"                            
[102] "/quality/mask/r60m/x"                              
[103] "/quality/mask/r60m/y"                              
[104] "/quality/mask/r60m/b01"                            
[105] "/quality/mask/r60m/b09"                            
[106] "/quality/mask/r60m/b10"                            

Happily, we see our target MDArray name in there /quality/l1c_quicklook/r10m/tci.

Actual data

Finally let’s get some data out. We can obtain the MDArray now by full name and find out some properties.

reticulate::py_require("gdal")
gdal <- reticulate::import("osgeo.gdal")
gdal$UseExceptions()

dsn <- "ZARR:\"/vsizip//vsicurl/https://eopf-public.s3.sbg.perf.cloud.ovh.net/eoproducts/S02MSIL1C_20230629T063559_0000_A064_T3A5.zarr.zip\""

ds <- gdal$OpenEx(dsn, gdal$OF_MULTIDIM_RASTER)
rg <- ds$GetRootGroup()
mdname <- "/quality/l1c_quicklook/r10m/tci"
md <- rg$OpenMDArrayFromFullname(mdname)
md$GetDimensionCount()

[1] 3

Traverse the dimensions to get their sizes and names.

vapply(md$GetDimensions(), \(.d) .d$GetSize(), 0L)

[1]     3 10980 10980

vapply(md$GetDimensions(), \(.d) .d$GetName(), "")

[1] "band" "y"    "x"   

Explore some metadata attributes.

mnames <- vapply(md$GetAttributes(), \(.a) .a$GetName(), "")

(meta <- setNames(lapply(md$GetAttributes(), \(.a) unlist(.a$Read())), mnames))

$long_name
[1] "TCI: True Color Image"

$`proj:bbox`
[1]  300000 4490220  409800 4600020

$`proj:epsg`
[1] 32636

$`proj:shape`
[1] 10980 10980

$`proj:transform`
[1]      10       0  300000       0     -10 4600020       0       0       1

$`proj:wkt2`
[1] "PROJCS[\"WGS 84 / UTM zone 36N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",33],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32636\"]]"

We see that the CRS information is in there, but sadly while the geotransform of the data is maintained when we convert to classic raster the CRS does not make the journey (the geotransform is just the image bbox intermingled with its resolution in a mathematical abstraction used in matrix manipulation).

A multidim raster can be converted to classic form, either in whole or after applying a $GetView() operation that acts like numpy ‘[:]’ array subsetting.

When converting to classic raster we specify the x, then y dimensions of the dataset, which from multdim convention are in reverse order. (Note that dimension 0 is the “3rd” dimension in normal thinking, here that dimension is “band” or a dimension for each of red, green, blue in the quicklook image).

cc <- md$AsClassicDataset(2L, 1L)
tr <- unlist(cc$GetGeoTransform())
dm <- c(cc$RasterXSize, cc$RasterYSize)
cc$GetSpatialRef() ## empty, so we retrieve from the attributes
crs <- cc$GetMetadata()[["proj:wkt2"]]

Now, I used reproj_extent to convert the dataset’s bbox to one in longlat, but I won’t share that here, we’ll just use the result so our map is a little more familiar. The source data xy bbox is xmin: 300000 ymin: 4490220 xmax: 409800 ymax:4600020, and we take small section of that which is this in longitude latitude:

# xmin,xmax,ymin,ymax converted below to bbox 
ex <- c(31.689, 31.774, 40.665, 40.724)

To warp the imagery to this region we first need to set the CRS properly on the dataset, so translate to a temporary VRT and use that dataset for the next step.

dsvrt <- gdal$Translate(tempfile(fileext = ".vrt", tmpdir = "/vsimem"), cc, options = c("-a_srs", crs))
tf <- tf <- "/vsimem/result.tif"
ww <- gdal$Warp(tf, dsvrt,  dstSRS = "EPSG:4326", outputBounds = ex[c(1, 3, 2, 4)])
ww$Close()

[1] 0

Finally we can read the image, plot it, obtain some contextual data and be assured that our map is correct.

library(gdalraster)
data <- read_ds(new(GDALRaster, tf))

library(osmdata)
x <- opq(ex[c(1, 3, 2, 4)]) |> add_osm_feature(key = "highway") |> osmdata_sf()

library(sf)
plot_raster(data)
plot(x$osm_lines[0], add = TRUE, col = "hotpink")

Summary

We had a look at GDAL multidimensional API, converting a 3D dataset to classic raster, augmenting the missing CRS information on a virtual copy and then warping an image to a familiar map context.

There is ongoing work to bring the full GDAL API to R in {gdalraster}, and in-development version of gdalraster to add the GDALMultiDimRaster class and helpers: https://github.com/mdsumner/gdalraster/tree/multidimnew.

Please get in touch if any of this is of interest!

Late Bonus Section: Bluelink ocean model

This is something I want to use VRT to encapsulate as one whole dataset, here just a test on one file (there are 174 of these ocean_temp files, up to June 2024 at time of writing).

Note that we are using ‘/fileServer/’ from Thredds, not ‘/dodsC/’. GDAL uses userfaultd as a trick to read NetCDF remotely.

from osgeo import gdal
dsn = "/vsicurl/https://thredds.nci.org.au/thredds/fileServer/gb6/BRAN/BRAN2023/daily/ocean_temp_2010_01.nc"
ds = gdal.OpenEx(dsn, gdal.OF_MULTIDIM_RASTER)
rg = ds.GetRootGroup()
rg.GetMDArrayNames()
# ['xt_ocean', 'yt_ocean', 'st_ocean', 'Time', 'nv', 'st_edges_ocean', 'average_T1', 'average_T2', 'average_DT', 'Time_bnds', 'temp']

temp = rg.OpenMDArrayFromFullname("//temp")
[d.GetName() for d in temp.GetDimensions()]
#['Time', 'st_ocean', 'yt_ocean', 'xt_ocean']
[d.GetSize() for d in temp.GetDimensions()]
#[31, 51, 1500, 3600]

I don’t know why xarray can seemingly read netcdf from S3, but not from normal url? Maybe just something fsspec I need to understand?

I also noticed that if the file is local the array names at root level are like this:

['//xt_ocean', '//yt_ocean', '//st_ocean', '//Time', '//nv', '//st_edges_ocean', '//average_T1', '//average_T2', '//average_DT', '//Time_bnds', '//temp']

gdalraster::gdal_version()

[1] "GDAL 3.11.0dev-dd6009a0eb, released 2018/99/99"
[2] "3110000"                                       
[3] "20189999"                                      
[4] "3.11.0dev-dd6009a0eb"                          

devtools::session_info()

─ Session info ───────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.4.2 (2024-10-31)
 os       Ubuntu 20.04.6 LTS
 system   x86_64, linux-gnu
 ui       X11
 language (EN)
 collate  en_AU.UTF-8
 ctype    en_AU.UTF-8
 tz       Australia/Hobart
 date     2025-03-12
 pandoc   3.2 @ /usr/lib/rstudio-server/bin/quarto/bin/tools/x86_64/ (via rmarkdown)

─ Packages ───────────────────────────────────────────────────────────────────
 package     * version     date (UTC) lib source
 bit           4.5.0.1     2024-12-03 [2] CRAN (R 4.4.2)
 bit64         4.6.0-1     2025-01-16 [2] CRAN (R 4.4.2)
 cachem        1.1.0       2024-05-16 [2] CRAN (R 4.4.1)
 class         7.3-23      2025-01-01 [4] CRAN (R 4.4.2)
 classInt      0.4-11      2025-01-08 [2] CRAN (R 4.4.2)
 cli           3.6.4       2025-02-13 [2] CRAN (R 4.4.2)
 codetools     0.2-20      2024-03-31 [4] CRAN (R 4.4.1)
 curl          6.2.1       2025-02-19 [2] CRAN (R 4.4.2)
 DBI           1.2.3       2024-06-02 [2] CRAN (R 4.4.1)
 devtools      2.4.5       2022-10-11 [2] CRAN (R 4.4.1)
 digest        0.6.37      2024-08-19 [2] CRAN (R 4.4.1)
 e1071         1.7-16      2024-09-16 [2] CRAN (R 4.4.1)
 ellipsis      0.3.2       2021-04-29 [2] CRAN (R 4.4.1)
 evaluate      1.0.3       2025-01-10 [2] CRAN (R 4.4.2)
 fastmap       1.2.0       2024-05-15 [2] CRAN (R 4.4.1)
 fs            1.6.5.9000  2024-12-18 [2] Github (r-lib/fs@8d40528)
 gdalraster  * 1.12.0.9250 2025-03-10 [1] local
 generics      0.1.3       2022-07-05 [2] CRAN (R 4.4.1)
 glue          1.8.0       2024-09-30 [2] CRAN (R 4.4.1)
 here          1.0.1       2020-12-13 [2] CRAN (R 4.4.1)
 htmltools     0.5.8.1     2024-04-04 [2] CRAN (R 4.4.1)
 htmlwidgets   1.6.4       2023-12-06 [2] CRAN (R 4.4.1)
 httpuv        1.6.15      2024-03-26 [2] CRAN (R 4.4.1)
 httr2         1.1.0       2025-01-18 [2] CRAN (R 4.4.2)
 jsonlite      1.9.0       2025-02-19 [2] CRAN (R 4.4.2)
 KernSmooth    2.23-26     2025-01-01 [4] CRAN (R 4.4.2)
 knitr         1.49        2024-11-08 [2] CRAN (R 4.4.2)
 later         1.3.2       2023-12-06 [2] CRAN (R 4.4.1)
 lattice       0.22-6      2024-03-20 [4] CRAN (R 4.4.1)
 lifecycle     1.0.4       2023-11-07 [2] CRAN (R 4.4.1)
 lubridate     1.9.4       2024-12-08 [2] CRAN (R 4.4.2)
 magrittr      2.0.3       2022-03-30 [2] CRAN (R 4.4.1)
 Matrix        1.7-0       2024-04-26 [2] CRAN (R 4.4.1)
 memoise       2.0.1       2021-11-26 [2] CRAN (R 4.4.1)
 mime          0.12        2021-09-28 [2] CRAN (R 4.4.1)
 miniUI        0.1.1.1     2018-05-18 [2] CRAN (R 4.4.1)
 nanoarrow     0.6.0       2024-10-13 [2] CRAN (R 4.4.2)
 osmdata     * 0.2.5       2023-08-14 [2] CRAN (R 4.4.2)
 pkgbuild      1.4.4       2024-03-17 [2] CRAN (R 4.4.1)
 pkgload       1.4.0       2024-06-28 [2] CRAN (R 4.4.1)
 png           0.1-8       2022-11-29 [2] CRAN (R 4.4.1)
 profvis       0.4.0       2024-09-20 [2] CRAN (R 4.4.1)
 promises      1.3.0       2024-04-05 [2] CRAN (R 4.4.1)
 proxy         0.4-27      2022-06-09 [2] CRAN (R 4.4.1)
 purrr         1.0.4.9000  2025-02-19 [2] Github (tidyverse/purrr@9c8beb4)
 R6            2.6.1       2025-02-15 [2] CRAN (R 4.4.2)
 rappdirs      0.3.3       2021-01-31 [2] CRAN (R 4.4.1)
 Rcpp          1.0.14      2025-01-12 [2] CRAN (R 4.4.2)
 remotes       2.5.0       2024-03-17 [2] CRAN (R 4.4.1)
 reticulate  * 1.41.0      2025-02-24 [2] CRAN (R 4.4.2)
 rlang         1.1.5       2025-01-17 [2] CRAN (R 4.4.2)
 rmarkdown     2.29        2024-11-04 [2] CRAN (R 4.4.2)
 rprojroot     2.0.4       2023-11-05 [2] CRAN (R 4.4.1)
 rstudioapi    0.16.0      2024-03-24 [2] CRAN (R 4.4.1)
 sessioninfo   1.2.2       2021-12-06 [2] CRAN (R 4.4.1)
 sf          * 1.0-19      2024-11-05 [2] CRAN (R 4.4.2)
 shiny         1.9.1       2024-08-01 [2] CRAN (R 4.4.1)
 timechange    0.3.0       2024-01-18 [2] CRAN (R 4.4.1)
 units         0.8-5       2023-11-28 [2] CRAN (R 4.4.1)
 urlchecker    1.0.1       2021-11-30 [2] CRAN (R 4.4.1)
 usethis       3.0.0       2024-07-29 [2] CRAN (R 4.4.1)
 vctrs         0.6.5       2023-12-01 [2] CRAN (R 4.4.1)
 withr         3.0.2       2024-10-28 [2] CRAN (R 4.4.1)
 wk            0.9.4       2024-10-11 [2] CRAN (R 4.4.1)
 xfun          0.50        2025-01-07 [2] CRAN (R 4.4.2)
 xml2          1.3.6       2023-12-04 [2] CRAN (R 4.4.1)
 xtable        1.8-4       2019-04-21 [2] CRAN (R 4.4.1)
 yaml          2.3.10      2024-07-26 [2] CRAN (R 4.4.1)

 [1] /perm_storage/home/mdsumner/R/x86_64-pc-linux-gnu-library/4.4
 [2] /usr/local/lib/R/site-library
 [3] /usr/lib/R/site-library
 [4] /usr/lib/R/library

─ Python configuration ───────────────────────────────────────────────────────
 python:         /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf/bin/python3
 libpython:      /perm_storage/home/mdsumner/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/libpython3.11.so
 pythonhome:     /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf:/perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf
 virtualenv:     /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf/bin/activate_this.py
 version:        3.11.11 (main, Feb 12 2025, 14:51:05) [Clang 19.1.6 ]
 numpy:          /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf/lib/python3.11/site-packages/numpy
 numpy_version:  2.2.3
 osgeo:          /perm_storage/home/mdsumner/.cache/R/reticulate/uv/cache/archive-v0/twFzciVwGCsaiQXG6_cWf/lib/python3.11/site-packages/osgeo
 
 NOTE: Python version was forced by py_require()

──────────────────────────────────────────────────────────────────────────────

reticulate::py_require()

══════════════════════════ Python requirements ══════════════════════════

── Current requirements ─────────────────────────────────────────────────

 Python:   [No Python version specified. Will default to '3.11.11']
 Packages: numpy, gdal

── R package requests ───────────────────────────────────────────────────

R package  Python packages                           Python version      
reticulate numpy                                                         

Simple grid with four values

Take a grid, 2x2 with values 1,2,3,4 that sums to 10 and warp it to a new size.

## target
dm <- c(2, 2)

g1 <- matrix(c(1, 2, 3, 4), dm[2], dm[1])
e1 <- c(-1, 1, -1, 1)

library(vapour)
library(terra)

terra 1.8.29

## we need this so we can use MEM: via dsn::mem()
Sys.setenv(GDAL_MEM_ENABLE_OPEN = "YES")


## dsn::mem generates a Memory raster, 
## gdal_raster_data is the warper, here we get identity
sg <- gdal_raster_data(dsn::mem(g1, extent = e1, projection = "EPSG:4326"))

unique(sg[[1]])

[1] 1 2 3 4

sum(sg[[1]])

[1] 10

## spatialize for easy plotting
tor <- function(x) {
 
  dm <- attr(x, "dimension")[2:1]
  r <- terra::rast(terra::ext(attr(x, "extent")), ncols = dm[1], nrows = dm[2], crs = attr(x, "projection"), 
                   vals = x[[1]])
  r
}

r <- "sum"

##   (always use mem() "live" to avoid the garbage collector, and only for Float64 I'm afraid)


## now use the same extent but reduce pixel size
tg <- gdal_raster_data(dsn::mem(g1, extent = e1, projection = "EPSG:4326"), target_ext = e1,
                         target_dim = dm * 8, resample = r)

str(tg)

List of 1
 $ : num [1:256] 0.0156 0.0156 0.0156 0.0156 0.0156 ...
 - attr(*, "dimension")= num [1:2] 16 16
 - attr(*, "extent")= num [1:4] -1 1 -1 1
 - attr(*, "projection")= chr "GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AU"| __truncated__

So we have a lot more values as we resized from 2x2 to 16x16.

It still looks the same, but the overall quantity has been distributed.

plot(tor(tg), main = r)
## draw boundaries on
abline(v = vaster::x_corner(dm * 8, e1), h = vaster::y_corner(dm * 8, e1))

print(sum(tg[[1]]))

[1] 10

## our sum of 10 was distributed across 256 pixels
unique(tg[[1]])

[1] 0.015625 0.031250 0.046875 0.062500

Now a different example, an actual map projection change.

But, gawd … there’s a bug here. WIP

## now try reprojecting our unit 1x1 longlat grid to LAEA centred a few degrees east and north
crs <- "+proj=laea +lon_0=5 +lat_0=0"
## reproject the extent, we use a densified boundary to find the new extent
prex <- reproj::reproj_extent(c(-1, 1, -1, 1), crs, source = "EPSG:4326")
newg <- gdal_raster_data(dsn::mem(g1, extent = e1, projection = "EPSG:4326"), 
                         target_ext = prex, target_res = c(10000, 10000), target_crs = crs, resample = "sum", options = "-tap")
plot(tor(newg))

sum(newg[[1]])

[1] 10

And what about trying to return, of course this can only be approximate (I want to know how this compares to other tools.)

## what if we go the other way, 

newdm <- attr(newg, "dimension")
newext <- attr(newg, "extent")
x <- gdal_raster_data(dsn::mem(matrix(newg[[1]],newdm[2], byrow = TRUE),  extent = newext, projection = crs), 
                 target_dim = dm, target_ext = e1, target_crs = "EPSG:4326", resample = "sum")
plot(tor(x))

sum(x[[1]])

[1] 9.496864

Get raster data from online

We can read satellite imagery or elevation with a central point and a buffer (in metres).

library(ceramic)

Loading required package: terra

terra 1.7.23

pt <- cbind(147.3257, -42.8826)

(im <- cc_location(pt, buffer = c(15000, 25000)))

class       : SpatRaster 
dimensions  : 960, 576, 3  (nrow, ncol, nlyr)
resolution  : 52.08333, 52.08333  (x, y)
extent      : 16385222, 16415222, -5319119, -5269119  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
colors RGB  : 1, 2, 3 
names       : red, green, blue 
min values  :   0,     0,    0 
max values  : 255,   255,  252 

(el <- cc_elevation(pt, buffer = c(15000, 25000)))

class       : SpatRaster 
dimensions  : 960, 576, 1  (nrow, ncol, nlyr)
resolution  : 52.08333, 52.08333  (x, y)
extent      : 16385222, 16415222, -5319119, -5269119  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
name        :  lyr.1 
min value   :   -1.9 
max value   : 1264.9 

op <- par(mfcol = c(1, 2))
plot(el, legend = F); plot(im, add = TRUE); plot(el, legend = F, 
                                                 col = hcl.colors(64)[tail(seq_len(64), 45)])

par(op)

These raster objects are in terra ‘SpatRaster’ format (older ceramic used raster package).

These use the Mapbox ‘mapbox.satellite’ and ‘mapbox-terrain-rgb’ tile servers.

Get raster tiles from online

If we want the actual tiles, we can use the original get_tiles() function. (In older versions cc_location() and cc_elevation() would invoke get_tiles, but no longer).

pt <- cbind(147.3257, -42.8826)

imtiles <- get_tiles(pt, buffer = c(15000, 25000))

Preparing to download: 24 tiles at zoom = 12 from 
https://api.mapbox.com/v4/mapbox.satellite/

str(imtiles)

List of 3
 $ files : chr [1:24] "/perm_storage/home/mdsumner/.cache/.ceramic/api.mapbox.com/v4/mapbox.satellite/12/3722/2586.jpg" "/perm_storage/home/mdsumner/.cache/.ceramic/api.mapbox.com/v4/mapbox.satellite/12/3723/2586.jpg" "/perm_storage/home/mdsumner/.cache/.ceramic/api.mapbox.com/v4/mapbox.satellite/12/3724/2586.jpg" "/perm_storage/home/mdsumner/.cache/.ceramic/api.mapbox.com/v4/mapbox.satellite/12/3725/2586.jpg" ...
 $ tiles :List of 2
  ..$ tiles:'data.frame':   24 obs. of  2 variables:
  .. ..$ x: int [1:24] 3722 3723 3724 3725 3722 3723 3724 3725 3722 3723 ...
  .. ..$ y: int [1:24] 2586 2586 2586 2586 2587 2587 2587 2587 2588 2588 ...
  .. ..- attr(*, "out.attrs")=List of 2
  .. .. ..$ dim     : Named int [1:2] 4 6
  .. .. .. ..- attr(*, "names")= chr [1:2] "x" "y"
  .. .. ..$ dimnames:List of 2
  .. .. .. ..$ x: chr [1:4] "x=3722" "x=3723" "x=3724" "x=3725"
  .. .. .. ..$ y: chr [1:6] "y=2586" "y=2587" "y=2588" "y=2589" ...
  ..$ zoom : int 12
  ..- attr(*, "class")= chr "tile_grid"
 $ extent: num [1:4] 16385222 16415222 -5319119 -5269119

eltiles <- get_tiles(pt, buffer = c(15000, 25000))

Preparing to download: 24 tiles at zoom = 12 from 
https://api.mapbox.com/v4/mapbox.satellite/

head(gsub(ceramic::ceramic_cache(), "",  eltiles$files))

[1] "/api.mapbox.com/v4/mapbox.satellite/12/3722/2586.jpg"
[2] "/api.mapbox.com/v4/mapbox.satellite/12/3723/2586.jpg"
[3] "/api.mapbox.com/v4/mapbox.satellite/12/3724/2586.jpg"
[4] "/api.mapbox.com/v4/mapbox.satellite/12/3725/2586.jpg"
[5] "/api.mapbox.com/v4/mapbox.satellite/12/3722/2587.jpg"
[6] "/api.mapbox.com/v4/mapbox.satellite/12/3723/2587.jpg"

And to see what tiles we have we can materialize their footprint in wk::rct() form, this is way more efficient than having to use polygons.

zoomtiles <- ceramic::ceramic_tiles(imtiles$tiles$zoom)
## sub out the ones we just triggered
zoomtiles <- dplyr::filter(zoomtiles, fullname %in% imtiles$files)
rc <- ceramic::tiles_to_polygon(zoomtiles)
plot(rc)
plot(ext(im), add = TRUE)  ## see the image from above is not the tiles, but the buffer around a point

plot(read_tiles(pt, buffer = c(15000, 25000)))  ## but we can read thos tiles exactly

Preparing to download: 24 tiles at zoom = 12 from 
https://api.mapbox.com/v4/mapbox.satellite/

plot(rc, add = TRUE, border = "white")

Diverse query inputs

Finally, we can use a point and buffer to get imagery, or we can use a spatial object, currently supported are objects from {geos}, {wk}, {terra}, {sf}, {sp}, {raster}, and {stars}.

Here’s an example.

wkarea <- wk::rct(xmin = 5e6, ymin = -2e6, xmax = 2e6, ymax = 3e6, crs = "+proj=laea +lon_0=25 +lat_0=31")
im2 <- cc_location(wkarea)
el2 <- cc_elevation(wkarea)
op <- par(mfcol = c(1, 2))
plot(el2, legend = F); plot(im2, add = TRUE); plot(el2, col = grey.colors(128), legend = FALSE)

par(op)

(Please note that the result is still in Mercator, the query can be in any projection but we’re not matching that here, only its extent - ceramic version 0.8.0 is merely a stepping stone to some of the things we can do better with GDAL).

Previous versions of cc_location() and cc_elevation() were stuck with only ‘zoom’ and ‘max_tiles’ arguments, these make sense when you restrict exactly to the available tiles but when you just want a given area and a resolution, ‘dimension’ is more appropriate.

By default, the dimension is chosen relative to the graphics device. But, we can also specify exactly what we want. (Use zero for one of the dimensions to let the system figure out an appropriate size for the query you have).

cc_location(wkarea)

class       : SpatRaster 
dimensions  : 960, 776, 3  (nrow, ncol, nlyr)
resolution  : 7094.906, 7094.906  (x, y)
extent      : 4203326, 9708973, 105375.6, 6916485  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
colors RGB  : 1, 2, 3 
names       : red, green, blue 
min values  :   0,     0,    0 
max values  : 255,   255,  255 

cc_location(wkarea, dimension = c(400, 700))

class       : SpatRaster 
dimensions  : 700, 400, 3  (nrow, ncol, nlyr)
resolution  : 13766.3, 9730.157  (x, y)
extent      : 4203326, 9709847, 105375.6, 6916485  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
colors RGB  : 1, 2, 3 
names       : red, green, blue 
min values  :   0,     2,    0 
max values  : 255,   255,  255 

cc_location(wkarea, dimension = c(1024, 0))

class       : SpatRaster 
dimensions  : 1267, 1024, 3  (nrow, ncol, nlyr)
resolution  : 5377.463, 5377.463  (x, y)
extent      : 4203326, 9709847, 103240.4, 6916485  (xmin, xmax, ymin, ymax)
coord. ref. : WGS 84 / Pseudo-Mercator (EPSG:3857) 
source(s)   : memory
colors RGB  : 1, 2, 3 
names       : red, green, blue 
min values  :   0,     1,    0 
max values  : 255,   255,  255 

Some other improvements

• it’s faster, for loading reads data from image servers directly with the GDAL warper API
• we have separation of tile downloading and interaction from raster loading via GDAL
• we don’t materialize tiles as polygons, we have the tile description, or the compact rct representation
• new functions read_tiles(), unpack_rgb(),

See the full list of changes here: https://hypertidy.github.io/ceramic/news/index.html.

ceramic is my final CRAN package that had direct dependencies on rgdal, rgeos, or maptools - in some ways it was the most challenging for me and had loomed as a problem for some time. But, I’ve progressed my own tool kit well in the time and I learnt a lot with this update.

A future version will probably make the separation (are we tiles, or are we loading raster?) more complete, at any rate there are better ways to organize these things now. There are data sources, data readers and writers, data structures, and algorithms. This work is part of a family of tools that aims to make that orthogonality real and accessible.

Degenerate Rectilinear Coordinates

I have taken to saying this phrase to describe a particular situation in array handling. I get confused responses, or normally just no response or feedback on the term itself. I think it’s extremely important though. Python communities raced up to where R was a few years back and just sped on by without any nod to what R folks had learned, I think that’s a huge failure and I take it personally.

Here’s what it is. Let’s say we have a pretty low resolution grid of the world, we break up the entire range of longitude and latitude into intervals, 360 for meridians and 180 for parallels.

(This is neat because in whole number terms, this is exactly the right number).

So, define a grid, we make it a farily typical 1-degree-per-pixel grid of the entire globe flattened unceremoniously from an
angular coordinate system “longitude and latitude” to one where we just plot those numbers in x,y on a flat plane.

## define a grid in -180,180 -90,90 (360x180)
dm <- c(360, 180)
xlim <- c(-180, 180)
ylim <- c(-90, 90)

## create a grid
dat <- matrix(c(0, 1, -1, 0), 2L, 2L)[rep(1:2, each = 180), rep(1:2, each = 90)]
## we don't get much from this because 0,1 0,1
## and the world isn't 0,360 0,180 anyway (that would have been handy!)
#image(dat)


xs <- seq(xlim[1], xlim[2], length.out = dm[1] + 1)
ys <- seq(ylim[1], ylim[2], length.out = dm[2] + 1)
image(xs, ys, dat, asp = 1)
maps::map(add = TRUE)

What happened there? Well, dat is a matrix, a 2D array in R. It has 360 rows and 180 columns (but yes, we are treating the world map as if it had 360 columns, 180 rows - but that is not this blogpost). Just don’t question it right now, but be assured we are treating the world as having 360 unique whole number longitudes, and 180 unique whole number latitudes.

We can image() that, by which I mean draw a pixel map on the screen of all the values in the matrix as if they were a field of little rectangles.

image(dat)

But, that’s very boring because we don’t have any idea where anything is in 0,1 0,1 space.

What if we use xlim and ylim, these are plot() arguments in R (and exist in image() too).

image(dat, xlim = xlim, ylim = ylim)

Ok that was a trick, our matrix is drawn as a tiny dot right at 0,0, because xlim/ylim is about the plot not the data.

No more tricks, image() has 3 arguments x,y,z for normal usage.

image(x = xs, y = ys, z = dat, asp = 1)

Why does that work? We essentially have an x coordinate and a y coordinate for every pixel edge. We have one for the left side of the leftmost pixel column, and the second one is the right side of the left most pixel, and so on. We used dm[1] + 1 and dm[2] + 1 up there exactly because we wanted xlim and ylim to be the very extremes of the data.

Cool, explained - a matrix has a coordinate array for each side, we plot it like a map with those. Done.

We are so so so not done.

I’m not here to explain how image() works in R, but that’s kind of necessary because I want people who don’t use R to also read this.

What is Rectilinear?

Rectilinear means varying in one dimension. When applied to axis coordinates it just means you have a position along the axis for each step. These can vary in how big each step is, and that is exactly when we say it’s rectilinear, because otherwise it’s regular.

Our ys above are a regular list of coordinates, each one is exactly the same distance apart (distance in the frame we are using, forget about the actual Earth).

op <- par(mfrow = c(2, 1))
plot(ys)
range(diff(ys))

[1] 1 1

plot(diff(ys))

See? They just plod along with exactly the same delta between each, in this case the step size is 1 but it could be anything.

What do you mean Degenerate?

Isn’t this a bad word, pejorative? Well no, it means something very specific.

It means even though there is a quite a lot of data, the information is confused or confounded. The required information could be expressed very compactly, but it’s actually obscured by the details.

Tiles

Tiles are a way of organizing rasters, a tile may be 256x256, 512x512 (typically powers of 2 for sensible reasons), or they may be of higher dimensions 256x256x8 - this is really a private detail for a storage format, such as a file. The data values stored in a tile are a nother matter, these are like variables - we might have 1 variable (say elevation) stored in a double floating point value, or we might have 3 variables of byte values for storing an RGB image. These are independent concepts to the size of the tile, and strictly there may be no variables at all, just the abstract idea of the tiling.

Another kind of block, or tile is a the scanline of a raster. Imagine reading each row of the raster, from the top row to the bottom. Each line is its own kind of degenerate tile. We can read from a raster this way no matter what its internal tiling is, just software might be opening several tiles and reading just one line from each. So, best if we match our “tile attack” to the native structure of the data.

Here we find a large, tiled raster, obtain its internal tiling, and use that to update a copy of the same file.

Please note how we copy the source file, then write to that. There’s probably other software better suited to doing this atm, this post simply aims to air the topic a little in an R context.

Here we use only temporary files that we download, and copy as needed. In practice, you should set this up to work across different physical disks, and for better workflow we need the ability to open an empty file to write to (WIP).

raster_url <- "ftp://ftp.data.pgc.umn.edu/elev/dem/setsm/REMA/mosaic/v1.1/200m/REMA_200m_dem.tif"

readfrom_file <- tempfile(pattern = "readfrom", fileext = ".tif")
writeto_file <- tempfile(pattern = "writeto", fileext = ".tif")
## file size is 1.3Gb
curl::curl_download(raster_url, readfrom_file)

## this now makes a copy of the 1.3Gb file, so we have two of them
fs::file_copy(readfrom_file, writeto_file)

Now we want the tiling

info <- vapour::vapour_raster_info(readfrom_file)
tiling <- list(dimension = info$dimXY, tiles = info$tilesXY)
fac <- 100
fac * tiling$tiles
if (tiling$tiles[2] == 1) {
  ## let's take fac scanlines at a time
  tiling$tiles[2] <- fac
}
calc_steps <- function(dimension, tiles) {
  bounds_x <- seq(0, dimension, by = tiles)
  steps_x <-  rep(tiles, length.out = length(bounds_x)-1)
  dangle_x <- sum(steps_x) -dimension
  if (dangle_x > 0) steps_x[length(steps_x)] <- steps_x[length(steps_x)] - dangle_x
  
  list(head(bounds_x, -1), steps_x)
}
x_step <- calc_steps(tiling$dimension[1], tiling$tiles[1])
y_step <- calc_steps(tiling$dimension[2], tiling$tiles[2])
y_step
system.time({
for (i in seq_along(x_step[[1]])) {
  startx <- x_step[[1]][i]
  countx <- x_step[[2]][i]
  for (j in seq_along(y_step[[1]])) {
  starty <- y_step[[1]][j]
  county <- y_step[[2]][j]
    
  ## now read
  offset <- c(startx, starty)
  dimension <- c(countx, county)
  vals <- vapour:::vapour_read_raster_block(readfrom_file, offset = offset, dimension = dimension, band_output_type = info$datatype, band = 1)[[1L]]
  
  ## do something to the values
  if (any(na.omit(vals) > info$nodata_value)) {
    vals <- vals * -1
    vapour:::vapour_write_raster_block(writeto_file, vals, offset, dimension, band = 1, overwrite = TRUE)
  }
  }
}

The GDAL C++ API warping library

GDAL is complex. It deals with very many different formats, and has many tools. In terms of this post, it has very low-level development facilities, written in C++. This means you can usually go as deep as you need into a geospatial problem by writing C++ code against the library directly. Often you don’t need C++ and can use Python, and sometimes you can use R depending on what is exposed there. Sometimes you need to modify GDAL itself, and you can do anything you want then(!), which is how open source is supposed to work.

Key feature of the gdalwarp_lib.cpp GDALWarp() C++ function, versus the GDALWarpDirect(), GDALWarpIndirect() and the lower level GDALWarpMulti() and GDALWarpImage() functions.

When we use gdalwarp_lib, we get handling of multiple-zoom level sources without intervention on our own.

Most use cases I see of the warper facilities are for whole-sale conversion of large datasets, multiple input files converted to a another projection in a large output file or series of tiles.

The warper is also a generalized RasterIO

(rasterio is a famous python package for using GDAL’s raster facilities, it’s not what we mean here).

RasterIO is a C++ function in the GDAL library, its job is to take a source data set (i.e. a path to GeoTIFF) and to provide you with a window of pixels from that source. A window can be a subset, the entire raster, or a resampling (i.e. fewer pixels than native) of either the entire or a subset of the raster.

It’s really, cool and using it looks like this:

err = rasterBand->RasterIO(GF_Read, 
                           Xoffset, Yoffset, 
                           nXSize, nYSize,
                           &double_scanline[0], 
                           outXSize, outYSize, 
                           GDT_Float64,
                           0, 0, &psExtraArg);

There is a lot going on in that function call, but in short the key parts are (these are my argument-variable-names):

• rasterBand->RasterIO() is the way we read a pixel values, we have a raster band and we call the member function RasterIO()
• Xoffset, Yoffset is the first column and row we should consider for reading (0,0 if we start at the top left corner)
• nXSize, nYSize
• outXSize, outYSize is the dimension of the window we get out

That last bits, the two kinds of Size is the magic, we can ask to start at a particular row,column and read out a given number of pixels in x and y. But, not only that we can specifiy where to end in the source. If nXSize and outXSize are not equal in value then we have asked for a resampling of the source “only read every nXSize / outXSize values in the x direction. If they are equal, just read every one. Note also that we might ask for an outX/YSize that is larger than the source nXSize/nYSize - and this would give us a resampling to higher resolution. This is really where the resampling algorithm comes in. ‘Nearest neighbour’ would give us copies of pixels, ‘Bilinear’ and interpolation between source pixels for new pixels in between. Other algorithms include ‘Cubic’, ‘Lanczos’, ‘Average’, ‘Sum’.

This is the key behind the fast and lazy reads provided by the terra and sf packages in R, this was originally available also in rgdal and raster made heavy use of it.

But, what is the offset and the size? We really want to think in geographic coordinates, and this is exactly what crop() does for example. We get a discretized crop, not an exact one because we only get to read by this raster-based mechanism - we are bound to the size and alignment of the source pixels.

Under the hood, functions like crop() do the following:

offsets/scale vs. xlim,ylim

The other arguments in RasterIO.

• GF_Read controls the mode we are into (read or write or update)
• GDT_Float64 controls the type of data we get out (64 bit doubles here, GDAL will auto-convert if the source is different type)
• 0, 0, &psExtraArg are further details we won’t discuss (though, the way resampling is done is controlled in the extra args options).

What are its limitations:

• only one source at a time, you can’t resample from multiple rasters at once (we are using offset/size indexing so the rasters must all be the same shape, you can read multiple bands)

What is interesting about RasterIO vs. Warping is that we would never ever use warping for pure graphics. It doesn’t make sens

Enter the WuTang Warper!

Doing discretized extent raster math is boring. With the warper we don’t need to do it.

This is because the warper is for changing projections, this is an entire re-modelling of raster data. (in the 2000s it was possible to do but still quite slow and problematic, hence you have very slowly changing standards and perceptions of what is possible/normal/appropriate in software comunities etc.).

This is a global longlat data set, but we want a local projection so we adopt one by finding its specification.

If we were using RasterIO we’d have to do this kind of math to run the index function.

But with the warper, we can use an extent (but we still need to align it, or we won’t get what the RasterIO facility would faithfully give).

rgl

The rgl package has long provided interactive 3D graphics for R. The neat thing for me about 3D graphics is the requirement for mesh forms of data, and the fact that meshes are extremely useful for very many tasks. When we plot data in 3D we necessarily have to convert the usual spatial types into mesh forms. You can see me discuss that in more detail in this talk.

The mesh3d format

Here is an example of a mesh3d object, it stores two polygonal areas in a form ready for 3D graphics.

mesh0 <- structure(list(vb = structure(c(0, 0, 0, 1, 0, 1, 0, 1, 0.75, 
                                1, 0, 1, 1, 0.8, 0, 1, 0.5, 0.7, 0, 1, 0.8, 0.6, 0, 1, 0.69, 
                                0, 0, 1, 0.2, 0.2, 0, 1, 0.5, 0.2, 0, 1, 0.5, 0.4, 0, 1, 0.3, 
                                0.6, 0, 1, 0.2, 0.4, 0, 1, 1.1, 0.63, 0, 1, 1.23, 0.3, 0, 1), .Dim = c(4L, 14L)), 
               it = structure(c(1L, 8L, 12L, 9L, 8L, 1L, 7L, 6L, 5L, 
                                5L, 4L, 3L, 2L, 1L, 12L, 9L, 1L, 7L, 5L, 3L, 2L, 2L, 12L, 11L, 
                                10L, 9L, 7L, 5L, 2L, 11L, 10L, 7L, 5L, 5L, 11L, 10L, 6L, 7L, 
                                14L, 14L, 13L, 6L), .Dim = c(3L, 14L)), 
               primitivetype = "triangle", 
               material = list(), 
               normals = NULL, 
               texcoords = NULL), 
               class = c("mesh3d", "shape3d"))


str(mesh0)

List of 6
 $ vb           : num [1:4, 1:14] 0 0 0 1 0 1 0 1 0.75 1 ...
 $ it           : int [1:3, 1:14] 1 8 12 9 8 1 7 6 5 5 ...
 $ primitivetype: chr "triangle"
 $ material     : list()
 $ normals      : NULL
 $ texcoords    : NULL
 - attr(*, "class")= chr [1:2] "mesh3d" "shape3d"

(It’s not obvious about the polygons, please bear with me).

The following characterizes the structure.

• two matrix arrays vb and it
• vb has 4 rows and 14 columns, and contains floating point numbers
• it has 3 rows and 14 columns, and contains integers (starting at 1)
• a primitivetype which is “triangle”
• an empty list of material propertes (this is the missing link for the polygons)
• a NULL value for normals and texcoords, these won’t be discussed further (but see ?quadmesh::quadmesh for texture coordinates from spatial)
• a class, this object is a mesh3d and inherits from shape3d

The vb array is the vertices, these are the corner coordinates of the elements of the mesh.

plot(t(mesh0$vb), main = "t(vb) - vertices", xlab = "X", ylab = "Y")

The elements of this mesh are triangles, and these are specified by the index array it. Elements of a mesh are called primitives, hence the primitivetype here.

plot(t(mesh0$vb), main = "t(vb[, it]) - primitives", xlab = "X", ylab = "Y")
polygon(t(mesh0$vb[, rbind(mesh0$it, NA)]), col = rgb(0.6, 0.6, 0.6, 0.5))

Transpose

These matrix arrays are transpose the way we usually use them in R, for now just remember that you must t()ranspose them for normal plotting, e.g. plot(t(mesh0$vb[1:2, ])) will give the expected scatter plot of the vertices. The reason these arrays are transpose is because each coordinate value is then contiguous in memory, each Y value is right next to its counterpart X, and Z (and W), and vb[it, ] provides a flat vector of XYZW values in a continuous block - this is a very important efficiency, and help explains why computer graphics use elements in a mesh form like this.

Colours

Unsurprisingly, if we set the material property to a constant we get a constant colour.

widgetfun <- function() {
  view3d(0, phi = 8)
  rglwidget()
}
mesh0$material$color <- "red"
library(rgl)
clear3d()
shade3d(mesh0, lit = FALSE); 
widgetfun()

In the usual R way our singleton colour value is magically recycled across every part of the shape, and it’s all red. But, is it recycled by vertices or by primitive? Until recently it was only possible to tell by trying (or reading the source code).

Here I think it’s easy to see that the two colours are specified at the vertices, and they bleed across each triangle accordingly. We also get a warning that the behaviour has recently changed.

clear3d()
mesh0$material$color <- c("firebrick", "black")
material3d(lit = FALSE)
shade3d(mesh0)
widgetfun()

The default is to meshColor = "vertices", so let’s specify faces.

clear3d()
mesh0$material$color <- c("firebrick", "dodgerblue")
material3d(lit = FALSE)
shade3d(mesh0, meshColor = "faces")
widgetfun()

Sometimes we get neighbouring triangles with the same colour, so let’s also add the edges.

mesh0$vb[3, ] <- 0.01  ## vertical bias avoids z-fighting
## material properties here override the recycling of internal colours
## onto edges
wire3d(mesh0, lwd = 5, color = "black")
widgetfun()

If we go a bit further we can see the original arrangement for this shape, two individual polygons that share a single edge.

This only works because I happen to know how this was created, and I know how this control of behaviour occurs in new rgl.

There are 12 triangles in the first polygon, and 2 in the second. (The original polygons can be seen here (left panel)).

clear3d()
mesh0$material$color <- rep(c("firebrick", "dodgerblue"), c(12, 2))
shade3d(mesh0, meshColor = "faces", lit = FALSE)
widgetfun()

If we treat the colours as applying to each vertex, then we needed to propagate it to each vertex around each face (triangle), and this is what rgl now calls legacy behaviour.

clear3d()
mesh0$material$color <- rep(rep(c("firebrick", "dodgerblue"), c(12, 2)), each = 3)
shade3d(mesh0, meshColor = "legacy", lit = FALSE)
widgetfun()

We cannot recreate this effect with meshColor = "vertices", because each of our vertices is actually unique. (It could be done by making the vb array every repeated vertex, and updating the index array but I can’t summon this up atm).

clear3d()
mesh0$material$color <- rep_len(c("firebrick", "dodgerblue"), length.out = ncol(mesh0$vb))
shade3d(mesh0, meshColor = "vertices", lit = FALSE)
widgetfun()

Primitives

The other kind of element supported by mesh3d is a quad, specified by an ib array with 4 rows (ib versus it, 4 vertices versus 3) and the primitivetype = "quad".

The it values are an index into, i.e. the column number of the vertex array. The vertices, or coordinates, are stored by column in this structure, whereas normally we would store a coordinate per row.

When I first explored mesh3d I was looking at a quad type mesh - and I was completely confused. Both vb and ib had four rows, and so while I understood that a quad must have 4 vertices (4 index values for every primitive), I did not understand why the vertices also had four rows.

(There are other kinds of primitives in common use are edge, point, tetrahedron - but rgl has no formal class for these - in practice the edge type is referred to as segment in rgl, and tetrahedra are approximated by enclosing their shape with triangles).

Why does the vertex array have 4 rows?

All mesh3d objects have a vb array, and it always includes 4 rows.

The reason there are 4 rows in the vertex array is that these are homogeneous coordinates which …

are ubiquitous in computer graphics because they allow common vector operations such as translation, rotation, scaling and perspective projection to be represented as a matrix by which the vector is multiplied

… yeah. For our purposes just think

X, Y, Z in the usual sense and set W = 1.

(Do not set W = 0 because your data will vanish to infinity when plotted with rgl, which is what those math folks are saying more or less).

QUADS

Now let’s get a quad type mesh from the real world.

## remotes::install_github("hypertidy/ceramic")
library(ceramic)
topo <- cc_elevation(raster::extent(-72, -69, -34, -32), zoom = 6)

Preparing to download: 1 tiles at zoom = 6 from 
https://api.mapbox.com/v4/mapbox.terrain-rgb/

qm <- quadmesh::quadmesh(topo)

str(qm)

List of 8
 $ vb             : num [1:4, 1:60225] -8015493 -3761925 0 1 -8014270 ...
  ..- attr(*, "dimnames")=List of 2
  .. ..$ : chr [1:4] "x" "y" "z" "1"
  .. ..$ : NULL
 $ ib             : int [1:4, 1:59732] 1 2 277 276 2 3 278 277 3 4 ...
 $ primitivetype  : chr "quad"
 $ material       : list()
 $ normals        : NULL
 $ texcoords      : NULL
 $ raster_metadata:List of 7
  ..$ xmn  : num -8015493
  ..$ xmx  : num -7680393
  ..$ ymn  : num -4028537
  ..$ ymx  : num -3761925
  ..$ ncols: int 274
  ..$ nrows: int 218
  ..$ crs  : chr "+proj=merc +lon_0=0 +k=1 +x_0=0 +y_0=0 +R=6378137 +units=m +no_defs"
 $ crs            : chr "+proj=merc +lon_0=0 +k=1 +x_0=0 +y_0=0 +R=6378137 +units=m +no_defs"
 - attr(*, "class")= chr [1:3] "quadmesh" "mesh3d" "shape3d"

This topographic raster from near Santiago is now a mesh3d subclassed to quadmesh. This adds two properties raster_metadata and crs, which under limited conditions allows reconstruction of the original raster data. To drop back to a generic mesh3d the easiest is to reproject the data.

##remotes::install_github("hypertidy/reproj")
library(reproj)
qm_ll <- reproj(qm, "+proj=longlat +datum=WGS84")

Warning in reproj.quadmesh(qm, "+proj=longlat +datum=WGS84"): quadmesh raster
information cannot be preserved after reprojection, dropping to mesh3d class

This is a lossless reprojection, as it is equivalent to sf::sf_project(t(qm$vb[1:2, ]), from = qm$crs, to = "+proj=longlat +datum=WGS84") or with rgdal::project(, qm$crs, inv = TRUE).

We can plot this in the usual way with rgl, or see upcoming features in the mapdeck package.

clear3d()
shade3d(qm_ll, lit = TRUE, col = "grey")
aspect3d(1, 1, 0.1); 
view3d(0, phi = -60)
rglwidget()