Skip to main content

hypertidy-blog

VRT is: Closure, Names, Placement, Assertion, Deferral, Emission

vrt
gdal
virtualization
zarr

GDAL VRT has three features in one XML file: indirection, metadata edits, and operation chains. There is a deeper design though, in six properties.
Nov 6, 2026
Michael Sumner

GDAL past the familiar view: a field guide

GDAL
python
xarray
warping
VRT
multidimensional

What the rasterio window onto GDAL leaves out: two data models, dataset names as a language, VRT as a recipe, a warper whose defaults decide results, and a checklist for reviewing a plan that lowers to GDAL.
Oct 30, 2026
Michael Sumner

Bytes in place

virtualization
zarr
tables
gdal
news

A virtual store is not a format. It is a table of byte ranges plus a plan, and every format is a projection of that table. A position on virtual stores, catalogs and the path from research data to production, with a snapshot of where the work stands in October 2026.
Oct 23, 2026
Michael Sumner

The recipe is the dataset

GDAL
VRT
COG
REMA
Antarctica
browser

A 3 MB VRT makes 5,800 Antarctic COGs behave like one dataset, in GDAL and in a browser with no server, at every scale from a continent to a valley.
Oct 14, 2026
Michael Sumner

Where the line falls: tensors, tables and four layers

zarr
arrays
tables
stac
news

Arguments about tensors versus tables assume one line through the ecosystem. Storage, indexing, topology and rendering each draw it in a different place, so the same dataset is a tensor from one layer and a table from another.
Oct 13, 2026
Michael Sumner

Warp and cell: two worlds that were never separate

spatial
zarr
webgl

Two browser viewers, two assumptions about what a picture of data is made of, and why the gaps between them fall where they do.
Oct 6, 2026
Michael Sumner

Small contracts, long lives: altarr and the shape of lazy arrays in R

altarr
zarr
gdal
r-packages
news

One small fetch contract under R’s own indexing ties together zaro, tidync, ndr, gdalraster, GDAL7 and the browser tools. What that says about where Python’s array stack is heading, and about sustaining R itself.
Oct 5, 2026
Michael Sumner

Paired bars. Region 401 x 301 x 31: x[i, j, k] element at a time 4.38 s and 192 fetch calls; altarr_extract planned 0.21 s and 1 fetch call. 2000 scattered points: loop of x[i, j, k] 6.01 s and 287 fetch calls; x[cbind(i, j, k)] planned 0.15 s and 1 fetch call.

What R does today with an ALTREP that has a dim and no class

code
news
zarr
altarr
r-packages

Base R indexing can already drive planned reads of remote chunked arrays. One form does it today; a small hook would bring the other.
Oct 1, 2026
Michael Sumner

A mapview for modern R: Arrow in, any CRS out

mapview
arrow
deck.gl
spatial
R

A design sketch for a modern mapview: GeoArrow as the data plane, a renderer-agnostic scene spec, and a path that starts on deck.gl.
Sep 30, 2026
Michael Sumner

Distance, perimeter and area in sf and terra: which number do you get?

In May 2025 a question on R-sig-Geo asked why sf::st_perimeter() gave a different answer from sf::st_length(sf::st_boundary()) for the same small land parcel in France, with…
Sep 24, 2026
Michael Sumner

The road to +proj=spilhaus

projections
PROJ
GDAL
news

Spilhaus went viral in 2018, shipped in ArcGIS in 2020, and landed in PROJ in 2025. That last step may not have been noticed widely enough. There’s a picture of how our software landscape actually works.
Sep 20, 2026
Michael Sumner

R speaks Zax SQL: dbplyr and the tensor-as-table model

code
news
zarr
xrdbi
dbplyr
Earthmover announced Zax and Zax-SQL this week: a new query engine for chunked multidimensional arrays, fronted by SQL over the PostgreSQL wire protocol and Arrow Flight…
Sep 17, 2026
Michael Sumner

rasterize with rust in r

code
news
The controlledburn-rs project is a Rust library to rasterize abstractly.
Sep 8, 2026
Michael Sumner

Writing controlledburn output in parallel

controlledburn
rasterization
Antarctica
GDAL
Zarr
mirai

The serial COG write loop from the last post, replaced by 24 workers writing disjoint Zarr chunks. The 31-billion-cell grid on disk in 28 seconds of 9.5 MB.
Sep 3, 2026
Michael Sumner

Rasterizing ice-free Antarctica

controlledburn
rasterization
Antarctica
GDAL

Sparse rasterization of 25,954 Antarctic rock outcrop polygons onto a 31-billion-cell grid — in under a second.
Aug 19, 2026
Michael Sumner

s2 guide

s2
wk
spatial
rgl

A field guide to the s2 R package: what the three type families are, how to move between them, how to get raster and vector data into cell groupings, how to plot on a real sphere with rgl, and where geodesic edge densification fits.
Aug 10, 2026
Michael Sumner

Icechunk is coming - versioned cloud-native Zarr, GDAL, and R

code
news
Icechunk is a transactional storage engine for Zarr and provides git-style versioning for large scientific datasets. This post is about Icechunk from a GDAL perspective and…
Jun 26, 2026
Michael Sumner

Grids: regular, rectilinear, curvilinear

gdal
raster
netcdf
projections
theory

A grid taxonomy rooted in coordinate storage, not CRS. A companion to Detecting hidden grids in curvilinear coordinates.
May 28, 2026
Michael Sumner

R, GDAL, Zarr, and the Missing Recipe Format for Raster Datasets

code
news
r-packages
python
zarr
gdal
netcdf
The virtual-Zarr format that lets xarray describe 16,000 files in a tiny store doesn’t exist in R yet - but the pieces are in place, this post tries to show where they fit…
May 27, 2026
Michael Sumner

Check it out: gdalraster

r-packages
gdal
raster
vector
The gdalraster package provides a comprehensive interface to the GDAL library for R. It handles raster I/O, vector I/O, the virtual filesystem, coordinate transformation…
Mar 24, 2026
Michael Sumner

Check it out: reproj

r-packages
projections
The reproj package provides a single generic function, reproj(), for transforming coordinates between map projections. It works with matrices, data frames, and anything else…
Mar 24, 2026
Michael Sumner

Check it out: geos

r-packages
vector
The geos package provides a direct R interface to the GEOS library. It operates on a geos_geometry vector type and speaks wk natively — any wk-handleable geometry can be…
Mar 24, 2026
Michael Sumner

Check it out: proj_trans_create and wk_transform

r-packages
projections
The PROJ R package provides proj_trans_create(), which creates a reusable coordinate transformation object that plugs directly into wk’s wk_transform(). This is the…
Mar 24, 2026
Michael Sumner

Check it out: wk

r-packages
vector
The wk package is a foundation for geometry handling in R. It’s a lightweight, dependency-free geometry column with CRS support, vctrs integration, and a handler-based…
Mar 24, 2026
Michael Sumner

What’s on geodata.ucdavis.edu and how to access it

code
news
raster
vector
gdal
spatial
The R package geodata provides functions for downloading climate, elevation, soil, crop, species occurrence, and administrative boundary data. All of its raster and vector…
Mar 17, 2026
Michael Sumner

Map Reprojection and Map Edges

code
projections
maps
spatial
A recent issue on the tmap package reports a white patch appearing at the top of a map rendered in an Albers Equal Area projection. The basemap tiles don’t fill the whole…
Feb 20, 2026
Michael Sumner

Detecting hidden grids in curvilinear coordinates

gdal
raster
netcdf
projections
A lot of gridded data arrives with two-dimensional longitude and latitude arrays: one value per pixel, stored explicitly. This is the standard representation for curvilinear…
Feb 7, 2026
Michael Sumner

R vs JavaScript Datetime Parsing

code
news
Datetime parsing across R and JavaScript has subtle but dangerous differences. Both languages make assumptions that can silently produce incorrect results, and critically…
Dec 30, 2025
Michael Sumner

Raster logic without pixels

raster
vaster
Every geospatial package that works with grids has raster logic inside. The old {raster} package in R established powerful abstraction functions and now {terra} includes an…
Dec 21, 2025
Michael Sumner

Coordinates broken in NetCDF

netcdf
gdal
There’s a NetCDF file at this URL:
Sep 4, 2025
Michael Sumner

GDAL multidim and cloud-ready ZARR

gdal
zarr
netcdf
I have been working on a better understanding of the GDAL multidimensional model and to do that really needs a closer look at the GDAL API itself.
Jun 30, 2025
Michael Sumner

Conservative regridding with GDAL (?)

gdal
raster
Can GDAL do conservative re-gridding? For cases of regular grid to regular grid, yes I think it can.
Dec 11, 2024
Michael Sumner

IDEA - data and software

overview
Dec 5, 2024
Michael Sumner

Plot at native resolution, with R

raster
gdal
Plot native, something I should have done long ago.
Dec 4, 2024
Michael D. Sumner

GDAL and image tiles, the {ceramic} package

gdal
r-packages
ceramic
A new version of {ceramic} is now on CRAN, version 0.8.0.
Apr 22, 2023
Michael D. Sumner

Degenerate Rectilinear (WIP)

netcdf
raster
There are a lot of problems when it comes to array representation in data formats and software. There is a whole family of complex issues with sometimes subtle or even…
Dec 9, 2022
Michael D. Sumner

GDAL raster read/write by blocks

gdal
raster
A block is another word for a tile, a tile in a small-ish raster window within a larger raster. Tiles can be very clever, such as 256x256, they make a nice way to organize…
May 4, 2022
Michael D. Sumner

GDAL warper with R

news
code
There’s several meanings floating around when you say the “GDAL warper”. It can mean
Apr 25, 2022
Michael D. Sumner

mesh3d - recent changes in rgl workhorse format

mesh
rgl
PLEASE NOTE (April 2022): this post has been migrated from an old site, and some details may have changed. There might an update to this post to reflect the rgl package as…
May 29, 2019
Michael D. Sumner

GDAL in R

gdal
overview
manifold
gis
automation
R
For some time I have used GDAL as a standard tool in my kit, I was introduced to the concept by the rgdal package authors and it slowly dawned on me what it meant to have a g…
Sep 1, 2017
Michael D. Sumner

Web services for scientific data in R

netcdf
r-packages
NOTE: this post has been resurrected from a 2017 post (April 2022).
Jul 25, 2017
Michael D. Sumner

R spatial in 2017

overview
This document is a broad overview of what I see as most relevant to future spatial, for 2017 and beyond. I’ve tried to be as broad as possible, without going into too much…
Jan 10, 2017
Michael D. Sumner

GIS for 3D in R

mesh
rgl
projections
GIS data structures are not well suited for generalization, and visualizations and models in 3D require pretty forceful and ad hoc approaches.
Dec 28, 2015
Michael D. Sumner

R matrices and image

raster
In R, matrices are ordered row-wise:
Apr 17, 2014
No matching items