Where the line falls: tensors, tables and four layers

Arguments about geospatial and scientific data often come down to tensors versus tables: Zarr against Parquet, datacubes against feature collections, raster against vector. This essay argues that there is no single line between them. Storage, indexing, topology and rendering each draw the line in a different place, so the same dataset is a tensor from one layer and a table from another. Seeing the four lines separately makes several recurring design problems easier to name, and suggests that the two sides need each other more than they compete.

One model underneath

Almost every dataset in this field can be described the same way: a set of values over an index space, plus a map from each index to a place. A sea surface temperature grid has an index of (row, column, time) and a map that turns a row and column into a longitude and latitude. A table of ship positions has an index of row numbers and a map that is simply two of its columns.

The map comes in two kinds.

Implicit. The map is a recipe: an origin and a cell size, an affine geotransform, a start and a step, a rule that turns a HEALPix cell id into a cell on the sphere. Nothing per cell is stored; any cell’s place can be computed.
Explicit. The map is stored: arrays of longitude and latitude, a column of geometries, a table of vertices and the triangles that connect them.

Tensors and tables are the two ends of this scale. A tensor is a table whose key is implicit. Its index is complete and cartesian, so every combination of row, column and time exists and its coordinates follow from the recipe. A table is a one-dimensional tensor whose key is explicit. Its rows are indexed by number, the meaningful coordinates are ordinary columns, and nothing is implied about rows that are absent.

The interesting data sits between the ends. A curvilinear ocean model grid has a complete index but no recipe for its coordinates, so they are stored. A table of satellite granules can have a column whose every value is itself an array: a “dimensional column”. A ragged collection of float profiles is a table of profiles, each holding a short tensor.

None of this is new. Visualization software settled the same taxonomy in the 1990s. VTK’s dataset types run from fully implicit to fully explicit: ImageData (an origin and spacing), RectilinearGrid (one coordinate vector per axis), StructuredGrid (explicit coordinates on a complete index), UnstructuredGrid (explicit points and cells) and PolyData (points, lines and polygons). Geographic software split along the same scale into raster and vector, and for a long time treated the two halves as separate worlds with separate tools.

Four layers, four lines

A data system has at least four layers, and each has its own version of the tensor and table split. Communities tend to see the whole ecosystem divided along the line their own layer draws.

Layer	Tensor side	Table side	What the line really is
Storage	Zarr, Icechunk, netCDF and HDF5, COG	Parquet, Iceberg, Arrow, GeoPackage	Chunked by index space, or by rows
Index	Implicit maps: affine transforms, ranges, HEALPix ids	Explicit maps: coordinate columns, geometry columns, spatial trees over stored shapes	Whether a place is computed or looked up
Topology	Structured: neighbours implied by the index	Unstructured: neighbours recorded as vertices, edges and faces	Whether connectivity is implied or stored
Rendering	Warp: an image is moved to the view as a whole	Cell: each cell or feature becomes geometry and is drawn	When the map is applied: late, on the GPU, or early, as geometry

Storage is where the argument is usually held, and it is the least fundamental of the four. Zarr chunks values by blocks of index space; Parquet chunks them by groups of rows. Either can carry the other three layers. UGRID mesh connectivity is routinely stored as integer arrays in netCDF or Zarr, and the index of a collection of raster files is routinely stored as a table.

Index is the layer that decides how a coordinate becomes a position, and back. It is where the implicit and explicit maps live. Xarray’s recent work on pluggable indexes makes this layer explicit in code: since version 2025.6 an index is a replaceable component, and a CoordinateTransformIndex computes coordinates from a function rather than storing them.

Topology is about neighbours. In a regular grid the neighbours of a cell follow from its index; in a triangular mesh they have to be recorded. A HEALPix grid is structured in this sense even though it is not rectangular.

Rendering is where the two kinds of map meet the screen. A warp-based viewer keeps an image whole and moves it into the view’s projection, applying the map as late as possible. A cell-based viewer derives a geometry for each cell or feature and draws those, applying the map early. Both draw the same data, and in practice both are converging on the same datasets.

Objects that change sides

Because each layer draws its own line, the same object is a tensor from one layer and a table from another. This is why debates about the split rarely settle: the participants are standing in different layers.

Object	As a tensor	As a table
Triangular mesh (UGRID)	Connectivity arrays and values over a face dimension, in netCDF or Zarr	Tables of vertices, edges and triangles
Ocean float profiles	Contiguous ragged arrays (CF conventions)	One row per observation, or per profile with an array column
HEALPix field	A one-dimensional array over cell ids, with an implicit map	A sorted table of cell id and value, possibly sparse
Mosaic of COG files	Tiles of one large implicit grid	A tile index with one row per file and its footprint
Chunk byte references	The chunk grid of a Zarr array	Rows of path, offset and length (kerchunk’s Parquet references)
Vector data cube	An n-D array with one dimension indexed by geometries	A feature table with a time dimension per feature
Curvilinear model grid	Values on a complete (i, j) index	One stored longitude and latitude per cell, with no recipe

Two of these deserve a closer look.

A mesh is arrays to the model that writes it and tables to the software that edits it. Neither view is wrong. They are the same relations with the joins either implied by array position or written out as keys.

Byte references are a table whose only purpose is to describe a tensor. Each row says where one chunk of an array lives in some file. It is the clearest case of the two sides needing each other, and it comes back in the cases below.

Case: catalogs are tables in disguise

STAC, the SpatioTemporal Asset Catalog, is written as a tree of JSON documents. Normalized, it is three or four tables, joined by keys that the JSON leaves implicit in its nesting:

Table	Key	What a row holds
Collections	collection id	Extent, license, summaries
Items	item id, collection id	Datetime, footprint geometry, properties
Assets	asset key, item id	Link, media type, roles; with the projection extension, `proj:shape` and `proj:transform`
Bands	band index, asset	Scale, offset, nodata, statistics (`raster:bands`, `eo:bands`)

The tree form makes this structure hard to see in four ways.

Joins become position. An asset belongs to whatever item encloses it. There is no column to join on until the tree is flattened.
One viewpoint is built in. The tree is item-centric. “Every red-band asset sharing this transform” is a natural query on a long assets table and an awkward walk through documents.
The schema is per document. Extensions may add fields anywhere, so there is no table-wide schema to normalize against.
Identity becomes addresses. Links are URLs rather than keys.

stac-geoparquet goes a long way: one row per item, in a columnar format that query engines read directly. Assets are typically kept as a nested column keyed by asset name, so the tree survives one level down and the schema varies from collection to collection.

Once normalized, an asset row with proj:shape and proj:transform is an array descriptor: a shape plus an implicit map from index to place. Add a table of byte references keyed by asset and chunk, and the catalog and a virtual array store turn out to share one relational schema, viewed from opposite ends.

Catalog relation	Array-store relation
Collection	Repository, or a top-level group
Item	A group per scene
Asset, band	An array
Byte references	A chunk manifest

This matters for a live design question: how to store Level 2 data, the per-scene products that do not share a grid. A talk at CNG Forum 2026 by Tom Nicholas (Earthmover) and Sean Harkins (Development Seed) proposes keeping many scenes, of any shape and projection, as separate groups in one Icechunk repository, alongside a STAC-like metadata table, so that catalog and data change in one atomic commit. The motivation is real and well argued: when a catalog and its files live in separate systems, a failed job can leave files no search will find, and a reprocessing run can leave catalog entries pointing at deleted files. The talk shows two prototypes for the table, one storing columns as one-dimensional Zarr arrays read through DataFusion, and one placing an Apache Iceberg table’s pointer in the repository’s commit metadata. It also reports that virtualizing the full Harmonized Landsat Sentinel-2 archive this way implies about 7.7 billion repository nodes, with commit cost currently growing with node count, and that the fix is in progress.

Seen through the layers, this design records the scene-to-array relation twice: once as the repository’s hierarchy, and once in the catalog table. The normalized view above suggests an alternative worth exploring alongside it, in which the catalog, the array descriptors and the byte references are all ordinary tables, and arrays are what their rows point to. Which side should hold the transaction is an open question, and both approaches are likely to teach something.

Case: self-indexing formats and references

A recent DuckDB post describes a database file with no data in it: only view definitions that query Parquet files elsewhere, on object storage or over HTTPS. The file stays a few hundred kilobytes however large the data grows, and acts as a shareable catalog. Each view’s reference is the text of a query, such as read_parquet('s3://...') over a file pattern. There are no byte ranges in it.

That works because Parquet indexes itself. A Parquet file’s footer records where every row group and column chunk sits, with minimum and maximum statistics. A reader that knows only the file can still fetch just the bytes a query needs. References at the granularity of whole files are enough.

Many array formats are not like this. In HDF5, and so in netCDF-4, locating a chunk means walking internal B-trees, which costs many small reads before any data arrives. That is why byte-reference harvesting exists: kerchunk, VirtualiZarr, Icechunk’s virtual chunks, and tools that ask GDAL’s multidimensional API for chunk locations all build, ahead of time, the table of path, offset and length that the format does not offer cheaply. Read that way, a harvested reference table is the footer those formats are missing.

The idea is older than any of these. A GDAL VRT is a small file of references and transforms over data that lives elsewhere, and has been since the early 2000s. The line runs from VRT through kerchunk to DuckDB views: a catalog that describes data instead of holding it.

All of these share one requirement that is easy to forget: the referenced data must not change. A view over a file pattern silently returns different results when files are added or rewritten. A byte reference into a file that has been replaced points at the wrong bytes. References are safe when what they point to is pinned: a versioned object, a snapshot, a content-addressed chunk, or a fixed list of immutable files.

Case: what to do with a swath

A satellite swath is the hardest case for the tensor side. Each granule has its own footprint, often in the sensor’s own geometry, with a stored longitude and latitude for every pixel. Granules overlap, and no two share a grid. There are three honest ways to hold a collection of them.

One group per granule, plus a catalog. Every granule keeps its own arrays and its own explicit coordinates, and a table indexes them. Nothing is lost or resampled. The granules never become one array, and any cross-granule view has to be assembled when it is requested.
Resample to a common grid. Every granule is binned or interpolated onto one grid, producing a cube. This is how Level 3 products are made. It answers many questions well, but it is a different product: the original observations, and their individual footprints, are gone.
A global implicit index. Assign every observation the id of the discrete global grid cell it falls in (HEALPix, S2, H3 or similar), and store rows of (cell, time, value), with whatever per-observation detail is needed. The map from index to place is implicit and shared by every granule. Observations are not resampled. Overlaps simply become several rows for one cell. The result is a table and a sparse tensor over a global cell dimension at the same time.

The third option sits squarely between the two sides, which is why neither side tends to propose it. It also fits the idea of dimensional columns: one row per granule or scan line, carrying its footprint, time and geolocation recipe as ordinary columns, with arrays as the payload of each row.

Who works on which layer

Most projects cover one layer well, some two. A partial and certainly incomplete map:

Layer	Python	Rust and TypeScript	R and others
Storage	zarr-python, Icechunk, VirtualiZarr, kerchunk	Icechunk core, icechunk-js, async-tiff, obstore	GDAL multidimensional API, gdalraster, ncdf4, Rarr; hypertidy’s blocklist and altarr
Index	xarray pluggable indexes: rasterix, xproj, xvec, xdggs	geoarrow-rs	stars; hypertidy’s vaster and grout
Topology	UXarray, xugrid, meshio	spade, CDT (C++)	MDAL, CGAL; sf and wk for features; hypertidy’s meshcore, silicate and trianglewins
Rendering	GeoVista (on PyVista and VTK), lonboard	deck.gl-raster, gridlook	mapview, tmap

The xarray index work is the closest anyone has come to writing the index layer down as code. Meshes are the next frontier there: UXarray and xugrid treat a mesh as a grid object beside xarray rather than as an index within it, and whether a mesh can be an index is open. On the rendering side, GeoVista carries VTK’s dataset taxonomy into earth science directly, and is perhaps the clearest statement of the model in this essay from the visualization end.

Open questions

These are questions, not conclusions. Each could be answered by building something small.

Should references be open tables? If byte references were ordinary Parquet or Iceberg tables, every query engine could read them, and a reference table could double as the catalog. What would be lost relative to references held inside an array store?
Where should the transaction live? Consistency between catalog and data needs one commit protocol. An array store can host a table, or a table format can point into arrays. Which composes better with the engines people already use?
Can a mesh be an index? If a mesh could be an index in the way an affine transform now is in xarray, meshes and rasters would share selection, alignment and concatenation.
What is the smallest shared index contract across languages? Python now has one in xarray. R has no pluggable equivalent: the map from index to place is tied to the classes of individual packages. A small contract for computed coordinates would let packages interoperate without sharing classes.
How should pinning be expressed? Every reference system depends on immutable sources, but there is no common way to state that a reference is pinned, or to check it.
Is the swath problem a table problem? If collections of granules are tables first, with arrays as the payload of rows, how much of the Level 2 difficulty goes away, and what new difficulty appears?