hypertidy-blog – IDEA - data and software

raadtools, software to extract maps of ocean properties and values at points-in-time

raadtools > 10 years old R package

R function	Purpose
readsst()	global sea surface temperature
readice()	polar sea ice concentrations
readghrsst()	high resolution sea surface temperature
read_adt/ugos/vgos_daily()	global altimetry, sea height, surface currents
read_chla_daily()	global ocean colour
readtopo()	global or local bathymetry
…

other functions we don’t have, yet …

software we want

features we want	raadtools 🤔	<new tool>
users don’t download files	✅	✅
data is up to date	✅	✅
we can add new data	✅	✅
you can add new data	❌	✅
use outside AAD without Mike or Ben	❌	✅
use outside of R	❌	✅
scale up on super computing	❌	✅
robust to research/local outage	❌	✅
available offline on Nuyina	❌	??

Python support

Address entire data cubes, with one line of code e.g. daily data 1993 to November 2024

import xarray; <some settings>

ds = xarray.open_dataset('s3://vzarr/SEALEVEL_GLO_PHY_L4.parquet', <more settings>)

<xarray.Dataset> Size: 574GB
Dimensions:    (time: 11538, latitude: 720, longitude: 1440)
Coordinates:
  * latitude   (latitude) float32 3kB -89.88 -89.62 -89.38 ... 89.38 89.62 89.88
  * longitude  (longitude) float32 6kB -179.9 -179.6 -179.4 ... 179.6 179.9
  * time       (time) datetime64[ns] 92kB 1993-01-01 1993-01-02 ... 2024-11-25
Data variables:
    adt        (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
    sla        (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
    ugos       (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
    ugosa      (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
    vgos       (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
    vgosa      (time, latitude, longitude) float64 96GB dask.array<chunksize=(1, 50, 50), meta=np.ndarray>
Attributes: (12/44) ...

How are we doing this

The old and new tools reflect user-demand, tell us your ideas!
Modern tech: cloud-native and efficient public-available files
Data curation and cataloguing tools: {bowerbird}, STAC, VirtualiZarr
Exploring best-practice usage in Python xarray, odc, and in R terra, gdalraster, rsi
Contributing to software libraries GDAL.org, and community with AADC, SCAR, rOpenSci, Pangeo, Radiant Earth, Opendatacube, Digital Earth Antarctica