-
Notifications
You must be signed in to change notification settings - Fork 50
Open
Labels
use case 🌎Real-world use caseReal-world use casevirtual references 👻Involves virtual kerchunk/virtualizarr chunk referencesInvolves virtual kerchunk/virtualizarr chunk references
Description
NOAA's UFS Replay could be an interesting public dataset to demo Icechunk with. Its big, >1PB!
It is available in two formats, both of which could be interesting to explore via virtual datasets:
- A Zarr v2 dataset on Google Cloud Storage (
gs://noaa-ufs-gefsv13replay/ufs-hr1) - A collection of NetCDF files on AWS S3 (
s3://noaa-ufs-gefsv13replay-pds/)
My thinking is that this dataset could be a good stress test for PB scale Icechunk datasets and virtual datasets at scale.
cc @TomNicholas and @timothyas
Known blockers:
TomNicholas, timothyas and norlandrhagen
Metadata
Metadata
Assignees
Labels
use case 🌎Real-world use caseReal-world use casevirtual references 👻Involves virtual kerchunk/virtualizarr chunk referencesInvolves virtual kerchunk/virtualizarr chunk references