Skip to content

feat: init datatree support - #45

Draft
ljstrnadiii wants to merge 1 commit into
carbonplan:mainfrom
ljstrnadiii:feat/datatree-support
Draft

feat: init datatree support#45
ljstrnadiii wants to merge 1 commit into
carbonplan:mainfrom
ljstrnadiii:feat/datatree-support

Conversation

@ljstrnadiii

Copy link
Copy Markdown

Summary

Closes #44

👋 I wanted to give this a shot and definitely used AI a lot for this. Happy to keep this in draft as a point of discussion or take any guidance if you are amenable to these changes!

[AI-based Description Below]
This PR adds first-pass datatree support to zarr-layer and improves CRS/proj4 handling for datasets that don’t provide a complete projection definition inline.

What Changed

  • Added datatree discovery from root consolidated metadata:
    • New ZarrStore.discoverDatasets(...) finds child groups containing the target variable.
    • Supports v2 and v3 metadata paths.
  • Added multi-store/multi-mode handling in ZarrLayer:
    • Initializes one ZarrStore + mode per discovered child dataset.
    • Handles lifecycle across all child modes/stores (init, selector updates, projection changes, cleanup).
    • Uses child dataset bounds for render culling when available.
  • Added datatree query behavior:
    • Point queries pick the matching child dataset by bounds.
    • Region queries can query multiple matching child datasets and merge results.
  • Improved CRS inference and proj4 resolution:
    • New resolveProj4 option (true | false | callback) in public types/options.
    • CRS inference from proj:* and spatial_ref metadata (including WKT parsing).
    • Optional online EPSG proj4 lookup fallback when proj4 is missing.
  • Added wkt-parser type declaration.
  • Updated README:
    • Added resolveProj4 option docs.
    • Added datatree section including fast-path note:
      • Fast when child multiscales layout metadata is present in root consolidated metadata.
      • Slower fallback when bounds must be derived by reading per-child coordinate arrays.

Assumptions / Current Constraints

  • Child dataset footprints are assumed non-overlapping.
  • scale_factor / add_offset are assumed consistent across child datasets.
  • Root consolidated metadata is required for datatree discovery.
  • In datatree mode, layer-level crs/proj4/bounds overrides are ignored so each child resolves its own spatial metadata.

Validation

  • npm run typecheck
  • npm run build

@vercel

vercel Bot commented Mar 19, 2026

Copy link
Copy Markdown

@ljstrnadiii is attempting to deploy a commit to the carbonplan Team on Vercel.

A member of the Team first needs to authorize it.

@ljstrnadiii

ljstrnadiii commented Mar 19, 2026

Copy link
Copy Markdown
Author
video.mov

@james-willis

Copy link
Copy Markdown
Contributor

one usecase I imagine this serving is UTM CRSes. But those are likely to have at least a tiny bit of overlap or buffering. What will happen in this case?

Comment thread src/types.ts
Comment on lines +143 to +149
/**
* CRS resolver for missing proj4 definitions.
* - `true` (default behavior when omitted): try online EPSG lookup (https://epsg.io/<code>.proj4)
* - `false`: disable online lookup fallback
* - function: custom resolver callback
*/
resolveProj4?: boolean | ResolveProj4

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a way to bundle these into the package? In java we use a geotools extension: https://docs.geotools.org/stable/userguide/library/referencing/extension.html

arbitrary site access is not ideal.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kyle Barron just published a sweet and relatively small package of codes that I've been wondering if we should try implementing here! https://developmentseed.org/deck.gl-raster/api/epsg/

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will update to use this tomorrow

Comment thread src/types.ts
}

/** Describes a child dataset discovered within a datatree root. */
export interface DatasetDescriptor {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its it ok to assume the child has the same CRS?

Comment thread src/wkt-parser.d.ts
@@ -0,0 +1,4 @@
declare module 'wkt-parser' {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pardon my ignorance. Are we supposed to commit these ".d.ts" files?

Comment thread src/zarr-layer.ts
Comment on lines +748 to +753
// Use first child store for shared dimension metadata in layer-level state.
this.zarrStore = this.datatreeStores[0]
rootStore.cleanup()

// Use first store for shared metadata (dimensions, fill_value, etc.)
const firstDesc = this.datatreeStores[0].describe()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worry about this kind of assumed sharing of information across stores in a tree

Comment thread src/zarr-store.ts
Comment on lines +25 to +26
const EPSG_AUTHORITY_REGEX = /AUTHORITY\["EPSG","(\d+)"\]/g
const PROJ4_EXTENSION_REGEX = /EXTENSION\["PROJ4","([^"]+)"\]/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been doing the same thing but im not sure if its ok to parse CRS WKT with regex. @paleolimbot ideas?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's theoretically whitespace allowed between the tokens but other than that I think it it would be unlikely to pick up some other portion of the CRS string.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support datatree roots with consolidated metadata

4 participants