ClimateDataFetcher

ClimateDataFetcher (cdf) is a climate data tool with CLI, HTTP API backend, and optional Dash UI workflows.

Current Status

• Release: 1.0.0
• Application type: CLI + HTTP API + optional Dash UI
• Core commands: list-sources, describe-source, fetch, ndvi, api, ui
• Config check command: validate-config
• Sources: open_meteo, noaa_cdo, nasa_gibs, sentinel_hub, planetary_computer
• Outputs: JSON, GeoJSON, CSV, PNG, GeoTIFF, manifest.json, logs/fetch.log
• Tests: 103 passed
• Coverage: 87% (target >= 86% met)
• Rust acceleration: enabled and verified (RUST_AVAILABLE=True)
• NOAA long-range behavior: auto-chunked date windows (default 31 days per chunk)
• Fetch UX: built-in progress bars (--progress / --no-progress)
• UI design: refreshed Dash theme with responsive layout and accessibility-focused styling

Quick Start

pip install -e ".[dev]"
cdf --help
cdf list-sources

# Optional geospatial export dependencies
pip install -e ".[geotiff]"

# Optional UI dependencies
pip install -e ".[ui]"

Docker (Preferred)

Build smallest API image (no UI dependency):

docker-build.bat

Build with Dash UI included:

docker-build.bat --ui

Run API container:

docker-run-api.bat

Run UI container (requires image built with --ui):

docker-run-ui.bat

Stop both containers:

docker-stop.bat

Disk cleanup when needed:

docker image prune -f
docker container prune -f
docker volume prune -f

Helper Scripts

Windows (.bat):

• setup.bat install environment and dependencies (--with-geotiff optional)
• build.bat run install + tests (--dist optional for wheel/sdist)
• run.bat run CLI (--background optional)
• stop.bat stop background run started by run.bat --background
• run-ui.bat run Dash UI (--background optional)
• stop-ui.bat stop background UI run
• docker-build.bat build container image (--ui optional)
• docker-run-api.bat run API container
• docker-run-ui.bat run UI container
• docker-stop.bat stop/remove API + UI containers

Unix/macOS (.sh):

• setup.sh
• build.sh
• run.sh
• stop.sh
• run-ui.sh
• stop-ui.sh
• docker-build.sh
• docker-run-api.sh
• docker-run-ui.sh
• docker-stop.sh

CLI Commands

• cdf list-sources
• cdf describe-source <source>
• cdf validate-config [--source <source|all>]
• cdf fetch --source ... --bbox ... --since ... --until ...
• cdf ndvi --nir <nir.png> --red <red.png> [--format json,csv]
• cdf api [--host 127.0.0.1] [--port 8000]
• cdf ui [--host 127.0.0.1] [--port 8501] [--debug]

HTTP API (v1.0)

Start backend server:

cdf api --host 127.0.0.1 --port 8000

Endpoints:

• GET /health
• GET /sources
• GET /sources/{name}
• POST /validate-config
• POST /fetch
• POST /ndvi

Fetch Examples

# Open-Meteo (no auth)
cdf fetch `
  --source open_meteo `
  --bbox "-10,35,40,70" `
  --since 2024-01-01 `
  --until 2024-01-03 `
  --verbose `
  --out .\output_open_meteo

# NOAA CDO (requires NOAA_CDO_TOKEN in .env)
cdf fetch `
  --source noaa_cdo `
  --bbox "-87.9,41.6,-87.5,42.1" `
  --since 2023-06-01 `
  --until 2023-06-07 `
  --out .\output_noaa

# NASA GIBS imagery
cdf fetch `
  --source nasa_gibs `
  --bbox "-10,35,40,70" `
  --since 2024-04-01 `
  --until 2024-04-07 `
  --format png `
  --resolution high `
  --out .\output_gibs

# NASA GIBS imagery + GeoTIFF export
cdf fetch `
  --source nasa_gibs `
  --bbox "-10,35,40,70" `
  --since 2024-04-01 `
  --until 2024-04-07 `
  --format png,geotiff `
  --resolution high `
  --out .\output_gibs_geotiff

# Sentinel Hub imagery (requires SENTINEL_HUB_CLIENT_ID / SENTINEL_HUB_CLIENT_SECRET)
cdf fetch `
  --source sentinel_hub `
  --bbox "-10,35,40,70" `
  --since 2024-04-01 `
  --until 2024-04-07 `
  --format png `
  --resolution high `
  --out .\output_sentinel_hub

# Planetary Computer STAC metadata
cdf fetch `
  --source planetary_computer `
  --bbox "-10,35,40,70" `
  --since 2024-04-01 `
  --until 2024-04-07 `
  --format json,csv,geojson `
  --out .\output_planetary_computer

Output Structure

output/
├── manifest.json
├── data/
│   ├── <source>_<since>_<until>.json
│   ├── <source>_<since>_<until>.geojson
│   ├── <source>_<since>_<until>.csv
│   ├── <source>_<date>.png
│   └── <source>_<date>.tif
└── logs/
    └── fetch.log

Configuration

Create .env from .env.example:

NOAA_CDO_TOKEN=your_token_here
# Optional v0.2 tuning
# HTTP_RETRY_ATTEMPTS=3
# HTTP_RETRY_BASE_DELAY=0.5
# HTTP_RETRY_MAX_DELAY=5.0
# NOAA_CDO_CHUNK_DAYS=31
# SENTINEL_HUB_CLIENT_ID=
# SENTINEL_HUB_CLIENT_SECRET=
# SENTINEL_HUB_CLOUD_COVERAGE=80
# PLANETARY_PC_COLLECTION=sentinel-2-l2a
# PLANETARY_PC_LIMIT=25

Validate config before a run:

cdf validate-config
cdf validate-config --source noaa_cdo
cdf validate-config --source sentinel_hub --live

NDVI example (local image analysis only; no network):

cdf ndvi `
  --nir .\bands\nir.png `
  --red .\bands\red.png `
  --format json,csv `
  --out .\output_ndvi

Testing

python -m pytest -q tests -p no:tmpdir -p no:cacheprovider
python -m pytest tests --cov=climatedatafetcher --cov-report=term-missing -p no:tmpdir -p no:cacheprovider

Screenshot Assets (Sanitized)

High-Resolution Images (First)

• 720x720
• 1080x1080
• 1440x1440
• 1920x1920
• 2560x2560
• 3840x3840

CLI Screenshot

Caption: cdf --help CLI output in terminal for command discovery.

UI Screenshots

Caption: v1.0 Dash UI Fetch tab with source fallback, compact bbox/date layout, and live log panel.

Caption: v1.0 Dash UI Validate Config tab.

Caption: v1.0 Dash UI NDVI tab.

Docs

• Implementation status: status.md
• Source details: docs/sources.md
• Contributing guide: docs/contributing.md
• Architecture notes: docs/architecture.md
• Planning spec (original): ClimateDataFetcher_Architecture.md