Skip to content

Conversation

@rhaegar325
Copy link
Collaborator

Add memory validation to write() method and corresponding unit tests.

Changes

Memory Validation in write()

  • Estimate data size before loading with 1.5x overhead factor
  • Detect existing Dask client and check cluster memory limits
  • Raise MemoryError with helpful message when data exceeds available memory
  • Auto-close Dask client when data exceeds single worker memory but fits in system memory
  • Gracefully handle missing dask.distributed package (lazy import)

Unit Tests

  • Add TestCMIP6CMORiserWrite class covering:
    • Missing attribute validation
    • System memory checks
    • Dask cluster memory checks
    • Output file format and structure
    • Logging output

Memory Check Logic

Condition Action
Data > total cluster memory Raise MemoryError
Data > worker memory, < cluster total Close client, use local memory
Data > system available memory Raise MemoryError
Data < available memory Proceed with write

@codecov
Copy link

codecov bot commented Dec 5, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@207f119). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #134   +/-   ##
=======================================
  Coverage        ?   56.49%           
=======================================
  Files           ?       18           
  Lines           ?     2101           
  Branches        ?        0           
=======================================
  Hits            ?     1187           
  Misses          ?      914           
  Partials        ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants