Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
691a645
picking up the pieces
May 28, 2026
12a79e1
Merge branch 'main' into mapset
May 28, 2026
d98b37a
Fixes to close #57
May 28, 2026
f01223c
Initial layout for multiple field/file handling
May 28, 2026
c765902
Support for grad and laplacians
May 28, 2026
ffe674e
Adding create bounds support, and fixing a gui issue that resulted in…
May 28, 2026
6f1d965
Layout changes. Tooltips. Improving fonts.
May 29, 2026
822eee5
Include output to nc/zarr with rechunking support
May 29, 2026
779ad00
Substantial reworking of cf interface testing and cleaning up of some…
May 30, 2026
1e0bf4d
Renamed main menu elements
May 30, 2026
bd47c65
Fix inability to plot some single lat/lon ranges - closes #59
Jun 3, 2026
283cfd4
Adding GUI hooks for regridding. Adding esmpy dependency
Jun 4, 2026
1a82289
Missing scaffolding for pre-existing grids
Jun 4, 2026
002f039
Slight change to selection controller labelling
Jun 6, 2026
d4d3c0e
Standard regridding purports to work
Jun 6, 2026
92b1516
Added support for differencing
Jun 6, 2026
52fb814
Refactored cf and plotting code into cf_interface sub-module
Jun 8, 2026
ac0224b
Refactoring
Jun 8, 2026
7b0bb3d
Tidying
Jun 8, 2026
5a1a932
Interim refactoring regridding commit. Need to work on replay
Jun 8, 2026
89d2900
Support for copying the status line and for replaying last operations
Jun 8, 2026
18d89d9
Regridding via replay seems to work
Jun 8, 2026
fc00ae8
Improving field handling in the GUI and ensuring the correct fields g…
Jun 8, 2026
d7b22e0
Fixed regridding to target fields
Jun 8, 2026
eb4bcb7
Add command line support for file opening. Closes #61
Jun 8, 2026
90e4316
Fixing metadata in differences
Jun 8, 2026
fe78475
Adding test data file
Jun 9, 2026
ab5606a
Updated tests
Jun 9, 2026
198ebae
Prov support
Jun 9, 2026
8b86f69
Better provenance
Jun 9, 2026
d5bc9c9
Improving readabilty of prov
Jun 9, 2026
fdd9814
Fixed an off by one issue in sliders and an issue in sub-spacing, may…
Jun 9, 2026
f154ce7
Initial vector support
Jun 9, 2026
13e195c
Fixed some overplotting bugs
Jun 9, 2026
914c0d4
Merged main into twofile
Jun 9, 2026
2ae1439
Starting a clean up
Jun 9, 2026
c211979
refactor: extract worker message router and status handler
Jun 9, 2026
65a8855
refactor: extract remote replay and plot helpers
Jun 9, 2026
1e263c3
refactor: organize main_window helpers into components package
Jun 9, 2026
ddc3650
refactor: extract remote flow orchestration from main window
Jun 9, 2026
06e54e5
Extract SSH auth and replay utility helpers from CFVMain
Jun 9, 2026
4bb7c79
Cleaning up imports in main_window for clarity and brevity
Jun 10, 2026
4ba5851
Added filtering operations and slight improvmeent to property display
Jun 10, 2026
761ba07
Bumping version
Jun 10, 2026
ede9c42
Developer documentation (and a minor refactoring that resulted from l…
Jun 10, 2026
3531118
Woops missing tool for CI
Jun 10, 2026
a93472a
Hopefully fixes the ancient ubuntu plantuml problem
Jun 10, 2026
8dcd230
CI sucks
Jun 10, 2026
e121e94
AI control of CI sucks. Human required
Jun 10, 2026
136fb24
tests
Jun 10, 2026
f16c679
Interim commit while rebuilding mac app
Jun 10, 2026
c5e93cb
Build docs
Jun 10, 2026
195322f
Revert "Build docs"
Jun 10, 2026
801c84d
Revert "Interim commit while rebuilding mac app"
Jun 10, 2026
6b235ef
Revert "tests"
Jun 10, 2026
e12b935
Restore mac-app documentation after rollback
Jun 10, 2026
a525721
tests: use real cf I/O in remote worker tests
Jun 10, 2026
d600e1f
tests: use real cf fields in worker save-script test
Jun 10, 2026
abd41f2
tests: add vector overplot dispatch regression guards
Jun 10, 2026
df2179a
ui: import filtering symbols directly in dialogs
Jun 10, 2026
1e7fad7
Improving load time
Jun 10, 2026
d79fbf9
Packaging changes
Jun 10, 2026
03ef167
Fixing the current branch requirements and adding the new examples
Jun 11, 2026
9f95769
Fixing broken test and environment consistency
Jun 11, 2026
702459f
Italics as well as red for non-saved fields
Jun 11, 2026
bfd46c7
environment fixes, startup first field default change
Jun 11, 2026
ba2dd89
Zarr writes don't work given integer attributes. cf.write problem I t…
Jun 11, 2026
ca68791
Fixed some test issues around paramiko and logging
Jun 12, 2026
dd90941
working on zarr opening
Jun 12, 2026
1580d16
Now reads remote zarr too
Jun 12, 2026
2e8799d
Starting some dev notes
Jun 12, 2026
e9036b5
Fix an a another bounds/precision error in snapping to the coordinate…
Jun 12, 2026
29efa68
Improved attribution. Starting dev docs
Jun 13, 2026
ea6d11e
sliders from 2-d coordinates
davidhassell Jun 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions .github/workflows/uml-svg-sync.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
name: UML SVG Sync Check

on:
push:
branches:
- main
pull_request:
branches:
- main

jobs:
verify-uml-svgs:
name: Verify UML SVGs are up to date
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v6.0.2
with:
fetch-depth: 0

- name: Detect UML source changes
id: changed
run: |
set -euo pipefail

if [[ "${{ github.event_name }}" == "pull_request" ]]; then
base_ref="origin/${{ github.base_ref }}"
git fetch origin "${{ github.base_ref }}" --depth=1
diff_range="$base_ref...${{ github.sha }}"
else
before_sha="${{ github.event.before }}"
if [[ -z "$before_sha" || "$before_sha" == "0000000000000000000000000000000000000000" ]]; then
echo "uml_sources_changed=true" >> "$GITHUB_OUTPUT"
exit 0
fi
diff_range="$before_sha...${{ github.sha }}"
fi

changed_files="$(git diff --name-only "$diff_range" || true)"
if echo "$changed_files" | grep -Eq '^docs/uml/.*\.(pu|puml)$'; then
echo "uml_sources_changed=true" >> "$GITHUB_OUTPUT"
else
echo "uml_sources_changed=false" >> "$GITHUB_OUTPUT"
fi

- name: Skip note
if: steps.changed.outputs.uml_sources_changed != 'true'
run: echo "No .pu/.puml source changes detected; skipping UML SVG sync check."

- name: Install Graphviz and Java
if: steps.changed.outputs.uml_sources_changed == 'true'
run: |
sudo apt-get update
sudo apt-get install -y graphviz default-jre-headless curl

- name: Install latest PlantUML
if: steps.changed.outputs.uml_sources_changed == 'true'
run: |
jar_path="$RUNNER_TEMP/plantuml.jar"
bin_path="$RUNNER_TEMP/plantuml"

curl -fsSL -o "$jar_path" \
"https://github.com/plantuml/plantuml/releases/latest/download/plantuml.jar"
printf '%s\n' '#!/usr/bin/env bash' > "$bin_path"
printf '%s\n' 'set -euo pipefail' >> "$bin_path"
printf 'exec java -Djava.awt.headless=true -jar %q "$@"\n' "$jar_path" >> "$bin_path"
chmod +x "$bin_path"
"$bin_path" -version

- name: Regenerate UML SVG diagrams
if: steps.changed.outputs.uml_sources_changed == 'true'
env:
PLANTUML_BIN: ${{ runner.temp }}/plantuml
run: ./tools/generate_uml_svgs.sh

- name: UML render check passed
if: steps.changed.outputs.uml_sources_changed == 'true'
run: echo "PlantUML sources rendered to SVG successfully."
20 changes: 19 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,30 @@ limited to the following:

#### UML Diagrams

- `docs/uml/alpha_core_window.puml`
Rendered diagrams (GitHub-friendly):

- `docs/uml/svg/CoreWindowCurrentClass.svg`
- `docs/uml/svg/MainWindowComponentMap.svg`
- `docs/uml/svg/CoreWindowGuiWorkerSignals.svg`
- `docs/uml/svg/CoreWindowPhase3OptionsSequence.svg`
- `docs/uml/svg/RemoteWorkerWarmupSequence.svg`
- `docs/uml/svg/RemoteAccessModuleCurrent.svg`

PlantUML sources:

- `docs/uml/alpha_core_window.pu`
- `docs/uml/main_window_component_map.pu`
- `docs/uml/core_window_gui_worker_signals.puml`
- `docs/uml/core_window_options_sequence.puml`
- `docs/uml/remote_worker_warmup_sequence.puml`
- `docs/uml/remote_access_module.puml`

Regenerate SVGs after editing source diagrams:

```bash
./tools/generate_uml_svgs.sh
```

#### Architecture Notes

- `docs/architecture/core_window_refactor.md`
Expand Down
Binary file added data/da193a_25_3hr__198807_2days.nc
Binary file not shown.
Binary file added data/vector_eg.nc
Binary file not shown.
99 changes: 62 additions & 37 deletions docs/architecture/core_window_refactor.md
Original file line number Diff line number Diff line change
@@ -1,57 +1,82 @@
# Core Window Refactor Summary
# Core and Main Window Architecture

Date: 2026-03-08
Branch: `refactor/core-window-phase1-phase2`
This note reflects the current split between presentation (`CFVCore`) and
worker-backed orchestration (`CFVMain`).

## Overview
## Current architecture

`xconv2/core_window.py` was refactored in three phases to reduce class size and separate concerns by feature. The core window now acts as a coordinator that delegates behavior to focused UI controllers.
The app is now intentionally layered:

## Final Structure
- `xconv2/core_window.py` (`CFVCore`):

- `xconv2/core_window.py`
- Coordinator for window lifecycle, top-level wiring, and extension hooks used by `CFVMain`.
- Pure GUI composition, controller wiring, menu/status plumbing, cache manager UI,
recent-file UX, and local settings/state helpers.

- `xconv2/ui/settings_store.py`
- Settings load/save, validation, legacy migration, recent-file persistence, and default save-path helpers.
- `xconv2/main_window.py` (`CFVMain`):

- `xconv2/ui/menu_controller.py`
- Menu bar construction and recent-menu refresh logic.
- Worker process lifecycle (`QProcess`), IPC request submission, worker message
routing integration, and orchestration methods called from menu/actions.

- `xconv2/ui/selection_controller.py`
- Slider and collapse UI behavior, selected-range labels, and plot-summary enable/disable logic.
- `xconv2/worker_message_router.py`:

- `xconv2/ui/field_metadata_controller.py`
- Field list population, field detail display, property parsing, property table dialog, and CSV export.
- Parses worker stdout protocol lines and dispatches by message type.
- `WorkerStatusHandler` centralizes `STATUS:` behavior and plot/task completion
side effects.

- `xconv2/ui/plot_view_controller.py`
- Plot panel widgets, render/update of plot image, resize/aspect fitting, and save code/save plot button flows.
- `xconv2/main_window_components/*.py`:

- `xconv2/ui/contour_options_controller.py`
- Contour options dialog, annotation chooser, color-scale chooser, and scale-preview rendering.
- Feature-specific helper modules used by thin `CFVMain` wrappers.
- Current modules:
- `plot_ops.py`
- `remote_flow_ops.py`
- `remote_auth_ops.py`
- `replay_ops.py`

## UML Artifacts
- `xconv2/ui/*.py` controllers:

- `docs/uml/core_window.puml`
- `docs/uml/core_window_gui_worker_signals.puml`
- `docs/uml/core_window_options_sequence.puml`
- `MenuController`, `SelectionController`, `FieldMetadataController`,
`PlotViewController`, `ContourOptionsController`, `LineplotOptionsController`,
`VectorOptionsController`, `SettingsStore`.

These diagrams provide both class-level structure and key sequence flows for interaction paths.
## Why this layout

## Implementation Notes
- `CFVCore` stays presentation-focused and can be reasoned about as a GUI shell.
- `CFVMain` keeps worker orchestration in one host class, while high-churn feature
logic moves into focused component modules.
- Method names in `CFVMain` remain stable wrappers, preserving test seams and
monkeypatch points for integration-style tests.

- Public method names on `CFVCore` were retained where practical, with internal delegation added to controllers.
- Worker orchestration remains in `xconv2/main_window.py`.
- Dialog and feature behavior was preserved while moving logic into separate files.
## UML

## Refactor Commit History
Class map:

- `2fb1273` - refactor: extract menu and settings from core window
- `9ba1ddd` - refactor: extract selection and field metadata controllers
- `1925cea` - refactor: extract plot view and contour options controllers
![Core and main window class map](../uml/svg/CoreWindowCurrentClass.svg)

## Next Steps
Source: `docs/uml/alpha_core_window.pu`

- Add targeted unit tests for each controller class to reduce reliance on broad integration testing.
- Introduce narrow host protocols (typing `Protocol`) for controller dependencies to reduce coupling.
- Optionally split `ContourOptionsController` into smaller dialog-builder components if it continues to grow.
Main-window component map:

![Main window component map](../uml/svg/MainWindowComponentMap.svg)

Source: `docs/uml/main_window_component_map.pu`

GUI-worker protocol and routing flow:

![GUI worker protocol and routing flow](../uml/svg/CoreWindowGuiWorkerSignals.svg)

Source: `docs/uml/core_window_gui_worker_signals.puml`

Options flow (worker range fetch -> dialog -> render):

![Options flow sequence](../uml/svg/CoreWindowPhase3OptionsSequence.svg)

Source: `docs/uml/core_window_options_sequence.puml`

## Notes for future refactors

- Keep `CFVMain` methods as stable delegating wrappers when extracting logic,
so tests can continue patching host methods.
- Prefer module-level helper imports (as used now) over many per-function imports
for readability at the top of `main_window.py`.
- If additional host classes are introduced later, re-evaluate whether some
component helpers should become reusable classes or protocols.
120 changes: 49 additions & 71 deletions docs/architecture/remote_cache_and_pruning.md
Original file line number Diff line number Diff line change
@@ -1,99 +1,77 @@
# Remote Caching and Pruning Summary
# Remote Cache and Pruning (Current)

This note summarizes the remote caching work in xconv2, how users should use it, and important caveats.
This note reflects the current cache behavior in the refactored UI/worker layout.

## Scope

The caching behavior described here applies to remote opens that go through the worker-side filesystem path and then call `cf.read(..., filesystem=fs)`.
Cache settings are captured in remote configuration UI and propagated through the
descriptor used for worker-side remote sessions.

## What Is Implemented
Primary code paths:

### 1) Cache configuration captured in the GUI
- `xconv2/ui/dialogs.py` (cache config controls)
- `xconv2/remote_access.py` (`create_filesystem`)
- `xconv2/cache_utils.py` (usage + pruning)
- `xconv2/core_window.py` (cache manager + prune/flush actions)

Remote Configuration includes cache controls:
## Implemented behavior

- Memory cache strategy: `None`, `Block`, `Readahead`, `Whole-File`
- Block size (MB)
- RAM buffer (MB)
- Disk cache mode: `Disabled`, `Blocks`, `Files`
- Disk location
- Disk limit (GB)
- Disk expiry (`Never`, `1 day`, `7 days`, `30 days`)
### 1) Cache controls in configuration

These values are stored in the configuration state and passed through to the worker descriptor.
The remote configuration dialog persists disk cache options:

### 2) Worker-side cache application
- `disk_mode`: `Disabled`, `Blocks`, `Files`
- `disk_location`
- `disk_limit_gb`
- `disk_expiry`: `Never`, `1 day`, `7 days`, `30 days`

When the worker prepares a remote session, it now applies selected cache options to the filesystem:
These are included in the worker descriptor payload (`cache` object).

- Disk cache mode `Blocks` -> fsspec `blockcache`
- Disk cache mode `Files` -> fsspec `filecache`
- Memory strategy `Block` -> `cache_type="bytes"`
- Memory strategy `Readahead` -> `cache_type="readahead"`
- Memory strategy `Whole-File` -> `cache_type="all"`
- Memory strategy `None` -> no memory cache defaults injected
### 2) Worker-side cache preparation

Disk cache wrapper options include:
When worker control tasks prepare/open a remote session, filesystem creation goes
through `xconv2.remote_access.create_filesystem(...)`.

- cache storage path from GUI
- expiry seconds derived from GUI expiry label
- block size (for blockcache)
If `disk_mode` is `blocks` or `files`, the worker first prunes the configured
cache directory using `prune_disk_cache(...)`, then constructs protocol-specific
filesystem wrappers via `RemoteFileSystemFactory`.

### 3) Automatic pruning
### 3) Pruning policy

Before applying disk cache wrappers, xconv2 now prunes the configured cache location using:
`xconv2.cache_utils.prune_disk_cache(...)` applies:

- expiry policy (remove files older than configured expiry)
- size policy (if still above configured limit, remove oldest files first)
- expiry pruning (`mtime` older than configured expiry)
- size-limit pruning (oldest files first until under limit)

After file removal, the fsspec cache metadata index file is rewritten so removed payload entries are dropped.
It then rewrites fsspec cache metadata entries to drop payloads that no longer
exist, and returns a summary (`removed_files`, `removed_bytes`, totals).

### 4) Cache Manager in xconv menu
### 4) Cache Manager UI

A new xconv menu item opens a cache manager dialog.
`Xconv -> Manage Cache...` opens `CacheManagerDialog`.

The dialog provides:
Available operations:

- current cache configuration summary
- measured disk cache usage (bytes and file count)
- `Refresh`
- `Prune Cache` (apply policy-based pruning)
- `Flush Cache` (delete all cache payloads)
- Refresh cache summary and per-remote rows
- Prune configured cache (`_prune_configured_disk_cache`)
- Flush configured cache (`_flush_configured_disk_cache`)
- Flush per-remote location rows where enabled

Both prune and flush release any active remote session first.
Before prune/flush, the UI attempts to release active remote worker sessions to
avoid stale handles.

## How Users Should Use It
## Operational notes

## Recommended defaults
- Pruning is not background/continuous; it runs during filesystem creation and
when triggered manually from the cache manager.
- Size enforcement is file-level best effort.
- SSH rows in cache manager are currently marked non-flushable in the per-remote
table because SSH disk-cache behavior is constrained by current remote-fs path.
- If external processes modify the cache directory concurrently, metadata/index
consistency is best-effort and repaired on later prune/open cycles.

- Start with memory strategy `Readahead` or `Block`.
- Use disk mode `Blocks` for random-access NetCDF/HDF-style workloads.
- Set a dedicated disk cache location with enough space.
- Set a realistic size limit and non-zero expiry (for example `7 days`).
## UML

## Operational workflow
![Remote access module](../uml/svg/RemoteAccessModuleCurrent.svg)

1. Open xconv -> Configure Remote.
2. Set cache options in the Cache Configuration section.
3. Open remote data normally.
4. Use xconv -> Manage Cache... to inspect usage.
5. Use `Prune Cache` periodically, or `Flush Cache` when you want a hard reset.

## Caveats and Notes

- Disk limit enforcement is best-effort and file-based.
- Pruning uses payload file mtimes for age decisions.
- Metadata coherence is maintained for the writable cache index, but this is still a pragmatic cleanup approach.
- `disk_limit_gb` and expiry are now active for pruning, but eviction is not continuous/background; pruning occurs when sessions are prepared and when manually requested.
- `None` memory strategy disables injected read-cache defaults, but any lower-level library buffering still applies.
- If cache location is shared externally or modified concurrently, cache index consistency may degrade until next open/prune cycle.

## Developer Pointers

Main implementation points:

- `xconv2/cache_utils.py`: usage reporting, expiry parsing, pruning
- `xconv2/ui/remote_file_navigator.py`: cache strategy mapping and filesystem wrapping
- `xconv2/worker.py`: cache descriptor propagation into worker-side filesystem creation
- `xconv2/core_window.py`: cache manager UI and prune/flush actions
- `xconv2/ui/menu_controller.py`: xconv menu entry for cache manager
Source: `docs/uml/remote_access_module.puml`
Loading
Loading