Parity-mode pycurvelets regression tests fail from PR #30 baseline (TMEQ_RUN_CURVELETS=1, TMEQ_VALIDATE_MATLAB=1)

[Phil-Dua]

Hi team — reporting reproducible parity-mode failures in the pycurvelets MATLAB-to-Python regression tests introduced via [PR #30](#30). These tests were skipped in the PR by default, I was debuggingruff check . for the linting errors. Solved all the errors with 9 skipped tests and 1 warning (numpy core deprecation). I manually ran those 9 tests withTMEQ_RUN_CURVELETS=1 TMEQ_VALIDATE_MATLAB=1 uv run pytest -q -ra --tb=no tests/test_get_ct.py tests/test_process_image.py tests/test_new_curv.py and details are below.

Tagging @don04lee since this appears to originate in the pycurvelets conversion / parity path.

---

Scope

These are parity-mode tests (normally skipped unless environment flags are enabled):

• tests/test_get_ct.py
• tests/test_process_image.py
• tests/test_new_curv.py

---

Reproduction

PYTHONPATH=src TMEQ_RUN_CURVELETS=1 TMEQ_VALIDATE_MATLAB=1 \
pytest -q -ra --tb=no \
tests/test_get_ct.py \
tests/test_process_image.py \
tests/test_new_curv.py

---

Result

17 failed, 12 passed, 14 warnings

The dominant failure pattern is row-count / shape mismatches vs MATLAB reference outputs.

Examples:

• density_df length mismatch: 925 vs 622
• Row count mismatch in process_image feature outputs (e.g. 925 vs 622, 1298 vs 1196)
• new_curv center array shape mismatches (e.g. (925, 2) vs (622, 2))

---

Example failure excerpt

FAILED tests/test_get_ct.py::test_get_ct_matches_expected_results
AssertionError: density_df length mismatch: 925 vs 622

FAILED tests/test_process_image.py::test_process_image_returns_fiber_features[real1_default_no_boundary]
AssertionError: Row count mismatch: 925 vs 622

FAILED tests/test_new_curv.py::test_new_curv_matches_matlab_reference[real1_keep0.05_radius10]
AssertionError:
Not equal to tolerance rtol=0.05, atol=15
Curvelet centers differ from MATLAB reference
(shapes (925, 2), (622, 2) mismatch)

---

Environment

OS

macOS 15.6.1
Darwin 24.6.0 (arm64)

Python

3.11.14

uv

0.10.3

Key packages

{
  "numpy": "2.4.2",
  "scipy": "1.17.0",
  "pandas": "3.0.0",
  "pytest": "9.0.2",
  "curvelops": "0.23.4",
  "matplotlib": "3.10.8",
  "scikit_image": "0.26.0",
  "sklearn": "1.8.0"
}

---

Additional notes

Warnings observed:

• DeprecationWarning from curvelops import path (numpy.core.multiarray)
• RuntimeWarning: overflow encountered in cast in process_image.py reconstruction save path

[Phil-Dua]

The issue appears to start at merge commit 80ee91c, which merged PR #30 into main.

80ee91c^1 (parent before merge) does not yet include these parity suites (test_get_ct.py, test_process_image.py).
At 80ee91c, parity-enabled runs reproduce failures consistently.

[dlee489]

PYTHONPATH=src TMEQ_RUN_CURVELETS=1 TMEQ_VALIDATE_MATLAB=1 \
pytest -q -ra --tb=no \
tests/test_get_ct.py \
tests/test_process_image.py \
tests/test_new_curv.py
.............................                                                                                                                              [100%]
======================================================================== warnings summary ========================================================================
../../../../../opt/anaconda3/envs/pycurvelets/lib/python3.10/site-packages/curvelops/curvelops.py:12
  /opt/anaconda3/envs/pycurvelets/lib/python3.10/site-packages/curvelops/curvelops.py:12: DeprecationWarning: numpy.core.multiarray is deprecated and has been renamed to numpy._core.multiarray. The numpy._core namespace contains private NumPy internals and its use is discouraged, as NumPy internals can change without warning in any release. In practice, most real-world usage of numpy.core is to access functionality in the public NumPy API. If that is the case, use the public NumPy API. If not, you are using NumPy internals. If you would still like to access an internal attribute, use numpy._core.multiarray.normalize_axis_index.
    from numpy.core.multiarray import normalize_axis_index  # type: ignore

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
29 passed, 1 warning in 107.12s (0:01:47)

This has been tested in the main branch -- doesn't seem to be reproducible? Are there any local changes that you've made that could perhaps affect the results?

The main confusing thing to me about the errors you're getting are that they're row/col discrepancies, and that should really only happen if the inputs/outputs are being compared to wrong values?

[dlee489]

I've talked with @Phil-Dua, it seems that there's local uncommitted changes that modify the behavior of the functionalities between the main branch and the local branch.

Even running the tests on the local branch passes all the tests, so it seems that there are local uncommitted changes. @Phil-Dua will confirm and close when he identifies the problem.

[Phil-Dua]

This is very weird. I restored all the files in src/pycurvelets from commit 73b6a93 and even earlier 80ee91c. I get the exact same failing results:
Result: 17 failed, 12 passed, 14 warnings
Same failure pattern persists (row-count / center-count mismatches), e.g.:
density_df length mismatch: 925 vs 622
Curvelet centers differ ... (shapes (925, 2), (622, 2) mismatch)

[Phil-Dua]

@dlee489 do you have an idea where those patterns might be coming from?

[dlee489]

Kind of feel like we've went over this earlier, but I'll restate it -- could you go to the main branch and rerun the tests? If it fails there, then there's a possibility a file is being lost through gitignore, but I doubt it -- especially since it passed in other people's setups during the PR.

If I go to the local branch, git pull, and then rerun the tests, I pass all of them. I'm assuming when you say "restore all the files in src/pycurvelets" you mean within this branch. I really recommend double checking if there's any local uncommitted changes.

[Phil-Dua]

Now resolved, it's not about pulling the old source files, it's something else.

The parity failures were caused by a local FFTW 2.1.5 build that was configured with --enable-float (single-precision). CurveLab and curvelops expect double-precision FFTW; linking against the float build caused the FDCT to misinterpret memory, producing NaN and overflow values in the curvelet coefficients. These NaN values survived thresholding (since NaN * False = NaN and np.flatnonzero treats NaN as non-zero), creating spurious curvelets and the row-count mismatches we were seeing.

Fix: deleted the float-precision FFTW, rebuilt in double precision (the default), and rebuilt CurveLab + curvelops from scratch. All 29 parity tests pass now. Updated install.sh to detect float-precision FFTW builds and automatically trigger a rebuild.

[hinerm]

Updated install.sh to detect float-precision FFTW builds and automatically trigger a rebuild.

@Phil-Dua did this get merged to main? If so we can close this issue