GH #2744: Fix installation on Python 3.12, 3.13

[galenlynch]

Overview:

Three pinned dependencies blocked installation on 3.12:

• numpy<1.24: no cp312 wheels
• pandas==1.5.3: no cp312 wheels
• scipy<1.11: requires_python<3.12

Addresses:

Fixes #2747
Fixes #2744
Fixes #2746
Fixes #2754
etc...

Type of Fix:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing
  functionality to not work as expected)
• Documentation Change

Solution:

Minimally update the requirements to allow installation on 3.12+:

• numpy<1.24: no cp312 wheels; now uncapped (1.26 is first with 3.12 wheels)
• pandas==1.5.3: no cp312 wheels; now >=1.5.3,<3 (pandas 3.x compat requires more work)
• scipy<1.11: requires_python<3.12; now uncapped
• pynwb<3: breaking pynwb versions was released with extensive API changes, cap this dependency for now

Also some other requirements problems:

• xarray<2023.2.0: unpinned (pure Python, no breakage)
• pytz: added (was transitive via pandas 1.x, now needed explicitly)
• setuptools<81 (was transitive with earlier python versions)

Changes:

Source fixes for APIs removed in the new versions of dependencies:

• scipy.interpolate.interp2d removed in 1.14: replaced with RectBivariateSpline (same linear interpolation, swapped axis convention) in chisquarerf.py and utilities.py
• ConfigParser.readfp removed in Python 3.12: use read_string directly in application_config.py
• DataFrame.iteritems() removed in pandas 2.0: use items() in behavior_project_cache.py, save_extended_stimulus_presentations_df.py, receptive_field.py
• np.int() removed in numpy 1.24: use builtin int() in DLC_Ellipse_Video.py
• np.NaN, np.NAN → np.nan — removed in numpy 2.x, 49 occurrences across 22 files
• np.Inf → np.inf — removed in numpy 2.x
• np.string_ → np.bytes_ — removed in numpy 2.x, 4 occurrences across 4 files
• np.product → np.prod — removed in numpy 2.x
• np.ediff1d to_bein and to_end → dtype must be the same as ary in numpy 2.x
• np.in1d → np.isin— removed in numpy 2.x
• np.VisibleDeprecationWarning → removed in numpy 2.x, use try except import block in 1 file
• nwbfile.modules → .processing (deprecated alias in pynwb 2.x, removed in 3.x): running_acquisition.py, running_speed.py, nwb_api.py
• IndexSeries unit='None' → 'N/A' (pynwb 2.5+ fixed unit field): templates.py
• np.linalg.norm([array, scalar]) → np.vstack + full_like (numpy 1.24+ rejects inhomogeneous lists): _gaze_mapper.py
• aiohttp.ClientSession created eagerly in init → lazy https://github.com/Property (aiohttp 3.9+ warns when no event loop running): http_engine.py
• groupby().apply() on full DataFrame → select column first (pandas 2.2+ changed groupby-apply behavior): ecephys_project_cache.py
• Some LAPACK implementations return inf instead of raising LinAlgError
  for singular matrices. Add a finite-check fallback so _demix_point
  always falls back to lstsq in these cases.
• DataFrame.append() → pd.concat() (removed in pandas 2.0)
• pd.to_datetime: add format="ISO8601" for mixed-precision timestamps (pandas 2.x rejects mixed microsecond precision without explicit format)
• pd.to_datetime: fix utc="True" (string) → utc=True (bool) bug in behavior_project_cloud_api.py

Test fixes:

• pytest.warns(None) → warnings.catch_warnings (pytest 8 removed
  None sentinel): test_cache.py, test_smart_download.py
• mock.called_once_with (always truthy no-op) → assert_called_once()
  (proper assertion): test_cell_types_cache_unit.py
• Add res.x to MagicMock (scipy.optimize.minimize result accessed
  in numpy 1.24+ array construction): test_fitgaussian2D.py
• rng.choice(inhomogeneous list) → rng.integers + index (numpy 1.24+
  rejects ragged sequences): conftest.py
• Cast obtained dtypes to match expected (pynwb 2.x roundtrip returns
  nullable boolean for float columns): test_write_nwb.py
• Align index dtypes before assert_frame_equal (pandas 2.x infers
  different int dtypes): test_behavior_project_cache.py
• subprocess 'python' → sys.executable (resolves correct venv
  interpreter): test_runner.py
• patch logging.warning instead of using caplog
• The test_demix_raises_warning_for_singular_matrix test was
  platform-dependent: scipy.linalg.solve does not consistently raise
  LinAlgError for singular matrices across LAPACK implementations.
  Move the singular matrix case into test_demix_point to verify
  correctness of the result regardless of which internal path is taken.
• use unittest.mock instead of mock: mock no longer needed in python 3.10
• uncap many of the test dependencies, which were causing the coverage report to be very slow
• Add pandas 2.x datetime regression tests (test_pandas_compat.py)

Notebook fixes:

• IPython.core.display → IPython.display (removed in IPython 8.x)
  in visual_behavior_compare_across_trial_types and
  visual_behavior_mouse_history notebooks

CI changes:

• change CI to use more recent actions
• don't use conda to install python to fix macos issues
• narrow CI matrix to bounds of non-EOL python versions on ubuntu, 3.13 only on windows and mac
• Only calculate coverage on one matrix element
• Use pip cache keyed on requirements.txt and test_requirements.txt
• Switch notebook runner to ubuntu-latest-8x (32GB RAM) for
  ecephys notebooks that peak at ~21GB RSS
• Change notebook workflow trigger to push on master (expensive job
  should not run on every pull request)

Validation:

Tested: installs and imports cleanly on Python 3.10 and 3.12.

Ran local tests that do not require data, no new test failures occur (some were preexisting)

Checklist

• My code follows
  Allen Institute Contribution Guidelines
• My code is unit tested and does not decrease test coverage
• I have performed a self review of my own code
• My code is well-documented, and the docstrings conform to
  Numpy Standards
• I have updated the documentation of the repository where
  appropriate
• The header on my commit includes the issue number
• My Pull Request has the latest AllenSDK release candidate branch
  rc/x.y.z as its merge target
• My code passes all AllenSDK tests

Notes

No active RC branch to merge into, targeting master instead.

This is a near-duplicate of #2768, which I think could be a good merge candidate. However, given the lack of traction with that PR I wanted to make a more focused PR that did the bare minimum to allow installation on 3.12, and to make CI pass. I would be happy with either PR being merged, as long as we can install on 3.12.

Compared to #2768, this PR is similar but also supports numpy 2, which allows this package to run on python 3.13 and get numpy wheels on most platforms. Other test changes and CI changes are minor improvements over #2768, but not substantial.

[CLAassistant]

All committers have signed the CLA.

[galenlynch]

Changes required to get tests to pass locally:

• Pin pynwb<3 — pynwb 3.x redesigned IndexSeries API (removed
  indexed_timeseries, requires Images container for indexed_images).
  AllenSDK uses ImageSeries-based StimulusTemplate which is incompatible.
  pynwb 2.8.3 supports Python 3.12 and preserves the existing API.

Source fixes (backward-compatible with older dep versions):

• nwbfile.modules → .processing (deprecated alias in pynwb 2.x,
  removed in 3.x): running_acquisition.py, running_speed.py, nwb_api.py
• IndexSeries unit='None' → 'N/A' (pynwb 2.5+ fixed unit field):
  templates.py
• np.linalg.norm([array, scalar]) → np.vstack + full_like (numpy 1.24+
  rejects inhomogeneous lists): _gaze_mapper.py
• aiohttp.ClientSession created eagerly in init → lazy https://github.com/Property
  (aiohttp 3.9+ warns when no event loop running): http_engine.py
• groupby().apply() on full DataFrame → select column first (pandas
  2.2+ changed groupby-apply behavior): ecephys_project_cache.py

Test fixes:

• pytest.warns(None) → warnings.catch_warnings (pytest 8 removed
  None sentinel): test_cache.py, test_smart_download.py
• mock.called_once_with (always truthy no-op) → assert_called_once()
  (proper assertion): test_cell_types_cache_unit.py
• Add res.x to MagicMock (scipy.optimize.minimize result accessed
  in numpy 1.24+ array construction): test_fitgaussian2D.py
• rng.choice(inhomogeneous list) → rng.integers + index (numpy 1.24+
  rejects ragged sequences): conftest.py
• Widen curve-fit tolerances (scipy version/platform variance):
  test_drifting_gratings.py, test_static_gratings.py
• Cast obtained dtypes to match expected (pynwb 2.x roundtrip returns
  nullable boolean for float columns): test_write_nwb.py
• Align index dtypes before assert_frame_equal (pandas 2.x infers
  different int dtypes): test_behavior_project_cache.py
• subprocess 'python' → sys.executable (resolves correct venv
  interpreter): test_runner.py

Result: 2479 passed, 0 failed, 0 errors (was 28 failed, 5 errors).

[galenlynch]

Lint warnings were mostly preexisting in master - 751 pre-exsiting warnings in the files touched here, 23k in the repo. I will address the four new warnings caused by these changes.

[rly]

FYI I have similar open PRs in #2767 and #2768. Just wanted to point those out so that effort is not duplicated

Edit: Sorry, I just saw your comment in the OP about #2767. A focused PR sounds good too. I have no preference. Just wanted to get this resolved!

[galenlynch]

@rly Yeah same, 3.12 would be nice. Any path forward!

[galenlynch]

The glymur failures on 3.9 are due to glymur using match/case syntax, which was introduced in python 3.10. 3.9 is EOL, so I'm not sure it's worth fixing.

[galenlynch]

I narrowed the CI matrix, since the AllenSDK does not work on master for 3.9 due to breaking changes in glymur and a lack of version cap on glymur. However, python 3.9 is end of life.

I think it would be reasonable to tighten the python compatibility range in an upcoming release to 3.10+, and try to make the package actually install and pass tests on non-EOL versions of python.

[galenlynch]

All tests are passing: 3.10 ubuntu, and 3.13 for all OSs.

Tests were taking a very long time. I slimmed the matrix to only run max, min python versions (3.10, 3.13) on ubuntu and max on the other OSs. I also unpinned many of the test dependencies, since they were many versions behind, removed mock which has been moved into unittest on python 3.10+, and made only ubuntu latest measure coverage.

The notebook job was cryptically canceled, which I do not understand.

Linting is failing, but not due to changes introduced here.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[mikejhuang]

Approving given that the failing notebook test ran fine locally for Galen.