Pull common parts of skymatch from romancal and jwst into stcal #310

WilliamJamieson · 2024-10-17T17:40:56Z

This PR pulls the common parts of skymatch from romancal and jwst into stcal to be shared.

Note, no tests are included because it is well tested by the step tests in both romancal and jwst.

Tasks

update or add relevant tests
update relevant docstrings and / or docs/ page
Does this PR change any API used downstream? (if not, label with no-changelog-entry-needed)
- write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
- run regression tests with this branch installed ("git+https://github.com/<fork>/stcal@<branch>")
  - jwst regression test
  - romancal regression test

news fragment change types...

changes/<PR#>.apichange.rst: change to public API
changes/<PR#>.bugfix.rst: fixes an issue
changes/<PR#>.general.rst: infrastructure or miscellaneous change

codecov · 2024-10-17T17:46:42Z

Codecov Report

Attention: Patch coverage is 48.29545% with 455 lines in your changes missing coverage. Please review.

Project coverage is 79.42%. Comparing base (5bbdaa1) to head (1801b3a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/stcal/skymatch/skyimage.py	42.75%	316 Missing ⚠️
src/stcal/skymatch/skymatch.py	54.21%	125 Missing ⚠️
src/stcal/skymatch/skystatistics.py	72.54%	14 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #310      +/-   ##
==========================================
- Coverage   82.57%   79.42%   -3.16%     
==========================================
  Files          57       61       +4     
  Lines       10272    11241     +969     
==========================================
+ Hits         8482     8928     +446     
- Misses       1790     2313     +523

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

emolter

This looks great, thanks William! I just had a few questions about whether some of the ruff and mypy problems could be fixed instead of ignored.

I also had a few questions about the code itself, but those also exist on jwst main, so they're technically beyond the scope of this PR. But I'm still curious to know the answers to those and I figure now is as good a time as any to ensure the code is written as cleanly as possible

src/stcal/skymatch/skyimage.py

emolter · 2025-01-10T14:31:01Z

src/stcal/skymatch/skyimage.py

+              The functionality to support this conversion is not yet
+              implemented and at this moment `convf` is ignored.


Are there plans to implement this? If not, perhaps now is a good time to remove the option

Based on ancient (>9 years old) discussions with relevant people, this feature, in theory, may not be needed for JWST. In practice, however, we don't know. I suggest leaving this alone as is.

This is actually being used in places

Can you please point me to where? If that's the case, this comment in the docstring should be removed

Running as simple grep indicates its use:

https://github.com/WilliamJamieson/stcal/blob/d0976cb9afdd97b717bf30da08a0f8136ea84d91/src/stcal/skymatch/skymatch.py#L407-L414

https://github.com/WilliamJamieson/stcal/blob/d0976cb9afdd97b717bf30da08a0f8136ea84d91/src/stcal/skymatch/skymatch.py#L427-L436

Thats all logging but I'm not qualified to tell if we want to keep any parts of those logs

I don't think "used in logging" qualifies as being used given that convf is hard-coded to equal 1.0 and therefore has no effect on the scaling reported in the log messages. So I agree with Mihai we should leave this as-is, including the docstring

src/stcal/skymatch/skyimage.py

src/stcal/skymatch/skymatch.py

src/stcal/skymatch/skystatistics.py

WilliamJamieson · 2025-01-10T15:40:39Z

This looks great, thanks William! I just had a few questions about whether some of the ruff and mypy problems could be fixed instead of ignored.

I also had a few questions about the code itself, but those also exist on jwst main, so they're technically beyond the scope of this PR. But I'm still curious to know the answers to those and I figure now is as good a time as any to ensure the code is written as cleanly as possible

I agree they should be fixed. However, as you said they are beyond the scope of this PR. I think for making it possible to trace the history of changes it makes sense to move the code in as close a state as possible from JWST to stcal and then do a follow on PR making cleanups and addressing the issues uncovered here.

src/stcal/skymatch/skyimage.py

emolter · 2025-01-13T18:23:15Z

I agree they should be fixed. However, as you said they are beyond the scope of this PR.

IMO making the code adhere to the more stringent style rules in stcal remains within the scope. I don't much like ignoring them. I think the downstream testing should assuage any concerns that those changes might break something.

I agree that my comment RE the potentially unnecessary use of @property can be ignored for now though.

src/stcal/skymatch/skyimage.py

emolter

Thanks for all the updates! Before merging I think it would be good to update spacetelescope/jwst#8901 such that we can see if the changes are likely to work well with jwst. Perhaps you were already planning to do so.

WilliamJamieson · 2025-02-11T22:36:19Z

Thanks for all the updates! Before merging I think it would be good to update spacetelescope/jwst#8901 such that we can see if the changes are likely to work well with jwst. Perhaps you were already planning to do so.

I am not finished updating everything yet.

WilliamJamieson · 2025-02-12T18:03:46Z

@emolter I believe this is ready now.

WilliamJamieson · 2025-02-12T18:06:10Z

docs/conf.py

+    ("py:class", "stsci.imagestats.ImageStats"),  # intersphinx isn't working here
+    ("py:class", "spherical_geometry.polygon.SphericalPolygon"), # intersphinx isn't working here


These should be working with the intersphinx links I added. I believe the fix may require some fixes in the documentation for those packages.

It is possible something is off in the stcal docs causing the link failure. Please feel free to push changes to attempt to fix this.

emolter

All my previous questions have been answered now.

I don't think the changes have been so large that it would be helpful for me to go through it again, but let me know if you'd like me to.

So this looks good to me pending fixing the bug that is causing the failing jwst regtests

This is from jwst commit 2491b40a25f42bf2c10d242178434fe6c7d75847

WilliamJamieson requested a review from a team as a code owner October 17, 2024 17:40

This was referenced Oct 17, 2024

Switch skymatch to use stcal spacetelescope/romancal#1465

Draft

Use skymatch from stcal spacetelescope/jwst#8901

Open

nden requested a review from mcara October 22, 2024 17:25

github-actions bot added ramp_fitting testing jump installation labels Dec 19, 2024

WilliamJamieson force-pushed the feature/move_skymatch branch from c7e296c to 9ab3734 Compare December 19, 2024 19:21

github-actions bot added installation and removed ramp_fitting testing jump installation labels Dec 19, 2024

WilliamJamieson force-pushed the feature/move_skymatch branch 2 times, most recently from 9ed58df to cfaf7eb Compare December 20, 2024 14:51

WilliamJamieson force-pushed the feature/move_skymatch branch 3 times, most recently from 2aee80f to 6202d38 Compare January 3, 2025 17:57

emolter reviewed Jan 10, 2025

View reviewed changes

kmacdonald-stsci approved these changes Jan 13, 2025

View reviewed changes

src/stcal/skymatch/skyimage.py Show resolved Hide resolved

WilliamJamieson force-pushed the feature/move_skymatch branch 6 times, most recently from eb4b7d8 to 2e6b1c9 Compare February 11, 2025 22:02

WilliamJamieson force-pushed the feature/move_skymatch branch 3 times, most recently from 96da6e5 to 46a88e2 Compare February 11, 2025 22:10

emolter reviewed Feb 11, 2025

View reviewed changes

src/stcal/skymatch/skyimage.py Outdated Show resolved Hide resolved

src/stcal/skymatch/skyimage.py Outdated Show resolved Hide resolved

WilliamJamieson force-pushed the feature/move_skymatch branch 2 times, most recently from b5c4516 to d0976cb Compare February 11, 2025 22:24

emolter requested changes Feb 11, 2025

View reviewed changes

WilliamJamieson force-pushed the feature/move_skymatch branch from d0976cb to 705f544 Compare February 11, 2025 22:34

WilliamJamieson force-pushed the feature/move_skymatch branch from 705f544 to f41b760 Compare February 12, 2025 16:28

github-actions bot added the documentation Improvements or additions to documentation label Feb 12, 2025

WilliamJamieson requested a review from emolter February 12, 2025 18:03

WilliamJamieson commented Feb 12, 2025

View reviewed changes

emolter reviewed Feb 12, 2025

View reviewed changes

WilliamJamieson force-pushed the feature/move_skymatch branch 2 times, most recently from dbcfe42 to ba04e8e Compare February 14, 2025 17:42

WilliamJamieson added 3 commits February 14, 2025 13:05

Copy skymatch code from JWST

068d754

This is from jwst commit 2491b40a25f42bf2c10d242178434fe6c7d75847

Update changes

6f06512

Transfer documentation from JWST

1801b3a

WilliamJamieson force-pushed the feature/move_skymatch branch from ba04e8e to 1801b3a Compare February 14, 2025 18:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pull common parts of skymatch from romancal and jwst into stcal #310

Pull common parts of skymatch from romancal and jwst into stcal #310

WilliamJamieson commented Oct 17, 2024 •

edited

Loading

codecov bot commented Oct 17, 2024 •

edited

Loading

emolter left a comment

emolter Jan 10, 2025

mcara Jan 13, 2025

WilliamJamieson Feb 11, 2025

emolter Feb 11, 2025

WilliamJamieson Feb 11, 2025

WilliamJamieson Feb 11, 2025

emolter Feb 12, 2025

WilliamJamieson commented Jan 10, 2025

emolter commented Jan 13, 2025

emolter left a comment

WilliamJamieson commented Feb 11, 2025

WilliamJamieson commented Feb 12, 2025

WilliamJamieson Feb 12, 2025

emolter left a comment

		The functionality to support this conversion is not yet
		implemented and at this moment `convf` is ignored.

		("py:class", "stsci.imagestats.ImageStats"), # intersphinx isn't working here
		("py:class", "spherical_geometry.polygon.SphericalPolygon"), # intersphinx isn't working here

Pull common parts of skymatch from romancal and jwst into stcal #310

Are you sure you want to change the base?

Pull common parts of skymatch from romancal and jwst into stcal #310

Conversation

WilliamJamieson commented Oct 17, 2024 • edited Loading

Tasks

codecov bot commented Oct 17, 2024 • edited Loading

Codecov Report

emolter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WilliamJamieson commented Jan 10, 2025

emolter commented Jan 13, 2025

emolter left a comment

Choose a reason for hiding this comment

WilliamJamieson commented Feb 11, 2025

WilliamJamieson commented Feb 12, 2025

Choose a reason for hiding this comment

emolter left a comment

Choose a reason for hiding this comment

WilliamJamieson commented Oct 17, 2024 •

edited

Loading

codecov bot commented Oct 17, 2024 •

edited

Loading