Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slow processing of L2 uncertainties (top priority) #80

Open
oceancolorcoder opened this issue Sep 19, 2023 · 9 comments
Open

Slow processing of L2 uncertainties (top priority) #80

oceancolorcoder opened this issue Sep 19, 2023 · 9 comments

Comments

@oceancolorcoder
Copy link
Contributor

Ideas to pursue:

Reduce the size of Z17 by confining it to protocol geometries (DA).

Build LUTs in lieu of full calculations for given conditions (AR).
@oceancolorcoder
Copy link
Contributor Author

oceancolorcoder commented Sep 20, 2023 •
edited
Loading

Pathway discussed Sept 20: calculate partial LUT for min/max parameters (e.g., wind, AOT, etc.) impact on difference between M99 and Z17 rather than calculated Z17 for each spectrum or LUTs for all possible parameterizations.

@ARamsay17

@oceancolorcoder oceancolorcoder mentioned this issue Oct 21, 2023
Closed
@oceancolorcoder
Copy link
Contributor Author

oceancolorcoder commented Oct 21, 2023 •
edited
Loading

pySAS, M99, turning off satellite convolution to Aqua reduces timing from to 2 min.

The snag is in instrument.rrsHyperUNCFACTORY is quite slow, ~160s per ensemble/xSlice:

Processing: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
Process Single Level
ProcessL2: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
stationsEnsemblesReflectance
Binning datasets to ensemble time interval.
Convolving MODIS Aqua (ir)radiances in the slice
67 spectra in slice (ensemble).
7 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.3 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 160.2 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -9.523629031717784e-05; offset(nLw) = -0.04426776425734717

… [Repeat for each ensemble]

Filtering reflectance spectra for negative values.

0.0% of Rrs_HYPER spectra flagged
0.0% of nLw_HYPER spectra flagged
Plotting Rrs
Plotting Rrs
Plotting Es
Plotting Li
Plotting Lt
Process Single Level: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf - SUCCESSFUL
L2 file produced:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Output SeaBASS for HDF:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Writing PDF Report...
Level 1A
Level 1AQC
Adding deglitching plots...
Level 1B
Adding interpolation plots...
Level 1BQC
Adding spectral filter plots
Level 2
Adding radiometry plots
Adding ocean color product plots
processFilesSingleLevel, single file - DONE
Time elapsed: 14 minutes

@oceancolorcoder
Copy link
Contributor Author

oceancolorcoder commented Oct 21, 2023 •
edited
Loading

Without Aqua convolution takes < 1 minute:

Processing: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
Process Single Level
ProcessL2: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L1BQC/FRM4SOC2_FICE22_NASA_20220719_080000_L1BQC.hdf
stationsEnsemblesReflectance
Binning datasets to ensemble time interval.
67 spectra in slice (ensemble).
7 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.2 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.3 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -9.523629031717784e-05; offset(nLw) = -0.04426776425734717
80 spectra in slice (ensemble).
8 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.1 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.4 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -9.679655711474021e-06; offset(nLw) = -0.033131963419616335
78 spectra in slice (ensemble).
8 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.1 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.3 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = -2.845188536517354e-05; offset(nLw) = -0.034560316042568845
76 spectra in slice (ensemble).
8 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.2 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.4 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = 1.2613785225814207e-06; offset(nLw) = -0.030815234070160086
49 spectra in slice (ensemble).
5 spectra remaining in slice to average after filtering to lowest 10.0%.
Calculating M99 glint correction with complete LUT
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 4.2 s
Reading : Data/hybrid_reference_spectrum_p1nm_resolution_c2020-09-21_with_unc.nc
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 6.467318590049452e-14 percent) to accomodate our method.
warnings.warn(
/Users/daurin/anaconda3/envs/hypercp/lib/python3.11/site-packages/comet_maths/linear_algebra/matrix_calculation.py:286: UserWarning: One of the provided covariance matrix is not positivedefinite. It has been slightly changed (maximum difference of 4.311545726699635e-14 percent) to accomodate our method.
warnings.warn(
Uncertainty Update Elapsed Time: 0.4 s
Perform similarity spectrum residual NIR subtraction.
offset(rrs) = 8.215846949638485e-05; offset(nLw) = -0.020145435286546286
Filtering reflectance spectra for negative values.
0.0% of Rrs_HYPER spectra flagged
0.0% of nLw_HYPER spectra flagged
Plotting Rrs
Plotting Rrs
Plotting Es
Plotting Li
Plotting Lt
Process Single Level: /Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf - SUCCESSFUL
L2 file produced:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Output SeaBASS for HDF:
/Users/daurin/Projects/HyperCP/Sample_Data/pySAS_Factory/L2/FRM4SOC2_FICE22_NASA_20220719_080000_L2.hdf
Writing PDF Report...
Level 1A
Level 1AQC
Adding deglitching plots...
Level 1B
Adding interpolation plots...
Level 1BQC
Adding spectral filter plots
Level 2
Adding radiometry plots
Adding ocean color product plots
processFilesSingleLevel, single file - DONE
Time elapsed: 1 minutes

oceancolorcoder added a commit that referenced this issue Oct 21, 2023
@oceancolorcoder
Troubleshooting #80 and #95. Revised Sample_Data layout for clarity. …
17b856b 
…See Issues comments and updated Sample_Data and README for details
@oceancolorcoder
Copy link
Contributor Author

Also, while Z17 is apparently not the logjam here, I have so far found that updating Z17 has only increased its size, considering the newly added AOD to 0.5. From 2.47 GB, it goes to 3.28 GB. I can bring it back down to 2.26 GB by truncating AOD of 0 and SZA of 0 (i.e., starting at 0.05 and 10, respectively). Does not seem worth it yet, but will continue to experiment.

@oceancolorcoder
Copy link
Contributor Author

Two issues to address: Z17 uncertainties at L2 are too slow (i.e., processing time is nearly. as long as data collection time). For this we need a LUT for M99 - Z17.

The second issue is that the convolutions to satellite bands are excessively slow, making even M99 processing interminable. Setting created to forgo convolution uncertainties until a resolution can be identified.

@oceancolorcoder
Copy link
Contributor Author

One confusing element about the slowness of running Z17 uncertainties is that when zhangWrapper runs Z17 for uncertainty estimates, each run takes much longer than the run of Z17 to do the glint correction:

...
6 spectra in slice (ensemble).
Wavelengths extend beyond model limits. Truncating to 350 - 1000 nm.
Calculating Zhang glint correction.
Zhang17 Elapsed Time: 9.1 s
Uncertainty_Analysis.zhangWrapper. Wind: 2.1 AOT: 0.09 Cloud: 0.0 SZA: 33.5 SST: 13.1 SSS: 35.6
Uncertainty_Analysis.zhangWrapper. VZA: 40.0 RelAz: 135.0
zhangWrapper Z17 Elapsed Time: 35.8 s
Uncertainty_Analysis.zhangWrapper. Wind: 2.1 AOT: 0.09 Cloud: 0.0 SZA: 33.7 SST: 15.7 SSS: 34.8
Uncertainty_Analysis.zhangWrapper. VZA: 40.0 RelAz: 133.5
zhangWrapper Z17 Elapsed Time: 35.5 s
...

@oceancolorcoder oceancolorcoder changed the title Slow processing of L2 uncertainties Slow processing of L2 uncertainties (top priority) Jun 28, 2024
@oceancolorcoder
Copy link
Contributor Author

The LUTs are built, but AR needs to reconcile differences between LUT results and full model runs.

@juanchossn
Copy link
Contributor

juanchossn commented Nov 27, 2024 •
edited
Loading

Issues related to the way in which interpolation is performed - e.g. linear inteprolation applied to things that logarithmic dependence. @raphidoc and @ARamsay17 to coordinate on this.

@ARamsay17
Copy link
Contributor

@juanchossn @oceancolorcoder @raphidoc the new zhang17 Lookup table can be found on the box: HyperCP Project Team -> Data -> Zhang_LUTs -> Z17_LUT_new.nc

I will test on my end. The code to fully utilise the LUT is in PR #308 which is waiting to be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@juanchossn @oceancolorcoder @ARamsay17