Add drawers temperature trend in the DQM

[alessandromontanari]

feat(src/nectarchain/dqm):

• camera_monitoring.py: add temperature trends with values for all drawers

This PR adds the temperature trends for all the drawers as output of the processor camera_monitoring.py in the DQM, following the same logic as for the other outputs. This is necessary both to have the plots produced by the DQM and to write in the database serving the web app.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 61 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.01%. Comparing base (265b63a) to head (4e19fa3).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/nectarchain/dqm/camera_monitoring.py	0.00%	40 Missing ⚠️
src/nectarchain/dqm/dqm_summary_processor.py	0.00%	21 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #310      +/-   ##
==========================================
- Coverage   49.01%   49.01%   -0.01%     
==========================================
  Files          85       87       +2     
  Lines        7850     8012     +162     
==========================================
+ Hits         3848     3927      +79     
- Misses       4002     4085      +83     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hashkar]

@alessandromontanari I get:
File "/Users/hashkar/Desktop/nectarcam/nectarchain/src/nectarchain/dqm/start_dqm.py", line 306, in
main()
File "/Users/hashkar/Desktop/nectarcam/nectarchain/src/nectarchain/dqm/start_dqm.py", line 276, in main
NESTED_DICT[NESTED_DICT_KEYS[dict_num]] = p.get_results()
^^^^^^^^^^^^^^^
File "/Users/hashkar/Desktop/nectarcam/nectarchain/src/nectarchain/dqm/trigger_statistics.py", line 116, in get_results
"Wrong times": np.array(self.event_wrong_id),
^^^^^^^^^^^^^^^^^^^
AttributeError: 'TriggerStatistics' object has no attribute 'event_wrong_id'. Did you mean: 'event_wrong_times'?

This is what @jlenain was fixing with 7298b03 and [3bc95c1]?

[alessandromontanari]

@alessandromontanari I get: File "/Users/hashkar/Desktop/nectarcam/nectarchain/src/nectarchain/dqm/start_dqm.py", line 306, in main() File "/Users/hashkar/Desktop/nectarcam/nectarchain/src/nectarchain/dqm/start_dqm.py", line 276, in main NESTED_DICT[NESTED_DICT_KEYS[dict_num]] = p.get_results() ^^^^^^^^^^^^^^^ File "/Users/hashkar/Desktop/nectarcam/nectarchain/src/nectarchain/dqm/trigger_statistics.py", line 116, in get_results "Wrong times": np.array(self.event_wrong_id), ^^^^^^^^^^^^^^^^^^^ AttributeError: 'TriggerStatistics' object has no attribute 'event_wrong_id'. Did you mean: 'event_wrong_times'?

This is what @jlenain was fixing with 7298b03 and [3bc95c1]?

Yes exactly, event_wrong_id does not exist. I'm fixing it in #308, and then maybe I can pull that change in this PR too once it is merged...

[jlenain]

@alessandromontanari I get: File "/Users/hashkar/Desktop/nectarcam/nectarchain/src/nectarchain/dqm/start_dqm.py", line 306, in main() File "/Users/hashkar/Desktop/nectarcam/nectarchain/src/nectarchain/dqm/start_dqm.py", line 276, in main NESTED_DICT[NESTED_DICT_KEYS[dict_num]] = p.get_results() ^^^^^^^^^^^^^^^ File "/Users/hashkar/Desktop/nectarcam/nectarchain/src/nectarchain/dqm/trigger_statistics.py", line 116, in get_results "Wrong times": np.array(self.event_wrong_id), ^^^^^^^^^^^^^^^^^^^ AttributeError: 'TriggerStatistics' object has no attribute 'event_wrong_id'. Did you mean: 'event_wrong_times'?
This is what @jlenain was fixing with 7298b03 and [3bc95c1]?

Yes exactly, event_wrong_id does not exist. I'm fixing it in #308, and then maybe I can pull that change in this PR too once it is merged...

This has been fixed in PR #313, which is now merged onto main, thanks a lot to @alessandromontanari !

Let me rebase it here, and fix the current conflict with main (which comes from PR #309 since both PRs works on the same part of the code).

[jlenain]

Rebase done and conflicts fixed.

[alessandromontanari]

Rebase done and conflicts fixed.

Hi @jlenain and @hashkar, I'm rerunning in local after the rebase. I had to change a bit _create_hdu in dqm_summary_processor.py to write 2D arrays into BinTableHDU -- we need it for the temperature trend for each drawer. I should be able to push a fix ~soon.

[alessandromontanari]

Hi @jlenain and @hashkar, I believe I fixed what was missing for the temperature trends. I had to rework a bit in camera_monitoring.py how to plot the temperature values vs the time points. Now there is a line for each drawer. I also had to rework _create_hdu in dqm_summary_processor.py, which now checks the data type (to write into the Table for the fits) and writes the data into a ImageHDU (instead of a BinTableHDU) if the array is 2D.
Now the trends look like this:

@jlenain, did we specifically need BinTableHDU for the DB? ImageHDU is different, but I struggled a lot with the astropy Table, and this was the only solution that worked. I don't think this should be a problem for the web app, we just need to write the appropriate function to read the 2D (or N-D) datasets, and it should be ok....

[jlenain]

@jlenain, did we specifically need BinTableHDU for the DB? ImageHDU is different, but I struggled a lot with the astropy Table, and this was the only solution that worked. I don't think this should be a problem for the web app, we just need to write the appropriate function to read the 2D (or N-D) datasets, and it should be ok....

No, I don't think so. I just used BinTableHDU because up to now, we did not need to store more complex objects in the DQM FITS file. For the DB, as long as we are able to parse the FITS file back, it should not be an issue, since ZODB can cope with any kind of python objects.

[alessandromontanari]

Hi @jlenain and @hashkar, is there anything left to test on this PR? Otherwise, can we merge it?

[jlenain]

Hi @jlenain and @hashkar, is there anything left to test on this PR? Otherwise, can we merge it?

Hi @alessandromontanari,

Sorry, I wanted to test it locally, and everything runs fine. I'll merge this PR. Thanks !

[jlenain]

Thanks, @alessandromontanari !