[Bug]: DISP-S1 Query Job in historical processing must validate all CSLC bursts

[philipjyoon]

Checked for duplicates

Yes - I've already checked

Describe the bug

On rare occasion, less than 0.1% of the time, CMR may return incomplete amount of data. Seems to be a transient issue.

When running in historical mode the PCM query code currently filters out any bursts that don’t belong in burst patterns but does not validate that all patterns have sufficiently been retrieved. It does this when we’re in forward mode for triggering rule. We had assumed that CMR query result would always be consistent and since we’d had done the burst pattern grouping ahead of time in the database file, we wouldn't need to further validate. This is a bad assumption and could bite us if CMR has a large inconsistency issue in the future.

The current behavior is that the query and download jobs will succeed but SCIFLO will fail because the coverage is consistent. In fact, the query job should fail in historical processing mode.

The current work-around is to simply resubmit the query job and verify that all CSLC bursts are returned from CMR. This can be inspected by comparing the query job log output to disp_s1_burst_db_tool.py

What did you expect?

n/t

Reproducible steps

1.
2.
3.
...

Environment

- Version of this software [e.g. vX.Y.Z]
- Operating System: [e.g. MacOSX with Docker Desktop vX.Y]
...