TriOS L0 to L1A assumes 255 spectral channels

[davidbenncsiro]

By the time we reach ProcessL1aTriOS.py, line 368, it is possible for len(my_arr) != len(ds_dt) to be true in the case where the number of spectral channels is not 255.

That is the case for some PyTriOS data I convert to .mlb format. I'm assuming this varies with RAMSES model.

For now, to get past this, in my working copy I "sacrifice" the first channel:

        rec_arr = np.rec.fromarrays(my_arr[1:], dtype=ds_dt)

I also tried basing wl, so ds_dt, on the length of my_arr:

        # create spectral and wavelength arrays
        my_arr = np.array(data).transpose()

        wl = []
        for i in range(1,len(my_arr)+1):
            wl.append(str(round((c0 + c1*(i+1) + c2*(i+1)**2 + c3*(i+1)**3), 2)))

but that led to other required downstream changes for L1AQC to L1B processing. I did not complete those modifications (locally) but I'm happy to contribute to a solution if it seems feasible and helps.

So, ultimately my question is: do you think it's reasonable for HyperCP to cater for variable RAMSES spectral channels, e.g. 255 vs 256?

[juanchossn]

Thanks David Benn for posting this issue. This is certainly possible, let us discuss it in our next team meeting, and we'll be back to you.

[davidbenncsiro]

Thank you @juanchossn!

[juanchossn]

@davidbenncsiro , could you share your TriOS RAW datafile + calibration file with us? This will allow us to reproduce the issue. Please see our contact emails in the README, thanks!

[juanchossn]

@davidbenncsiro Could you please share if possible the ancillary and config files you are using as well? Thanks!

[davidbenncsiro]

Sorry for the delay @juanchossn. Here are the files. TriOS files.zip

As you will see from some of the headers, we're still settling on some metadata values.

[oceancolorcoder]

I took a look at your data and the problem is a mismatch in the expected number of pixels based in the calibration and characterization files and the pixels in the raw .mlb data. I am going to bring this up with the team tomorrow and get back to you as soon as I can -- hopefully before I leave for the holidays the day after.

[davidbenncsiro]

Thanks @oceancolorcoder

[oceancolorcoder]

@davidbenncsiro No solid answers today, sorry. TriOS (presumably within MSDA-XE, but I'm no TriOS expert) is doing something strange in which most of the files I see have 256 pixels expected in the calibration and characterization files but only 255 in the raw (.mlb) files, hence HyperCP expects 255. Your files have 256 pixels in the raw file. I've put out inquiries with people who know more about TriOS datasets than I do and will follow up asap. Unfortunately, that won't happen until after the holidays.

[davidbenncsiro]

No worries @oceancolorcoder. Talk in the New Year. I hope you have a good break.

[oceancolorcoder]

Hi @davidbenncsiro, I wanted to drop a note that we are still working with TriOS trying to get to the bottom of your mismatched pixels issue. Will let you know when we know more.
Dirk

[davidbenncsiro]

Thanks @oceancolorcoder. Appreciate it!