Bug in L1A merging

[PennyHow]

There is a bug in pypromice that has been discovered in the intermediary merging step between L1 and L1A (the merged L1 dataset). If a L0tx file is empty then it is passed through L0toL1 without being caught and causes a bug when it comes to merging with other L1 datasets.

2024-11-11 08:06:26,882; INFO; pypromice.process.aws; L0 data successfully loaded from NUK_P_300534064388790_1.txt2024-11-11 

08:06:26,914; INFO; pypromice.process.aws; L0 data successfully loaded from NUK_P_300534063814740_2.txt2024-11-11 

08:06:26,929; INFO; pypromice.process.aws; Level 1 processing...

Traceback (most recent call last):  
File "/home/aws/.conda/envs/aws-operational-processing/bin/get_l2", line 8, in <module>    
sys.exit(main())  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/pypromice/process/get_l2.py", line 78, in main
    _ = get_l2(  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/pypromice/process/get_l2.py", line 57, in get_l2
    aws.getL1() 
 File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/pypromice/process/aws.py", line 146, in getL1
    self.L1 = [toL1(item, self.vars) for item in self.L0]  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/pypromice/process/aws.py", line 146, in <listcomp>
    self.L1 = [toL1(item, self.vars) for item in self.L0]  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/pypromice/process/L0toL1.py", line 62, in toL1  
    ds['t_u_interp'] = interpTemp(ds['t_u'], vars_df)  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/pypromice/process/L0toL1.py", line 299, in interpTemp
    temp_interp = temp.interpolate_na(dim='time', max_gap=max_interp)  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/xarray/core/dataarray.py", line 3598, in interpolate_na
    return interp_na(  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/xarray/core/missing.py", line 357, in interp_na
    index = get_clean_interp_index(self, dim, use_coordinate=use_coordinate)  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/xarray/core/missing.py", line 292, in get_clean_interp_index
    offset = type(index[0])(1970, 1, 1)  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5389, in __getitem__ 
   return getitem(key)  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/pandas/core/arrays/datetimelike.py", line 381, in __getitem__
    result = cast("Union[Self, DTScalarOrNaT]", super().__getitem__(key))  
File "/home/aws/.conda/envs/aws-operational-processing/lib/python3.10/site-packages/pandas/core/arrays/_mixins.py", line 284, in __getitem__
    result = self._ndarray[key]IndexError: index 0 is out of bounds for axis 0 with size 0

Proposed solution/s

• Make sure the toml test in aws-l0 checks for empty L0 files
• Make sure that pypromice removes empty L0 files when loading
• OR make sure empty datasets are not included in the L1A merging