Verify usage of FillValue vs. _FillValue

[gerritholl]

Several readers contain references to a FillValue NetCDF attribute. Normally this should be _FillValue. We should check whether those FillValues are a typo and should be _FillValue instead. Considering that FillValue is non-standard, it would probably be good to add a note to the sourcecode that it is like this in the data and there is no mistake in satpy.

• FCI L1C data loading. Fixed by Fix fill value removal for non-integer FCI data #3060 .

  satpy/satpy/readers/fci_l1c_nc.py

  Lines 359 to 361 in b6b4ef1

  	fv = attrs.pop(
  	"FillValue",
  	default_fillvals.get(data.dtype.str[1:], np.float32(np.nan)))

• FCI L1C calibration.
  

  satpy/satpy/readers/fci_l1c_nc.py

  Lines 707 to 714 in b6b4ef1

  	if v == v.attrs.get("FillValue",
  	default_fillvals.get(v.dtype.str[1:])):
  	logger.error(
  	"{:s} set to fill value, cannot produce "
  	"brightness temperatures for {:s}.".format(
  	v.attrs.get("long_name",
  	"at least one necessary coefficient"),
  	measured))

  
  

  satpy/satpy/readers/fci_l1c_nc.py

  Lines 730 to 734 in b6b4ef1

  	if cesi == cesi.attrs.get(
  	"FillValue", default_fillvals.get(cesi.dtype.str[1:])):
  	logger.error(
  	"channel effective solar irradiance set to fill value, "
  	"cannot produce reflectance for {:s}.".format(measured))

• AGRI L1 data loading.

  satpy/satpy/readers/agri_l1.py

  Lines 71 to 73 in b6b4ef1

  	data.attrs.pop("FillValue", None)
  	data.attrs.pop("Intercept", None)
  	data.attrs.pop("Slope", None)

• GHI L1 data loading.

  satpy/satpy/readers/ghi_l1.py

  Lines 73 to 75 in b6b4ef1

  	data.attrs.pop("FillValue", None)
  	data.attrs.pop("Intercept", None)
  	data.attrs.pop("Slope", None)

  (also in the unit test:

  satpy/satpy/tests/reader_tests/test_ghi_l1.py

  Line 56 in b6b4ef1

  "FillValue": np.array(-65535.0),

  and several more occurences)
• FY4 base, several times:

  satpy/satpy/readers/fy4_base.py

  Line 101 in b6b4ef1

  fill_value = data.attrs.get("FillValue")

  satpy/satpy/readers/fy4_base.py

  Line 141 in b6b4ef1

  ds_info["fill_value"] = data.attrs["FillValue"].item()

  satpy/satpy/readers/fy4_base.py

  Line 154 in b6b4ef1

  data.attrs["_FillValue"] = data.attrs["FillValue"].item()

  satpy/satpy/readers/fy4_base.py

  Line 170 in b6b4ef1

  data.data = da.where(data.data == data.attrs["FillValue"].item(), np.nan, data.data)

• MERSI. This one is probably such in the data.

  satpy/satpy/readers/mersi_l1b.py

  Lines 177 to 178 in b6b4ef1

  	fill_value = attrs.pop("_FillValue", np.nan) if self.platform_name in ["FY-3A", "FY-3B"] else \
  	attrs.pop("FillValue", np.nan) # covered by valid_range

• MWS data loading.

  satpy/satpy/readers/mws_l1b.py

  Lines 202 to 204 in b6b4ef1

  	fv = attrs.pop(
  	"FillValue",
  	default_fillvals.get(data.dtype.str[1:], np.nan))

This can lead to bugs such as #3058, but it might also be originally like this in the data. The cases quoted above are not easily caught by unit tests, as .pop(...) never fails and would need to be explicitly tested against.

[djhoese]

AGRI L1:

variables:
        float CALChannel02(phony_dim_0) ;
                CALChannel02:valid_range = 0.f, 1.5f ;
                CALChannel02:FillValue = -65535.f ;

[djhoese]

MERSI L1B (tf2019071182739.FY3D-X_MERSI_0250M_L1B.HDF):

  variables:
        ushort EV_250_Emissive_b24(phony_dim_5, phony_dim_6) ;
                EV_250_Emissive_b24:units = "mW/ (m2 cm-1 sr)" ;
                EV_250_Emissive_b24:Intercept = 0.f ;
                EV_250_Emissive_b24:Slope = 0.01f ;
                EV_250_Emissive_b24:band_name = "Band 24" ;
                EV_250_Emissive_b24:valid_range = 0, 4095 ;
                EV_250_Emissive_b24:FillValue = 65535 ;