Config and mapping files for ADPUPA prepBUFR #17

PraveenKumar-NOAA · 2024-12-17T21:07:11Z

This PR adds following config/ and mapping/ files for the ADPUPA prepBUFR:

config/bufr_bufr_backend_adpupa_prepbufr.yaml
config/bufr_script_backend_adpupa_prepbufr.yaml
mapping/bufr_adpupa_prepbufr.py
mapping/bufr_adpupa_prepbufr_mapping.yaml

All of the four jobs (bufr2netcdf, script2netcdf, bufr_backend, script_backend) were run and tested using serial and parallel execution with nproc=2.

nicholasesposito

Some things need to be addressed.

nicholasesposito · 2024-12-17T21:53:29Z

dump/mapping/bufr_adpupa_prepbufr.py

+#!/usr/bin/env python3
+import os
+import sys
+sys.path.append('/scratch1/NCEPDEV/da/Praveen.Singh/HERA/bufr-query/build/lib')


Get rid of these sys.path.append lines.

@nicholasesposito thanks, removed these lines.

nicholasesposito · 2024-12-17T21:54:50Z

dump/mapping/bufr_adpupa_prepbufr.py

+        log_method(message)
+
+
+#def Compute_dateTime(cycleTimeSinceEpoch, hrdr):


dateTime should stay in. For the regular mapping yaml, @emilyhcliu and I discussed and said that DHR should determine dateTime if the script is not being used, BUT the script is being used, dateTime should be computed using the script.

@nicholasesposito added a Python subroutine to compute dateTime in the Python script.

dump/mapping/bufr_adpupa_prepbufr.py

xincjin-NOAA · 2024-12-18T01:58:08Z

dump/mapping/bufr_adpupa_prepbufr.py

+    logging(comm, 'DEBUG', f'Do ps ObsError calculations')
+    poe = container.get('variables/pressureError')
+    psoe = np.full(poe.shape[0], poe.fill_value)
+    psoe = np.where(cat == 0, poe, psoe)


Is it possible to define these (and the corresponding container.replace) as a method? There so many repeat codes here.

dump/mapping/bufr_adpupa_prepbufr.py

…, hrdr) subroutine

PraveenKumar-NOAA · 2025-01-16T15:38:30Z

@nicholasesposito @xincjin-NOAA @BrettHoover-NOAA @emilyhcliu @rmclaren: please review this PR, thanks!

rmclaren · 2025-01-16T22:13:35Z

dump/mapping/bufr_adpupa_prepbufr.py

+        log_method(message)
+
+
+def Compute_dateTime(cycleTimeSinceEpoch, hrdr):


pep 8 naming def _compute_datetime(cycleTimeSinceEpoch, hrdr)

@PraveenKumar-NOAA could you update the function name to comply with the prep8 naming convention?

@rmclaren done, thanks!

rmclaren · 2025-01-16T22:13:45Z

dump/mapping/bufr_adpupa_prepbufr.py

+    return dateTime
+
+
+def Mask_typ_for_var(typ, var):


@rmclaren done, thanks!

rmclaren · 2025-01-16T22:15:32Z

dump/mapping/bufr_adpupa_prepbufr.py

+def _make_description(mapping_path, cycle_time, update=False):
+    description = bufr.encoders.Description(mapping_path)
+
+    ReferenceTime = np.int64(calendar.timegm(time.strptime(str(int(cycle_time)), '%Y%m%d%H')))


pep 8- reference_time

str(int(cycle_time)) ??

Pease address the reviewer's comments. Thanks.

@rmclaren done, thanks!

rmclaren · 2025-01-16T22:18:56Z

dump/mapping/bufr_adpupa_prepbufr.py

+                longName=var['longName']
+            )
+
+        description.add_global(name='datetimeReference', value=str(ReferenceTime))


Should you format the reference time in some way or is it just seconds since January 1970?

Basd on IODA convention table:
https://docs.google.com/spreadsheets/d/1fHb2QIHOwgp4d6plyn9Dj9HZFZsDRxcPaUV6hdTb5BU/edit?gid=46200857#gid=46200857

Look for the global attribute tab.
The dateTimeReference format is string with ISO form.
ISO-8601 string like 2022-02-15T12:00:00Z

@rmclaren done, thanks!

rmclaren · 2025-01-16T22:23:25Z

dump/mapping/bufr_adpupa_prepbufr.py

+
+    logging(comm, 'DEBUG', f'Do DateTime calculation')
+    hrdr = container.get('variables/obsTimeMinusCycleTime')
+    hrdr_paths = container.get_paths('variables/obsTimeMinusCycleTime')


you never use this result... same for other paths

@rmclaren done, thanks!

rmclaren · 2025-01-16T22:23:50Z

dump/mapping/bufr_adpupa_prepbufr.py

+
+    logging(comm, 'DEBUG', f'Change longitude range from [0,360] to [-180,180]')
+    lon = container.get('variables/longitude')
+    lon_paths = container.get_paths('variables/longitude')


@rmclaren done, thanks!

rmclaren · 2025-01-16T22:24:03Z

dump/mapping/bufr_adpupa_prepbufr.py

+
+    logging(comm, 'DEBUG', f'observationType')
+    typ = container.get('variables/observationType')
+    typ_paths = container.get_paths('variables/observationType')


@rmclaren done, thanks!

rmclaren · 2025-01-16T22:25:33Z

dump/mapping/bufr_adpupa_prepbufr.py

+
+    logging(comm, 'DEBUG', f'Do ps calculation')
+    pob = container.get('variables/pressure')
+    pob_paths = container.get_paths('variables/pressure')


@rmclaren done, thanks!

rmclaren · 2025-01-16T22:25:38Z

dump/mapping/bufr_adpupa_prepbufr.py

+    logging(comm, 'DEBUG', f'Do tsen and tv calculation')
+    tpc = container.get('variables/temperatureEventProgramCode')
+    tob = container.get('variables/airTemperature')
+    tob_paths = container.get_paths('variables/airTemperature')


@rmclaren done, thanks!

nicholasesposito · 2025-01-22T17:04:38Z

dump/mapping/bufr_adpupa_prepbufr_mapping.yaml

+    # ObsError: specificHumidity
+    - name: "ObsError/specificHumidity"
+      source: variables/specificHumidityError
+      longName: "Specific Humidity Error"


Does this have a unit?

@nicholasesposito thanks, added unit for specificHumidity.

nicholasesposito

Changes requested for global attributes.
One other question

nicholasesposito · 2025-01-22T17:08:52Z

dump/mapping/bufr_adpupa_prepbufr_mapping.yaml

+  type: netcdf
+
+  globals:
+    - name: "data_format"


Standard is dataOriginalFormatSpec

@nicholasesposito, thanks. Fixed.

nicholasesposito · 2025-01-22T17:09:17Z

dump/mapping/bufr_adpupa_prepbufr_mapping.yaml

+      type: string
+      value: "prepbufr"
+
+    - name: "data_type"


Standard is "platforms"

@nicholasesposito, thanks. Fixed.

nicholasesposito · 2025-01-22T17:09:37Z

dump/mapping/bufr_adpupa_prepbufr_mapping.yaml

+      type: string
+      value: "ADPUPA"
+
+    - name: "subsets"


We don't need this.

@nicholasesposito, thanks. Fixed.

nicholasesposito · 2025-01-22T17:10:32Z

dump/mapping/bufr_adpupa_prepbufr_mapping.yaml

+      type: string
+      value: "ADPUPA"
+
+    - name: "data_provider"


Standard is "dataProviderOrigin"

@nicholasesposito, thanks. Fixed.

nicholasesposito · 2025-01-22T17:11:02Z

dump/mapping/bufr_adpupa_prepbufr_mapping.yaml

+
+    - name: "data_provider"
+      type: string
+      value: "U.S. NOAA"


Change to "U.S. NOAA NCEP"

@nicholasesposito, thanks. Fixed.

nicholasesposito · 2025-01-22T17:13:18Z

dump/mapping/bufr_adpupa_prepbufr_mapping.yaml

+      type: string
+      value: "U.S. NOAA"
+
+    - name: "data_description"


Standard is "description"

@nicholasesposito thanks, done.

emilyhcliu · 2025-01-22T18:28:16Z

Tested bufr2netcdf conversion and found a problem with MPI - The total number of obs from the MPI runs are not consistent with the run without MPI. Issue #46 opened for this in bufr-query.

The number of obs in output:

MPI = 0  ----> 270600
MPI = 1  ----> 270600
MPI = 2 ----> 261786
MPI = 4 ----> 250079
MPI = 8 ---->  239868

We are losing more data in the output as we increase the number of MPIs

nicholasesposito · 2025-01-22T19:09:23Z

Tested bufr2netcdf conversion and found a problem with MPI - The total number of obs from the MPI runs are not consistent with the run without MPI. Issue #46 opened for this in bufr-query.

The number of obs in output:
MPI = 0  ----> 270600
MPI = 1  ----> 270600
MPI = 2 ----> 261786
MPI = 4 ----> 250079
MPI = 8 ---->  239868
We are losing more data in the output as we increase the number of MPIs

@emilyhcliu this may be the same issue as NOAA-EMC/bufr-query#40

PraveenKumar-NOAA · 2025-01-22T19:17:39Z

@nicholasesposito yes, this is a similar issue and associated with the group_by_field variable. Looking into the details.

emilyhcliu · 2025-02-10T19:43:04Z

@PraveenKumar-NOAA Please address the reviews' comments, get approved, and then we can merge this to SPOC.

… per reviewer comments

rmclaren · 2025-02-11T05:14:37Z

dump/mapping/bufr_adpupa_prepbufr.py

@@ -163,30 +163,30 @@ def _make_obs(comm, input_path, mapping_path, cycle_time):

    logging(comm, 'DEBUG', f'Do DateTime calculation')
    hrdr = container.get('variables/obsTimeMinusCycleTime')
-    hrdr_paths = container.get_paths('variables/obsTimeMinusCycleTime')
+    #hrdr_paths = container.get_paths('variables/obsTimeMinusCycleTime')


Delete commented out code, this example and the following ones..

@rmclaren thanks, deleted all the commented lines.

rmclaren · 2025-02-11T05:15:52Z

dump/mapping/bufr_adpupa_prepbufr.py

    cycleTimeSinceEpoch = np.int64(calendar.timegm(time.strptime(str(int(cycle_time)), '%Y%m%d%H')))
-    dateTime = Compute_dateTime(cycleTimeSinceEpoch, hrdr)
+    dateTime = _compute_datetime(cycleTimeSinceEpoch, hrdr)


result of dateTime doesn't seem to be used

@rmclaren thanks, reading dateTime now.

rmclaren · 2025-02-11T05:16:19Z

dump/mapping/bufr_adpupa_prepbufr.py

    logging(comm, 'DEBUG', f'dateTime min/max = {dateTime.min()} {dateTime.max()}')

    logging(comm, 'DEBUG', f'Change longitude range from [0,360] to [-180,180]')
    lon = container.get('variables/longitude')
-    lon_paths = container.get_paths('variables/longitude')
+    #lon_paths = container.get_paths('variables/longitude')
    lon[lon>180] -= 360

    logging(comm, 'DEBUG', f'observationType')
    typ = container.get('variables/observationType')


typ doesn't seem to be used

@rmclaren thanks, fixed.

Config and mapping files for ADPUPA prepBUFR

76603e0

PraveenKumar-NOAA requested review from emilyhcliu, nicholasesposito, BrettHoover-NOAA and xincjin-NOAA December 17, 2024 21:07

nicholasesposito reviewed Dec 17, 2024

View reviewed changes

xincjin-NOAA reviewed Dec 18, 2024

View reviewed changes

dump/mapping/bufr_adpupa_prepbufr.py Show resolved Hide resolved

Removed sys.path lines and added Compute_dateTime(cycleTimeSinceEpoch…

c3d7889

…, hrdr) subroutine

PraveenKumar-NOAA requested a review from rmclaren January 16, 2025 15:33

Praveen Singh and others added 5 commits January 16, 2025 10:08

Added ReferenceTime to the Python script

036604f

Renamed and relocated the bufr and script backend yamls

2eb25ff

Removed extra files

53f0685

Updated version

c668895

Merge branch 'develop' into feature/adpupa_prepbufr_NEW

bb3f9e8

rmclaren reviewed Jan 16, 2025

View reviewed changes

nicholasesposito reviewed Jan 22, 2025

View reviewed changes

Changed the Python script and mapping file for the adpupa_prepbufr as…

5404783

… per reviewer comments

rmclaren reviewed Feb 11, 2025

View reviewed changes

Removed commented lines/ paths

5367dfd

		log_method(message)


		#def Compute_dateTime(cycleTimeSinceEpoch, hrdr):

		log_method(message)


		def Compute_dateTime(cycleTimeSinceEpoch, hrdr):

Config and mapping files for ADPUPA prepBUFR #17

Are you sure you want to change the base?

Config and mapping files for ADPUPA prepBUFR #17

Conversation

PraveenKumar-NOAA commented Dec 17, 2024 • edited Loading

nicholasesposito left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PraveenKumar-NOAA commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicholasesposito left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilyhcliu commented Jan 22, 2025

nicholasesposito commented Jan 22, 2025 • edited Loading

PraveenKumar-NOAA commented Jan 22, 2025

emilyhcliu commented Feb 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PraveenKumar-NOAA commented Dec 17, 2024 •

edited

Loading

nicholasesposito commented Jan 22, 2025 •

edited

Loading