Mid-year startup with DGLC or CISM can lead to unexpected non-zero delta

@ekluzek

With stub-glacier we didn't have this problem, but it's coming up with either DGLC or CISM. Two CMI6-SSP tests that passed now fail.

Two ctsm_sci tests that are failing unexpectedly are:

SMS_Ld5.f09_g17.ISSP245Clm50BgcCrop.derecho_intel.clm-ciso_dec2050Start ( RUN)
SMS_Ld5.f09_t232.ISSP245Clm60BgcCropCrujra.derecho_intel.clm-ciso_dec2050Start  ( RUN)

With this in cesm.log:

dec1863.hsn.de.hpc.ucar.edu 1497:  set_curr_delta: found unexpected non-zero delta mid-year: ERROR in AnnualFluxDr
dec1863.hsn.de.hpc.ucar.edu 1497:  ibbler.F90 at line 277
dec1875.hsn.de.hpc.ucar.edu 1547:  Negative conc. in ch4tran. c,j,deficit (mol):         233           1
dec1875.hsn.de.hpc.ucar.edu 1547:   1.118330921757848E-003
dec1875.hsn.de.hpc.ucar.edu 1547:  Negative conc. in ch4tran. c,j,deficit (mol):         233           2
dec1875.hsn.de.hpc.ucar.edu 1547:   1.710347228052210E-003
dec1786.hsn.de.hpc.ucar.edu 1403: Image              PC                Routine            Line        Source             
dec1786.hsn.de.hpc.ucar.edu 1403: cesm.exe           0000000000FC55C6  shr_abort_abort           110  shr_abort_mod.F90
dec1786.hsn.de.hpc.ucar.edu 1403: cesm.exe           000000000059D8BE  endrun_vanilla             63  abortutils.F90
dec1786.hsn.de.hpc.ucar.edu 1403: cesm.exe           000000000059D91D  endrun_write_poin          95  abortutils.F90
dec1786.hsn.de.hpc.ucar.edu 1403: cesm.exe           0000000000741B52  set_curr_delta            275  AnnualFluxDribbler.F90
dec1786.hsn.de.hpc.ucar.edu 1403: cesm.exe           0000000000DC1B73  dyn_cnbal_patch           436  dynConsBiogeochemMod.F90
dec1786.hsn.de.hpc.ucar.edu 1403: cesm.exe           0000000000871460  dynamicareaconser         761  CNVegetationFacade.F90
dec1786.hsn.de.hpc.ucar.edu 1403: cesm.exe           0000000000624083  dynsubgrid_driver         352  dynSubgridDriverMod.F90

Originally posted by @ekluzek in #4110 (comment)

There are plenty of DecStart tests that DO PASS. So what is different about the above CMIP6-SSP cases?

All of these pass, including the last one that is a SSP2-4.5 case just with a lower resolution.

ERI_D_Ld20.f10_f10_mg37.IHistClm60Bgc.derecho_intel.clm-default--clm-midDecStart--clm-allowTransientYearJump
ERP_D.f10_f10_mg37.IHistClm60Bgc.derecho_gnu.clm-decStart
ERP_D.f10_f10_mg37.IHistClm60Bgc.derecho_gnu.clm-decStart--clm-matrixcnOn_ignore_warnings
ERP_D.f10_f10_mg37.IHistClm60Bgc.derecho_intel.clm-decStart
ERP_D_Ld10_P64x2.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-ciso_decStart
ERP_D_Ld10_P64x2.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-ciso_decStart--clm-matrixcnOn_ignore_warnings
ERP_D_Ld5.f10_f10_mg37.I2000Clm60Sp.derecho_intel.clm-decStart--clm-nofireemis
ERP_D_P128x1_Ld26.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-crop--clm-midDecStart--clm-RxCropCalsAdaptGGCMI
ERP_P64x2_D_Ld10.f10_f10_mg37.IHistClm50SpG.derecho_intel.clm-glcMEC_decrease--clm-nofireemis
ERS_D_Ld10.f10_f10_mg37.IHistClm50Sp.derecho_intel.clm-collapse_pfts_78_to_16_decStart_f10
ERS_D_Ld12.f10_f10_mg37.I1850Clm50BgcCropG.derecho_intel.clm-glcMEC_spunup_inc_dec_bgc
ERS_D_Ld5.f10_f10_mg37.IHistClm50BgcQian.derecho_intel.clm-ciso_bombspike1963DecStart
ERS_D_Ld5.f10_f10_mg37.IHistClm50BgcQian.derecho_intel.clm-ciso_bombspike1963DecStart--clm-matrixcnOn_ignore_warnings
PEM_D_P128x1_Ld26.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-crop--clm-midDecStart--clm-RxCropCalsAdaptGGCMI
SMS_D_Ld3_PS.f09_g17.I1850Clm60BgcNoAnthroRtm.derecho_intel.clm-decStart1851_noinitial--clm-matrixcnOn
SMS_D_Ld3_PS.f09_g17.I1850Clm60SpNoAnthroRtm.derecho_intel.clm-decStart1851_noinitial--clm-nofireemis
SMS_D_Ld3_PS.f09_t232.I1850Clm60SpCrujraNoAnthroRtm.derecho_intel.clm-decStart1851_noinitial--clm-nofireemis
SMS_Ld3_PS.f09_g17.IHistClm50BgcCrop.derecho_intel.clm-f09_dec1990Start_GU_LULCC
SMS_Ld3_PS.f09_g17.IHistClm50BgcCrop.derecho_intel.clm-f09_dec1990Start_GU_LULCC--clm-matrixcnOn_ignore_warnings
SMS_Ld5.f10_f10_mg37.ISSP245Clm60BgcCropCrujra.derecho_gnu.clm-ciso_dec2050Start

Same fail seen with:

SMS_Ld5.f09_t232.SSP245_DATM%CRUJRA2024b_CLM60%BGC-CROP_SICE_SOCN_MOSART_CISM2%GRIS-EVOLVE_SWAV_SESP.derecho_intel.clm-ciso_dec2050Start

The dribbler that's failing is this one from CNvegStateType.f90

    if (get_for_testing_allow_non_annual_changes()) then
       allows_non_annual_delta = .true.
    else if (use_cndv) then
       allows_non_annual_delta = .true.
    else
       allows_non_annual_delta = .false.
    end if
    this%dwt_dribbler_patch = annual_flux_dribbler_patch( &
         bounds = bounds, &
         name = 'dwt', &
         units = 'fractional area', &
         allows_non_annual_delta = allows_non_annual_delta)

So this fail should only happen for Bgc cases fro one. But, it seems really odd now that it would fail by adding a GLC model.

There's three bare-soil points that fail. Based on the lat/lons they all look like they are likely in Greenland.

dec1629.hsn.de.hpc.ucar.edu 1572: iam = 1444: gridcell longitude    =  346.2500000
dec1629.hsn.de.hpc.ucar.edu 1572: iam = 1444: gridcell latitude     =   81.5183246
dec1556.hsn.de.hpc.ucar.edu 1403: iam = 1275: gridcell longitude    =  342.5000000
dec1556.hsn.de.hpc.ucar.edu 1403: iam = 1275: gridcell latitude     =   79.6335079
dec1559.hsn.de.hpc.ucar.edu 1497: iam = 1369: gridcell longitude    =  346.2500000
dec1559.hsn.de.hpc.ucar.edu 1497: iam = 1369: gridcell latitude     =   80.5759162

So it's something about baresoil points in Greenland. Is there something going on with the downscaling of data over these points that BGC has trouble with? Causing some sort of jump maybe at the first time step?

OK, this all happens at startup and I think what must be going on is that for 2050 there must be a couple points that aren't glacier, but still land and so get turned into bare-soil points and it gets filled with the closest DWT dribbler values to start with from use_init_interp. And then it just doesn't like it being a positive value? So is this unique to the starting year? I'm testing if it goes away if GLC_TWO_WAY_COUPLING==FALSE.

Looking at the line causing the failure in dynConsBiogeochemMod.F90, it looks like the failure is in the dribbler for dwt, which is computed as:

dwt(p) = patch%wtgcell(p)-prior_weights%pwtgcell(p)

where prior_weights is set near the start of dynSubgrid_driver.

This is especially puzzling, but the good news is that this should be easier to trace than if the problem was happening with the dribbler of an actual state variable.

I'd suggest printing patch%wtgcell(p) for all of the patches in one of the offending grid cells at many points throughout dynSubgrid_driver to see where that changes. (Note that call prior_weights%set_prior_weights(bounds_clump) should set the prior_weights variable to the value of patch%wtgcell(p) at the start of dynSubgrid_driver, but you could confirm that, too.)

Next: need milestone and/or assignee

For sciences uses, we'll likely want to support user running CLM6_SSP CMIP6 scenarios. Do we just need these runs to be in stub glacier compsets?

At this point, this doesn't seem like high priority, with the caveat that we'll likely need to support some kind of capability to run future scenarios in the CLM6 release.

For sciences uses, we'll likely want to support user running CLM6_SSP CMIP6 scenarios. Do we just need these runs to be in stub glacier compsets?

At this point, this doesn't seem like high priority, with the caveat that we'll likely need to support some kind of capability to run future scenarios in the CLM6 release.

I'm fine with this assessment given numerous competing priorities, but I do want to add: Given how unclear it is to me why this is happening, it is hard to say that this problem is restricted to certain scenarios. It seems very possible that this error will come up in other situations as well... it's just hard to say at this point. I'm fine with what seems to be the plan of waiting and seeing if this happens more generally before making it a priority.

I spent some time investigating this in the course of looking into a similar-looking issue that turned out to be from a different cause (ESCOMP/CISM-wrapper#152). I think I understand the cause of this error now, and I'll submit a fix for it shortly.

I looked into this by examining what was happening in one of the problem grid cells in SMS_Ld5.f09_g17.ISSP245Clm50BgcCrop.derecho_intel.clm-ciso_dec2050Start (on processor 1343). The issue is:

Initially, the input datasets specified that this grid cell had 100% glacier and slightly greater than 0 crop area
The crop area got removed to bring the total to 100%, leading to just 100% glacier
Then, in the glacier initialization block in the driver, when we get fields from GLC, glacier area was set to less than 100%
This led to a growth in natveg. But crop area was still 0 at this point (because the logic for shrinking glacier dictates that it's all replaced by natural veg). Prior weights were set here.
Then in the main dyn subgrid driver code in the first time step, crop area was re-read. This led to crop area becoming non-zero again. This time, since glacier was < 100%, there was enough natveg to accommodate this non-zero crop area.
So there is now a difference from prior weights: crop area was 0 in prior weights, but non-zero to start the first time step.
This causes the dynamic landunit code to want to do some adjustments, but this runs into the check that there shouldn't be any area changes outside of the year boundary.

In general, I think this problem can arise when the glacier area on the surface dataset or initial conditions file is larger than what GLC dictates in the first time step, and the natural veg area isn't enough to accommodate this larger glacier area because of the presence of other landunits (such as crop in the above case): in this situation, the other landunits get squeezed down in this block:

CTSM/src/main/clm_driver.F90

Lines 238 to 250 in 1531f6f

    
           if (need_glacier_initialization) then 
        
              !$OMP PARALLEL DO PRIVATE (nc, bounds_clump) 
        
              do nc = 1, nclumps 
        
                 call get_clump_bounds(nc, bounds_clump) 
        
                 call glc2lnd_inst%update_glc2lnd_fracs( & 
        
                      bounds = bounds_clump) 
        
                 call dynSubgrid_wrapup_weight_changes(bounds_clump, glc_behavior) 
        
              end do 
        
              !$OMP END PARALLEL DO 
        
           end if

but then regrow in the first call to dynSubgrid_driver.

In a typical run that starts on Jan 1, this causes a bit of oddity but nothing catastrophic, but when starting mid-year (as in CTSM's decStart tests) it causes an abort because CTSM doesn't allow mid-year area changes. I spent quite a while trying to come up with a better general way to handle model initialization that would solve this problem, but everything I could come up with felt like it caused at least as many new problems as it solved, or would involve significant additional code complexity for marginal benefit. So I think the fix will be a kludge to handle this specific situation of a mid-year start.

Only do file-based land cover change if is_beg_curr_year

See #4281 for a fix

Navigation Menu

Repository navigation

Mid-year startup with DGLC or CISM can lead to unexpected non-zero delta #4165

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if (need_glacier_initialization) then
	!$OMP PARALLEL DO PRIVATE (nc, bounds_clump)
	do nc = 1, nclumps
	call get_clump_bounds(nc, bounds_clump)

	call glc2lnd_inst%update_glc2lnd_fracs( &
	bounds = bounds_clump)

	call dynSubgrid_wrapup_weight_changes(bounds_clump, glc_behavior)

	end do
	!$OMP END PARALLEL DO
	end if

Mid-year startup with DGLC or CISM can lead to unexpected non-zero delta #4165

Description

Activity

ekluzek commented on Aug 10, 2026

ekluzek commented on Aug 10, 2026

ekluzek commented on Aug 11, 2026

ekluzek commented on Aug 11, 2026

ekluzek commented on Aug 11, 2026

billsacks commented on Aug 11, 2026

slevis-lmwg commented on Aug 26, 2026

wwieder commented on Aug 27, 2026

billsacks commented on Aug 27, 2026

billsacks commented on Oct 22, 2026

billsacks commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions