Offline partitioning of mesh in weekly tests

[Matthew Walker (mattatmet)]

PR Summary

Sci/Tech Reviewer: mo-marqh
Code Reviewer: Benjamin Went (@MetBenjaminWent)

Overview

Issue #575 details the problems being faced by large ensembles of lfric_atm jobs with large rank counts. The main issue is the initialisation of the model which requires significant amounts of data, putting strain on the parallel file system. A component of this problem is the initialisation of the mesh which currently requires each rank to read in a single mesh file. When each rank of each ensemble is reading the same file at approximately the same time it can put significant strain on metadata servers. Prepartitioning the mesh in advance such that each rank is reading a different file can help to alleviate this problem.

The infrastructure for prepartioning and using prepartioned meshes already exists in the code. This PR makes use of this by integrating this functionality into the rose-stem lfric_atm_ex1a_weekly testing suite through the addition of two new workflows. Each uses the mesh task to prepartition the mesh for C224 and C896 resolutions.

To make use of the OpenMP parallelism in the cubedsphere_mesh_generator.F90 which prepartitions our mesh, mesh_cpus and mesh_memory task values have been passed to populate_graph_sections.cylc. These values can therefore be set in the task definition files, allowing tuning depending on mesh resolution.

Performance Tests

For these tests, an additional timing calliper has been added around the mesh initialisation section of gungho_model_mod.F90. Using this timer with Vernier, we were able to show the performance improvement that this change allows. We ran four standard C896 lfric_atm jobs plus four lfric_atm jobs with a prepartitioned mesh (generated in the mesh step of the workflow). The former had an average Max Total time for the mesh initialisation calliper of $12.34 \pm 0.13$ seconds and the later was $0.57 \pm 0.01$ seconds.

Related Issues

• This PR is related to off line partitioning for meshes #569
• linked Prepartitioning with threading race condition fix lfric_core#450

Code Quality Checklist

• I have performed a self-review of my own code
• My code follows the project's style guidelines
• Comments have been included that aid understanding and enhance the readability of the code
• My changes generate no new warnings
• All automated checks in the CI pipeline have completed successfully

Testing

• I have tested this change locally, using the LFRic Apps rose-stem suite
• If any tests fail (rose-stem or CI) the reason is understood and acceptable (e.g. kgo changes)
• I have added tests to cover new functionality as appropriate (e.g. system tests, unit tests, etc.)
• Any new tests have been assigned an appropriate amount of compute resource and have been allocated to an appropriate testing group (i.e. the developer tests are for jobs which use a small amount of compute resource and complete in a matter of minutes)

trac.log

Test Suite Results - lfric_apps - pr_688_dev_apps/run2

Suite Information

Item	Value
Suite Name	pr_688_dev_apps/run2
Suite User	matthew.walker
Workflow Start	2026-08-03T13:28:00
Groups Run	developer

Dependency	Reference	Main Like
casim	MetOffice/casim@2026.07.1	True
jules	MetOffice/jules@2026.07.1	True
lfric_apps	mattatmet/lfric_apps@offline_partition	False
lfric_core	MetOffice/lfric_core@2026.07.1	True
moci	MetOffice/moci@2026.07.1	True
SimSys_Scripts	MetOffice/SimSys_Scripts@77a5166	True
socrates	MetOffice/socrates@2026.07.1	True
socrates-spectral	MetOffice/socrates-spectral@2026.07.1	True
ukca	MetOffice/ukca@2026.07.1	True

Task Information

✅ succeeded tasks - 1216

Security Considerations

• I have reviewed my changes for potential security issues
• Sensitive data is properly handled (if applicable)
• Authentication and authorisation are properly implemented (if applicable)

Performance Impact

• Performance of the code has been considered and, if applicable, suitable performance measurements have been conducted

AI Assistance and Attribution

• Some of the content of this change has been produced with the assistance of Generative AI tool name (e.g., Met Office Github Copilot Enterprise, Github Copilot Personal, ChatGPT GPT-4, etc) and I have followed the Simulation Systems AI policy (including attribution labels)

Documentation

• Where appropriate I have updated documentation related to this change and confirmed that it builds correctly

PSyclone Approval

• If you have edited any PSyclone-related code (e.g. PSyKAl-lite, Kernel interface, optimisation scripts, LFRic data structure code) then please contact the TCD Team

Sci/Tech Review

• I understand this area of code and the changes being added
• The proposed changes correspond to the pull request description
• Documentation is sufficient (do documentation papers need updating)
• Sufficient testing has been completed

(Please alert the code reviewer via a tag when you have approved the SR)

Code Review

• All dependencies have been resolved
• Related Issues have been properly linked and addressed
• CLA compliance has been confirmed
• Code quality standards have been met
• Tests are adequate and have passed
• Documentation is complete and accurate
• Security considerations have been addressed
• Performance impact is acceptable

[iboutle]

A couple of minor comments

[iboutle]

Thanks - looks good to me now!

[James Bruten (james-bruten-mo)]

rose-stem changes look good

[mo-marqh]

minor task and group changes requested, to maintain usage consistency

[Matthew Walker (mattatmet)]

minor task and group changes requested, to maintain usage consistency

Thanks for the comments mo-marqh, I've made those changes and pushed them. Cheers.

[mo-marqh]

i have found a couple of further issues that i think require some thought.
i wonder if James Bruten (@james-bruten-mo) might also give us perspective on this?

i tried running this code using a custom decomposition task, alongside another task:
tasks_lfric_atm.cylc

{% if task_ns.conf_name.startswith("nwp_gal9-C896_MG_op_720t") and '24x16' in  task_ns.conf_name %}

    {% do task_dict.update({
        "mpi_parts": 12544,
        "xios_nodes": 7,
        "mpi_parts_xios" : 392,
        "panel_decomp": "custom",
        "xproc": 112,
        "yproc": 56,
    }) %}
{% endif %}

&&
groups_lfric_atm.cylc

    "lfric_atm_ex1a_performance_runstats": [
        "lfric_atm_nwp_gal9-C896_MG_op_720t_opDiag_32x32_perfRun1_ex1a_cce_production-32bit",
        "lfric_atm_nwp_gal9-C896_MG_op_720t_opDiag_24x16_perfRun1_ex1a_cce_production-32bit",
    ],

this fails in 2 different ways:

1. custom partitioning is not passed through from the task to the mesh generation
   • this can be fixed with a bit more environment variabling and population of graph sections
2. only 1 mesh generation task is triggered
   • but there are now different meshes being requested, the same resolution, but different partitioning
   • i think that the logic flow that decides how many mesh generation tasks to generate needs to adapt to become pre-partition aware, and to create N tasks, for the different pre-partition strategies
   • James Bruten (@james-bruten-mo) :
     • does this make logical sense?
     • is the configuration fairly extensible here?

[mo-marqh]

noting that i tried to fix

custom partitioning is not passed through from the task to the mesh generation

but this fails on a single task. Mesh generation runs, and makes many mesh files, but not all, and fails with

`Program received signal SIGSEGV: Segmentation fault - invalid memory reference.`

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:
#0  0x14a6ee653dbf in ???
#1  0x469edc in __global_mesh_map_collection_mod_MOD_get_global_mesh_map
        at mesh/global_mesh_map_collection_mod.F90:178
#2  0x4a287c in __generate_local_objects_mod_MOD_generate_local_objects
        at support/generate_local_objects_mod.f90:240
#3  0x4041f5 in MAIN__._omp_fn.0
        at /var/tmp/pbs.9255527.ehz100/reimplement_lessMDSLoad-stable_vn3dot2_performance_evaluation_Jul26_runs/run7/mesh/gnu_fast-debug-64bit/mesh_tools/cubedsphere_mesh_generator.f90:768
#4  0x14a6efa1dafd in gomp_thread_start
        at ../../../cpe-gcc-12.1.0-202208101649.1dfb26392197c/libgomp/team.c:129
#5  0x14a6f05c36e9 in ???
#6  0x14a6ee72149e in ???
#7  0xffffffffffffffff in ???
nidb1819: rank 0 died from signal 11 and dumped core
[FAIL] mkdir -p $MESH_DIR ; mpiexec -n 1 --cpu-bind=depth --depth=$OMP_NUM_THREADS $BIN_DIR/cubedsphere_mesh_generator mesh_generation.nml # return-code=139
2026-08-11T09:16:42Z CRITICAL - failed/EXIT

Using these code additions to the config:

custom partitioning patch code

--- a/rose-stem/app/mesh/opt/rose-app-C896_MG_op.conf
+++ b/rose-stem/app/mesh/opt/rose-app-C896_MG_op.conf
@@ -20,3 +20,6 @@ partition_mesh=.true.
 max_stencil_depth=9
 n_partitions=$PARTITION_PARTS
 partition_range=0,$FINAL_RANK
+panel_decomposition='$PANEL_DECOMP'
+panel_xproc=$XPROC
+panel_yproc=$YPROC
--- a/rose-stem/templates/graph/populate_graph_sections.cylc
+++ b/rose-stem/templates/graph/populate_graph_sections.cylc
@@ -154,6 +154,15 @@
             {% if "mesh_memory" in task_values %}
                 {% do mesh_dict.update({"mesh_memory": task_values["mesh_memory"]}) %}
             {% endif %}
+            {% if "panel_decomp" in task_values %}
+                {% do mesh_dict.update({"mesh_panel_decomp": task_values["panel_decomp"]}) %}
+            {% endif %}
+            {% if "xproc" in task_values %}
+                {% do mesh_dict.update({"mesh_xproc": task_values["xproc"]}) %}
+            {% endif %}
+            {% if "yproc" in task_values %}
+                {% do mesh_dict.update({"mesh_yproc": task_values["yproc"]}) %}
+            {% endif %}
             {% do tasks_to_run.update({mesh_run_task: mesh_dict}) %}
         {% endif %}
     {% endif %}
diff --git a/rose-stem/templates/runtime/generate_runtime_mesh.cylc b/rose-stem/templates/runtime/generate_runtime_mesh.cylc
index 1ef150c3..f61c7fb2 100644
--- a/rose-stem/templates/runtime/generate_runtime_mesh.cylc
+++ b/rose-stem/templates/runtime/generate_runtime_mesh.cylc
@@ -30,6 +30,9 @@
             PARTITION_PARTS       = {{ partition_parts }}
             FINAL_RANK            = $((PARTITION_PARTS-1))
             OMP_NUM_THREADS       = {{task_values["mesh_cpus"]}}
+            PANEL_DECOMP          = {{task_values["mesh_panel_decomp"]}}
+            XPROC                 = {{task_values["mesh_xproc"]}}
+            YPROC                 = {{task_values["mesh_yproc"]}}
 
         [[[directives]]]
     {{ set_task_resources(task_values["mesh_cpus"], task_values["mesh_memory"]) }}

[mo-marqh]

I suggest we provide a multi-partition test configuration within the test suite, using C48, that is extensible to larger resolutions for irregular performance testing.

I suggest a group, containing four tasks:

• nwp_gal9-C48_MG_op_run1
• nwp_gal9-C48_MG_op_run2
• nwp_gal9_ragged-C48_MG_op_run1
• nwp_gal9_ragged-C48_MG_op_run2

This should result in 2 different meshes being created and each patched into the two relevant tasks.

The mesh creation within the
https://github.com/MetOffice/lfric_apps/blob/main/rose-stem/templates/graph/populate_graph_sections.cylc#L83
and the
https://github.com/MetOffice/lfric_apps/blob/main/rose-stem/templates/runtime/generate_runtime_mesh.cylc
should be fairly extensible to meet this requirement

This group can be added to the weekly tests, to maintain coverage at a low iteration rate.

[iboutle]

I suggest we provide a multi-partition test configuration within the test suite, using C48, that is extensible to larger resolutions for irregular performance testing.

I suggest a group, containing four tasks:

• nwp_gal9-C48_MG_op_run1
• nwp_gal9-C48_MG_op_run2
• nwp_gal9_ragged-C48_MG_op_run1
• nwp_gal9_ragged-C48_MG_op_run2

This should result in 2 different meshes being created and each patched into the two relevant tasks.

The mesh creation within the https://github.com/MetOffice/lfric_apps/blob/main/rose-stem/templates/graph/populate_graph_sections.cylc#L83 and the https://github.com/MetOffice/lfric_apps/blob/main/rose-stem/templates/runtime/generate_runtime_mesh.cylc should be fairly extensible to meet this requirement

This group can be added to the weekly tests, to maintain coverage at a low iteration rate.

No particular objection to this, but I'm not really sure what this is adding above the C224_MG_op test that we're already running in the weekly group - what is the difference between run1 and run2 here?

[iboutle]

Just a note here as well - #702 went on main yesterday. When this is merged up to the head of main, please ensure all references to GAL9 are changed to GAL10 in any task names

[mo-marqh]

I suggest we provide a multi-partition test configuration within the test suite, using C48, that is extensible to larger resolutions for irregular performance testing.
I suggest a group, containing four tasks:

• nwp_gal9-C48_MG_op_run1
• nwp_gal9-C48_MG_op_run2
• nwp_gal9_ragged-C48_MG_op_run1
• nwp_gal9_ragged-C48_MG_op_run2

This should result in 2 different meshes being created and each patched into the two relevant tasks.
The mesh creation within the https://github.com/MetOffice/lfric_apps/blob/main/rose-stem/templates/graph/populate_graph_sections.cylc#L83 and the https://github.com/MetOffice/lfric_apps/blob/main/rose-stem/templates/runtime/generate_runtime_mesh.cylc should be fairly extensible to meet this requirement
This group can be added to the weekly tests, to maintain coverage at a low iteration rate.

No particular objection to this, but I'm not really sure what this is adding above the C224_MG_op test that we're already running in the weekly group - what is the difference between run1 and run2 here?

run1 & run2 here are the same configuration, run twice, to gather run to run variability.

It seems useful to me to ensure that the _op configurations make the list of unique meshes, but don't over-duplicate.

So, it could be useful to use run1 & run2 for development and debugging, but then perhaps they're not needed for the actual submission for merger, and can just be a single run each.

This might make the test code cleaner
thanks iboutle

[iboutle]

I suggest we provide a multi-partition test configuration within the test suite, using C48, that is extensible to larger resolutions for irregular performance testing.
I suggest a group, containing four tasks:

• nwp_gal9-C48_MG_op_run1
• nwp_gal9-C48_MG_op_run2
• nwp_gal9_ragged-C48_MG_op_run1
• nwp_gal9_ragged-C48_MG_op_run2

This should result in 2 different meshes being created and each patched into the two relevant tasks.
The mesh creation within the https://github.com/MetOffice/lfric_apps/blob/main/rose-stem/templates/graph/populate_graph_sections.cylc#L83 and the https://github.com/MetOffice/lfric_apps/blob/main/rose-stem/templates/runtime/generate_runtime_mesh.cylc should be fairly extensible to meet this requirement
This group can be added to the weekly tests, to maintain coverage at a low iteration rate.

No particular objection to this, but I'm not really sure what this is adding above the C224_MG_op test that we're already running in the weekly group - what is the difference between run1 and run2 here?

run1 & run2 here are the same configuration, run twice, to gather run to run variability.

It seems useful to me to ensure that the _op configurations make the list of unique meshes, but don't over-duplicate.

So, it could be useful to use run1 & run2 for development and debugging, but then perhaps they're not needed for the actual submission for merger, and can just be a single run each.

This might make the test code cleaner thanks iboutle

Thanks mo-marqh - I'm still struggling to follow exactly what you're wanting to test here though. If it's the generation of the OP meshes, then that should be done as a task in the mesh generation app, rather than hijacking lfric_atm.

[mo-marqh]

Thanks mo-marqh - I'm still struggling to follow exactly what you're wanting to test here though. If it's the generation of the OP meshes, then that should be done as a task in the mesh generation app, rather than hijacking lfric_atm.

that's fair iboutle
I think I am suggesting adding tests that test the test infrastructure, not that test lfric_atm

I think it could be useful for Matthew Walker (@mattatmet) to have this use case to code against, but then it can likely be removed once it is working, rather than adding to the lfric_atm test cases on main

[James Bruten (james-bruten-mo)]

Thanks for sorting this, it looks good. A couple of thoughts,

Suggested change

	{% set mesh_run_task = "run_mesh_"~
	task_values["resolution"]~"_"~
	task_values["mpi_parts"]~"P_"~
	task_values["panel_decomp"]~"_"~
	task_values["xproc"]~"X_"~
	task_values["yproc"]~"Y_"~
	task_ns.platform~"_"~
	site_vars["mesh_build"][task_ns.platform] %}
	{% set mesh_run_task = "run_mesh_"~
	task_values["resolution"]~"-"~
	task_values["mpi_parts"]~"P-"~
	task_values["panel_decomp"]~"-"~
	task_values["xproc"]~"X-"~
	task_values["yproc"]~"Y_"~
	task_ns.platform~"_"~
	site_vars["mesh_build"][task_ns.platform] %}

Could we split these sections by "-" rather than "_".

I also wonder whether it would be better to save this string as a new variable in the task_values dictionary, and then reading that in the 2 generate runtime files, rather than having this line of code repeated?

[Matthew Walker (mattatmet)]

Thanks for these comments James, I've made some changes, let me know what you think.

[iboutle]

Just a note here as well - #702 went on main yesterday. When this is merged up to the head of main, please ensure all references to GAL9 are changed to GAL10 in any task names

Just thinking more about this - it would probably be safe to make this change on the branch now. Each task is a new task, so naming each new task as gal10 instead of gal9 would be consistent with what we want at the head of main, will work as-is, and should merge cleanly with the head of main...

[James Bruten (james-bruten-mo)]

Thanks Matt, looks good

[Matthew Walker (mattatmet)]

Update: after looking into the seg fault Mark found with one of the custom partitioning jobs, I think I narrowed it down to a race condition in the mesh generator that was only triggered by very high partition numbers. I've addressed that in core and linked the PR (MetOffice/lfric_core#450) to this one.

[mo-marqh]

many thanks Matthew Walker (@mattatmet)

this PR now addresses all of the initial intent and the scope creep from review

Minimal new testing is added, with only 1 new test, a C224 with offline partitioning, whilst existing tests are adapted.

This change passes Sci/Tech review

[mo-marqh]

ready for code review Benjamin Went (@MetBenjaminWent)