MPI Split communicator. #318

mo-joshuacolclough · 2025-10-17T09:41:01Z

No description provided.

* hotfix/1.6.2: For CI lumi-g cce, set CMAKE_PARALLEL_BUILD_LEVEL=1 (already so in develop) More resources for CI lumi-g cce etrans benchmark: specify LINKER_LANGUAGE Fortran for static linking transi: link against etrans if enabled etrans: fix library target names in CPU build Version bump to 1.6.2

…comm.

mo-joshuacolclough

(From discussion with @MarekWlasak)

mo-joshuacolclough · 2025-10-17T09:41:58Z

tests/trans/test_example.F90

+! which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+!
+
+program test_example


Suggested change

program test_example

program test_split_mpi_comm

Also the name of the file then.

mo-joshuacolclough · 2025-10-17T09:45:23Z

tests/trans/test_example.F90

+
+integer(kind=JPIM) :: num_spectral_elements, num_grid_points
+integer(kind=JPIM) :: g_num_spectral_elements, g_num_grid_points  ! global
+integer(kind=JPIM) :: mode_index


Suggested change

integer(kind=JPIM) :: mode_index

integer(kind=JPIM) :: local_spectral_coefficient_index

Should use type prefix? E.g K. Standards

Perhaps not strictly needed.

mo-joshuacolclough · 2025-10-17T09:47:45Z

src/transi/transi_module.F90

Transi changes in sep PR?

For now I like it here.

mo-joshuacolclough · 2025-10-17T09:53:05Z

tests/trans/test_example.F90

+real(kind=JPRM), allocatable :: spectral_field(:,:)
+real(kind=JPRM), allocatable :: grid_point_field(:,:,:)
+
+real(kind=JPRM), allocatable :: g_grid_point_field(:)


Comment on dimension difference? Local testing variable - grid point field to output to file.

mo-joshuacolclough · 2025-10-17T09:54:53Z

tests/trans/test_example.F90

+call MPI_Comm_rank(MPI_COMM_WORLD, rank, ierror)
+
+split_colour = get_split_group()
+split_key = rank


Suggested change

split_key = rank

split_key = rank ! Global rank

mo-joshuacolclough · 2025-10-17T10:11:52Z

tests/trans/test_example.F90

+! Get function space sizes
+call trans_inq(KSPEC2=num_spectral_elements, KGPTOT=num_grid_points)
+call trans_inq(KSPEC2G=g_num_spectral_elements, KGPTOTG=g_num_grid_points)
+print*,"Num spec = ", num_spectral_elements, "| Num grid points = ", num_grid_points, g_num_grid_points


Suggested change

print*,"Num spec = ", num_spectral_elements, "| Num grid points = ", num_grid_points, g_num_grid_points

print*,"R", world_rank, "C", split_rank, "Num spec = ", num_spectral_elements, "| Num grid points = ", num_grid_points, g_num_grid_points

mo-joshuacolclough · 2025-10-17T10:28:41Z

tests/trans/test_example.F90

+! Allocate a global field.
+allocate(g_grid_point_field(g_num_grid_points))
+
+! Make displacement arrays
+allocate(displs(num_ranks))
+displs = 0
+do i=2, num_ranks
+  displs(i) = displs(i - 1) + grid_partition_sizes(i - 1)
+end do
+print*,"displs => ", displs(:)
+


Suggested change

! Allocate a global field.

allocate(g_grid_point_field(g_num_grid_points))

! Make displacement arrays

allocate(displs(num_ranks))

displs = 0

do i=2, num_ranks

displs(i) = displs(i - 1) + grid_partition_sizes(i - 1)

end do

print*,"displs => ", displs(:)

if (rank == 1) then

! Allocate a global field.

allocate(g_grid_point_field(g_num_grid_points))

! Make displacement arrays

allocate(displs(num_ranks))

displs = 0

do i=2, num_ranks

displs(i) = displs(i - 1) + grid_partition_sizes(i - 1)

end do

print*,"displs => ", displs(:)

end if

mo-joshuacolclough · 2025-10-17T10:35:23Z

tests/trans/test_multiple_res.F90

+#include "dir_trans.h"
+#include "trans_inq.h"
+
+integer(kind=JPIM), parameter, dimension(2) :: truncations = [79, 188]


Check if memory usage increases with same resolution on both.

mo-joshuacolclough · 2025-10-17T10:37:16Z

tests/trans/test_multiple_res.F90

Remove (?) once understood.

Possibly it can be merged with the split_comm case later, where you would also have more than one handle available.

mo-joshuacolclough · 2025-10-17T10:43:42Z

tests/transi/transi_test_example.c

Rename from example

samhatfield · 2025-10-17T13:53:29Z

Looks like a good start Josh, thanks. I should have mentioned before, but all PRs must be from branches started from develop (as well as targeting develop). It looks like you've started from main, hence the apparent conflicts with CMakeLists.txt. I'd recommend rebasing all of your commits on top of develop before continuing to save yourself some headaches later.

mo-joshuacolclough · 2025-10-17T15:42:34Z

Thanks @samhatfield - we are working off of the tagged release we are using currently, to avoid any potential complications with using develop in our codebase... (Happy to deal with the headaches at a later date... :) ) Just while we get things working. I don't intend for this to get merged, just for sharing at the moment.

We are pleased so far that the changes needed seem to be quite minimal!

samhatfield · 2025-10-20T08:29:46Z

Thanks @samhatfield - we are working off of the tagged release we are using currently, to avoid any potential complications with using develop in our codebase... (Happy to deal with the headaches at a later date... :) ) Just while we get things working. I don't intend for this to get merged, just for sharing at the moment.

We are pleased so far that the changes needed seem to be quite minimal!

That's fair enough! Okay then, let's discuss what you've done so far in our next call.

wdeconinck

Nice progress on building up some test cases.
You can keep the transi developments here for now. I find it useful as a discussion point. We can always split the PR in pieces later.

wdeconinck · 2025-10-21T13:51:50Z

src/transi/transi.h

 */
 int trans_init(void);

+int trans_set_mpi_comm(const MPI_Fint mpi_user_comm);


Please change this to use a standard int as to not #include <mpi.h>. We don't want to expose MPI in transi; especially as ectrans can be completely compiled without MPI.

Suggested change

int trans_set_mpi_comm(const MPI_Fint mpi_user_comm);

int trans_set_mpi_comm(int);

wdeconinck · 2025-10-21T13:54:33Z

src/transi/CMakeLists.txt

There is no need to link transi with MPI; changes in this file can be reverted.

wdeconinck · 2025-10-21T14:06:51Z

tests/trans/test_multiple_res.F90

+  call MPI_Gather(grid_partition_size_local, 1, MPI_INT, &
+                  grid_partition_sizes, 1, MPI_INT, &
+                  0, MPI_COMM_WORLD, ierror)


I think it should be possible to turn this into an MPL_GATHERV call, and I don't think you need

use mpl_data_module ! probably you can add what is needed to the ONLY part of MPL_MODULE use mpl_mpif

wdeconinck · 2025-10-21T14:07:59Z

tests/trans/test_multiple_res.F90

+  call MPI_Gatherv(grid_point_field(:,1,1), num_grid_points, MPI_FLOAT, &
+                   g_grid_point_field, grid_partition_sizes, displs, MPI_FLOAT, &
+                   0, MPI_COMM_WORLD, ierror)


Also here replace with MPL_GATHERV should be used.

wdeconinck · 2025-10-21T14:14:15Z

tests/CMakeLists.txt

+    LIBS      ectrans_test
+    MPI 1
+    LINKER_LANGUAGE C
+    ENVIRONMENT TRANS_USE_MPI=1 )


Because MPI is used within this file, we need add MPI::MPI_C to the LIBS argument here.
I would only add the searching for MPI C library here:

find_package( MPI COMPOMENTS C ) if (MPI_C_FOUND) ecbuild_add_test( TARGET ectrans_test_transi_example ... LIBS ectrans_test MPI::MPI_C ... ) endif()

wdeconinck · 2025-10-21T14:16:45Z

tests/trans/test_example.F90

+! which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
+!
+
+program test_example


Also the name of the file then.

wdeconinck · 2025-10-21T14:22:47Z

CMakeLists.txt

 ecbuild_add_option( FEATURE MPI
                    DESCRIPTION "Support for MPI distributed memory parallelism"
-                    REQUIRED_PACKAGES "MPI COMPONENTS Fortran CXX"
+                    REQUIRED_PACKAGES "MPI COMPONENTS Fortran C CXX"


This could be reverted with find_package( MPI COMPONENTS C ) further down for the tests only. (see another comment)

wdeconinck and others added 9 commits July 22, 2025 10:53

Add dummy tests

ae88a4d

Remove MPL_SET_DEFAULT_COMM and set default prior to MPL_INIT call

28774d7

Implement set_mpi_comm. Add transi example with split communicator

281b875

Add mpi includes in CMake

ae81b10

Fix test include

2dc9ebb

Check that if set_mpi_comm is called again, that it is with the same …

1947405

…comm.

Ensure logic is correct - only do check if MPL_INIT is called

4a0d487

add MPI to libs

277053a

github-actions bot added the contributor label Oct 17, 2025

mo-joshuacolclough commented Oct 17, 2025

View reviewed changes

wdeconinck reviewed Oct 21, 2025

View reviewed changes

	integer(kind=JPIM) :: mode_index
	integer(kind=JPIM) :: local_spectral_coefficient_index

	print*,"Num spec = ", num_spectral_elements, "\| Num grid points = ", num_grid_points, g_num_grid_points
	print*,"R", world_rank, "C", split_rank, "Num spec = ", num_spectral_elements, "\| Num grid points = ", num_grid_points, g_num_grid_points

	int trans_set_mpi_comm(const MPI_Fint mpi_user_comm);
	int trans_set_mpi_comm(int);

MPI Split communicator. #318

Are you sure you want to change the base?

MPI Split communicator. #318

Uh oh!

Conversation

mo-joshuacolclough commented Oct 17, 2025

Uh oh!

mo-joshuacolclough left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samhatfield commented Oct 17, 2025

Uh oh!

mo-joshuacolclough commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samhatfield commented Oct 20, 2025

Uh oh!

wdeconinck left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mo-joshuacolclough commented Oct 17, 2025 •

edited

Loading