I think that

self.job.launcher.env_vars = {'MASTER_PORT': '1234'}

or

self.job.launcher.env_vars['MASTER_PORT] = '1234'

is the best and matches the test's env_vars in the syntax.

@casparvl I guess that your suggestion is for cases that the scheduler does not forward the environment to the compute nodes, right? If that's the case, maybe we would just need a configuration parameter to tell ReFrame to export the test's env_vars when launching the job. Do you see a case that you would need to pass a different set of environment variables to the launcher?

@casparvl I guess that your suggestion is for cases that the scheduler does not forward the environment to the compute nodes, right?

Hmm, that was actually not my problem. What you're talking about (I think) is exporting variables from the submission environment to the batch job. What I'm talking about is exporting the environment from the job's head-node of the allocation, to the processes launched by a parallel launcher like srun or mpirun. The trouble is that there is no launcher-agnostic way to do that, and it would be nice if ReFrame would abstract that away, so that we can define e.g. in the test:

self.job.launcher.env_vars['MASTER_PORT] = '1234'
self.executable = python
self.executable_opts = ['pytorch_benchmark.py']

And that, when configured with e.g. mpirun as parallel launcher, ReFrame would create the following job script:

#SBATCH ...
module load ...
mpirun -x MASTER_PORT=1234 python pytorch_benchmark.py

But, when configured with srun as parallel launcher, ReFrame would created:

#SBATCH ...
module load ...
srun --export=MASTER_PORT=1234 python pytorch_benchmark.py

I.e. in that way, we can write tests that can tell the parallel launcher to export an environment variable to the parallel processes being launched without the test developer having to know which parallel launcher will be used.

Thanks @casparvl for the clarification; it makes sense. I was confused at the beginning.

@vkarak, we discussed modifying the JobLauncher base class and implementing MPI-specific launchers, like an IntelMPI mpirun (impi_mpirun) and an OpenMPI mpirun (ompi_mpirun).

After thinking about this some more, doesn't this just shift the reuse problem from a test definition into the config files?

If I'm not mistaken, in order to use these MPI-specific launchers, a user would have to have multiple system.partition.launchers, meaning new system:partitions. This seems inconvenient.

Am I thinking about this correctly?

Edit: Maybe this approach is fine... system.partition.launcher already implies a single launcher is configured. I'm just thinking about situations where a user may want to use either OpenMPI or IntelMPI (or others) on a single partition.

Edit: Maybe this approach is fine... system.partition.launcher already implies a single launcher is configured. I'm just thinking about situations where a user may want to use either OpenMPI or IntelMPI (or others) on a single partition.

Yes, this is a general assumption of the system partitions, so we are not doing anything different. The key problem we are trying to solve here is that exporting variables is specific to the MPI launcher flavor. We will not replace the generic mpirun launcher, though it won't support/interpret the discussed env_vars. If users want to export variables at the launcher level, would then have to pick a more specific launcher for their partition.