|
1 | | -# ICON |
2 | | -- a unified next-generation global numerical weather prediction and climate modelling system |
3 | | -- Atmosphere and Ocean components |
4 | | -- in Grid Point Space on a Geodesic Icosahedral Grid |
5 | | -- MPI Application |
6 | | - |
7 | | - |
8 | | - |
9 | | - |
10 | | - |
11 | | -## IPM Result |
12 | | - |
13 | | - |
14 | | - |
15 | | -- Why 20 MPI Rank Having more MPI_WAIT time than others? |
16 | | -- How to Reduce MPI Call time and Balance MPI? |
17 | | - |
18 | | -## Intel® Trace Analyzer and Collector Detailed MPI Profiling Result |
19 | | - |
20 | | - |
21 | | - |
22 | | - |
23 | | - |
24 | | -* Intel® Trace Analyzer and Collector Detailed MPI Profiling |
25 | | - * MPI Message Matrix |
26 | | - |
27 | | - |
28 | | - |
29 | | -* Intel® Trace Analyzer and Collector Detailed MPI Profiling |
30 | | - * MPI Trace |
31 | | - |
32 | | - |
33 | | - |
34 | | - |
35 | | - |
36 | | - |
37 | | - |
38 | | -* Intel® Trace Analyzer and Collector Detailed MPI Profiling |
39 | | - * MPI Pattern 1 |
40 | | - |
41 | | -__MPI Pattern 1 Barrier__ |
42 | | - |
43 | | - |
44 | | - |
45 | | - |
46 | | - |
47 | | -_Ocean Ranks always Waiting for Atmosphere Ranks_ |
48 | | - |
49 | | -* Intel® Trace Analyzer and Collector Detailed MPI Profiling |
50 | | - * MPI Pattern 2 |
51 | | - |
52 | | - |
| 1 | +--- |
| 2 | +title: ICON |
| 3 | +description: |
| 4 | +published: true |
| 5 | +date: 2022-04-23T08:06:22.718Z |
| 6 | +tags: |
| 7 | +editor: markdown |
| 8 | +dateCreated: 2022-03-21T02:22:59.229Z |
| 9 | +--- |
53 | 10 |
|
54 | | - |
55 | | - |
56 | | -* Ocean Procs Tuning |
57 | | - * 20 Ocean\, 140 Atmosphere Ranks @ Niagara |
58 | | - * 36 Ocean\, 476 Atmosphere Ranks @ Bridges2 |
59 | | - * block:block |
60 | | - |
61 | | - |
62 | | - |
63 | | -## ICON Serial Code Tuning |
64 | | - |
65 | | - |
66 | | - |
67 | | -- Nproma Tuning |
68 | | -- Vectorization and SIMD |
69 | | -- _AVX512 with Intel_ |
70 | | - |
71 | | - |
72 | | - |
73 | | - |
74 | | - |
75 | | -## ICON Things can be Improve |
76 | | - |
77 | | -GPU Version |
78 | | - |
79 | | -Increase MPI affinity with slurm at bridges\-2 |
80 | | - |
81 | | - |
82 | | - |
83 | | - |
84 | | - |
85 | | -## Lessons learned |
| 11 | +# ICON |
86 | 12 |
|
87 | | -* Always notice the Communication Pattern of a MPI Application |
88 | | - * When a MPI application separates its ranks to do the different jobs\, the load balance must be taking into consider\. |
89 | | -* A Detailed MPI Trace Profile is Important |
90 | | - * A detailed profiling result can tell you more about the data dependency and the bottleneck\. |
91 | | -* Different Architecture have different Vectorization Ability |
| 13 | +## Prepare |
92 | 14 |
|
| 15 | +```bash |
| 16 | +git clone https://gitlab.dkrz.de/icon-scc-isc22/icon-scc |
| 17 | +cd /path/to/icon-scc |
| 18 | +git submodule init |
| 19 | +git submodule update |
| 20 | +``` |
| 21 | +## How to run |
| 22 | +### spack compile |
| 23 | +```bash |
| 24 | +spack install -j (nproc) -vvvv icon% [email protected] |
| 25 | +``` |
| 26 | +There are some varients: |
| 27 | +- debug |
| 28 | +- cuda |
| 29 | +- openmp |
| 30 | + |
| 31 | +### run copy scripts |
| 32 | +```bash |
| 33 | +cd {ICON_BUILD_DIR} |
| 34 | +export ICON_DIR={ICON_DIR} |
| 35 | +# Copy runscript-related files when building out-of-source: |
| 36 | +if test $(pwd) != $(cd "${ICON_DIR}"; pwd); then |
| 37 | + echo "Copying runscript input files from the source directory..." |
| 38 | + rsync -uavz ${ICON_DIR}/run . --exclude='*.in' --exclude='.*' --exclude='standard_*' |
| 39 | + ln -sf -t run/ ${ICON_DIR}/run/standard_* |
| 40 | + ln -sf set-up.info run/SETUP.config |
| 41 | + rsync -uavz ${ICON_DIR}/externals . --exclude='.git' --exclude='*.f90' --exclude='*.F90' --exclude='*.c' --exclude='*.h' --exclude='*.Po' --exclude='tests' --exclude='rrtmgp*.nc' --exclude='*.mod' --exclude='*.o' |
| 42 | + rsync -uavz ${ICON_DIR}/make_runscripts . |
| 43 | + ln -sf ${ICON_DIR}/data |
| 44 | + ln -sf ${ICON_DIR}/vertical_coord_tables |
| 45 | +fi |
| 46 | +``` |
| 47 | +### Gen sbatch |
| 48 | +```bash |
| 49 | +cd {ICON_BUILD_DIR} |
| 50 | +export ICON_DIR={ICON_DIR} |
| 51 | +cd {ICON_BUILD_DIR}/run |
| 52 | +$ICON_DIR/utils/mkexp/mkexp standard_experiments/scc.config CO2=2850 |
| 53 | +``` |
| 54 | +OK if |
| 55 | +```bash |
| 56 | +Script directory: '/mnt/nfs4/node1/home/qinfr/spack/opt/spack/linux-ubuntu20.04-zen/gcc-6.4.0/icon-2021-isc-scc-hw7pyldsuxsug2jrnmhdulvk5knzbzw6/experiments/exp_scc2850/scripts' |
| 57 | +Data directory: '/mnt/nfs4/node1/home/qinfr/spack/opt/spack/linux-ubuntu20.04-zen/gcc-6.4.0/icon-2021-isc-scc-hw7pyldsuxsug2jrnmhdulvk5knzbzw6/experiments/exp_scc2850/outdata' |
| 58 | +Work directory: '/mnt/nfs4/node1/home/qinfr/spack/opt/spack/linux-ubuntu20.04-zen/gcc-6.4.0/icon-2021-isc-scc-hw7pyldsuxsug2jrnmhdulvk5knzbzw6/experiments/exp_scc2850/work' |
| 59 | +``` |
| 60 | +### Modify sbatch |
| 61 | +In `experiments/exp_scc2850/scripts/exp_scc2850.run_start` |
| 62 | +- FIX SLURM args |
| 63 | +- FIX path |
| 64 | + - no `/build/` |
| 65 | + - no `/home/qinfr` |
| 66 | +```git |
| 67 | +- BUILD_DIR=/home/qinfr/spack/opt/spack/linux-ubuntu20.04-zen/gcc-6.4.0/icon-2021-isc-scc-hw7pyldsuxsug2jrnmhdulvk5knzbzw6/BUILD |
| 68 | ++ BUILD_DIR=/mnt/nfs4/node1/home/qinfr/spack/opt/spack/linux-ubuntu20.04-zen/gcc-6.4.0/icon-2021-isc-scc-hw7pyldsuxsug2jrnmhdulvk5knzbzw6/ |
| 69 | ++ export PATH={cdo-1.9.10_BUILD_DIR}/bin:$PATH |
| 70 | +... |
| 71 | +``` |
| 72 | + |
| 73 | + |
| 74 | + |
| 75 | +- Subsitute all `/home/qinfr` to `/mnt/nfs4/node1/home/qinfr/` |
| 76 | +### Run |
| 77 | +```bash |
| 78 | +sbatch exp_scc2850.run_start |
| 79 | +``` |
93 | 80 | ## Tips |
94 | 81 | ### How to check if compiled code uses SSE and AVX instructions? |
95 | 82 | [https://stackoverflow.com/questions/47878352/how-to-check-if-compiled-code-uses-sse-and-avx-instructions](https://stackoverflow.com/questions/47878352/how-to-check-if-compiled-code-uses-sse-and-avx-instructions) |
|
0 commit comments