Skip to content

Commit 52a30e8

Browse files
doscherdacmcknigh
andauthored
Add updated Grid files (#28)
Co-authored-by: David Doscher <[email protected]> Co-authored-by: cmcknigh <[email protected]>
1 parent edb68fc commit 52a30e8

File tree

5 files changed

+366
-0
lines changed

5 files changed

+366
-0
lines changed

README.md

+2
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ For ROCm installation procedures and validation checks, see:
2020
- [Docker](/cholla/docker/)
2121
- [Bare Metal](/cholla/baremetal/)
2222
- [CP2K](/cp2k-docker/)
23+
- [Grid](/grid/)
24+
- [Docker](/grid/docker/)
2325
- [HPCG in Docker](/hpcg-docker/)
2426
- [LAMMPS in Docker](/lammps-docker/)
2527
- [OpenFOAM in Docker](/openfoam-docker/)

grid/README.md

+87
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Grid
2+
3+
## Overview
4+
Grid is a library for lattice QCD calculations that employs a high-level data parallel approach while using a number of techniques to target multiple types of parallelism. The library currently supports MPI, OpenMP and short vector parallelism. The SIMD instructions sets covered include SSE, AVX, AVX2, FMA4, IMCI and AVX512. Recent releases expanded this support to include GPU offloading. The code requires at least one AMD GPU to run.
5+
6+
The source code is available on [github](https://github.com/paboyle/Grid).
7+
8+
## Single-Node Server Requirements
9+
10+
| CPUs | GPUs | Operating Systems | ROCm™ Driver | Container Runtimes |
11+
| ---- | ---- | ----------------- | ------------ | ------------------ |
12+
| X86_64 CPU(s) | AMD Instinct MI200 GPU(s) <br> AMD Instinct MI100 GPU(s) <br> Radeon Instinct MI50(S) | Ubuntu 20.04 <br> Ubuntu 22.04 <BR> RHEL8 <br> RHEL9 <br> SLES 15 sp4 | ROCm v5.x compatibility |[Docker Engine](https://docs.docker.com/engine/install/) <br> [Singularity](https://sylabs.io/docs/) |
13+
14+
For ROCm installation procedures and validation checks, see:
15+
* [ROCm Documentation](https://rocm.docs.amd.com)
16+
* [AMD Lab Notes ROCm installation notes](https://github.com/amd/amd-lab-notes/tree/release/rocm-installation).
17+
* [ROCm Examples](https://github.com/amd/rocm-examples)
18+
19+
## Build Recipes
20+
- [Docker/Singularity Build](/grid/docker/)
21+
22+
23+
## Running Grid Benchmarks
24+
These examples are using the Container built using [Grid Docker Build](/grid/docker/).
25+
26+
Grid has many examples within the project `/opt/grid/bin/` directory.
27+
28+
You can run the examples using the command syntax below, where
29+
`<gpus>` is the number of GPUs to use
30+
`./benchmark/gpu_bind.sh` defines the CPU to GPU NUMA mapping
31+
`<benchmark-to-run>` is the name of the benchmark to run
32+
`<threads>` is the number of threads to use
33+
`<i>` is the MPI configuration
34+
35+
```
36+
mpirun -np <gpus> ./benchmark/gpu_bind.sh <benchmark-to-run> \
37+
--accelerator-threads <threads> --mpi <i> --shm 2048
38+
```
39+
The `./benchmark/gpu_bind.sh` command may need to be updated for your specific CPU / GPU combination. Use the `lscpu` command, to determine your NUMA configuration, and map the values back into the gpu_bind.sh command.
40+
41+
A combination of values for the MPI configuration, number of threads, gpus and other input parameters should be tested to achieve the highest throughput for your specific system.
42+
43+
### Examples:
44+
45+
* 1 GPU using Benchmark_ITT
46+
```
47+
mpirun -np 1 /benchmark/gpu_bind.sh Benchmark_ITT --accelerator-threads 1 --mpi 1.1.1.1 --shm 2048
48+
```
49+
50+
* 4 GPUs using Benchmark_ITT
51+
```
52+
mpirun -np 4 /benchmark/gpu_bind.sh Benchmark_ITT --accelerator-threads 4 --mpi 4.1.1.1 --shm 2048
53+
```
54+
55+
* 8 GPUs using Benchmark_ITT
56+
```
57+
mpirun -np 8 /benchmark/gpu_bind.sh Benchmark_ITT --accelerator-threads 8 --mpi 8.1.1.1 --shm 2048
58+
```
59+
60+
Each MPI rank will bind to a particular unique GPU (1 rank per device) and strong-scale the problem accordingly.
61+
62+
63+
## Licensing Information
64+
Your access and use of this application is subject to the terms of the applicable component-level license identified below. To the extent any subcomponent in this container requires an offer for corresponding source code, AMD hereby makes such an offer for corresponding source code form, which will be made available upon request. By accessing and using this application, you are agreeing to fully comply with the terms of this license. If you do not agree to the terms of this license, do not access or use this application.
65+
66+
The application is provided in a container image format that includes the following separate and independent components:
67+
|Package | License | URL|
68+
|---|---|---|
69+
|Ubuntu| Creative Commons CC-BY-SA Version 3.0 UK License |[Ubuntu Legal](https://ubuntu.com/legal)|
70+
|CMAKE|OSI-approved BSD-3 clause|[CMake License](https://cmake.org/licensing/)|
71+
|OpenMPI|BSD 3-Clause|[OpenMPI License](https://www-lb.open-mpi.org/community/license.php)<br /> [OpenMPI Dependencies Licenses](https://docs.open-mpi.org/en/v5.0.x/license/index.html)|
72+
|OpenUCX|BSD 3-Clause|[OpenUCX License](https://openucx.org/license/)|
73+
|ROCm|Custom/MIT/Apache V2.0/UIUC OSL|[ROCm Licensing Terms](https://rocm.docs.amd.com/en/latest/release/licensing.html)|
74+
|Grid|GPL V2|[Grid](https://github.com/paboyle/Grid)<br >[Grid License](https://github.com/paboyle/Grid/blob/develop/LICENSE)|
75+
76+
77+
Additional third-party content in this container may be subject to additional licenses and restrictions. The components are licensed to you directly by the party that owns the content pursuant to the license terms included with such content and is not licensed to you by AMD. ALL THIRD-PARTY CONTENT IS MADE AVAILABLE BY AMD “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT.
78+
79+
## Disclaimer
80+
The information contained herein is for informational purposes only, and is subject to change without notice. In addition, any stated support is planned and is also subject to change. While every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale.
81+
82+
## Notices and Attribution
83+
© 2022-2023 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, Instinct, Radeon Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc.
84+
85+
Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. Docker, Inc. and other parties may also have trademark rights in other terms used herein. Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.
86+
87+
All other trademarks and copyrights are property of their respective owners and are only mentioned for informative purposes.

grid/docker/Dockerfile

+145
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
ARG IMAGE="rocm/dev-ubuntu-22.04:5.6-complete"
2+
3+
FROM ${IMAGE}
4+
5+
ARG GRID_BRANCH="develop"
6+
ARG UCX_BRANCH="v1.14.1"
7+
ARG OMPI_BRANCH="v4.1.5"
8+
9+
RUN rm /etc/apt/sources.list.d/* \
10+
&& apt-get update \
11+
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
12+
ca-certificates \
13+
git \
14+
ssh \
15+
make \
16+
vim \
17+
nano \
18+
libtinfo* \
19+
initramfs-tools \
20+
libelf-dev \
21+
numactl \
22+
curl \
23+
wget \
24+
tmux \
25+
build-essential \
26+
autoconf \
27+
automake \
28+
libtool \
29+
pkg-config \
30+
libnuma* \
31+
gfortran \
32+
flex \
33+
hwloc \
34+
libmpfr-dev \
35+
libelf1 \
36+
tar \
37+
libssl-dev \
38+
zlib1g-dev \
39+
python3 \
40+
&& apt-get clean
41+
42+
# Note: only need to define the GFX arch that you are using
43+
RUN echo "gfx900" > /opt/rocm/bin/target.lst \
44+
&& echo "gfx906" >> /opt/rocm/bin/target.lst \
45+
&& echo "gfx908" >> /opt/rocm/bin/target.lst \
46+
&& echo "gfx90a" >> /opt/rocm/bin/target.lst
47+
48+
RUN chmod a+r /opt/rocm/bin/target.lst
49+
50+
ENV ROCM_PATH=/opt/rocm
51+
ENV HIP_PATH=$ROCM_PATH/hip \
52+
HCC_HOME=$ROCM_PATH/hcc
53+
54+
WORKDIR /tmp
55+
56+
# Install UCX
57+
RUN cd /tmp/ \
58+
&& echo git clone https://github.com/openucx/ucx.git -b ${UCX_BRANCH} \
59+
&& git clone https://github.com/openucx/ucx.git -b ${UCX_BRANCH} \
60+
&& ls -lsa \
61+
&& cd ucx \
62+
&& ./autogen.sh \
63+
&& mkdir build \
64+
&& cd build \
65+
&& ../contrib/configure-release --prefix=/opt/ucx \
66+
--with-rocm=$ROCM_PATH --without-knem \
67+
--without-xpmem --without-cuda \
68+
--enable-optimizations \
69+
--disable-logging \
70+
--disable-debug \
71+
--enable-assertions \
72+
--enable-params-check \
73+
--disable-examples \
74+
&& make -j 8 \
75+
&& make install \
76+
&& cd /tmp \
77+
&& rm -rf ucx
78+
79+
# Install OpenMPI
80+
RUN cd /tmp \
81+
&& echo git clone --recursive https://github.com/open-mpi/ompi.git -b ${OMPI_BRANCH} \
82+
&& git clone --recursive https://github.com/open-mpi/ompi.git -b ${OMPI_BRANCH} \
83+
&& ls -lsa \
84+
&& cd ompi \
85+
&& ./autogen.pl\
86+
&& mkdir build \
87+
&& cd build \
88+
&& ../configure --prefix=/opt/ompi --with-ucx=/opt/ucx \
89+
--enable-mca-no-build=btl-uct \
90+
--with-pmix=internal \
91+
--enable-mpi \
92+
--enable-mpi-fortran=yes \
93+
--disable-debug \
94+
--disable-man-pages \
95+
&& make -j 8 \
96+
&& make install \
97+
&& cd /tmp \
98+
&& rm -rf ompi
99+
100+
# Adding OpenMPI and UCX to PATH
101+
ENV PATH=/opt/ompi/bin:/opt/ucx/bin:$PATH
102+
103+
#
104+
# Build app
105+
#
106+
ENV GRID_PATH=/opt/grid
107+
108+
RUN git clone -b ${GRID_BRANCH} https://github.com/paboyle/Grid.git \
109+
&& cd Grid \
110+
&& ./bootstrap.sh \
111+
&& mkdir build \
112+
&& cd build \
113+
&& ../configure \
114+
--prefix=$GRID_PATH \
115+
--enable-unified=no \
116+
--enable-accelerator=hip \
117+
--enable-setdevice \
118+
--enable-alloc-cache \
119+
--enable-shm=nvlink \
120+
--enable-comms=mpi3-auto \
121+
--enable-simd=GPU \
122+
--enable-gen-simd-width=64 \
123+
--disable-accelerator-cshift \
124+
--disable-fermion-reps \
125+
CXX=hipcc \
126+
MPICXX=mpicxx \
127+
CPPFLAGS="-I${ROCM_PATH}/include -I${ROCM_PATH}/include/roctracer -std=c++17 " \
128+
LDFLAGS="-L${ROCM_PATH}/lib -L${ROCM_PATH}/roctracer/lib -lroctracer64 -lroctx64 -std=c++17" \
129+
&& make -j 8 \
130+
&& make install \
131+
&& cd /tmp \
132+
&& rm -rf Grid
133+
134+
COPY ./benchmark $GRID_PATH/benchmark/.
135+
RUN chmod a+rx $GRID_PATH/benchmark/*.sh \
136+
&& ln -s $GRID_PATH/benchmark/ /benchmark
137+
138+
WORKDIR /benchmark
139+
140+
ENV PATH=$GRID_PATH/bin:$PATH \
141+
OMPI_ALLOW_RUN_AS_ROOT=1 \
142+
OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1 \
143+
OMPI_MCA_pml=ucx
144+
145+
CMD ["/bin/bash"]

grid/docker/README.md

+116
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# Grid Docker Build Instructions
2+
3+
4+
## Inputs
5+
Possible `build-arg` for the Docker build command
6+
7+
- ## IMAGE
8+
Default: `rocm/dev-ubuntu-22.04:5.6-complete`
9+
Docker Tags found:
10+
- [ROCm Ubuntu 22.04](https://hub.docker.com/r/rocm/dev-ubuntu-22.04)
11+
- [ROCm Ubuntu 20.04](https://hub.docker.com/r/rocm/dev-ubuntu-20.04)
12+
> ***Note:***
13+
> The `*-complete` version has all the components required for building and installation.
14+
15+
- ## UCX_BRANCH
16+
Default: `v1.14.1`
17+
Branch/Tag found: [UXC repo](https://github.com/openucx/ucx)
18+
19+
- ## OMPI_BRANCH
20+
Default: `v4.1.5`
21+
Branch/Tag found: [OpenMPI repo](https://github.com/open-mpi/ompi)
22+
23+
- ## GRID_BRANCH
24+
Default: `develop`
25+
Branch/Tag found: [Grid repo](https://github.com/paboyle/Grid)
26+
27+
28+
## Building Container
29+
Download the [Dockerfile related files](/grid/docker/).
30+
31+
To build the default configuration:
32+
```
33+
docker build -t mycontainer/grid -f /path/to/Dockerfile .
34+
```
35+
> Notes:
36+
>- `mycontainer` is an example container name.
37+
>- the `.` at the end of the build line is important. It tells Docker where your build context is located.
38+
>- `-f /path/to/Dockerfile` is only required if your docker file is in a different directory than your build context, if you are building in the same directory it is not required.
39+
40+
To run a custom configuration, include one or more customized build-arg
41+
42+
*DISCLAIMER:* This Docker build has only been validated using the default values. Using a different base image or branch may result in build failures or poor performance.
43+
44+
```
45+
docker build \
46+
-t mycontainer/grid \
47+
-f /path/to/Dockerfile \
48+
--build-arg IMAGE=rocm/dev-ubuntu-22.04:5.5-complete \
49+
--build-arg UCX_BRANCH=master \
50+
--build-arg OMPI_BRANCH=main
51+
.
52+
```
53+
54+
## Running a Grid Container
55+
Both Docker and Singularity can be run interactively or as a single command.
56+
57+
To run the [Grid Benchmarks](/grid/README.md#running-grid-benchmarks), replace the `<Grid Command>` the examples in [Running Grid Benchmarks](/grid/README.md#running-grid-benchmarks) section of the Grid readme. The commands can be run directly in an interactive session as well.
58+
59+
### Docker
60+
If you want to save files generated during the run, add `-v $(pwd):/host` before `mycontainer/grid` in the following commands. Files may then be copied or saved between the container /host directory and the users working directory.
61+
62+
#### Docker Interactive
63+
```
64+
docker run --rm -it --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined \
65+
mycontainer/grid /bin/bash
66+
```
67+
#### Docker Non-Interactive
68+
```
69+
docker run --rm -it --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined \
70+
mycontainer/grid <Grid Command>
71+
```
72+
73+
### Singularity
74+
If you want to save files generated during the run, add `--binf $(pwd):/host` before `mycontainer/grid` in the following commands. Files may then be copied or saved between the container /host directory and the users working directory.
75+
#### Build Singularity image from Docker
76+
To build a Singularity image from the locally created docker file do the following:
77+
```
78+
singularity build grid.sif docker-daemon://mycontainer/grid:latest
79+
```
80+
81+
#### Singularity Interactive
82+
To launch a Singularity image build locally.
83+
```
84+
singularity shell --no-home --writable-tmpfs --pwd /benchmark grid.sif
85+
```
86+
87+
#### Singularity Non-Interactive
88+
To launch a Singularity image build locally.
89+
```
90+
singularity run --no-home --writable-tmpfs -pwd /benchmark grid.sif <Grid Command>
91+
```
92+
93+
## Licensing Information
94+
Your access and use of this application is subject to the terms of the applicable component-level license identified below. To the extent any subcomponent in this container requires an offer for corresponding source code, AMD hereby makes such an offer for corresponding source code form, which will be made available upon request. By accessing and using this application, you are agreeing to fully comply with the terms of this license. If you do not agree to the terms of this license, do not access or use this application.
95+
96+
The application is provided in a container image format that includes the following separate and independent components:
97+
|Package | License | URL|
98+
|---|---|---|
99+
|Ubuntu| Creative Commons CC-BY-SA Version 3.0 UK License |[Ubuntu Legal](https://ubuntu.com/legal)|
100+
|CMAKE|OSI-approved BSD-3 clause|[CMake License](https://cmake.org/licensing/)|
101+
|OpenMPI|BSD 3-Clause|[OpenMPI License](https://www-lb.open-mpi.org/community/license.php)<br /> [OpenMPI Dependencies Licenses](https://docs.open-mpi.org/en/v5.0.x/license/index.html)|
102+
|OpenUCX|BSD 3-Clause|[OpenUCX License](https://openucx.org/license/)|
103+
|ROCm|Custom/MIT/Apache V2.0/UIUC OSL|[ROCm Licensing Terms](https://rocm.docs.amd.com/en/latest/release/licensing.html)|
104+
|Grid|GPL V2|[Grid](https://github.com/paboyle/Grid)<br >[Grid License](https://github.com/paboyle/Grid/blob/develop/LICENSE)|
105+
106+
Additional third-party content in this container may be subject to additional licenses and restrictions. The components are licensed to you directly by the party that owns the content pursuant to the license terms included with such content and is not licensed to you by AMD. ALL THIRD-PARTY CONTENT IS MADE AVAILABLE BY AMD “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT.
107+
108+
## Disclaimer
109+
The information contained herein is for informational purposes only, and is subject to change without notice. In addition, any stated support is planned and is also subject to change. While every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale.
110+
111+
## Notices and Attribution
112+
© 2022-2023 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, Instinct, Radeon Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc.
113+
114+
Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. Docker, Inc. and other parties may also have trademark rights in other terms used herein. Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.
115+
116+
All other trademarks and copyrights are property of their respective owners and are only mentioned for informative purposes.

grid/docker/benchmark/gpu_bind.sh

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash
2+
3+
export HIP_VISIBLE_DEVICES=$OMPI_COMM_WORLD_LOCAL_RANK
4+
5+
case $OMPI_COMM_WORLD_LOCAL_RANK in
6+
[0]) cpus=0-15 ;;
7+
[1]) cpus=16-31;;
8+
[2]) cpus=32-47;;
9+
[3]) cpus=48-63;;
10+
[4]) cpus=64-79;;
11+
[5]) cpus=80-95;;
12+
[6]) cpus=96-111;;
13+
[7]) cpus=112-127;;
14+
esac
15+
16+
numactl $@

0 commit comments

Comments
 (0)