|
| 1 | +# Cholla |
| 2 | + |
| 3 | +## Overview |
| 4 | +Cholla is a static-mesh, GPU-native hydrodynamics simulation code that efficiently runs high-resolution simulations on massively-parallel computers. The code is written in a combination of C++, Cuda C, and HIP, and requires at least one AMD GPU to run. |
| 5 | + |
| 6 | +Cholla (Computational Hydrodynamics On ParaLLel Architectures) is a 3D GPU-based hydrodynamics code (Schneider & Robertson, ApJS, 2015). Cholla is designed to be run using (AMD or NVIDIA) GPUs, and can be run in serial mode using one GPU or with MPI for multiple GPUs. |
| 7 | + |
| 8 | +The source code is available on [github](https://github.com/cholla-hydro/cholla/). Cholla was designed for astrophysics simulations, and the current release includes the following physics: |
| 9 | +- compressible hydrodynamics in 1, 2, or 3 dimensions |
| 10 | +- optically-thin radiative cooling and photoionization heating, including the option to use the Grackle cooling library |
| 11 | +- static gravity with user-defined functions |
| 12 | +- FFT-based gas self-gravity |
| 13 | +- particle-mesh based particle gravity |
| 14 | +- cosmology |
| 15 | +- passive scalar tracking |
| 16 | + |
| 17 | +Cholla can be run using a variety of different numerical algorithms, allowing users to test the sensitivity of their results to the exact code configuration. Options include: |
| 18 | +- Exact, Roe, and HLLC Riemann solvers |
| 19 | +- 2nd and 3rd order spatial reconstruction with limiting in either primitive or conserved variables |
| 20 | +- a second order Van Leer integrator |
| 21 | + |
| 22 | +## Single-Node Server Requirements |
| 23 | + |
| 24 | +| CPUs | GPUs | Operating Systems | ROCm™ Driver | Container Runtimes | |
| 25 | +| ---- | ---- | ----------------- | ------------ | ------------------ | |
| 26 | +| X86_64 CPU(s) | AMD Instinct MI200 GPU(s) <br> AMD Instinct MI100 GPU(s) <br> Radeon Instinct MI50(S) | Ubuntu 20.04 <br> Ubuntu 22.04 <BR> RHEL8 <br> RHEL9 <br> SLES 15 sp4 | ROCm v5.x compatibility |[Docker Engine](https://docs.docker.com/engine/install/) <br> [Singularity](https://sylabs.io/docs/) | |
| 27 | + |
| 28 | +For ROCm installation procedures and validation checks, see: |
| 29 | +* [ROCm Documentation](https://rocm.docs.amd.com) |
| 30 | +* [AMD Lab Notes ROCm installation notes](https://github.com/amd/amd-lab-notes/tree/release/rocm-installation). |
| 31 | +* [ROCm Examples](https://github.com/amd/rocm-examples) |
| 32 | + |
| 33 | +## Build Recipes |
| 34 | +- [Docker/Singularity Build](/cholla/docker/) |
| 35 | +- [Baremetal Build](/cholla/baremetal/) |
| 36 | + |
| 37 | +## Running Cholla Benchmarks |
| 38 | +These examples are using the [Cholla Docker Build](/cholla/docker/), which build the binary file cholla.hydro.cholla-container based on environment variable `CHOLLA_MACHINE` set to `cholla-container` and was build/installed into `/opt/cholla`. |
| 39 | + |
| 40 | +Cholla has many examples within the project, `/path/to/cholla/examples/`, find the benchmark that you wish to run in there, or provide your own workload. |
| 41 | + |
| 42 | +You can run the examples using the command syntax below, where `#` is the number of GPUs to use and `<benchmark-to-run>` should be replaced with the full file path to the load details. |
| 43 | + |
| 44 | +``` |
| 45 | +mpirun -np # cholla.hydro.cholla-container <benchmark-to-run> |
| 46 | +``` |
| 47 | +### Examples: |
| 48 | +* 4 GPU 3D sound wave<br>`mpirun -np 4 cholla.hydro.cholla-container /opt/cholla/examples/3D/sound_wave.txt` |
| 49 | +* 8 GPU 3D Sound wave<br>`mpirun -np 8 cholla.hydro.cholla-container /opt/cholla/examples/3D/sound_wave.txt` |
| 50 | +* 4 GPU 3D sod<br> `mpirun -np 4 cholla.hydro.cholla-container /opt/cholla/examples/3D/sod.txt` |
| 51 | +* 8 GPU 3D sod<br> `mpirun -np 8 cholla.hydro.cholla-container /opt/cholla/examples/3D/sod.txt` |
| 52 | + |
| 53 | +Each MPI rank will bind to a particular unique GPU (1 rank per device) and strong-scale the problem accordingly. Please define `HIP_VISIBLE_DEVICES` to control which particular GPUs are available to Cholla. |
| 54 | + |
| 55 | + |
| 56 | +## Licensing Information |
| 57 | +Your access and use of this application is subject to the terms of the applicable component-level license identified below. To the extent any subcomponent in this container requires an offer for corresponding source code, AMD hereby makes such an offer for corresponding source code form, which will be made available upon request. By accessing and using this application, you are agreeing to fully comply with the terms of this license. If you do not agree to the terms of this license, do not access or use this application. |
| 58 | + |
| 59 | +The application is provided in a container image format that includes the following separate and independent components: |
| 60 | +|Package | License | URL| |
| 61 | +|---|---|---| |
| 62 | +|Ubuntu| Creative Commons CC-BY-SA Version 3.0 UK License |[Ubuntu Legal](https://ubuntu.com/legal)| |
| 63 | +|CMAKE|OSI-approved BSD-3 clause|[CMake License](https://cmake.org/licensing/)| |
| 64 | +|OpenMPI|BSD 3-Clause|[OpenMPI License](https://www-lb.open-mpi.org/community/license.php)<br /> [OpenMPI Dependencies Licenses](https://docs.open-mpi.org/en/v5.0.x/license/index.html)| |
| 65 | +|OpenUCX|BSD 3-Clause|[OpenUCX License](https://openucx.org/license/)| |
| 66 | +|ROCm|Custom/MIT/Apache V2.0/UIUC OSL|[ROCm Licensing Terms](https://rocm.docs.amd.com/en/latest/release/licensing.html)| |
| 67 | +|Cholla|MIT|[Cholla](https://github.com/cholla-hydro/cholla)<br >[Cholla License](https://github.com/cholla-hydro/cholla/blob/main/LICENSE.txt)| |
| 68 | +|HDF5|BSD-like(CUSTOM)|[HDF5 License](https://github.com/HDFGroup/hdf5/blob/develop/COPYING)| |
| 69 | + |
| 70 | + |
| 71 | +Additional third-party content in this container may be subject to additional licenses and restrictions. The components are licensed to you directly by the party that owns the content pursuant to the license terms included with such content and is not licensed to you by AMD. ALL THIRD-PARTY CONTENT IS MADE AVAILABLE BY AMD “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT. |
| 72 | + |
| 73 | +## Disclaimer |
| 74 | +The information contained herein is for informational purposes only, and is subject to change without notice. In addition, any stated support is planned and is also subject to change. While every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale. |
| 75 | + |
| 76 | +## Notices and Attribution |
| 77 | +© 2022-2023 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, Instinct, Radeon Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. |
| 78 | + |
| 79 | +Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. Docker, Inc. and other parties may also have trademark rights in other terms used herein. Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries. |
| 80 | + |
| 81 | +All other trademarks and copyrights are property of their respective owners and are only mentioned for informative purposes. |
0 commit comments