Building and Running ABACUS

Requirements

Please refer to installation guide. Below, we will discuss each required components:

• Compiler, including MPI support
• Math libs, including FFTW
• ELPA
• Cereal

Too long; don't read: see also dockerfile.* under abacus repositry root path for reference. The lines starting with RUN indicate commands.

Intel oneAPI toolkit

The Intel® oneAPI toolkit provides a complete toolchain. The Intel® oneAPI Base Toolkit contains Intel® oneAPI Math Kernel Library (aka MKL), providing a fast BLAS library. The Intel® oneAPI HPC Toolkit contains compilers with MPI Library.

Pros: you don't need to compile varios BLAS libraries by yourself.

Cons: large disk usage; potential backfire on non-Intel platform.

Below are minimum required components by ABACUS:

apt-get install -y  \
		intel-oneapi-compiler-dpcpp-cpp-and-cpp-classic \
		intel-oneapi-compiler-fortran \
		intel-oneapi-mkl-devel \
		intel-oneapi-mpi-devel

Alternatively, install the whole Toolkit by the links above (recommended), or by the commands below. Be sure to check disk free space before:

wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18236/l_BaseKit_p_2021.4.0.3422_offline.sh
sudo bash l_BaseKit_p_2021.4.0.3422_offline.sh
wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18211/l_HPCKit_p_2021.4.0.3347_offline.sh
sudo bash l_HPCKit_p_2021.4.0.3347_offline.sh

After installing, configure your system. To work at a Command Line Interface (CLI), you may configure the components of the oneAPI toolkits using environment variable script.

source /opt/intel/oneapi/setvars.sh

DO NOT forget to set env vars each time you create a new shell window! Most error happens here. To configure environment variables to be set up automatically, add the command above to your ~/.bashrc or ~/.zshrc, respectively. Check correctness by mpiicc --version, and it should show something like:

icc (ICC) 2021.1 Beta 20201112 Copyright (C) 1985-2020 Intel Corporation. All rights reserved.

The compilers we use are: mpiicc, mpiicpc and mpiifort, being counterparts to mpicc, mpicxx and gfortran.

Now, go to the ELPA part after completing the installation. Continue to read if you decide not using Intel oneAPI kit.

Compilers

gcc

Abacus requires a minimum gcc version of 4.9.2. Check by gcc --version. We recommend using the latest gcc. If the gcc is not qualified,

• Install from apt/yum: apt-get install gcc
• Install [Red Hat Developer Toolset](Red Hat Developer Toolset)

gfortran

ELPA requires a fortran compiler. apt-get install gfortran helps.

MPI

Abacus relies on MPI to scale up. This requires compilers with MPI wrapper, e.g. mpicc. We recommend using MPICH:

apt-get install mpich libmpich-dev

Math libs

BLAS

We use OpenBLAS.

git clone https://github.com/xianyi/OpenBLAS.git
cd OpenBLAS 
make -j9 FC=gfortran
make PREFIX=/usr/local install

ScaLAPACK

git clone https://github.com/darelbeida/scalapack.git -b v2.0.2-openblas --single-branch --depth=1 \
cd scalapack && make lib
cp libscalapack.a /usr/local/lib/ 

FFTW

The support to FFTW2 is out-dated in ABACUS; FFTW here only reference to FFTW3. Simply installing FFTW by apt-get will not install the MPI support part, i.e. a serial version of FFTW. To get FFTW with parallelism support, build from source:

wget http://www.fftw.org/fftw-3.3.9.tar.gz 
tar zxvf fftw-3.3.9.tar.gz 
cd fftw-3.3.9 
./configure --enable-mpi-fortran --enable-orterun-prefix-by-default FC=gfortran 
make -j9
make PREFIX=/usr/local install

ELPA

We use ELPA to solve eigenvalue.

 wget https://elpa.mpcdf.mpg.de/software/tarball-archive/Releases/2021.05.002/elpa-2021.05.002.tar.gz
tar xzf elpa-2021.05.002.tar.gz
cd elpa-2021.05.002 
mkdir build 
cd build 

The configure process varies between compilers.

For gcc compiler, use the configure command below:

../configure CFLAGS="-O3 -march=native -funsafe-loop-optimizations -funsafe-math-optimizations -ftree-vect-loop-version -ftree-vectorize" \
    FCFLAGS="-O2 -mavx" --disable-avx512 

Here's another one for Intel oneAPI(don't forget to run setvars.sh before!):

CC=mpiicc CXX=mpiicpc FC=mpiifort ../configure FCFLAGS="-mkl=cluster"

Build and install.

make -j9 
make PREFIX=/usr/local install
ln -s /usr/local/include/elpa-2021.05.002/elpa /usr/local/include/ 

Cereal

We use Cereal to serialize data. It is a header-only library.

git clone https://github.com/USCiLab/cereal.git \
cp -r cereal/include /usr/local