Colvar with Torch CV

[cchapellier]

Hi all,

I would like to run a eABF biasing simulation with Gromacs 2025.2 patched with Torch and Colvar. The end goal is to get a free energy landscape and get it converged in a couple days if possible (so far it takes me more than a week with a speed of 90 ns/day and my simulations get systematically interrupted. I am not using Colvar thought and was hoping I might be able to get faster with it). I am looking for feedback on past experience so that I set my environment and simulation parameters as best as possible for fast performance.

My environment has been built with the following Dockerfile:

FROM nvidia/cuda:12.8.1-cudnn-devel-ubuntu22.04
# Install system dependencies
RUN apt update && apt install -y \
    build-essential \
    git \
    wget \
    unzip \
    libboost-all-dev \
    libfftw3-dev \
    libssl-dev \
    libhwloc-dev \
    libxml2-dev \
    python3.10 \
    python3.10-dev \
    libpython3.10

WORKDIR /opt/

# Install CMake >= 3.28
ENV CMAKE_VERSION=4.1.0
RUN wget https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz && \
    tar -zxvf cmake-4.1.0.tar.gz && \
    cd cmake-4.1.0 && \
    ./bootstrap && \
    make && \
    make install

# Install libtorch
ENV TORCH_VERSION=2.7.1
ENV TORCH_ZIP=libtorch-cxx11-abi-shared-with-deps-${TORCH_VERSION}%2Bcu128.zip
RUN wget https://download.pytorch.org/libtorch/cu128/${TORCH_ZIP} -O libtorch.zip && \
    unzip libtorch.zip -d /opt/libtorch && \
    rm libtorch.zip
ENV Torch_DIR=/opt/libtorch/libtorch/share/cmake/Torch
ENV LD_LIBRARY_PATH=/opt/libtorch/lib:$LD_LIBRARY_PATH

# Install GROMACS 2025.2
ENV GMX_VERSION=2025.2
RUN wget https://ftp.gromacs.org/gromacs/gromacs-${GMX_VERSION}.tar.gz && \
    tar -xzf gromacs-${GMX_VERSION}.tar.gz && \
    rm gromacs-${GMX_VERSION}.tar.gz

RUN cd gromacs-${GMX_VERSION} && \
    mkdir build && cd build && \
    cmake .. \
        -DGMX_GPU=CUDA \
        -DGMX_MPI=OFF \
        -DCMAKE_INSTALL_PREFIX=/opt/gromacs \
        -DGMX_NNPOT=TORCH \
        -DPython_EXECUTABLE=/usr/bin/python3.10 \
        -DPython_INCLUDE_DIRS=/usr/include/python3.10 \
        -DPython_LIBRARIES=/usr/lib/x86_64-linux-gnu/libpython3.10.so \
        -DTorch_DIR=/opt/libtorch/libtorch/share/cmake/Torch \
        -DTORCH_CUDA_ARCH_LIST=8.9 && \
    make -j$(nproc) && \
    make install

# Add GROMACS to PATH
ENV PATH="/opt/gromacs/bin:$PATH"

I have access to L4 types of GPU. I didn't install OpenMPI because in the past (I was not using colvar), thread MPI with 8 OPenMP threads was the best combo for me for performance.

When I switch to colvar, do you recommend I use OpenMPI ? Do I need to allocate part of the computation to the GPUs and some to the CPUs ?

Thank you for your help and feedback !

[jhenin]

Hi Charlotte!

Just to clarify what you are doing now: you are running eABF, but not with Colvars, so I assume using Plumed?
What throughput do you get in a plain MD simulation on the same hardware?
Why do your simulations get interrupted?

I see no a priori reason to enable OpenMPI. You should keep a couple OpenMP threads for Colvars. 8 threads should be plenty if all the force calculations are offloaded to the GPU.

I recommend you run some benchmarks using Colvars, then it will be easier to discuss.

[cchapellier]

Hi Jerôme !

Thank you for your answer. Yes I am running it with plumed. It's working well, however I cannot run the biasing part on GPU yet (apparently they are working on it though so it is a matter of time). I have frequent interruptions, which I think are due to way our cluster is designed. My ressources seem to get reallocated every couple days, as in my logs I get a backup.sh: fork: retry: Resource temporarily unavailable error and then i cannot connect to the container with ssh anymore. It seems to me that it is a server type of failure but I am still investigating it.

I remember attending a conference during which you explained that now with Colvar you can get a similar speed on unbiased and biased trajectories so I wanted to try it out.

Thank you for your feedback on OpenMPI ! I'll come back after trying a few things then.