AMD GPU, moving from Windows to Linux Mint, my experiences, and questions

[AlfredSartan572]

After I already installed ROCm with the instructions below, I saw this entry in the AMD PyTorch install instructions:
    The recommended setup to get a PyTorch environment is through Docker, as it avoids potential installation issues.
    The tested, prebuilt image includes PyTorch, ROCm, and other dependencies.
Thanks AMD...
I did not switch to Docker, but continued installing all components individually.

system requirements:
    https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html
    - ldd --version => glibc 2.39     : must be 2.39
    - uname -srmv => 6.17.0-14-generic     : must be 6.14-1018 or newer
    - uname -srmv => 24.04.1-Ubuntu     : must be 24.04 or newer
(I am using Linux Mint 22.3, and this was OK for me)
(my GPU = 7900 XTX)

if not OK, then:
    (from: https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installryz/native_linux/install-ryzen.html)
    sudo apt update && sudo apt install linux-oem-24.04c

prerequisites:
    https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/prerequisites.html
    - groups => reboot to activate
    - disable IGPU in BIOS

if needed, uninstall ROCm first:
    sudo amdgpu-install --uninstall --rocmrelease=all
    sudo apt purge amdgpu-install
    sudo apt autoremove
    sudo reboot

remark:
All variants of AMD's install instructions fail for me, and all with the same error:
    alternative path /lib/rocmmod doesn't exist
I did not find a proper solution, so I created a symbolic link which "resolved" the issue:
    sudo ln -s /opt/rocm-7.2.0/lib/rocmmod /lib/rocmmod

quick-start instructions:
https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/quick-start.html
for the ROCm part, I skipped the following 2 instructions as those were already done in the prerequisites
    sudo apt install python3-setuptools python3-wheel
    sudo usermod -a -G render,video $LOGNAME
for the AMDGPU part, I skipped the first 2 instructions as those were already done in the ROCm part
    This gets installed to, for example: /lib/modules/6.17.0-14-generic/updates/dkms/
    this feels like the same issue as rocm-core using /lib/rocmmod instead of /opt/rocm-7.2.0/lib/rocmmod
    Maybe some environment variable is missing?

post install:
https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/post-install.html
=> issue: the instructions say to use "/opt/rocm/lib", but that doesn't exist. It is "/opt/rocm-7.2.0/lib"
    I kept AMD's instructions but created another symbolic link to "resolve" this:
        sudo ln -s /opt/rocm-7.2.0 /opt/rocm
=> update-alternatives did not work for me, but environment-modules did (and PATH too of course)
    Also check: https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/install-faq.html

Git:
if you don't already have it installed:
sudo apt install git

python:
should already be installed. If not, use Python 3.12

python venv:
as I install this to use comfyUI, I called my venv "comfyui-venv"

    sudo apt install python3.12-venv
    cd ~
    python3 -m venv comfyui-venv
    source comfyui-venv/bin/activate

PyTorch:
https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/native_linux/install-pytorch.html
as I already installed amdgpu and ROCm manually, I continued with "Option A" using pip.

ONNX Runtime:
https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/native_linux/install-onnx.html
(MIGraphX prerequisite satisfied)

TensorFlow:
https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installrad/native_linux/install-tensorflow.html

ComfyUI:
https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/advanced/advancedrad/linux/comfyui/installcomfyui.html

Flash Attention:
don't bother, it does not improve rendering times, but if you want to:
https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/model-acceleration-libraries.html
Installation works (4500 hipcc calls! took 3 hours!)
running with --use-flash-attention
=> slower than --use-quad-cross-attention
=> hundreds of these warnings: Flash Attention failed, using default SDPA: invalid argument for fmha_fwd

Question:

Did I forget to install something?
Are there other components I could install to improve performance?

Basic speed compare Windows to Linux:
Illustriuos workflow with 1 ksampler and 3 refiners:
    Windows : 45 s
    Linux : 35 s
Z-Image Turbo:
    Windows : 5.0 s
    Linux : 4.5 s
Wan 2.2 i2v, FP8, 5 sec, using lightx2v but with 6 steps instead of the usual 4:
    Windows
        480 x 608 : 211 s
        640 x 800 : 23 m
        768 x 960 : 52 m
        960 x 1200 : 3 h
    Linux
        480 x 608 : 209 s
        640 x 800 : 11 m
        768 x 960 : 20 m
        960 x 1200 : 53 m

my run_comfy.sh script:

#!/bin/bash
cd ~
source comfyui-venv/bin/activate

# enable or disable Manager sync
#cp config_online.ini ~/ComfyUI/user/__manager/config.ini
cp config_offline.ini ~/ComfyUI/user/__manager/config.ini

cd ~/ComfyUI

export PYTHONPATH=/opt/rocm-7.2.0/lib:$PYTHONPATH
export MIGRAPHX_MLIR_USE_SPECIFIC_OPS="attention"
export TORCH_BLAS_PREFER_HIPBLASLT=0

#export ROCM_USE_F16=1


# RDNA3
export HSA_OVERRIDE_GFX_VERSION=11.0.0

# I found the settings below on https://gist.github.com/alexheretic/d868b340d1cef8664e1b4226fd17e0d0

# triton
export TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1
export FLASH_ATTENTION_TRITON_AMD_ENABLE="TRUE"
## Significantly faster attn_fwd performance for wan2.2 workflows
export FLASH_ATTENTION_FWD_TRITON_AMD_CONFIG_JSON='{"BLOCK_M":128,"BLOCK_N":64,"waves_per_eu":1,"PRE_LOAD_V":false,"num_stages":1,"num_warps":8}'

# miopen
## Tell comfyui to *not* disable miopen/cudnn, otherwise upscale perf is much worse
export COMFYUI_ENABLE_MIOPEN=1
## miopen default find mode causes significant initial slowness, yields little or no benefit to workloads I tested
export MIOPEN_FIND_MODE=FAST


echo "Wan2                 : use option 1"
echo "Wan2, huge resolution: use option 3"
echo "SDXL                 : use option 5"
echo

echo "Please choose an option:"
echo "1) --use-quad-cross-attention"
echo "2) --use-quad-cross-attention --disable-pinned-memory"
echo "3) --use-quad-cross-attention --disable-pinned-memory --lowvram"
echo "4) --use-quad-cross-attention --disable-smart-memory"
echo "5) --use-pytorch-cross-attention"
echo "6) --use-pytorch-cross-attention --disable-smart-memory"
echo "7) --use-pytorch-cross-attention --disable-smart-memory --disable-pinned-memory"
echo "8) --use-pytorch-cross-attention --disable-pinned-memory"
echo "9) --use-pytorch-cross-attention --disable-pinned-memory --lowvram"


read -t 9 -p "Nr: " choice

if [ -z "$choice" ]; then
    # No input after 9 seconds, default to first option
    choice=1
fi
while ! [[ "$choice" =~ ^[1-9]$ ]]; do
    echo "Invalid selection. Please enter a number between 1 and 9."
    read -p "Nr: " choice
done

case "$choice" in
    1) args=" --use-quad-cross-attention";;
    2) args=" --use-quad-cross-attention --disable-pinned-memory";;
    3) args=" --use-quad-cross-attention --disable-pinned-memory --lowvram";;
    4) args=" --use-quad-cross-attention --disable-smart-memory";;
    5) args=" --use-pytorch-cross-attention";;
    6) args=" --use-pytorch-cross-attention --disable-smart-memory";;
    7) args=" --use-pytorch-cross-attention --disable-smart-memory --disable-pinned-memory";;
    8) args=" --use-pytorch-cross-attention --disable-pinned-memory";;
    9) args=" --use-pytorch-cross-attention --disable-pinned-memory --lowvram";;
esac

echo
echo "selected arguments: $args"
echo

python3 main.py --auto-launch $args

read -p "Press [Enter] to finish..."

Question:

are there other environment variables I should set?
are there other command line arguments I should use?