This issue was opened automatically by the Test Playbooks workflow after the test quick-train-lora failed on the main branch.
Failure scope
- Playbook:
pytorch-finetuning
- Test id:
quick-train-lora
- Device:
rx7900xt
- Operating system:
linux
- Runner labels:
self-hosted, Linux, rx7900xt
- Runner name:
CS-RORDMZ-DT71
- Commit:
eb0f0325fc56bdcdd679be10aab7bdc822119586
- Workflow run: https://github.com/amd/playbooks/actions/runs/29402226653
Hardware / OS to use to reproduce
Run the failing test on a machine that matches the runner labels above (OS = linux, device = rx7900xt). The repo's self-hosted runners already advertise these labels; if you reproduce locally, use the same OS family and the same AMD device class.
How to dispatch the same test from CI
Re-run only the failing playbook on the same matrix entry by triggering the workflow with the playbook id:
gh workflow run test-playbooks.yml --repo amd/playbooks -f playbook_id=pytorch-finetuning
The workflow's matrix narrows down to this (device, platform) combination automatically based on the playbook's tested_platforms.
How to run just this test locally
python .github/scripts/run_playbook_tests.py --playbook pytorch-finetuning --platform linux --device rx7900xt
The runner extracts test blocks from playbooks/*/pytorch-finetuning/README.md (the failing block starts around line 375).
Failing test (verbatim from the README)
- Setup:
source finetune-venv/bin/activate
- Timeout:
600s
import os
import subprocess
import sys
os.environ["QUICK_TRAIN"] = "1"
os.environ["QUICK_TRAIN_MODEL"] = "unsloth/gemma-3-4b-it"
r = subprocess.run([sys.executable, "train_lora.py"], timeout=600)
sys.exit(r.returncode)
Result
stderr (last lines)
bitsandbytes library load error: Configured ROCm binary not found at /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/bitsandbytes/libbitsandbytes_rocm83.so
Traceback (most recent call last):
File "/home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/bitsandbytes/cextension.py", line 320, in <module>
lib = get_native_library()
File "/home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/bitsandbytes/cextension.py", line 288, in get_native_library
raise RuntimeError(f"Configured {BNB_BACKEND} binary not found at {cuda_binary_path}")
RuntimeError: Configured ROCm binary not found at /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/bitsandbytes/libbitsandbytes_rocm83.so
Using the latest cached version of the dataset since databricks/databricks-dolly-15k couldn't be found on the Hugging Face Hub (offline mode is enabled).
Found the latest cached dataset configuration 'default' at /home/ubuntu/.cache/huggingface/datasets/databricks___databricks-dolly-15k/default/0.0.0/bdd27f4d94b9c1f951818a7da7fd7aeea5dbff1a (last modified on Tue Jul 14 17:56:31 2026).
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:03<00:03, 3.94s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.50s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:05<00:00, 2.71s/it]
Some parameters are on the meta device because they were offloaded to the cpu.
Tokenizing train dataset: 0%| | 0/6 [00:00<?, ? examples/s]
Tokenizing train dataset: 100%|██████████| 6/6 [00:00<00:00, 79.40 examples/s]
Building labels for train dataset: 0%| | 0/6 [00:00<?, ? examples/s]
Building labels for train dataset: 100%|██████████| 6/6 [00:00<00:00, 2687.79 examples/s]
Truncating train dataset: 0%| | 0/6 [00:00<?, ? examples/s]
Truncating train dataset: 100%|██████████| 6/6 [00:00<00:00, 3427.65 examples/s]
Tokenizing eval dataset: 0%| | 0/2 [00:00<?, ? examples/s]
Tokenizing eval dataset: 100%|██████████| 2/2 [00:00<00:00, 650.33 examples/s]
Building labels for eval dataset: 0%| | 0/2 [00:00<?, ? examples/s]
Building labels for eval dataset: 100%|██████████| 2/2 [00:00<00:00, 1349.08 examples/s]
Truncating eval dataset: 0%| | 0/2 [00:00<?, ? examples/s]
Truncating eval dataset: 100%|██████████| 2/2 [00:00<00:00, 1334.49 examples/s]
The model is already on multiple devices. Skipping the move to device specified in `args`.
0%| | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
File "/home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/train_lora.py", line 264, in <module>
trainer.train()
~~~~~~~~~~~~~^^
File "/home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/transformers/trainer.py", line 2325, in train
return inner_training_loop(
args=args,
...<2 lines>...
ignore_keys_for_eval=ignore_keys_for_eval,
)
File "/home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
File "/home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/trl/trainer/sft_trainer.py", line 1844, in training_step
return super().training_step(*args, **kwargs)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/transformers/trainer.py", line 4020, in training_step
loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
File "/home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/trl/trainer/sft_trainer.py", line 1781, in compute_loss
per_token_entropy = entropy_from_logits(shift_logits)
File "/home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/pytorch-finetuning/assets/finetune-venv/lib/python3.13/site-packages/trl/trainer/utils.py", line 506, in entropy_from_logits
chunk_entropy = -(torch.exp(logps) * logps).sum(-1)
~~~~~~~~~^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 130.00 MiB. GPU 0 has a total capacity of 23.98 GiB of which 0 bytes is free. Of the allocated memory 7.56 GiB is allocated by PyTorch, and 41.65 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)
0%| | 0/1 [00:06<?, ?it/s]
stdout (last lines)
Loading dataset...
QUICK_TRAIN=1: using non-gated model for smoke test: unsloth/gemma-3-4b-it
QUICK_TRAIN=1: using 1 step and a tiny dataset (smoke test).
Train samples: 6, Test samples: 2
Total selected samples: 8
Loading unsloth/gemma-3-4b-it...
Note: Model is stored as MXFP4 on Hugging Face but will be loaded as BF16 for training
(This is expected - the warning about MXFP4 is informational)
Base model loaded. Memory footprint: 8.60 GB
Gradient checkpointing enabled
Trainable params: 65,576,960 (1.50%)
Total params: 4,365,656,432
LoRA rank: 32
LoRA alpha: 64
Using bf16 mixed precision.
Starting LoRA Fine-tuning...
Model: unsloth/gemma-3-4b-it
Trainable parameters: 65,576,960
Effective batch size: 16
Learning rate: 0.0003
Quick smoke mode enabled: tiny dataset + max_steps=1
This issue is opened and deduplicated by .github/scripts/create_failure_issues.py. Close it once the failure is fixed; subsequent failures with the same scope will reopen a fresh issue.
This issue was opened automatically by the Test Playbooks workflow after the test
quick-train-lorafailed on themainbranch.Failure scope
pytorch-finetuningquick-train-lorarx7900xtlinuxself-hosted,Linux,rx7900xtCS-RORDMZ-DT71eb0f0325fc56bdcdd679be10aab7bdc822119586Hardware / OS to use to reproduce
Run the failing test on a machine that matches the runner labels above (OS =
linux, device =rx7900xt). The repo's self-hosted runners already advertise these labels; if you reproduce locally, use the same OS family and the same AMD device class.How to dispatch the same test from CI
Re-run only the failing playbook on the same matrix entry by triggering the workflow with the playbook id:
The workflow's matrix narrows down to this
(device, platform)combination automatically based on the playbook'stested_platforms.How to run just this test locally
The runner extracts test blocks from
playbooks/*/pytorch-finetuning/README.md(the failing block starts around line 375).Failing test (verbatim from the README)
source finetune-venv/bin/activate600sResult
1stderr (last lines)
stdout (last lines)
This issue is opened and deduplicated by
.github/scripts/create_failure_issues.py. Close it once the failure is fixed; subsequent failures with the same scope will reopen a fresh issue.