Skip to content

[CI] unsloth-llms-finetuning / quick-train-unsloth failed on r9700 (linux) #538

Description

@github-actions

This issue was opened automatically by the Test Playbooks workflow after the test quick-train-unsloth failed on the main branch.

Failure scope

  • Playbook: unsloth-llms-finetuning
  • Test id: quick-train-unsloth
  • Device: r9700
  • Operating system: linux
  • Runner labels: self-hosted, Linux, r9700
  • Runner name: MNB-UCICD-DT386
  • Commit: eb0f0325fc56bdcdd679be10aab7bdc822119586
  • Workflow run: https://github.com/amd/playbooks/actions/runs/29402226653

Hardware / OS to use to reproduce

Run the failing test on a machine that matches the runner labels above (OS = linux, device = r9700). The repo's self-hosted runners already advertise these labels; if you reproduce locally, use the same OS family and the same AMD device class.

How to dispatch the same test from CI

Re-run only the failing playbook on the same matrix entry by triggering the workflow with the playbook id:

gh workflow run test-playbooks.yml --repo amd/playbooks -f playbook_id=unsloth-llms-finetuning

The workflow's matrix narrows down to this (device, platform) combination automatically based on the playbook's tested_platforms.

How to run just this test locally

python .github/scripts/run_playbook_tests.py --playbook unsloth-llms-finetuning --platform linux --device r9700

The runner extracts test blocks from playbooks/*/unsloth-llms-finetuning/README.md (the failing block starts around line 416).

Failing test (verbatim from the README)

  • Setup: source unsloth-env/bin/activate
  • Timeout: 2400s
python test_unsloth_ci.py

Result

  • Exit code: -6

stderr (last lines)

 this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.vit.image_processing_vit`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
Accessing `is_flash_linear_attention_available` from `.models.vitmatte.image_processing_pil_vitmatte`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.vitmatte.image_processing_pil_vitmatte`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
Accessing `is_flash_linear_attention_available` from `.models.vitmatte.image_processing_vitmatte`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.vitmatte.image_processing_vitmatte`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
Accessing `is_flash_linear_attention_available` from `.models.vitpose.image_processing_pil_vitpose`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.vitpose.image_processing_pil_vitpose`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
Accessing `is_flash_linear_attention_available` from `.models.vitpose.image_processing_vitpose`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.vitpose.image_processing_vitpose`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
Accessing `is_flash_linear_attention_available` from `.models.vivit.image_processing_vivit`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.vivit.image_processing_vivit`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
Accessing `is_flash_linear_attention_available` from `.models.yolos.image_processing_pil_yolos`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.yolos.image_processing_pil_yolos`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
Accessing `is_flash_linear_attention_available` from `.models.yolos.image_processing_yolos`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.yolos.image_processing_yolos`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
Accessing `is_flash_linear_attention_available` from `.models.zoedepth.image_processing_pil_zoedepth`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.zoedepth.image_processing_pil_zoedepth`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
Accessing `is_flash_linear_attention_available` from `.models.zoedepth.image_processing_zoedepth`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.
[transformers|WARNING]Accessing `is_flash_linear_attention_available` from `.models.zoedepth.image_processing_zoedepth`. Returning `is_flash_linear_attention_available` instead. Behavior may be different and this alias will be removed in future versions.

Loading weights:   0%|          | 0/2130 [00:00<?, ?it/s]
Loading weights:  19%|█▉        | 406/2130 [00:00<00:00, 4044.76it/s]
Loading weights:  38%|███▊      | 811/2130 [00:01<00:03, 385.58it/s] 
Loading weights:  47%|████▋     | 992/2130 [00:02<00:04, 283.69it/s]
Loading weights:  52%|█████▏    | 1098/2130 [00:03<00:03, 277.86it/s]
Loading weights:  55%|█████▌    | 1172/2130 [00:03<00:03, 295.25it/s]
Loading weights:  58%|█████▊    | 1236/2130 [00:03<00:02, 310.71it/s]
Loading weights:  61%|██████    | 1293/2130 [00:03<00:02, 330.13it/s]
Loading weights:  63%|██████▎   | 1347/2130 [00:03<00:02, 342.09it/s]
Loading weights:  66%|██████▌   | 1397/2130 [00:04<00:03, 240.17it/s]
Loading weights:  67%|██████▋   | 1435/2130 [00:08<00:16, 42.16it/s] 
Loading weights:  69%|██████▊   | 1462/2130 [00:09<00:16, 41.25it/s]
Loading weights:  87%|████████▋ | 1848/2130 [00:09<00:01, 164.38it/s]
Loading weights: 100%|██████████| 2130/2130 [00:09<00:00, 222.17it/s]
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': 2}.
[transformers.trainer_utils|WARNING]The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': 2}.
==((====))==  Unsloth - 2x faster free finetuning | Num GPUs used = 1
   \\   /|    Num examples = 128 | Num Epochs = 1 | Total steps = 5
O^O/ \_/ \    Batch size per device = 1 | Gradient accumulation steps = 4
\        /    Data Parallel GPUs = 1 | Total batch size (1 x 4 x 1) = 4
 "-____-"     Trainable parameters = 20,649,984 of 8,016,806,432 (0.26% trained)
[transformers.trainer|WARNING]==((====))==  Unsloth - 2x faster free finetuning | Num GPUs used = 1
   \\   /|    Num examples = 128 | Num Epochs = 1 | Total steps = 5
O^O/ \_/ \    Batch size per device = 1 | Gradient accumulation steps = 4
\        /    Data Parallel GPUs = 1 | Total batch size (1 x 4 x 1) = 4
 "-____-"     Trainable parameters = 20,649,984 of 8,016,806,432 (0.26% trained)

  0%|          | 0/5 [00:00<?, ?it/s]terminate called after throwing an instance of 'c10::AcceleratorError'
  what():  CUDA error: an illegal memory access was encountered
Search for `hipErrorIllegalAddress' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3
Device-side assertion tracking was not enabled by user.
Exception raised from SetDevice at /__w/rockrel/rockrel/external-builds/pytorch/pytorch/c10/hip/HIPFunctions.cpp:334 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x7377caab605d in /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/unsloth-llms-finetuning/assets/unsloth-env/lib/python3.13/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x1293e (0x737670aa493e in /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/unsloth-llms-finetuning/assets/unsloth-env/lib/python3.13/site-packages/torch/lib/libc10_hip.so)
frame #2: c10::cuda::SetDevice(signed char, bool) + 0x51 (0x737670ae23b1 in /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/unsloth-llms-finetuning/assets/unsloth-env/lib/python3.13/site-packages/torch/lib/libc10_hip.so)
frame #3: <unknown function> + 0x2ca43 (0x737670abea43 in /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/unsloth-llms-finetuning/assets/unsloth-env/lib/python3.13/site-packages/torch/lib/libc10_hip.so)
frame #4: <unknown function> + 0x134198e (0x73767a14198e in /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/unsloth-llms-finetuning/assets/unsloth-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #5: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x5c9 (0x73767ea1a0c9 in /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/unsloth-llms-finetuning/assets/unsloth-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #6: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x337 (0x73767ea0f407 in /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/unsloth-llms-finetuning/assets/unsloth-env/lib/python3.13/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x829ce2 (0x737688029ce2 in /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/unsloth-llms-finetuning/assets/unsloth-env/lib/python3.13/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0xecdb4 (0x7378532ecdb4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #9: <unknown function> + 0x9caa4 (0x73785629caa4 in /lib/x86_64-linux-gnu/libc.so.6)
frame #10: <unknown function> + 0x129c6c (0x737856329c6c in /lib/x86_64-linux-gnu/libc.so.6)


stderr was truncated; see the workflow run artifacts for the full log.

stdout (last lines)

Removing cache: /home/ubuntu/actions-runner/_work/playbooks/playbooks/playbooks/supplemental/unsloth-llms-finetuning/assets/unsloth_compiled_cache
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
🦥 Unsloth Zoo will now patch everything to make training faster!
[09:06:51] ===== Unsloth CI Training Pipeline =====
[09:06:51] Python: 3.13.14 (main, Jun 11 2026, 03:02:07) [GCC 13.3.0]
[09:06:51] PyTorch: 2.11.0+rocm7.13.0
[09:06:51] Checking GPU availability...
[09:06:51] GPU available: AMD Radeon AI PRO R9700
[09:06:51] Loading model...
==((====))==  Unsloth 2026.7.2: Fast Gemma4 patching. Transformers: 5.5.0.
   \\   /|    AMD Radeon AI PRO R9700. Num GPUs = 1. Max memory: 31.859 GB. Platform: Linux.
O^O/ \_/ \    Torch: 2.11.0+rocm7.13.0. ROCm Toolkit: 7.13.99004. Triton: 3.6.0
\        /    Bfloat16 = TRUE. FA [Xformers = None. FA2 = False]
 "-____-"     Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Unsloth: QLoRA and full finetuning all not selected. Switching to 16bit LoRA.
[09:07:09] Loading dataset...
[09:07:09] Standardizing dataset format...
[09:07:09] Applying chat template...
[09:07:11] Prepared dataset size: 128
[09:07:11] Applying LoRA adapters...
[09:07:14] Setting up trainer...
[09:07:16] Starting training...


This issue is opened and deduplicated by .github/scripts/create_failure_issues.py. Close it once the failure is fixed; subsequent failures with the same scope will reopen a fresh issue.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions