LocalAI version:
Docker latest-gpu-intel latest-gpu-vulkan latest-gpu-hipblas
Environment, CPU architecture, OS, and Version:
CashyOS x86v3, Podman: LocalAI distributed server is running on a KVM running CachySO, with hw passthrough for an Intel Arc A380 gpu. Worker is running on a machine running CachyOS with a 7900XTX gpu. Trying to tun with model "qwen3.6-35b-a3b-apex" on rocm-llama-cpp backend
docker image id:sha256:20173838d297981a28f921c117aae936fb552addaed9467a81401d55d89c5cfd
Describe the bug
The server seems unable to set the Context Size for the worker's model.
To Reproduce
Setup LocalAI server with worker token set. Setup remote machine worker, with worker token. Schedule model to be run on remote worker. Set context size from the web UI to be as big as managable on the remote worker. Run chat with a function needing larger than default context size. Response gets truncated to 8192 Tokens.
Expected behavior
Set model context size should apply to all or be configure-able to the worker nodes.
Logs
12:20:42 PM.025 stderr 12.54.917.954 I slot release: id 2 | task 34807 | stop processing: n_tokens = 8191, truncated = 1
Additional context
The current behavior of setting model context size appears to only apply to the local machine running the main LocalAI server/administration.
LocalAI version:
Docker latest-gpu-intel latest-gpu-vulkan latest-gpu-hipblas
Environment, CPU architecture, OS, and Version:
CashyOS x86v3, Podman: LocalAI distributed server is running on a KVM running CachySO, with hw passthrough for an Intel Arc A380 gpu. Worker is running on a machine running CachyOS with a 7900XTX gpu. Trying to tun with model "qwen3.6-35b-a3b-apex" on rocm-llama-cpp backend
docker image id:sha256:20173838d297981a28f921c117aae936fb552addaed9467a81401d55d89c5cfd
Describe the bug
The server seems unable to set the Context Size for the worker's model.
To Reproduce
Setup LocalAI server with worker token set. Setup remote machine worker, with worker token. Schedule model to be run on remote worker. Set context size from the web UI to be as big as managable on the remote worker. Run chat with a function needing larger than default context size. Response gets truncated to 8192 Tokens.
Expected behavior
Set model context size should apply to all or be configure-able to the worker nodes.
Logs
12:20:42 PM.025 stderr 12.54.917.954 I slot release: id 2 | task 34807 | stop processing: n_tokens = 8191, truncated = 1Additional context
The current behavior of setting model context size appears to only apply to the local machine running the main LocalAI server/administration.