[QUESTION] Inquiry about DynamicGenerator: EOS not returned until max_new_tokens reached, despite stop_conditions

[keds-rnd]

OS

Linux

GPU Library

CUDA 12.x

Python version

3.10

Pytorch version

2.8.0+cu128

Model

gemm3 27b exl2

Describe the bug

Hello,

I'm a developer in South Korea using your framework, and I want to start by saying thank you for building such an excellent library like ExLlamaV2.

I am currently using a combination of ExLlamaV2 (ExLlamaV2DynamicJob) with FastAPI and Redis to handle multiple concurrent user requests.

I've observed an issue where, even when the model's response to a user query is shorter than max_new_tokens, the generation process seems to continue internally until it reaches max_new_tokens before finally reporting the End-of-Stream (EOS) status.

This persistence occurs even though I have explicitly set stop_conditions. (Currently, I'm mitigating this by forcibly closing the web connection when no data is received for a set period, to allow the next conversation to begin.)

I'm wondering if there's a recommended way to force the LLM to return the EOS status immediately after the output is logically complete, without having to wait until max_new_tokens has been fully generated.

Here is the code I use to create the job:

job = ExLlamaV2DynamicJob(
    input_ids=input_ids, max_new_tokens=max_new_tokens,
    stop_conditions=get_stop_conditions(PROMPT_FORMAT, tokenizer),
    gen_settings=ExLlamaV2Sampler.Settings(),
    filter_prefer_eos=True, identifier=job_id
)

The model I am using is Gemma 3.

Thank you for your time and assistance!

Best regards.

Reproduction steps

.

Expected behavior

.

Logs

No response

Additional context

No response

Acknowledgements

• I have looked for similar issues before submitting this one.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[turboderp]

Are you sure the stop conditions are right for the model's instruct format? The generator is supposed to stop when it encounters one of the stop tokens/strings, but if the model never emits one (or they're blocked because of the sampler settings, though they shouldn't be if you're using default settings) then all the generator can do is keep going until it reaches max_new_tokens.

[keds-rnd]

@turboderp

Hello,

Thank you for your previous quick response.

To reproduce and verify the issue I reported earlier, I ran the provided example file (examples/dynamic_gen.py) in my environment, making only minor adjustments to the model path and setting prompt_format = "gemma".

The main issue is that all my test jobs are ending with the status max_token, rather than the expected stop_token status shown in the library's main documentation/examples. This suggests the generation continues unnecessarily after the LLM has completed its response, causing the observed delay.

My Environment Details:

Model: Gemma3-27B-v1-6.0bpw-h8-exl2 (MikeRoz/TheDrummer_Fallen-Gemma3-27B-v1-6.0bpw-h8-exl2)

CUDA: 12.8

ExLlamaV2: 0.3.2

GPU: RTX 5090 32GB

Could you please investigate why the stop_conditions are not immediately triggering the EOS state and causing the job to terminate before reaching max_new_tokens?

Thank you for your time and support.

Best regards.

--ADD
Additionally, conversations are being handled smoothly and immediately in chat.py.
Thank you.

[keds-rnd]

I've successfully resolved the issue by modifying the gemma section of the get_stop_conditions function in util.py as shown below.

This change ensures that the token ID for <end_of_turn> is used when available, correctly triggering the stop condition and allowing the stream to end immediately upon completion of the LLM's response.

def get_stop_conditions(prompt_format, tokenizer):
    if prompt_format == "llama":
        return [tokenizer.eos_token_id]
    elif prompt_format == "llama3":
        return [tokenizer.single_id("<|eot_id|>")]
    elif prompt_format == "granite":
        return [tokenizer.eos_token_id, "\n\nQuestion:"]
    elif prompt_format == "gemma":
        eot_token_id = tokenizer.single_id("<end_of_turn>")

        if eot_token_id is not None:
             return [tokenizer.eos_token_id, eot_token_id]
        else:
             return [tokenizer.eos_token_id, "<end_of_turn>"]