LongVT-RL在videomme上tool_call频率非常低

[HiFiChang]

脚本如下，使用longvideotool/LongVT-RL预训练模型，评测videomme，发现几乎没有工具调用的过程

#!/bin/bash

# Environment variables
export OPENAI_API_BASE="http://localhost:8000/v1"
export OPENAI_MODEL_NAME="judge"
export OPENAI_BASE_URL="http://your-judge-server-ip:8000/v1"
export OPENAI_API_KEY="EMPTY"
export USE_LLM_JUDGE=False
export DECORD_EOF_RETRY_MAX=409600

TASK_NAME=videomme_reward_tool                # Evaluation task name
IS_QWEN3_VL=True              # Whether using Qwen3-VL model (True/False)
MAX_FRAME_NUM=${4:-768}     # Number of frames (Default:768)

# Path to MCP server for tool calling
MCP_PATH="./examples/video_tools/mcp_server.py"

# Activate conda environment
source /opt/conda/etc/profile.d/conda.sh
conda activate eval

# Start vLLM server
# Qwen3 VL does not need additional chat template
if [ "$IS_QWEN3_VL" == "False" ]; then
    vllm serve $CKPT_PATH \
        --chat-template ./examples/eval/tool_call_qwen2_5_vl.jinja \
        --tool-call-parser hermes \
        --enable-auto-tool-choice \
        --data-parallel-size 1 \
        --gpu-memory-utilization 0.8 \
        --trust-remote-code &
else
    vllm serve $CKPT_PATH \
        --tool-call-parser hermes \
        --enable-auto-tool-choice \
        --data-parallel-size 1 \
        --gpu-memory-utilization 0.8 \
        --trust-remote-code &
fi
sleep 240

# Run evaluation
accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval \
    --model async_openai \
    --model_args model_version=$CKPT_PATH,mcp_server_path=$MCP_PATH,fps=1,max_frames=$MAX_FRAME_NUM,max_pixels=50176,base_url=$OPENAI_API_BASE,api_key=$OPENAI_API_KEY,num_cpus=1,timeout=12000,is_qwen3_vl=$IS_QWEN3_VL \
    --tasks $TASK_NAME \
    --batch_size 1 \
    --output_path ./eval_logs \
    --log_samples \
    --include_path ./lmms_eval_tasks \
    --limit 100

详细信息如下
20260204_234358_results.json

20260204_234358_samples_videomme_reward_tool.json

[mwxely]

Hi @HiFiChang ,

Thanks for the detailed report! After reviewing your configuration, I identified several potential issues:

1. Task Version Mismatch

The experiments in the LongVT paper were conducted on VideoMME with subtitles. You are currently using videomme_reward_tool (without subtitles), which may lead to unexpected model behaviors.

2. Incorrect IS_QWEN3_VL Setting

The IS_QWEN3_VL argument was added to help future researchers who want to use Qwen3-VL as the base model to train new tool-calling models with our codebase. However, the released checkpoint longvideotool/LongVT-RL is fine-tuned from Qwen2.5-VL, and hence IS_QWEN3_VL should be set to False. This parameter controls how messages are formatted for the API. Using the wrong setting may cause tool calls to fail to trigger properly.

3. Limited Sample Size

With --limit 100, you are only evaluating 100 questions. In our experiments, tool calls are primarily triggered on long videos, but VideoMME contains a mix of short/medium/long videos. A random sample of 100 questions may not include enough long video cases to observe meaningful tool call behavior. Consider removing --limit or increasing it significantly for a more representative evaluation.

4. USE_LLM_JUDGE Setting

You have USE_LLM_JUDGE=False. While this does not affect tool call frequency, it may impact scoring accuracy. When enabled, LLM-as-judge serves as a fallback when exact match fails (e.g., when the model outputs natural language instead of just "A/B/C/D"). For MCQ tasks like VideoMME, the impact is relatively minor, but for open-ended tasks (e.g., VideoSIAH-Eval), USE_LLM_JUDGE=True is required for accurate evaluation.

Recommended Configuration

TASK_NAME=videomme_w_subtitle_reward_tool
IS_QWEN3_VL=False
export USE_LLM_JUDGE=True  # optional but recommended
# Remove --limit 100 or increase sample size

Please try these changes and let us know if the issue persists!

[HiFiChang]

Thanks for your response!
I have re-evaluated the performance by testing the full VideoMME (2,700 tasks) using the videomme_w_subtitle_reward_tool dataset, with Qwen3-8B serving as the LLM judge.
It seems that although I resolved the issues regarding tool call frequency, the final accuracy score was only 0.4.
Is this expected?

Here are the detailed results:
20260213_170040_samples_videomme_w_subtitle_reward_tool.json
20260213_170040_results.json

[mwxely]

Hi,

Thanks for the follow-up and for sharing the detailed result files! I went through both your results.json and samples JSONL carefully. Here is my analysis.

Configuration Verification

First, your model_args in results.json now looks correct:

model_version=...longvideotool--LongVT-RL/snapshots/05e0c569...,
mcp_server_path=./examples/video_tools/mcp_server.py,
fps=1, max_frames=768, max_pixels=50176,
is_qwen3_vl=False

This matches our recommended setup. The task is correctly set to videomme_w_subtitle_reward_tool, limit is null (full 2,700 samples), and format_score is 0.986, confirming that the model is generating well-formed outputs. So the configuration issues from before are resolved — the remaining gap is a scoring issue, not a model behavior issue.

Root Cause Analysis

To understand the gap, I traced our exact scoring pipeline (compute_score in custom_rewards/lmms_lab_recipe.py) against your 2,700 samples. The pipeline works in two stages:

1. Regex extraction (parse_mcq): tries to find an option letter (A/B/C/D) from the <answer> tag, then compares with ground truth.
2. LLM judge fallback: only called when regex extraction fails to match — i.e., when the model outputs a natural language answer instead of a letter, and USE_LLM_JUDGE=True.

I replicated parse_mcq() on all 2,700 of your extracted answers and cross-referenced with the acc_score in your samples. The breakdown:

Category	Count	acc_score	Judge needed?
Regex found correct letter	1,101	1.0	No
Model chose wrong letter directly (e.g. <answer>A</answer> when target is C)	609	0	Called, but model is genuinely wrong
NL answer, regex extracted a spurious letter (e.g. <answer>Ballet</answer> → regex picks up "B" from the capital letter)	346	0	Called as fallback
NL answer, no letter found at all (e.g. <answer>Pink</answer>)	641	0	Called as fallback

The key observation: for the last two categories (346 + 641 = 987 samples), the judge should be called as a fallback. But in your results, all 987 scored 0 — not a single one was rescued by the judge.

To verify whether this is reasonable, I cross-referenced the model's natural language answers against the actual MCQ option text from the input prompts. Among those 987 samples, at least 493 are trivially correct — the model's answer is a near-verbatim match with the correct option text:

Model Answer	Target	Correct Option Text	acc_score
Pink	D	Pink.	0
May 5th	B	May 5th.	0
Ballet	C	Ballet.	0
United States	D	United States.	0
Shamrock	D	Shamrock.	0
Hosting large parades	A	Hosting large parades.	0
She decides what size the head should be	C	She decides what size the head should be.	0

A 0/493 rescue rate on such trivial matches (differing only by a trailing period) is extremely unlikely even with a weak judge model. This strongly suggests that the LLM judge was not effectively invoked during your evaluation — either USE_LLM_JUDGE was not set to True, or the judge server was not reachable (in which case the code catches the exception and silently defaults to 0).

A further cross-check from results.json supports this: the two accuracy metrics are videomme_perception_score = 27.1% (based on the simpler extract_characters_regex) and acc_score = 40.8% (based on the more aggressive parse_mcq in compute_score). The gap between them (13.6%) is fully explained by parse_mcq's better regex patterns, with no additional boost from the judge — consistent with the judge not being active.

Estimated Impact

If the judge had been working, those 493 trivially correct samples alone would bring accuracy from 40.8% to at least 59.0% ((1101+493)/2700). The true number is likely higher, since some of the remaining 494 NL answers may also be semantically correct and recoverable by a capable judge.

Recommendation

1. Verify that USE_LLM_JUDGE is actually enabled and the judge server is reachable. This is the most critical step. Please double-check:

# USE_LLM_JUDGE is an environment variable, must be exported before launching lmms_eval
# Note: the code checks for the exact string "True" (case-sensitive)
export USE_LLM_JUDGE=True

# Judge server connection — make sure these point to a running vLLM/SGLang instance
export OPENAI_BASE_URL="http://your-judge-server-ip:port/v1"
export OPENAI_MODEL_NAME="judge"
export OPENAI_API_KEY="EMPTY"

You can verify the judge is reachable by running a quick curl test before starting evaluation:

curl $OPENAI_BASE_URL/models

2. Use a strong judge model. In our evaluation, we used Qwen/Qwen3-235B-A22B:

vllm serve Qwen/Qwen3-235B-A22B \
    --port <your_judge_port> \
    --gpu-memory-utilization 0.9 \
    --tensor-parallel-size 8 \
    --served-model-name "judge" \
    --trust-remote-code

If running a 235B model is not feasible, Qwen/Qwen2.5-72B-Instruct should also work (we used it for RL reward computation during training). For VideoMME where the judge only needs to match natural language answers to MCQ options, a 72B model should be sufficient.

3. Verify generation temperature. The generation_kwargs in your results contains max_new_tokens: 4096 and until: ["\n\n"] with no explicit temperature. Our task YAML (_default_template_yaml) also does not set temperature — it relies on the async_openai model's code-level default of temperature=0. This should be fine in practice, but you can verify by checking your vLLM server logs or explicitly adding temperature: 0 to your generation config.

Please try these changes (especially verifying the judge setup) and let us know the results!

[HiFiChang]

Thanks for your detailed analysis! I will carefully check mentioned potencial mistakes in my evaluation config!

[mwxely]

We are closing this issue as it appears resolved and we have not received further feedback. Please feel free to reopen or file a new issue if needed.

[xuzq23]

Thanks for your response! I have re-evaluated the performance by testing the full VideoMME (2,700 tasks) using the videomme_w_subtitle_reward_tool dataset, with Qwen3-8B serving as the LLM judge. It seems that although I resolved the issues regarding tool call frequency, the final accuracy score was only 0.4. Is this expected?

Here are the detailed results: 20260213_170040_samples_videomme_w_subtitle_reward_tool.json 20260213_170040_results.json

@HiFiChang Hello, I noticed that in the first result, it took 2488.3 seconds to process 100 samples, while in the second result, it only took 578 seconds to process all samples. How was this achieved?