ASR 模型评估错误

[zzhdbw]

当我执行代码：
python audio_evals/main.py \ --dataset KeSpeech \ --prompt simple-asr \ --model paraformer-large \ --limit 2000
每条都会出现如下错误，导致评估失败。

{"type": "error", "id": 1888, "data": {"info": "Traceback (most recent call last):\n File \"/home/zzh/code/EVAL/UltraEval-Audio/audio_evals/eval_task.py\", line 75, in _run\n score, ans = self._eval(\n File \"/home/zzh/code/EVAL/UltraEval-Audio/audio_evals/eval_task.py\", line 52, in _eval\n output = self.predictor.inference(prompt)\n File \"/home/zzh/code/EVAL/UltraEval-Audio/audio_evals/models/model.py\", line 75, in inference\n return super().inference(prompt, **kwargs)\n File \"/home/zzh/code/EVAL/UltraEval-Audio/audio_evals/models/model.py\", line 38, in inference\n return self._inference(prompt, **sample_params)\n File \"/home/zzh/code/EVAL/UltraEval-Audio/audio_evals/models/asr/paraformer.py\", line 33, in _inference\n self.process.stdin.write(f\"{prefix}{audio}\\n\")\nBrokenPipeError: [Errno 32] Broken pipe\n"}}

请问是环境的原因吗，我看源码中创建了一个虚拟环境

[UltraEval]

可以重新拉下代码，会打印出来具体错误

[zzhdbw]

作者您好，我又重新拉代码试了一下，执行了如下命令

export PYTHONPATH=$PWD:$PYTHONPATH

export HF_ENDPOINT=https://hf-mirror.com

CUDA_VISIBLE_DEVICES=0 python audio_evals/main.py \
    --dataset sample \
    --model paraformer-zh

报错如下：

{"type": "prompt", "id": 2, "data": {"content": [{"role": "user", "contents": [{"type": "audio", "value": "raw/TwinkStart/sample/sample/2.wav"}, {"type": "text", "value": "listen the audio, output the audio content with format {\"content\": \"\"}"}]}]}}
{"type": "error", "id": 2, "data": {"info": "Traceback (most recent call last):\n  File \"/home/zzh/code/ASR/UltraEval-Audio/audio_evals/eval_task.py\", line 75, in _run\n    score, ans = self._eval(\n  File \"/home/zzh/code/ASR/UltraEval-Audio/audio_evals/eval_task.py\", line 52, in _eval\n    output = self.predictor.inference(prompt)\n  File \"/home/zzh/code/ASR/UltraEval-Audio/audio_evals/models/model.py\", line 75, in inference\n    return super().inference(prompt, **kwargs)\n  File \"/home/zzh/code/ASR/UltraEval-Audio/audio_evals/models/model.py\", line 32, in inference\n    raise ValueError(\"struct input not match pre-train model\")\nValueError: struct input not match pre-train model\n"}}

搜了一下项目中的关键词，发现audio_evals/models/model.py文件中的如下代码会报错

  def inference(self, prompt: PromptStruct, **kwargs) -> str:
        if isinstance(prompt, list) and not self.is_chat:
            raise ValueError("struct input not match pre-train model")
        if isinstance(prompt, str) and self.is_chat:
            prompt = [{"role": "user", "contents": [{"type": "text", "value": prompt}]}]
        sample_params = deepcopy(self.sample_params)
        sample_params.update(kwargs)
        logger.debug(f"sample_params: {sample_params}\nprompt: {prompt}")
        return self._inference(prompt, **sample_params)

发现prompt类型是列表，值如下：

[{'role': 'user', 'contents': [{'type': 'audio', 'value': 'raw/TwinkStart/sample/sample/0.wav'}, {'type': 'text', 'value': 'listen the audio, output the audio content with format {"content": ""}'}]}]

同时self.is_chat = False导致，请问应该怎么修复一下呢

[UltraEval]

CUDA_VISIBLE_DEVICES=0 python audio_evals/main.py --dataset  --model paraformer-zh --prompt simple-asr

paraformer-zh 不是一个语音大模型需要带上指定prompt:--prompt simple-asr

[zzhdbw]

--prompt simple-asr

👌，加上后又报错了：

执行：

CUDA_VISIBLE_DEVICES=1 python audio_evals/main.py \
    --dataset sample \
    --model paraformer-zh \
    --prompt simple-asr

报错如下：

{"type": "error", "id": 9, "data": {"info": "Traceback (most recent call last):\n  File \"/home/zzh/code/ASR/UltraEval-Audio/audio_evals/eval_task.py\", line 75, in _run\n    score, ans = self._eval(\n  File \"/home/zzh/code/ASR/UltraEval-Audio/audio_evals/eval_task.py\", line 52, in _eval\n    output = self.predictor.inference(prompt)\n  File \"/home/zzh/code/ASR/UltraEval-Audio/audio_evals/models/model.py\", line 75, in inference\n    return super().inference(prompt, **kwargs)\n  File \"/home/zzh/code/ASR/UltraEval-Audio/audio_evals/models/model.py\", line 38, in inference\n    return self._inference(prompt, **sample_params)\n  File \"/home/zzh/code/ASR/UltraEval-Audio/audio_evals/models/asr/paraformer.py\", line 51, in _inference\n    raise RuntimeError(\"FireRedASR failed: {}\".format(result))\nRuntimeError: FireRedASR failed: Error:[Errno 2] No such file or directory: 'ffmpeg'\n"}}

{'error': 'division by zero', 'fail_rate(%d)': 100.0}

[UltraEval]

需要安装ffmpeg

sudo apt update
sudo apt install ffmpeg -y

[zzhdbw]

需要安装ffmpeg

sudo apt update
sudo apt install ffmpeg -y

好了好了，感谢🙏