[Cherry-Pick][XPU] fix zmq err catch(#7844)#8015
Conversation
* fix zmq err catch * fix unit * fix unit
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-06-08 11:23:27
📋 Review 摘要
PR 概述：Cherry-pick #7844 到 release/2.6，调整 ZMQ 解码异常日志、control command multipart 解析和 TCP socket 关闭逻辑。
变更范围：fastdeploy/inter_communicator/zmq_server.py
影响面 Tag：[Engine]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🔴 安全 | fastdeploy/inter_communicator/zmq_server.py:116 |
JSON 解码失败时输出原始 ZMQ payload，可能泄露 prompt/token ids 等请求内容 |
| 🔴 安全 | fastdeploy/inter_communicator/zmq_server.py:162 |
pickle 解码失败时输出原始二进制 payload，可能泄露用户请求或多模态输入内容 |
📝 PR 规范检查
标题 Tag 与 diff 主要影响面不匹配，当前变更是通用 ZMQ/Engine 通信修复，不是 XPU 专属路径；PR 描述也保留模板占位内容，建议替换为下面的完整文本。
标题建议（可直接复制）：
[Cherry-Pick][BugFix] Fix ZMQ decode error handling(#7844)
PR 描述建议（点击展开，可直接复制）
## Motivation
Cherry-pick #7844 to release/2.6. Fix ZMQ message decode error handling and reduce malformed control message handling issues.
## Modifications
- Add decode-error handling around `recv_json` and `recv_pyobj` in `fastdeploy/inter_communicator/zmq_server.py`.
- Allow `recv_control_cmd` to tolerate multipart messages with different frame counts and drop messages with insufficient frames.
- Unbind the TCP ZMQ server address before closing the socket.
## Usage or Command
N/A
## Accuracy Tests
N/A
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [x] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.总体评价
解码异常补充日志和 control command 容错方向是合理的，但失败日志不能输出原始 ZMQ payload；请先改为长度/异常类型等脱敏信息后再合入。
| try: | ||
| data_dict = self.socket._deserialize(msg, lambda buf: jsonapi.loads(buf)) | ||
| except (UnicodeDecodeError, ValueError) as e: | ||
| llm_logger.warning(f"recv_json decode failed, msg={msg}, err={e}") |
There was a problem hiding this comment.
这里不要把完整 msg 写入日志。recv_json 接收的是 ZmqClientBase.send_json() 封装后的完整请求字典，调用方在 engine_client.py 里会把 prompt_token_ids、prompt_tokens 等请求内容通过 ZMQ 发过来；一旦 JSON 解码失败，这行会把原始 payload 落盘到 warning 日志，命中 checklist 里的原始 prompt/token ids 日志泄露风险。建议只记录 len(msg)、消息来源/地址和异常类型，或先做字段级脱敏后再输出。
| try: | ||
| envelope = ForkingPickler.loads(data_bytes) | ||
| except (UnicodeDecodeError, ValueError, pickle.UnpicklingError) as e: | ||
| llm_logger.warning(f"recv_pyobj decode failed, msg={data_bytes}, err={e}") |
There was a problem hiding this comment.
这里同样不应记录完整 data_bytes。recv_pyobj 覆盖多模态请求/worker 输出等 pickle 对象路径，bytes 中可能包含 prompt、token ids 或多模态输入内容；解码异常时直接拼接到日志会泄露用户请求数据。建议改成只记录 payload 长度、异常类型和必要的 request/connection 元信息，避免输出原始二进制内容。
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release/2.6 #8015 +/- ##
==============================================
Coverage ? 71.59%
==============================================
Files ? 386
Lines ? 55715
Branches ? 8748
==============================================
Hits ? 39891
Misses ? 13005
Partials ? 2819
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thanks for your contribution! |
5 participants
Cherry-pick of #7844 (authored by @cmcamdy) to
release/2.6.devPR:#7844
Motivation
Modifications
Usage or Command
Accuracy Tests
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.