fix: replace unsafe cloudpickle deserialization with json.loads in rollout worker#1612
Closed
sebastiondev wants to merge 1 commit intoInternLM:mainfrom
Closed
Conversation
…llout worker Replace ray.cloudpickle.loads() deserialization of base64-encoded HTTP response data with json.loads() + np.array() to prevent arbitrary code execution from a compromised inference server. The routed_experts field in inference server responses was being decoded from base64 and then deserialized via ray.cloudpickle.loads(), which can execute arbitrary Python code during deserialization (CWE-502). Since the inference server communicates over HTTP, a compromised or malicious server endpoint could craft a pickle payload to achieve remote code execution on the training worker. The fix decodes the base64 data as JSON (which is safe to parse) and converts to numpy arrays, matching the behavior of the non-string code path already in place.
Author
|
Closing — this PR inadvertently included unrelated file deletions due to a working tree issue during automated submission. The underlying security finding (CWE-502: unsafe cloudpickle deserialization in rollout worker) is valid and will be resubmitted as a clean minimal fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces two instances of
ray.cloudpickle.loads()inxtuner/v1/ray/rollout/worker.pywith safejson.loads()+np.array()deserialization, preventing a potential remote code execution vulnerability.Vulnerability Details
CWE-502: Deserialization of Untrusted Data
In
RolloutWorker._process_response(), when the inference server returnsrouted_expertsas a base64-encoded string in the HTTP JSON response, the code was:ray.cloudpickle.loads()cloudpickle.loads()can execute arbitrary Python code during deserialization. Since the inference server communicates over HTTP (it runs as a separate process/Ray task listening on a network port), a compromised or malicious inference endpoint could craft a base64-encoded pickle payload in therouted_expertsfield that would execute arbitrary code on the training worker when deserialized.Affected lines: L575 and L586 (original)
Data Flow
Fix
Replace
ray.cloudpickle.loads()withjson.loads()+np.array(), which:torch.tensor()/np.array()Path 1 (no history, L575):
Path 2 (with history, L586):
Additional Finding (not included in this PR)
The
.github/workflows/claude-general.ymlworkflow has several security issues including direct${{ github.event.comment.body }}interpolation in prompt fields (expression injection risk),allowed_non_write_users: "*"allowing any user to trigger Claude with write permissions, and PAT tokens exposed in prompt context. These require theworkflowscope to fix and are noted here for the maintainers' awareness:${{ github.event.comment.body }}is interpolated directly into thepromptYAML field — should use environment variables insteadallowed_non_write_users: "*"allows any GitHub user to trigger the workflow — should be restricted to repo collaborators${{ steps.pr.outputs.token }}is interpolated directly in the prompt text visible to Claude — should be passed via environment variable onlyTesting
jsonandnumpywhich are already imported in the moduleray.put()) matches the existing non-string code path