fix: make default Slurm CPU headers portable#596
Conversation
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughThe PR removes the login-shell invocation ( ChangesSlurm Header and GPU Directive Refactoring
🎯 2 (Simple) | ⏱️ ~10 minutes Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
dpdispatcher/machines/slurm.py (1)
38-38: 🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick winAdd return type hint to
gen_script_header.The method should include a return type annotation
-> strto comply with the project's type-hint requirements. As per coding guidelines, "Always add type hints - Include proper type annotations in all Python code for better maintainability" for files matchingdpdispatcher/**/*.py.📝 Proposed fix
- def gen_script_header(self, job): + def gen_script_header(self, job) -> str:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@dpdispatcher/machines/slurm.py` at line 38, Add a return type annotation to the method signature of gen_script_header so it declares it returns a string (i.e., change def gen_script_header(self, job) to include -> str). Locate the gen_script_header method in the Slurm-related class (method name: gen_script_header) and update its signature to include the return type; no other behavioral changes are required. Ensure the change compiles with the existing codebase and update any stubs/tests if they assert signature compatibility.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_slurm_script_generation.py`:
- Around line 25-37: Add type annotations to the test helper _make_header:
change its signature to accept resource_updates: Optional[Dict[str, Any]] = None
and remove_resource_keys: Optional[List[str]] = None and declare the return type
as str (the header returned by Machine.gen_script_header). Also ensure the
necessary typing imports (Optional, Dict, Any, List) are present at the top of
the test file; leave the function body unchanged and keep using Machine,
Resources and SimpleNamespace as before.
---
Outside diff comments:
In `@dpdispatcher/machines/slurm.py`:
- Line 38: Add a return type annotation to the method signature of
gen_script_header so it declares it returns a string (i.e., change def
gen_script_header(self, job) to include -> str). Locate the gen_script_header
method in the Slurm-related class (method name: gen_script_header) and update
its signature to include the return type; no other behavioral changes are
required. Ensure the change compiles with the existing codebase and update any
stubs/tests if they assert signature compatibility.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 7df7e554-64d2-465d-9941-587b3a7b0c45
📒 Files selected for processing (2)
dpdispatcher/machines/slurm.pytests/test_slurm_script_generation.py
d2e9d18 to
ddc4482
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #596 +/- ##
==========================================
+ Coverage 48.33% 48.40% +0.07%
==========================================
Files 40 40
Lines 3958 3960 +2
==========================================
+ Hits 1913 1917 +4
+ Misses 2045 2043 -2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR improves the portability of generated Slurm job scripts by removing login-shell usage in the default Slurm header and ensuring GPU directives are only emitted when explicitly requested or configured, which helps CPU-only jobs run on stricter clusters.
Changes:
- Removed
-lfrom the Slurm shebang and dropped#SBATCH --parsablefrom the default header template. - Updated Slurm GPU header generation to omit GPU requests when
gpu_per_node == 0, unless an explicitcustom_gpu_lineis provided. - Refactored Slurm script-header tests to focus on
gen_script_headerdirectly and added CPU/GPU-specific assertions.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
dpdispatcher/machines/slurm.py |
Makes the default Slurm header more portable and fixes GPU directive emission logic for CPU jobs. |
tests/test_slurm_script_generation.py |
Simplifies header tests and adds coverage for CPU headers omitting GPU directives and for default GPU --gres behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
njzjz-bot
left a comment
There was a problem hiding this comment.
Thanks for the portability fix. The code direction and the added Slurm header tests look reasonable, and CI is green, but I think this PR still needs documentation/example updates before merging.
Blocking documentation gaps:
doc/context.mdstill says DPDispatcher submission scripts usebash -land therefore execute login-shell startup files. This PR changes the default Slurm header to#!/bin/bash, so that statement is no longer universally true for Slurm users and directly contradicts the new behavior.doc/examples/expanse.mdstill says Expanse needscustom_gpu_linebecause the default would emit--gres=gpu:0. With this PR, CPU Slurm jobs now omit the GPU directive by default, so the example text should be updated.examples/resources/expanse_cpu.jsonstill setskwargs.custom_gpu_lineto#SBATCH --gpus=0. If the purpose of this PR is to make CPU Slurm headers portable by default, this example should demonstrate the new default behavior by removing that override, unless Expanse specifically requires--gpus=0—in which case the docs should explain that distinction.examples/resources/template.slurmstill includes#!/bin/bash -land#SBATCH --parsable. Since this is the documented custom-header template example, please either update it to match the new recommended default or explicitly state that custom templates are user-controlled and may still opt into login shells / embedded--parsable.
The tests cover the Python behavior well enough for this small change (tests/test_slurm_script_generation.py passes locally; selected CLI/submission tests also pass in an editable venv), but the user-facing docs are now stale.
— OpenClaw 2026.4.22 (model: gpt-5.5)
|
This PR introduces an inconsistency among different types of Machines. Shell, Slurm, PBS, LSF, and Bohrium all used |
252b175 to
593e67f
Compare
Avoid invalid Slurm directives for CPU jobs while preserving the login-shell shebang used by other machine backends.
I've restored the line involving |
njzjz-bot
left a comment
There was a problem hiding this comment.
Thanks for the updates. The documentation/examples now match the final scope of this PR:
#SBATCH --parsableis removed from the default Slurm header and custom template example, whilesbatch --parsableis still used at submission time.- CPU Slurm jobs now omit the default GPU directive when
gpu_per_node == 0. - The Expanse example no longer relies on
custom_gpu_linefor the default CPU case, and documents when an explicit site-specific directive such as--gpus=0is still appropriate. - Keeping
#!/bin/bash -lin this PR is acceptable because it avoids changing login-shell behavior for only Slurm while the other Machine backends still usebash -l; handling login-shell consistency across all backends should be a separate PR.
Local validation:
git diff --check origin/master...HEADpytest tests/test_slurm_script_generation.py tests/test_argcheck.py tests/test_submit.py tests/test_run_submission.py -q→ 20 passed, 14 skippedruff check dpdispatcher/machines/slurm.py tests/test_slurm_script_generation.py
— OpenClaw 2026.4.22 (model: gpt-5.5)
Avoid login-shell and GPU directives for CPU Slurm jobs so generated scripts work on clusters with stricter compute-node environments.
Summary by CodeRabbit
Bug Fixes
Documentation
Tests