Skip to content

fix pdc resume bug - #4899

Open
lugimzzz wants to merge 3 commits into
PaddlePaddle:developfrom
lugimzzz:resume
Open

fix pdc resume bug#4899
lugimzzz wants to merge 3 commits into
PaddlePaddle:developfrom
lugimzzz:resume

Conversation

@lugimzzz

@lugimzzz lugimzzz commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

PR types

Others

PR changes

Others

Description

当前容错重启的时候需要手动修改yaml
开始训练需要设置

load_from_hf: True
ignore_load_lr_and_optim: True

开启容错续训的时候又需要:

load_from_hf: False
ignore_load_lr_and_optim: False

为了降低使用成本做此修改。ignore_load_lr_and_optim 这个开关用户根据实际续训需要设置为True或False,load_from_hf在续训的时候会自动设为False避免报错

@lugimzzz lugimzzz changed the title fix pdc resume bug 【WIP】fix pdc resume bug Aug 20, 2026
Comment thread paddleformers/trainer/argparser.py Outdated
logger.info(
f"init_step == 0 and user has defined resume_from_checkpoint: {user_defined_resume_from_checkpoint}"
)
if args.get("load_from_hf", False) and not args.get("ignore_load_lr_and_optim", False):

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

当前都是通过user_defined_resume_from_checkpoint来初始化权重的, load_from_hf 如果为True的话,ignore_load_lr_and_optim为需要设为True

@lugimzzz lugimzzz changed the title 【WIP】fix pdc resume bug fix pdc resume bug Aug 20, 2026
@Paddle-CI-Bot

Paddle-CI-Bot commented Aug 20, 2026

Copy link
Copy Markdown

PaddleFormers Log Analysis

Run #32448999558 · Attempt 1

日志分析报告

流水线名称 问题标签 修复建议 日志片段
CI_XPU / Clone-Linux / code-clone Runner 组缺失 CI 维护人员检查 GitHub Actions 中 HK-Clone runner group 是否存在且可用,恢复后 rerun 报错代码

失败的测试 case:

Clone-Linux / code-clone — Required runner group 'HK-Clone' not found
xpu_test — skipped(因 clone 步骤失败,未执行)

根本原因分析:
_clone_linux.ymlcode-clone job 指定了 runs-on: group: HK-Clone,但该 runner group 在当前 GitHub Actions 环境中不存在或不可用,导致 job 在排队阶段直接失败,下游 xpu_test 被跳过。

修复建议:

  1. CI 维护人员登录 GitHub 组织 Settings → Actions → Runner groups,确认 HK-Clone group 是否存在;若已删除或改名,需恢复该 group 或更新 _clone_linux.yml 中的 group 名称。
  2. 确认 HK-Clone group 中有在线的 self-hosted runner(状态为 Active/Idle)。
  3. 修复后对该 PR rerun 整个 CI_XPU workflow。

🔍 准确性记录:请点击评论底部 😊 图标,选择 👍(准确)或 👎(有误),将自动记录到 CI 监控系统

🔄 每次 Re-run 后自动更新

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants