Using pin_memory=true in the WSL2 environment will cause an error
#1032
liuwake
started this conversation in
Show and tell
Replies: 2 comments
-
|
Full error log see: Click to expand/collapseTraining command is /opt/miniconda3/envs/mm/bin/python /home/bc/code/mmyolo/mmyolo/.mim/tools/train.py configs/ppyoloe/ppyoloe_plus_s_fast_8xb8-80e_coco.py --launcher none.
09/26 10:12:09 - mmengine - WARNING - Failed to search registry with scope "mmyolo" in the "log_processor" registry tree. As a workaround, the current "log_processor" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmyolo" is a correct scope, or whether the registry is initialized.
09/26 10:12:09 - mmengine - INFO -
------------------------------------------------------------
System environment:
sys.platform: linux
Python: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 1010925727
GPU 0: NVIDIA GeForce RTX 4090
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.8, V11.8.89
GCC: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.0.1+cu118
PyTorch compiling details: PyTorch built with:
- GCC 9.3
- C++ Version: 201703
- Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX2
- CUDA Runtime 11.8
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
- CuDNN 8.7
- Magma 2.6.1
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,
TorchVision: 0.15.2+cu118
OpenCV: 4.10.0
MMEngine: 0.10.5
Runtime environment:
cudnn_benchmark: False
mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
dist_cfg: {'backend': 'nccl'}
seed: 1010925727
Distributed launcher: none
Distributed training: False
GPU number: 1
------------------------------------------------------------
09/26 10:12:10 - mmengine - INFO - Config:
_backend_args = None
_multiscale_resize_transforms = [
dict(
transforms=[
dict(scale=(
640,
640,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
640,
640,
),
type='LetterResize'),
],
type='Compose'),
dict(
transforms=[
dict(scale=(
320,
320,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
320,
320,
),
type='LetterResize'),
],
type='Compose'),
dict(
transforms=[
dict(scale=(
960,
960,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
960,
960,
),
type='LetterResize'),
],
type='Compose'),
]
backend_args = None
base_lr = 0.001
custom_hooks = [
dict(
ema_type='ExpMomentumEMA',
momentum=0.0002,
priority=49,
strict_load=False,
type='EMAHook',
update_buffers=True),
]
data_root = 'data/coco/'
dataset_type = 'YOLOv5CocoDataset'
deepen_factor = 0.33
default_hooks = dict(
checkpoint=dict(
interval=5, max_keep_ckpts=3, save_best='auto', type='CheckpointHook'),
logger=dict(interval=50, type='LoggerHook'),
param_scheduler=dict(
min_lr_ratio=0.0,
start_factor=0.0,
total_epochs=96,
type='PPYOLOEParamSchedulerHook',
warmup_epochs=5,
warmup_min_iter=1000),
sampler_seed=dict(type='DistSamplerSeedHook'),
timer=dict(type='IterTimerHook'),
visualization=dict(type='mmdet.DetVisualizationHook'))
default_scope = 'mmyolo'
env_cfg = dict(
cudnn_benchmark=False,
dist_cfg=dict(backend='nccl'),
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
img_scale = (
640,
640,
)
img_scales = [
(
640,
640,
),
(
320,
320,
),
(
960,
960,
),
]
launcher = 'none'
load_from = 'https://download.openmmlab.com/mmyolo/v0/ppyoloe/ppyoloe_pretrain/ppyoloe_plus_s_obj365_pretrained-bcfe8478.pth'
log_level = 'INFO'
log_processor = dict(by_epoch=True, type='LogProcessor', window_size=50)
max_epochs = 80
model = dict(
backbone=dict(
act_cfg=dict(inplace=True, type='SiLU'),
attention_cfg=dict(
act_cfg=dict(type='HSigmoid'), type='EffectiveSELayer'),
block_cfg=dict(
shortcut=True, type='PPYOLOEBasicBlock', use_alpha=True),
deepen_factor=0.33,
norm_cfg=dict(eps=1e-05, momentum=0.1, type='BN'),
type='PPYOLOECSPResNet',
use_large_stem=True,
widen_factor=0.5),
bbox_head=dict(
bbox_coder=dict(type='DistancePointBBoxCoder'),
head_module=dict(
act_cfg=dict(inplace=True, type='SiLU'),
featmap_strides=[
8,
16,
32,
],
in_channels=[
192,
384,
768,
],
norm_cfg=dict(eps=1e-05, momentum=0.1, type='BN'),
num_base_priors=1,
num_classes=80,
reg_max=16,
type='PPYOLOEHeadModule',
widen_factor=0.5),
loss_bbox=dict(
bbox_format='xyxy',
iou_mode='giou',
loss_weight=2.5,
reduction='mean',
return_iou=False,
type='IoULoss'),
loss_cls=dict(
alpha=0.75,
gamma=2.0,
iou_weighted=True,
loss_weight=1.0,
reduction='sum',
type='mmdet.VarifocalLoss',
use_sigmoid=True),
loss_dfl=dict(
loss_weight=0.125,
reduction='mean',
type='mmdet.DistributionFocalLoss'),
prior_generator=dict(
offset=0.5, strides=[
8,
16,
32,
], type='mmdet.MlvlPointGenerator'),
type='PPYOLOEHead'),
data_preprocessor=dict(
batch_augments=[
dict(
interval=1,
keep_ratio=False,
random_interp=True,
random_size_range=(
320,
800,
),
size_divisor=32,
type='PPYOLOEBatchRandomResize'),
],
bgr_to_rgb=True,
mean=[
0.0,
0.0,
0.0,
],
pad_size_divisor=32,
std=[
255.0,
255.0,
255.0,
],
type='PPYOLOEDetDataPreprocessor'),
neck=dict(
act_cfg=dict(inplace=True, type='SiLU'),
block_cfg=dict(
shortcut=False, type='PPYOLOEBasicBlock', use_alpha=False),
deepen_factor=0.33,
drop_block_cfg=None,
in_channels=[
256,
512,
1024,
],
norm_cfg=dict(eps=1e-05, momentum=0.1, type='BN'),
num_blocks_per_layer=3,
num_csplayer=1,
out_channels=[
192,
384,
768,
],
type='PPYOLOECSPPAFPN',
use_spp=True,
widen_factor=0.5),
test_cfg=dict(
max_per_img=300,
multi_label=True,
nms=dict(iou_threshold=0.7, type='nms'),
nms_pre=1000,
score_thr=0.01),
train_cfg=dict(
assigner=dict(
alpha=1,
beta=6,
eps=1e-09,
num_classes=80,
topk=13,
type='BatchTaskAlignedAssigner'),
initial_assigner=dict(
iou_calculator=dict(type='mmdet.BboxOverlaps2D'),
num_classes=80,
topk=9,
type='BatchATSSAssigner'),
initial_epoch=30),
type='YOLODetector')
num_classes = 80
optim_wrapper = dict(
optimizer=dict(
lr=0.001,
momentum=0.9,
nesterov=False,
type='SGD',
weight_decay=0.0005),
paramwise_cfg=dict(norm_decay_mult=0.0),
type='OptimWrapper')
param_scheduler = None
persistent_workers = True
resume = False
save_epoch_intervals = 5
strides = [
8,
16,
32,
]
test_cfg = dict(type='TestLoop')
test_dataloader = dict(
batch_size=1,
dataset=dict(
ann_file='annotations/instances_val2017.json',
data_prefix=dict(img='val2017/'),
data_root='data/coco/',
filter_cfg=dict(filter_empty_gt=True, min_size=0),
pipeline=[
dict(backend_args=None, type='LoadImageFromFile'),
dict(
height=640,
interpolation='bicubic',
keep_ratio=False,
type='mmdet.FixShapeResize',
width=640),
dict(_scope_='mmdet', type='LoadAnnotations', with_bbox=True),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
),
type='mmdet.PackDetInputs'),
],
test_mode=True,
type='YOLOv5CocoDataset'),
drop_last=False,
num_workers=2,
persistent_workers=True,
pin_memory=True,
sampler=dict(shuffle=False, type='DefaultSampler'))
test_evaluator = dict(
ann_file='data/coco/annotations/instances_val2017.json',
metric='bbox',
proposal_nums=(
100,
1,
10,
),
type='mmdet.CocoMetric')
test_pipeline = [
dict(backend_args=None, type='LoadImageFromFile'),
dict(
height=640,
interpolation='bicubic',
keep_ratio=False,
type='mmdet.FixShapeResize',
width=640),
dict(_scope_='mmdet', type='LoadAnnotations', with_bbox=True),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
),
type='mmdet.PackDetInputs'),
]
train_batch_size_per_gpu = 8
train_cfg = dict(max_epochs=80, type='EpochBasedTrainLoop', val_interval=5)
train_dataloader = dict(
batch_size=8,
collate_fn=dict(type='yolov5_collate', use_ms_training=True),
dataset=dict(
ann_file='annotations/instances_train2017.json',
data_prefix=dict(img='train2017/'),
data_root='data/coco/',
filter_cfg=dict(filter_empty_gt=True, min_size=0),
pipeline=[
dict(backend_args=None, type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='PPYOLOERandomDistort'),
dict(mean=(
103.53,
116.28,
123.675,
), type='mmdet.Expand'),
dict(type='PPYOLOERandomCrop'),
dict(prob=0.5, type='mmdet.RandomFlip'),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'flip',
'flip_direction',
),
type='mmdet.PackDetInputs'),
],
type='YOLOv5CocoDataset'),
num_workers=8,
persistent_workers=True,
pin_memory=True,
sampler=dict(shuffle=True, type='DefaultSampler'))
train_num_workers = 8
train_pipeline = [
dict(backend_args=None, type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(type='PPYOLOERandomDistort'),
dict(mean=(
103.53,
116.28,
123.675,
), type='mmdet.Expand'),
dict(type='PPYOLOERandomCrop'),
dict(prob=0.5, type='mmdet.RandomFlip'),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'flip',
'flip_direction',
),
type='mmdet.PackDetInputs'),
]
tta_model = dict(
tta_cfg=dict(max_per_img=300, nms=dict(iou_threshold=0.65, type='nms')),
type='mmdet.DetTTAModel')
tta_pipeline = [
dict(backend_args=None, type='LoadImageFromFile'),
dict(
transforms=[
[
dict(
transforms=[
dict(scale=(
640,
640,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
640,
640,
),
type='LetterResize'),
],
type='Compose'),
dict(
transforms=[
dict(scale=(
320,
320,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
320,
320,
),
type='LetterResize'),
],
type='Compose'),
dict(
transforms=[
dict(scale=(
960,
960,
), type='YOLOv5KeepRatioResize'),
dict(
allow_scale_up=False,
pad_val=dict(img=114),
scale=(
960,
960,
),
type='LetterResize'),
],
type='Compose'),
],
[
dict(prob=1.0, type='mmdet.RandomFlip'),
dict(prob=0.0, type='mmdet.RandomFlip'),
],
[
dict(type='mmdet.LoadAnnotations', with_bbox=True),
],
[
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
'pad_param',
'flip',
'flip_direction',
),
type='mmdet.PackDetInputs'),
],
],
type='TestTimeAug'),
]
val_batch_size_per_gpu = 1
val_cfg = dict(type='ValLoop')
val_dataloader = dict(
batch_size=1,
dataset=dict(
ann_file='annotations/instances_val2017.json',
data_prefix=dict(img='val2017/'),
data_root='data/coco/',
filter_cfg=dict(filter_empty_gt=True, min_size=0),
pipeline=[
dict(backend_args=None, type='LoadImageFromFile'),
dict(
height=640,
interpolation='bicubic',
keep_ratio=False,
type='mmdet.FixShapeResize',
width=640),
dict(_scope_='mmdet', type='LoadAnnotations', with_bbox=True),
dict(
meta_keys=(
'img_id',
'img_path',
'ori_shape',
'img_shape',
'scale_factor',
),
type='mmdet.PackDetInputs'),
],
test_mode=True,
type='YOLOv5CocoDataset'),
drop_last=False,
num_workers=2,
persistent_workers=True,
pin_memory=True,
sampler=dict(shuffle=False, type='DefaultSampler'))
val_evaluator = dict(
ann_file='data/coco/annotations/instances_val2017.json',
metric='bbox',
proposal_nums=(
100,
1,
10,
),
type='mmdet.CocoMetric')
val_num_workers = 2
vis_backends = [
dict(type='LocalVisBackend'),
]
visualizer = dict(
name='visualizer',
type='mmdet.DetLocalVisualizer',
vis_backends=[
dict(type='LocalVisBackend'),
])
widen_factor = 0.5
work_dir = './work_dirs/ppyoloe_plus_s_fast_8xb8-80e_coco'
/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mmcv/cnn/bricks/hsigmoid.py:35: UserWarning: In MMCV v1.4.4, we modified the default value of args to align with PyTorch official. Previous Implementation: Hsigmoid(x) = min(max((x + 1) / 2, 0), 1). Current Implementation: Hsigmoid(x) = min(max((x + 3) / 6, 0), 1).
warnings.warn(
09/26 10:12:10 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.
09/26 10:12:10 - mmengine - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(BELOW_NORMAL) LoggerHook
--------------------
after_load_checkpoint:
(49 ) EMAHook
--------------------
before_train:
(9 ) PPYOLOEParamSchedulerHook
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(NORMAL ) IterTimerHook
(VERY_LOW ) CheckpointHook
--------------------
before_train_epoch:
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) DistSamplerSeedHook
--------------------
before_train_iter:
(9 ) PPYOLOEParamSchedulerHook
(VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
--------------------
after_train_iter:
(9 ) PPYOLOEParamSchedulerHook
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(VERY_LOW ) CheckpointHook
--------------------
after_train_epoch:
(9 ) PPYOLOEParamSchedulerHook
(NORMAL ) IterTimerHook
(VERY_LOW ) CheckpointHook
--------------------
before_val:
(VERY_HIGH ) RuntimeInfoHook
--------------------
before_val_epoch:
(49 ) EMAHook
(NORMAL ) IterTimerHook
--------------------
before_val_iter:
(NORMAL ) IterTimerHook
--------------------
after_val_iter:
(NORMAL ) IterTimerHook
(NORMAL ) DetVisualizationHook
(BELOW_NORMAL) LoggerHook
--------------------
after_val_epoch:
(9 ) PPYOLOEParamSchedulerHook
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(VERY_LOW ) CheckpointHook
--------------------
after_val:
(VERY_HIGH ) RuntimeInfoHook
--------------------
before_save_checkpoint:
(49 ) EMAHook
--------------------
after_train:
(VERY_HIGH ) RuntimeInfoHook
(VERY_LOW ) CheckpointHook
--------------------
before_test:
(VERY_HIGH ) RuntimeInfoHook
--------------------
before_test_epoch:
(49 ) EMAHook
(NORMAL ) IterTimerHook
--------------------
before_test_iter:
(NORMAL ) IterTimerHook
--------------------
after_test_iter:
(NORMAL ) IterTimerHook
(NORMAL ) DetVisualizationHook
(BELOW_NORMAL) LoggerHook
--------------------
after_test_epoch:
(VERY_HIGH ) RuntimeInfoHook
(49 ) EMAHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
--------------------
after_test:
(VERY_HIGH ) RuntimeInfoHook
--------------------
after_run:
(BELOW_NORMAL) LoggerHook
--------------------
loading annotations into memory...
Done (t=6.26s)
creating index...
index created!
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.0.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.0.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stem.2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv_down.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv_down.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage1.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv_down.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv_down.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.blocks.1.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage2.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv_down.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv_down.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.blocks.1.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage3.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv_down.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv_down.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- backbone.stage4.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.spp.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.blocks.spp.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.reduce_layers.2.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.upsample_layers.0.0.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.upsample_layers.0.0.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.upsample_layers.1.0.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.upsample_layers.1.0.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.0.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.top_down_layers.1.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.downsample_layers.0.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.downsample_layers.0.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.downsample_layers.1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.downsample_layers.1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.0.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv2.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv2.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv2.rbr_dense.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv2.rbr_dense.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv2.rbr_1x1.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.blocks.0.conv2.rbr_1x1.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv3.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- neck.bottom_up_layers.1.0.conv3.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.0.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.0.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.1.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.1.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.2.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.cls_stems.2.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.0.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.0.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.1.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.1.conv.bn.bias:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.2.conv.bn.weight:weight_decay=0.0
09/26 10:12:22 - mmengine - INFO - paramwise_options -- bbox_head.head_module.reg_stems.2.conv.bn.bias:weight_decay=0.0
loading annotations into memory...
Done (t=0.26s)
creating index...
index created!
loading annotations into memory...
Done (t=0.26s)
creating index...
index created!
Loads checkpoint by http backend from path: https://download.openmmlab.com/mmyolo/v0/ppyoloe/ppyoloe_pretrain/ppyoloe_plus_s_obj365_pretrained-bcfe8478.pth
The model and loaded state dict do not match exactly
size mismatch for bbox_head.head_module.cls_preds.0.weight: copying a param with shape torch.Size([365, 96, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 96, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.0.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.head_module.cls_preds.1.weight: copying a param with shape torch.Size([365, 192, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 192, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.1.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.head_module.cls_preds.2.weight: copying a param with shape torch.Size([365, 384, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 384, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.2.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
missing keys in source state_dict: backbone.stage1.0.blocks.0.conv2.alpha, backbone.stage2.0.blocks.0.conv2.alpha, backbone.stage2.0.blocks.1.conv2.alpha, backbone.stage3.0.blocks.0.conv2.alpha, backbone.stage3.0.blocks.1.conv2.alpha, backbone.stage4.0.blocks.0.conv2.alpha
The model and loaded state dict do not match exactly
size mismatch for bbox_head.head_module.cls_preds.0.weight: copying a param with shape torch.Size([365, 96, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 96, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.0.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.head_module.cls_preds.1.weight: copying a param with shape torch.Size([365, 192, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 192, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.1.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
size mismatch for bbox_head.head_module.cls_preds.2.weight: copying a param with shape torch.Size([365, 384, 3, 3]) from checkpoint, the shape in current model is torch.Size([80, 384, 3, 3]).
size mismatch for bbox_head.head_module.cls_preds.2.bias: copying a param with shape torch.Size([365]) from checkpoint, the shape in current model is torch.Size([80]).
missing keys in source state_dict: backbone.stage1.0.blocks.0.conv2.alpha, backbone.stage2.0.blocks.0.conv2.alpha, backbone.stage2.0.blocks.1.conv2.alpha, backbone.stage3.0.blocks.0.conv2.alpha, backbone.stage3.0.blocks.1.conv2.alpha, backbone.stage4.0.blocks.0.conv2.alpha
09/26 10:12:23 - mmengine - INFO - Load checkpoint from https://download.openmmlab.com/mmyolo/v0/ppyoloe/ppyoloe_pretrain/ppyoloe_plus_s_obj365_pretrained-bcfe8478.pth
09/26 10:12:23 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
09/26 10:12:23 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
09/26 10:12:23 - mmengine - INFO - Checkpoints will be saved to /home/bc/code/mmyolo/work_dirs/ppyoloe_plus_s_fast_8xb8-80e_coco.
/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Traceback (most recent call last):
File "/home/bc/code/mmyolo/mmyolo/.mim/tools/train.py", line 123, in <module>
main()
File "/home/bc/code/mmyolo/mmyolo/.mim/tools/train.py", line 119, in main
runner.train()
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1777, in train
model = self.train_loop.run() # type: ignore
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mmengine/runner/loops.py", line 98, in run
self.run_epoch()
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mmengine/runner/loops.py", line 114, in run_epoch
for idx, data_batch in enumerate(self.dataloader):
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
data = self._next_data()
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
return self._process_data(data)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
data.reraise()
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/_utils.py", line 644, in reraise
raise exception
RuntimeError: Caught RuntimeError in pin memory thread for device 0.
Original Traceback (most recent call last):
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 34, in do_one_step
data = pin_memory(data, device)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 60, in pin_memory
return type(data)({k: pin_memory(sample, device) for k, sample in data.items()}) # type: ignore[call-arg]
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 60, in <dictcomp>
return type(data)({k: pin_memory(sample, device) for k, sample in data.items()}) # type: ignore[call-arg]
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 70, in pin_memory
return type(data)([pin_memory(sample, device) for sample in data]) # type: ignore[call-arg]
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 70, in <listcomp>
return type(data)([pin_memory(sample, device) for sample in data]) # type: ignore[call-arg]
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/torch/utils/data/_utils/pin_memory.py", line 55, in pin_memory
return data.pin_memory(device)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Traceback (most recent call last):
File "/opt/miniconda3/envs/mm/bin/mim", line 8, in <module>
sys.exit(cli())
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mim/commands/train.py", line 100, in cli
is_success, msg = train(
File "/opt/miniconda3/envs/mm/lib/python3.10/site-packages/mim/commands/train.py", line 261, in train
ret = subprocess.check_call(
File "/opt/miniconda3/envs/mm/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/miniconda3/envs/mm/bin/python', '/home/bc/code/mmyolo/mmyolo/.mim/tools/train.py', 'configs/ppyoloe/ppyoloe_plus_s_fast_8xb8-80e_coco.py', '--launcher', 'none']' returned non-zero exit status 1. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
However I have not modified the pin_memery in val_dataloader, it remained |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
mmyoloor so.pin_memory=truein theWSLenviroment will cause an error. So putting it toFalsewill fix it.Pytorchandpin_memory, see got the error: out of memory ,when invoke cuda in wsl2. microsoft/WSL#8447 (comment)This is the process I followed to discover and solve the problem
mim train mmyolo configs/ppyoloe/ppyoloe_plus_s_fast_8xb8-80e_coco.py --cfg-options train_dataloader.batch_size=16 train_dataloader.num_workers=8 optim_wrapper.optimizer.lr=0.00025Click to expand/collapse
pin memory.Torch,mmlab,mmyolowithpin memoryorTORCH_USE_CUDA_DSA.pin_memory=truein theWSL2environment will cause an error.mim train mmyolo configs/ppyoloe/ppyoloe_plus_s_fast_8xb8-80e_coco.py --cfg-options train_dataloader.batch_size=16 train_dataloader.num_workers=8 optim_wrapper.optimizer.lr=0.00025 train_dataloader.pin_memory=FalseBeta Was this translation helpful? Give feedback.
All reactions