MAIA(v2.0) Failed to download model weights problem in training #542

cywhale · 2023-02-07T09:25:08Z

cywhale
Feb 7, 2023

I had upgraded to MAIA 2.0.0 and novelty detection works well. But after that, when training, the maia job soon failed because

production.ERROR: Failed to download model weights from 'https://download.pytorch.org/models/resnet50-11ad3fa6.pth'. {"exception":"[object] (Exception(code: 0): Failed to download model weights from 'https://download.pytorch.org/models/resnet50-11ad3fa6.pth'. at /var/www/vendor/biigle/maia/src/Jobs/ObjectDetectionRequest.php:212)

Check the php code, it seems will check the file exist:

File::exists($to)

This 'resnet50-11ad3fa6.pth' acutally can be downloaded, then I downloaded and put it in storage/maia_jobs

docker exec c518fba8edca ls /var/www/storage/maia_jobs
#maia-33-novelty-detection
#maia-34-object-detection
#maia-35-object-detection
#maia-36-object-detection
#maia-37-object-detection
#mask_rcnn_coco.h5
#resnet50-11ad3fa6.pth

But after docker compose restart and re-run this job, it still failed for the same error cause.
I think I may miss some .env setup(? MAIA_MODEL_PATH or MAIA_MODEL_URL?) or other setup configurations I should try? Thanks.

My .env (about MAIA, as the same as .env.example):

MAIA_ANNOTATION_CANDIDATE_STORAGE_DISK=maia-ac
MAIA_TRAINING_PROPOSAL_STORAGE_DISK=maia-tp
VIDEOS_THUMBNAIL_STORAGE_DISK=video-thumbs

MAIA_REQUEST_QUEUE=gpu
MAIA_REQUEST_CONNECTION=gpu
MAIA_RESPONSE_QUEUE=default
MAIA_RESPONSE_CONNECTION=redis
MAIA_MAX_WORKERS=10
MAIA_PYTHON="/opt/conda/bin/python3"

mzur · 2023-02-07T12:09:00Z

mzur
Feb 7, 2023
Maintainer

Please try to run this:

docker exec c518fba8edca php -r "copy('https://download.pytorch.org/models/resnet50-11ad3fa6.pth', '/var/www/storage/maia_jobs/resnet-test.pth');"

Does it produce any warnings/errors? Maybe the path is not writable?

I think I may miss some .env setup(? MAIA_MODEL_PATH or MAIA_MODEL_URL?) or other setup configurations I should try?

No, this part should work out of the box. These two variables (MAIA_BACKBONE_MODEL_URL and MAIA_MODEL_URL) are only required if you want to host the model files yourself.

6 replies

mzur Feb 8, 2023
Maintainer

How strange... Yes, I deliberatly chose a different file name because I thought copy() fails if the file already exists. But in fact it will just overwrite the file. So we have two issues here, 1) File::exists() returns false even though the file exists and 2) copy() returns false as an error even though it worked in your test.

Maybe there is an issue with the file permissions of resnet50-11ad3fa6.pth? Can you run docker exec c518fba8edca rm -r /var/www/storage/maia_jobs/* without errors?

cywhale Feb 8, 2023
Author

Maybe there is an issue with the file permissions of resnet50-11ad3fa6.pth? Can you run docker exec c518fba8edca rm -r /var/www/storage/maia_jobs/* without errors?

I cannot run this for /* (" No such file or directory") but I can run
"docker exec c518fba8edca rm -r /var/www/storage/maia_jobs/resnet-test.pth" without errors

$ docker exec c518fba8edca rm -r /var/www/storage/maia_jobs/resnet-test.pth
$ docker exec c518fba8edca ls /var/www/storage/maia_jobs
maia-33-novelty-detection
maia-34-object-detection
maia-35-object-detection
maia-36-object-detection
maia-37-object-detection
maia-38-object-detection
mask_rcnn_coco.h5
resnet50-11ad3fa6.pth

mzur Feb 8, 2023
Maintainer

Then it looks like a permission issue. What is the output of docker exec c518fba8edca ls -la /var/www/storage/maia_jobs and ls -la storage/maia_jobs (outside the Docker container)?

cywhale Feb 8, 2023
Author

ls -la storage/maia_jobs got permission denied

$ docker exec c518fba8edca ls -la /var/www/storage/maia_jobs
total 351696
drwx------    8 1002     1002          4096 Feb  8 16:10 .
drwxr-xr-x   10 1002     1002          4096 Sep 21 11:17 ..
drwx------    2 1002     1002          4096 Feb  7 14:43 maia-33-novelty-detection
drwx------    3 1002     1002          4096 Feb  7 15:10 maia-34-object-detection
drwx------    3 1002     1002          4096 Feb  7 15:49 maia-35-object-detection
drwx------    3 1002     1002          4096 Feb  7 16:23 maia-36-object-detection
drwx------    3 1002     1002          4096 Feb  7 17:08 maia-37-object-detection
drwx------    3 1002     1002          4096 Feb  8 09:30 maia-38-object-detection
-rw-r--r--    1 1002     1002     257557808 Nov 11 10:58 mask_rcnn_coco.h5
-rw-r--r--    1 1002     1002     102540417 Jan 17  2022 resnet50-11ad3fa6.pth
$ ls -la storage/maia_jobs
ls: cannot open directory 'storage/maia_jobs': Permission denied

Then I tried to 'sudo chmod -R 755 ./maia_jobs' to make it with the same privileges as other subdirectories:

$ sudo chmod -R 755 ./maia_jobs/
$ ls -al
total 40
drwxr-xr-x 10 biigle biigle 4096  9月 21 11:17 .
drwxr-xr-x  8 root   root   4096  2月  7 14:34 ..
drwxr-xr-x  3 biigle biigle 4096  8月 14  2020 app
drwxr-xr-x  6 biigle biigle 4096  2月  7 14:36 framework
drwx------  2 biigle biigle 4096  9月 21 11:17 ifdos
drwxr-xr-x  2 biigle biigle 4096  8月 14  2020 images
drwxr-xr-x  2 biigle biigle 4096  4月 14  2021 imports
drwxr-xr-x  2 biigle biigle 4096  2月  8 09:30 logs
drwxr-xr-x  8 biigle biigle 4096  2月  8 16:10 maia_jobs
drwxr-xr-x  2 biigle biigle 4096  9月 21 11:14 reports

After chmod, I re-ran the training proposals. But unluckily I still cannot make it work for training. It still failed for the same cause.

I build the Biigle application at /opt/biigle. Is this the cause for this permission problem? Should I chown biigle:biigle for the whole biigle directory?

/opt$ ls -al
drwxr-xr-x  8 root          root          4096  2月  7 14:34 biigle

/opt$ ls -al biigle/
total 68
drwxr-xr-x  8 root   root   4096  2月  7 14:34 .
drwxr-xr-x 13 root   root   4096 12月 19  2020 ..
-rwxr-xr-x  1 root   root     65  8月 14  2020 artisan
drwxr-xr-x  4 root   root   4096  2月  7 16:39 build
drwxr-xr-x  2 root   root   4096  8月 16  2020 certificate
-rw-r--r--  1 root   root   2369  2月  7 14:34 docker-compose.yaml
drwxr-xr-x  4 root   root   4096  2月  7 11:19 docs
-rw-r--r--  1 root   root    188  4月  9  2021 .editorconfig
-rw-r--r--  1 root   root     56 11月 10 15:30 .env
-rw-r--r--  1 root   root     27  8月 14  2020 .env.example
drwxr-xr-x  8 root   root   4096  2月  7 11:29 .git
drwxr-xr-x  3 root   root   4096  9月 21 10:29 .github
-rw-r--r--  1 root   root     23  8月 14  2020 .gitignore
-rw-r--r--  1 root   root   1075  8月 14  2020 LICENSE
-rw-r--r--  1 root   root   1093  4月  9  2021 mkdocs.yml
-rw-r--r--  1 root   root   3937  9月 21 10:29 README.md
drwxr-xr-x 10 biigle biigle 4096  9月 21 11:17 storage

mzur Feb 9, 2023
Maintainer

I build the Biigle application at /opt/biigle. Is this the cause for this permission problem? Should I chown biigle:biigle for the whole biigle directory?

No, it only needs read/write permissions in the storage directory. This seems correct.

Is c518fba8edca the gpu-worker container or another one? Please check if the gpu-worker container runs as the biigle user.

cywhale · 2023-02-09T10:27:38Z

cywhale
Feb 9, 2023
Author

I found a redundant and wrong setting USER_ID in biigle/build/.env, which should be in biigle/.env (the USER_ID in biigle/.env is correct for biigle:biigle). I remove it, rebuild, got the same error cause for another .pth "faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth", then copy it into maia_jobs/, and re-ran training proposals, then it passed the .pth exist check. But now I encounter new fails.

(Even the permission seems correct after I modify my fault, the two .pth still need be copied manually into storage/maia_jobs)

(I hope that everytime the training proposal failed, we can re-submitt it so that I don't need to delete the job, re-run novelty and re-select the training proposal!)

The new error log is quite long, so I paste it as a new file in
https://github.com/cywhale/ODB/blob/master/ROV/biigle/log_MAIA2_20230209.log

It seems some internal causes, and I'm not sure what the true problem is. If there is any comments for furthur trial, please let me know. Thanks a lot for your support and help.

7 replies

mzur Feb 9, 2023
Maintainer

This looks like an actual bug! I suspect that the new batch size is forwarded as a string to the Python script. Then this error occurs because there is a division with an integer and a string. I will investigate.

If you want to test this, you could edit this line in your running Docker container and set int(params['batch_size']).

cywhale Feb 9, 2023
Author

If you want to test this, you could edit this line in your running Docker container and set int(params['batch_size']).

Ya, I modified TrainingRunner.py in /var/www/vendor/biigle/maia/src/resources/scripts/object-detection inside running docker (biigle/worker-dist) and commit it. I can confirm the file being modified as following after restart docker compose

'samples_per_gpu': int(params['batch_size']),

But re-run the training, I got the same error logs.

2023-02-10 00:54:35,924 - mmcv - INFO - 
roi_head.bbox_head.shared_fcs.1.bias - torch.Size([1024]): 
XavierInit: gain=1, distribution=uniform, bias=0 
 
/opt/conda/lib/python3.10/site-packages/mmdet/datasets/custom.py:181: UserWarning: CustomDataset does not support filtering empty gt images.
  warnings.warn(
Traceback (most recent call last):
  File \"/var/www/vendor/biigle/maia/src/config/../resources/scripts/object-detection/TrainingRunner.py\", line 143, in <module>
    output = runner.run()
  File \"/var/www/vendor/biigle/maia/src/config/../resources/scripts/object-detection/TrainingRunner.py\", line 121, in run
    train_detector(
  File \"/opt/conda/lib/python3.10/site-packages/mmdet/apis/train.py\", line 149, in train_detector
    data_loaders = [build_dataloader(ds, **train_loader_cfg) for ds in dataset]
  File \"/opt/conda/lib/python3.10/site-packages/mmdet/apis/train.py\", line 149, in <listcomp>
    data_loaders = [build_dataloader(ds, **train_loader_cfg) for ds in dataset]
  File \"/opt/conda/lib/python3.10/site-packages/mmdet/datasets/builder.py\", line 180, in build_dataloader
    sampler = GroupSampler(dataset,
  File \"/opt/conda/lib/python3.10/site-packages/mmdet/datasets/samplers/group_sampler.py\", line 21, in __init__
    size / self.samples_per_gpu)) * self.samples_per_gpu
TypeError: ufunc 'true_divide' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
 at /var/www/vendor/biigle/maia/src/Jobs/JobRequest.php:178)
(...skip)

If there is anything I should try, please let me know. I'll try it next week. Thanks a lot.

mzur Feb 9, 2023
Maintainer

Sorry but you have to apply the change in the gpu-worker container.

cywhale Feb 10, 2023
Author

My fault. Now Training proposal finally done and enter annotation stage. Thank you so much! Cheers^^

mzur Feb 10, 2023
Maintainer

Great, thanks. Then I know how to fix this bug.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BIIGLE

MAIA(v2.0) Failed to download model weights problem in training #542

{{title}}

Replies: 2 comments 13 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

BIIGLE

MAIA(v2.0) Failed to download model weights problem in training #542

cywhale Feb 7, 2023

Replies: 2 comments · 13 replies

mzur Feb 7, 2023 Maintainer

mzur Feb 8, 2023 Maintainer

cywhale Feb 8, 2023 Author

mzur Feb 8, 2023 Maintainer

cywhale Feb 8, 2023 Author

mzur Feb 9, 2023 Maintainer

cywhale Feb 9, 2023 Author

mzur Feb 9, 2023 Maintainer

cywhale Feb 9, 2023 Author

mzur Feb 9, 2023 Maintainer

cywhale Feb 10, 2023 Author

mzur Feb 10, 2023 Maintainer

cywhale
Feb 7, 2023

Replies: 2 comments 13 replies

mzur
Feb 7, 2023
Maintainer

mzur Feb 8, 2023
Maintainer

cywhale Feb 8, 2023
Author

mzur Feb 8, 2023
Maintainer

cywhale Feb 8, 2023
Author

mzur Feb 9, 2023
Maintainer

cywhale
Feb 9, 2023
Author

mzur Feb 9, 2023
Maintainer

cywhale Feb 9, 2023
Author

mzur Feb 9, 2023
Maintainer

cywhale Feb 10, 2023
Author

mzur Feb 10, 2023
Maintainer