WIP data-mover #2474
WIP data-mover #2474
Conversation
|
Initial version, both documentation and the actual tool WIP. Perfectly good for testing and feedback, though. Thank you :D |
|
Hola Dear Reviewers! I added plenty of you, and I'm happy to add more. The idea is that if you have time, you can also test the tool and try to break it :) Also, all the feedback on ease-of-use, how your tests worked out, etc is more than welcome. https://gitlab.ci.csc.fi/compen/hpc-environment/data-mover/-/issues |
| than would be practical in regular interactive login sessions. | ||
|
|
||
| Below is a guide for a simple scenario, moving data from Puhti project scratch | ||
| directory to corresponding project in Allas, and then back. Similar works with |
There was a problem hiding this comment.
similar what works? Unclear
There was a problem hiding this comment.
I guess you mean exact same instructions work on mahti, which is true. Lumi-O is slightly different, need to specify more
There was a problem hiding this comment.
These object storages are really bad from the perspective of traditional HPC use. The mapping between filesystem to object storage is far from 1-to-1, object storage is completely separate machine with it's own authentication and authorisation, there are many different transfer tools/clients, APIs, and object storage server configurations, all different and often incompatible, instead of OS just handling it... I started writing it all out, noticed that it would be a long article, wrote TLDR text (what it is now), and deleted the start of the more complete guide. This tool is supposed to be easy to use. If the documentation is long, it means the tool is not easy to use :)
There was a problem hiding this comment.
I'll see if I can redirect the reader quicker to more comprehensive docs for using other services than puhti and allas
There was a problem hiding this comment.
Ok. I reread the doc. Similar is exactly how it is. Very unclear, but truthfully so :)
There was a problem hiding this comment.
you could just say that easy instructions work the same way in Mahti. Cross service usage, using Lumi-O instead of allas is possible, please go read the advanced section if you are interested
docs/support/tutorials/data-mover.md
Outdated
|
|
||
| ## Moving data from Puhti to Allas | ||
|
|
||
| 1. Put the data in a single directory, for example |
There was a problem hiding this comment.
Interesting wording, I always thougth that users would take an existing folder and move that. Of course, one could also put data into a new one. Perhaps could be clarified though
There was a problem hiding this comment.
Good point. I think I have an idea how to fix that.
| ``` | ||
| dm import /scratch/project_<projid>/exampledir | ||
| ``` | ||
|
|
There was a problem hiding this comment.
What happens for the overlapping files in exampledir?
There was a problem hiding this comment.
How do you remove old exports?
There was a problem hiding this comment.
What do you mean by overlapping files?
Deleting an export from Allas means you moved something that you should have simply deleted in the first place :D
Ok, there is dm delete ..., says dm help :D There are also lot's of other dm subcommands, but my concern is to make this documentation short. If it is long, it looks like the tool is complicated and not easy to use.
| [Puhti web interface](https://puhti.csc.fi). Open "Cloud storage configuration" from the | ||
| "Tools" drop-down menu, and | ||
| [create Allas S3 rclone configuration for the project](../../computing/webinterface/file-browser.md#accessing-allas-and-lumi-o). | ||
|
|
There was a problem hiding this comment.
Ok. So this uses the the Open on demand style s3 configuration. This will be bit confusing for the old users of S3 allas.
https://csc-guide-preview.2.rahtiapp.fi/origin/data-mover/support/tutorials/data-mover