Skip to content

Commit 455be62

Browse files
NicoleLabrAvilaNicoleLabrAvila
authored
Merge pull request #97 from UCL-ARC/NicoleLabrAvila-patch-1
Create rclone.md
2 parents acb019d + e117fc3 commit 455be62

File tree

1 file changed

+287
-0
lines changed
  • mkdocs-project-dir/docs/Walkthroughs

1 file changed

+287
-0
lines changed

‎mkdocs-project-dir/docs/Walkthroughs/rclone.md

Lines changed: 287 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,287 @@
1+
# Rclone
2+
3+
[Rclone](https://rclone.org/) is a command-line program to manage files on cloud storage. This is a walkthrough for
4+
configuring **Rclone** to be used with Microsoft OneDrive but it also can be used with several other cloud systems like
5+
Dropbox, Google Drive, etc.
6+
7+
We have installed 2 versions of rclone in our clusters. ```rclone/1.61.0``` was used as reference for this walkthrough
8+
but the process is similar with other versions.
9+
10+
After loading the module with the command:
11+
12+
```
13+
module load rclone/1.61.0 ,
14+
```
15+
16+
the next step is to configure **RClone**. As the cloud systems usually have quite complicated authentication methods,
17+
these are kept in a config file. For example, the configuration file for **OneDrive** and **Dropbox** involve getting
18+
a **token** from **Microsoft** and **Dropbox** respectively, to link your account in the cloud system with **Rclone**.
19+
There are several ways to get this token and here is presented the easiest. If this method does not work for you, please
20+
contact our support: [[email protected]]
21+
22+
The method that allows an automatic way to get and add the token to the configuration file requires to uses
23+
[X-Forwarding](https://www.rc.ucl.ac.uk/docs/Supplementary/X-Forwarding/ )
24+
in your ssh connection. This will allow to the cluster to push an interactive window to your local computer over a network.
25+
In this window you will enter your credentials and connect to the account you want to link with **Rclone** (e.g. Microsoft
26+
One Drive or Dropbox). If you have activated MFA in your account, it will ask you for the respective code. Note that the
27+
connection to internet could not the best and it might take a while to load the page.
28+
29+
### X-Forwarding in Linux
30+
31+
If you are using linux, the X-Forwarding is very easy. Just add ```-X``` option to your ssh command like this:
32+
33+
```
34+
ssh -X [email protected]
35+
```
36+
### X-Forwarding in Mac
37+
38+
If you are using Mac, there is an app called ***XQuartz** you must donwload first and then use the ```-Y``` option with the
39+
ssh command to allow X-forwarding.
40+
41+
### X-Forwarding in Windows
42+
43+
If you are using Windows, you need to install an ```X``` server separately. The use **Xming** with ***Putty** is suggested
44+
to configure **Rclone**, although you can use your preferred method. You can download **Xming-fonts** free from
45+
[here](https://sourceforge.net/projects/xming/files/). Once you download it and install it, you must open **Xming** in your
46+
computer before connecting to ssh. The application is always working on the background so you will see anything opening after
47+
click on it.
48+
49+
Then, to connect to ssh we suggest to use **Putty** unless you know how to allow the X-forwarding with another method. To
50+
configure **Putty** to use x-Forwarding, you must look at the left side panel and go to ```SSH``` category. Then select ```X11```
51+
and in there tick on ```Enable X11 forwarding``` (the box must be blue). Then enter the details to connect to the cluster in
52+
the main window and that's all! If you are loading a previous saved session, then load the session first and then enable the
53+
X-Forwarding.
54+
55+
56+
After you have allowed X-Forwarding, you are ready to configure **Rclone**.
57+
58+
## Rclone and OneDrive
59+
60+
After loading the **Rclone** module, run in the command line:
61+
62+
```
63+
module load rclone/1.61.0
64+
rclone config
65+
```
66+
67+
The following menu and options will be displayed:
68+
69+
1.
70+
```
71+
No remotes found, make a new one?
72+
n) New remote
73+
s) Set configuration password
74+
q) Quit config
75+
n/s/q>
76+
```
77+
Select **n** and press **Enter** Key
78+
79+
2.
80+
```
81+
Enter name for new remote.
82+
name>
83+
```
84+
Enter the **name of your own choose for your remote** connection and press **Enter** key.
85+
86+
3. Something similar to this will be displayed. Please check what number corresponds to **Microsoft OneDrive**
87+
option and enter that one.
88+
```
89+
Option Storage.Type of storage to configure.
90+
Choose a number from below, or type in your own value.
91+
1 / 1Fichier \ (fichier)
92+
2 / Akamai NetStorage \ (netstorage)
93+
3 / Alias for an existing remote \ (alias)
94+
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean,
95+
Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Outscale,
96+
Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, Selectel, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others \ (s3)
97+
5 / Backblaze B2 \ (b2)
98+
6 / Better checksums for other remotes \ (hasher)
99+
7 / Box \ (box)
100+
8 / Cache a remote \ (cache)
101+
9 / Citrix Sharefile \ (sharefile)
102+
10 / Cloudinary \ (cloudinary)
103+
11 / Combine several remotes into one \ (combine
104+
12 / Compress a remote \ (compress)
105+
13 / Dropbox \ (dropbox)
106+
14 / Encrypt/Decrypt a remote \ (crypt)
107+
15 / Enterprise File Fabric \ (filefabric)
108+
16 / FTP \ (ftp
109+
17 / Files.com \ (filescom)
110+
18 / Gofile \ (gofile)
111+
19 / Google Cloud Storage (this is not Google Drive) \ (google cloud storage
112+
20 / Google Drive \ (drive)
113+
21 / Google Photos \ (google photos)
114+
22 / HTTP \ (http)
115+
23 / Hadoop distributed file system \ (hdfs)
116+
24 / HiDrive \ (hidrive)
117+
25 / ImageKit.io \ (imagekit)
118+
26 / In memory object storage system. \ (memory)
119+
27 / Internet Archive \ (internetarchive)
120+
28 / Jottacloud \ (jottacloud)
121+
29 / Koofr, Digi Storage and other Koofr-compatible storage providers \ (koofr)
122+
30 / Linkbox \ (linkbox)
123+
31 / Local Disk \ (local)
124+
32 / Mail.ru Cloud \ (mailru)
125+
33 / Mega \ (mega)
126+
34 / Microsoft Azure Blob Storage \ (azureblob)
127+
35 / Microsoft Azure Files \ (azurefiles)
128+
/ Microsoft OneDrive \ (onedrive)
129+
37 / OpenDrive \ (opendrive)
130+
38 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH) \ (swift)
131+
39 / Oracle Cloud Infrastructure Object Storage \ (oracleobjectstorage)
132+
40 / Pcloud \ (pcloud)
133+
41 / PikPak \ (pikpak)
134+
42 / Pixeldrain Filesystem \ (pixeldrain)
135+
43 / Proton Drive \ (protondrive)
136+
44 / Put.io \ (putio)
137+
45 / QingCloud Object Storage \ (qingstor)
138+
46 / Quatrix by Maytech \ (quatrix)
139+
47 / SMB / CIFS \ (smb)
140+
48 / SSH/SFTP \ (sftp)
141+
49 / Sia Decentralized Cloud \ (sia)
142+
50 / Storj Decentralized Cloud Storage \ (storj
143+
51 / Sugarsync \ (sugarsync)
144+
52 / Transparently chunk/split large files \ (chunker
145+
53 / Uloz.to \ (ulozto)
146+
54 / Union merges the contents of several upstream fs \ (union)
147+
55 / Uptobox \ (uptobox)
148+
56 / WebDAV \ (webdav)
149+
57 / Yandex Disk \ (yandex
150+
58 / Zoho \ (zoho)
151+
59 / iCloud Drive \ (iclouddrive)
152+
60 / premiumize.me \ (premiumizeme)
153+
61 / seafile \ (seafile)
154+
Storage>
155+
```
156+
In this case,**Microsoft OneDrive** corresponds to the option **36**, so enter **36** and press **Enter** key.
157+
158+
4.
159+
```
160+
Option client_id. OAuth Client Id.
161+
Leave blank normally.
162+
Enter a value. Press Enter to leave empty.
163+
client_id>
164+
```
165+
Press **Enter** key to leave empty.
166+
167+
5.
168+
```
169+
Option client_secret. OAuth Client Secret.
170+
Leave blank normally.
171+
Enter a value. Press Enter to leave empty.
172+
client_secret>
173+
```
174+
Press **Enter** key to leave empty.
175+
176+
6.
177+
```
178+
Option region. Choose national cloud region for OneDrive.
179+
Choose a number from below, or type in your own value of
180+
type string.
181+
Press Enter for the default (global).
182+
1 / Microsoft Cloud Global \ (global)
183+
2 / Microsoft Cloud for US Government \ (us)
184+
3 / Microsoft Cloud Germany (deprecated - try global region first). \ (de)
185+
4 / Azure and Office 365 operated by Vnet Group in China \ (cn)
186+
region>
187+
```
188+
Select **1** and press **Enter** Key
189+
190+
```
191+
Option tenant.ID of the service principal's tenant. Also called its directory ID.
192+
Set this if using- Client Credential flowEnter a value.
193+
Press Enter to leave empty.
194+
tenant>
195+
```
196+
Press **Enter** to leave empty
197+
198+
8.
199+
```
200+
Edit advanced config?
201+
y) Yes
202+
n) No (default)
203+
y/n>
204+
```
205+
Select **n** and press **Enter** key.
206+
207+
9.
208+
```
209+
Use web browser to automatically authenticate rclone with remote?
210+
* Say Y if the machine running rclone has a web browser you can use
211+
* Say N if running rclone on a (remote) machine without web browser access
212+
If not sure try Y. If Y failed, try N.
213+
y) Yes (default)
214+
n) No
215+
y/n>
216+
```
217+
Select **y** and press **Enter** key.
218+
219+
**If you have installed Xming and habilitated correctly X-Forwarding in Putty or your preferred method,
220+
then a window will open in your perosnal machine. There, you must log in, in this case into Microsoft using
221+
your UCL account (internal).**
222+
223+
10.
224+
```
225+
Option config_type.Type of connectionChoose a number from below,
226+
or type in an existing string value.
227+
Press Enter for the default (onedrive).
228+
1 / OneDrive Personal or Business \ (onedrive)
229+
2 / Root Sharepoint site \ (sharepoint) / Sharepoint site name or URL
230+
3 | E.g. mysite or https://contoso.sharepoint.com/sites/mysite \ (url)
231+
4 / Search for a Sharepoint site \ (search)
232+
5 / Type in driveID (advanced) \ (driveid)
233+
6 / Type in SiteID (advanced) \ (siteid) / Sharepoint server-relative path (advanced)
234+
7 | E.g. /teams/hr \ (path
235+
config_type>
236+
```
237+
Select **1** and press **Enter** key.
238+
239+
11.
240+
```
241+
Option config_driveid. Select drive you want to use
242+
Choose a number from below, or type in your own string value.
243+
Press Enter for the default (b!ra4Fi4CqS0GYT_qied1OV9A2pTgX9JxHhfZTS2MwNK-0r_cRHFddS7D8Yt0CozO-).
244+
1 / OneDrive (business) \ (b!ra4Fi4CqS0GYT_qied1OV9A2pTgX9JxHhfZTS2MwNK-0r_cRHFddS7D8Yt0CozO-)
245+
2 / PersonalCacheLibrary (business) \ (b!ra4Fi4CqS0GYT_qied1OV9A2pTgX9JxHhfZTS2MwNK9-7X_uAf71Qavjc1vd7Xis)
246+
config_driveid>
247+
```
248+
Select **1** and press **Enter** key.
249+
250+
12.
251+
```
252+
Drive OK Found drive "root" of type "business"
253+
URL: https://liveuclac-my.sharepoint.com/personal/<YOUR_UCL_ID>_ucl_ac_uk/Documents
254+
y) Yes (default)
255+
n) No
256+
y/n>
257+
```
258+
If the ULR is ok, then select **y** and press **Enter** key.
259+
260+
13.
261+
```
262+
Configuration complete.
263+
```
264+
265+
Then it will show you the remote you just added:
266+
267+
14.
268+
```
269+
Options:
270+
- type: onedrive
271+
- region: global
272+
- token:{"access_token":"eyJ0eXAiOiJKV1QiLCJub25jZSI6IkxmaXR6d2hKWkVaOGstRlZqZEh
273+
iOHFCUHZGMnFZSE8tQmpORV80d1lQWTQiLCJhbGciOiJSUzI1NiIsIng1dCI6IkNOdjBPSTNSd3FsSEZ
274+
FVm5hb01Bc2hDSDJYRSIsImtpZCI6IkNOdjBPSTNSd3FsSEZFVm5hb01Bc2hDSDJYRSJ9.eyJhdWQiOs iIwMDAwMDAwMy0wMDAwLTAwMDAtYzAwMC0wMDAwMDAwMDAwMDAiLCJpc3MiOiJodHRwczovL3N0cy53a W5kb3dzLm5ldC8xZmFmODhmZS1hOTk4LTRjNWItOTNjOS0yMTBhMTFkOWE1YzIvIiwiaWF0IjoxNzQ1N DIxNzE4LCJuYmYiOjE3NDU0MjE3MTgsImV4cCI6MTc0NTQyNTc4OCwiYWNjdCI6MCwiYWNyIjoiMSIsI mFjcnMiOlsicDEiLCJ1cm46dXNlcjpyZWdpc3RlcnNlY3VyaXR5aW5mbyJdLCJhaW8iOiJBWFFBaS84W kFBQUF1NjJxSG1Cc1YzRk5MaElFWmxwZ3lobjB6WGJzY0hlSGZWajJHdnFEcXhockI1WTl1bXF4Z3hGS UVsZ1k2Q1psRlBqSDd1V0NKbkR0cmNaME9FQjdQcmZtcE1oQjF4L3BENEJGY1RCOVUveEZ4TXlhWFVLZ Wt0VHplcURtUkxYNi9qM1l0em5JazQvaHpzUEpXdjRoY2c9PSIsImFtciI6WyJwd2QiLCJ3aWEiLCJtZ mEiXSwiYXBwX2Rpc3BsYXluYW1lIjoicmNsb25lIiwiYXBwaWQiOiJiMTU2NjVkOS1lZGE2LTQwOTItO DUzOS0wZWVjMzc2YWZkNTkiLCJhcHBpZGFjciI6IjEiLCJmYW1pbHlfbmFtZSI6IkxhYnJhIEF2aWxhI iwiZ2l2ZW5fbmFtZSI6Ik5pY29sZSIsImlkdHlwIjoidXNlciIsImlwYWRkciI6IjE5My42MC4yMzguM TEwIiwibmFtZSI6IkxhYnJhIEF2aWxhLCBOaWNvbGUiLCJvaWQiOiJmOWU4OTBlMS1iZDBhLTQzZjctY jRmNi03NWUyZTY4ODg3OWMiLCJvbnByZW1fc2lkIjoiUy0xLTUtMjEtMjkwMjI2NTYyMS0xMDYzMDI4N jIxLTIzODE1NjE0ODAtOTcwNjUxIiwicGxhdGYiOiI4IiwicHVpZCI6IjEwMDMyMDAxQzNBQjg4MjkiL CJyaCI6IjEuQVFVQV9vaXZINWlwVzB5VHlTRUtFZG1sd2dNQUFBQUFBQUFBd0FBQUFBQUFBQUFGQUY0R kFBLiIsInNjcCI6IkZpbGVzLlJlYWQgRmlsZXMuUmVhZC5BbGwgRmlsZXMuUmVhZFdyaXRlIEZpbGVzL lJlYWRXcml0ZS5BbGwgU2l0ZXMuUmVhZC5BbGwgcHJvZmlsZSBvcGVuaWQgZW1haWwiLCJzaWQiOiIwM DQwOTM1OS03OTQ2LTg5MjItNWU5MC1hMTljZGIyNmRiODgiLCJzdWIiOiJGakhnU21hNjBTSTFZWGk4M k1qMnEtR2RhQi1mS2ZYNTJpSEE3SGJsMUljIiwidGVuYW50X3JlZ2lvbl9zY29wZSI6IkVVIiwidGlkI joiMWZhZjg4ZmUtYTk5OC00YzViLTkzYzktMjEwYTExZDlhNWMyIiwidW5pcXVlX25hbWUiOiJza2d0b mwxQHVjbC5hYy51ayIsInVwbiI6InNrZ3RubDFAdWNsLmFjLnVrIiwidXRpIjoiWE9OQ3hPYW5sazJXV 0tRVnVONHJBQSIsInZlciI6IjEuMCIsIndpZHMiOlsiYjc5ZmJmNGQtM2VmOS00Njg5LTgxNDMtNzZiM Tk0ZTg1NTA5Il0sInhtc19mdGQiOiJOal9Zd1hsRG9pY0tyWnpUaS1wQWZPYjloR1RReEVFbW5ybXRHU kt6ekI0IiwieG1zX2lkcmVsIjoiMSAzMCIsInhtc19zdCI6eyJzdWIiOiJJOTdqOF9WWXljNnNhVkxWQ 1UtN3hIcEI3TVdzYWJ4cW9oNnZnbWttMl9nIn0sInhtc190Y2R0IjoxMzYwODcxMzQ1fQ.F-Ma2iOnL"}
275+
- drive_id: b!ra4Fi4CqS0GYT_qied1OV9A2pTgX9JxHhfZTS2MwNK-0r_cRHFddS7D8Yt0CozO-
276+
- drive_type: business
277+
Keep this "UCLDrive" remote?
278+
y) Yes this is OK (default)
279+
e) Edit this remote
280+
d) Delete this remote
281+
y/e/d>
282+
```
283+
If everything is ok., then select **y** and press **Enter** key.
284+
285+
Now you are ready to use **Rclone**!
286+
287+
If you are new using Rclone, please look [here](https://rclone.org/docs/#basic-syntax) the basic syntax.

0 commit comments

Comments
 (0)