Difference between revisions of "How to Transfer Data"
(Created page) |
m (rewording) |
||
| Line 16: | Line 16: | ||
The protocol nodes (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for large data transfers. However, they are only accessible via samba. On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box, type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem. | The protocol nodes (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for large data transfers. However, they are only accessible via samba. On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box, type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem. | ||
| − | If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/ | + | If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/</code>. You can now use rsync to copy files to your project's subdirectory. |
== From the World Wide Web == | == From the World Wide Web == | ||
Revision as of 14:25, 1 May 2019
There are different methods for transferring data to and from the DeepSense platform. Which method you use will depend from where you are transferring the data, as well as the size of the data.
Contents
To and From Your Personal Computer
Login Nodes
Since the two login nodes are the primary point of access for the platform, they may be in heavy use. We do not want to overload them unnecessarily for data transfer. Please only use this for small amounts of data.
The most common method for transferring data securely between machines will be scp. This is pretty straightforward to use.
One can also use rsync (see the man page). This has more options than scp, and can be used to sync files
between two machines. Note: do not use the -p, or -a options, as they preserve file permissions. This could cause problems with user quotas, as they are based on the owner/group of files.
Protocol Nodes
The protocol nodes (protocol1.deepsense.ca, protocol2.deepsense.ca) are specifically meant for large data transfers. However, they are only accessible via samba. On a Mac, open finder and hit ⌘-K, or use the menu Go -> Connect to Server. In the dialog box, type the address for either protocol node, and you can login. This will connect you to the /data filesystem.
If you want to use rsync to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at /Volumes/data/. You can now use rsync to copy files to your project's subdirectory.
From the World Wide Web
The standard tool for downloading data from websites is wget. Also available is curl. The two are compared in this StackExchange article.
Between DeepSense Filesystems
You may want to transfer data from your home directory to your data or scratch directories. To do this, you should not use the mv command. Please instead use the cp command (you can delete them from the original filesystem after). When files are copied to a new filesystem, new files are created with the proper group name. Using the mv command will keep the original group name, and can affect the quota reporting.
