How to Transfer Data - Revision history

Jnewport: /* From the World Wide Web */

2021-06-22T16:04:43Z

‎From the World Wide Web

Jnewport: /* From the World Wide Web */

2021-06-22T16:01:28Z

‎From the World Wide Web

Jnewport: /* From the World Wide Web */

2021-06-22T16:01:03Z

‎From the World Wide Web

Jnewport: /* Large Transfers */

2020-12-15T20:17:49Z

‎Large Transfers

Jnewport at 20:15, 15 December 2020

2020-12-15T20:15:35Z

Jnewport: /* Small Transfers */

2020-12-15T20:06:12Z

‎Small Transfers

Jnewport: /* Small Transfers */

2020-12-15T20:05:57Z

‎Small Transfers

Jnewport at 20:05, 15 December 2020

2020-12-15T20:05:07Z

Jnewport: Jnewport moved page Transferring Data to How to Transfer Data: Updating sidebar for clarity

2020-12-01T17:54:06Z

Jnewport moved page Transferring Data to How to Transfer Data: Updating sidebar for clarity

Jnewport: /* To and From Your Personal Computer */

2020-09-25T18:13:40Z

‎To and From Your Personal Computer

@@ Line 63: / Line 63: @@
 If you have URLs for multiple datasets, you can also use python code (or others) to download the files you need.  You can write a script that will look like this:
-<code>
+ <nowiki>import urllib
-import urllib
+urls=[ "url1", "url2", ...]
 for url in urls:
-   urllib.request.urlretrieve( url, filename=destination)
+   urllib.request.urlretrieve( url, filename=destination)</nowiki>
 Of course, you'll have to properly specify the filename <code>destination</code>.

@@ Line 65: / Line 65: @@
 <code>
 import urllib
-...
-urls=[ "url1", "url2", ... ]
+urls=[ "url1", "url2"]
 for url in urls:
    urllib.request.urlretrieve( url, filename=destination)

← Older revision		Revision as of 16:01, 22 June 2021
Line 59:		Line 59:

	The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].		The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].
		+
		+
		+	If you have URLs for multiple datasets, you can also use python code (or others) to download the files you need. You can write a script that will look like this:
		+
		+	<code>
		+	import urllib
		+	...
		+	urls=[ "url1", "url2", ... ]
		+	for url in urls:
		+	urllib.request.urlretrieve( url, filename=destination)
		+	</code>
		+
		+	Of course, you'll have to properly specify the filename <code>destination</code>.

@@ Line 54: / Line 54: @@
 == Large Transfers ==
-For large transfers (>100Gb), we suggest you email [mailto:support@deepsense.ca support@deepsense.ca] for assistance.  If you are local, you may be able to provide an external hard drive which we can plug in directly in our server room.  We also have special software that can be used to speed up large transfers.
+For large transfers (>100Gb), We generally find it is best to put the data on an external drive.  To make such arrangements, please email [mailto:support@deepsense.ca support@deepsense.ca].  We can then plug it in directly in our server room, and transfer the data for you.
 == From the World Wide Web ==
 The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].

@@ Line 48: / Line 48: @@
 Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.
-=== Large Transfers ===
+==== File Permissions ====
 For large transfers (>100Gb), we suggest you email [mailto:support@deepsense.ca support@deepsense.ca] for assistance.  If you are local, you may be able to provide an external hard drive which we can plug in directly in our server room.  We also have special software that can be used to speed up large transfers.

@@ Line 23: / Line 23: @@
 * P - same as --partial --progress.  Show progress while transferring, and keep partial files.
-'''Note''': We recommend always using the option <code>-p</code> (Using <code>-a</code> also invokes <code>-p</code>).  This ensures that everyone in your group should have the same permissions to the file as you do.
+'''Note''': We recommend always using the option <code>-p</code> (using <code>-a</code> also invokes <code>-p</code>).  This ensures that everyone in your group should have the same permissions to the file as you do.
 === Medium Size ===

@@ Line 7: / Line 7: @@
 For small transfers (<5Gb), you can use the two login nodes.  Since they are the primary point of access for the platform, they may be in heavy use.  We do not want to overload them unnecessarily for data transfer.  Please only use this for small amounts of data.
-The most common method for transferring data securely between machines will be <code>scp</code>.  This is pretty straightforward to use.
+The most common method for transferring data securely between machines will be <code>scp</code>.  This is pretty straightforward to use, however the destination files will have the wrong permissions set.  It will remove group permissions, so while you will be able to access the data, no one else in your group will be.  This is fine if you are the only one working on the project.
   '''Example''': scp -r /path/to/files/ username@login2.deepsense.ca:/data/projectdir/
@@ Line 23: / Line 23: @@
 * P - same as --partial --progress.  Show progress while transferring, and keep partial files.
 === Medium Size ===

← Older revision	Revision as of 17:54, 1 December 2020
(No difference)

@@ Line 3: / Line 3: @@
 == To and From Your Personal Computer ==
-=== Login Nodes ===
+=== Small Transfers ===
-Since the two login nodes are the primary point of access for the platform, they may be in heavy use.  We do not want to overload them unnecessarily for data transfer.  Please only use this for small amounts of data.
+For small transfers (<5Gb), you can use the two login nodes.  Since they are the primary point of access for the platform, they may be in heavy use.  We do not want to overload them unnecessarily for data transfer.  Please only use this for small amounts of data.
 The most common method for transferring data securely between machines will be <code>scp</code>.  This is pretty straightforward to use.
@@ Line 24: / Line 24: @@
-=== Protocol Nodes ===
+=== Medium Size ===
-The protocol nodes (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for large data transfers.  However, they are only accessible via samba.
+For medium sized transfers (between 5Gb and 100Gb), you should use the protocol nodes.  They (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for data transfers.  However, they are only accessible via samba.
 ==== Mac OSX ====
@@ Line 46: / Line 46: @@
 Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.
 == From the World Wide Web ==
 The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].