https://docs.deepsense.ca/api.php?action=feedcontributions&user=Jnewport&feedformat=atomDeepSense Docs - User contributions [en-ca]2024-03-29T05:41:17ZUser contributionsMediaWiki 1.31.1https://docs.deepsense.ca/index.php?title=How_to_Transfer_Data&diff=564How to Transfer Data2021-06-22T16:04:43Z<p>Jnewport: /* From the World Wide Web */</p>
<hr />
<div>There are different methods for transferring data to and from the DeepSense platform. Which method you use will depend from where you are transferring the data, as well as the size of the data.<br />
<br />
== To and From Your Personal Computer ==<br />
<br />
=== Small Transfers ===<br />
<br />
For small transfers (<5Gb), you can use the two login nodes. Since they are the primary point of access for the platform, they may be in heavy use. We do not want to overload them unnecessarily for data transfer. Please only use this for small amounts of data.<br />
<br />
The most common method for transferring data securely between machines will be <code>scp</code>. This is pretty straightforward to use, however the destination files will have the wrong permissions set. It will remove group permissions, so while you will be able to access the data, no one else in your group will be. This is fine if you are the only one working on the project.<br />
<br />
'''Example''': scp -r /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
One can also use <code>rsync</code> (see the [https://linux.die.net/man/1/rsync man page]). This has more options than <code>scp</code>, and can be used to sync files<br />
between two machines. <br />
<br />
'''Example''': rsync -azvhP /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
The rsync options above are:<br />
* a - archive mode, equal to rlptgoD (recursive, preserve links, times, permissions, group, owner, etc)<br />
* z - use compression when copying<br />
* v - verbose: list files copied<br />
* h - human readable: output numbers in human readable format<br />
* P - same as --partial --progress. Show progress while transferring, and keep partial files.<br />
<br />
'''Note''': We recommend always using the option <code>-p</code> (using <code>-a</code> also invokes <code>-p</code>). This ensures that everyone in your group should have the same permissions to the file as you do.<br />
<br />
=== Medium Size ===<br />
<br />
For medium sized transfers (between 5Gb and 100Gb), you should use the protocol nodes. They (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for data transfers. However, they are only accessible via samba. <br />
<br />
==== Mac OSX ====<br />
<br />
[[File:macSambaConnect.png|thumb|Connect via samba on OSX]]<br />
<br />
On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box (see image), type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem.<br />
<br />
If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/</code>. You can now use rsync to copy files to your project's subdirectory.<br />
<br />
'''Example''': rsync&nbsp;&#8209;rzvh&nbsp;/path/to/files/&nbsp;/Volumes/data/projectdir/<br />
<br />
==== Windows ====<br />
<br />
On windows computer, you should connect to <code>\\protocol1.deepsense.ca\data</code> or <code>\\protocol2.deepsense.ca\data</code>. To do this the first time, open a file explorer window. <br />
Right-click on This PC, and select "add a network location". In the wizard, click next and then select "Choose a custom network location" (this was the only option I saw). Highlight it, and click next. On the following screen, enter one of the addresses above, and click next. You may now enter a name for this location. Do so, and click next again. On the last screen, you should be able to look over your selections, and then click Finish. The name you chose should now be available under "This PC" in your file explorer. <br />
<br />
You may also have to change a SMB security level setting as follows (this was necessary in Windows 10):<br />
<br />
Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.<br />
<br />
==== File Permissions ====<br />
<br />
Unfortunately, samba won't preserve the proper file permissions. We find it strips the executable bit from any file that has it switched on. You can change an individual file by using <code>chmod ug+x filename</code>. If you want to change many files at once, and are unsure of how to, please send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).<br />
<br />
== Large Transfers ==<br />
<br />
For large transfers (>100Gb), We generally find it is best to put the data on an external drive. To make such arrangements, please email [mailto:support@deepsense.ca support@deepsense.ca]. We can then plug it in directly in our server room, and transfer the data for you.<br />
<br />
== From the World Wide Web ==<br />
<br />
The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].<br />
<br />
<br />
If you have URLs for multiple datasets, you can also use python code (or others) to download the files you need. You can write a script that will look like this:<br />
<br />
<nowiki>import urllib<br />
urls=[ "url1", "url2", ...]<br />
<br />
...<br />
<br />
for url in urls:<br />
urllib.request.urlretrieve( url, filename=destination)</nowiki><br />
<br />
Of course, you'll have to properly specify the filename <code>destination</code>.</div>Jnewporthttps://docs.deepsense.ca/index.php?title=How_to_Transfer_Data&diff=563How to Transfer Data2021-06-22T16:01:28Z<p>Jnewport: /* From the World Wide Web */</p>
<hr />
<div>There are different methods for transferring data to and from the DeepSense platform. Which method you use will depend from where you are transferring the data, as well as the size of the data.<br />
<br />
== To and From Your Personal Computer ==<br />
<br />
=== Small Transfers ===<br />
<br />
For small transfers (<5Gb), you can use the two login nodes. Since they are the primary point of access for the platform, they may be in heavy use. We do not want to overload them unnecessarily for data transfer. Please only use this for small amounts of data.<br />
<br />
The most common method for transferring data securely between machines will be <code>scp</code>. This is pretty straightforward to use, however the destination files will have the wrong permissions set. It will remove group permissions, so while you will be able to access the data, no one else in your group will be. This is fine if you are the only one working on the project.<br />
<br />
'''Example''': scp -r /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
One can also use <code>rsync</code> (see the [https://linux.die.net/man/1/rsync man page]). This has more options than <code>scp</code>, and can be used to sync files<br />
between two machines. <br />
<br />
'''Example''': rsync -azvhP /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
The rsync options above are:<br />
* a - archive mode, equal to rlptgoD (recursive, preserve links, times, permissions, group, owner, etc)<br />
* z - use compression when copying<br />
* v - verbose: list files copied<br />
* h - human readable: output numbers in human readable format<br />
* P - same as --partial --progress. Show progress while transferring, and keep partial files.<br />
<br />
'''Note''': We recommend always using the option <code>-p</code> (using <code>-a</code> also invokes <code>-p</code>). This ensures that everyone in your group should have the same permissions to the file as you do.<br />
<br />
=== Medium Size ===<br />
<br />
For medium sized transfers (between 5Gb and 100Gb), you should use the protocol nodes. They (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for data transfers. However, they are only accessible via samba. <br />
<br />
==== Mac OSX ====<br />
<br />
[[File:macSambaConnect.png|thumb|Connect via samba on OSX]]<br />
<br />
On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box (see image), type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem.<br />
<br />
If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/</code>. You can now use rsync to copy files to your project's subdirectory.<br />
<br />
'''Example''': rsync&nbsp;&#8209;rzvh&nbsp;/path/to/files/&nbsp;/Volumes/data/projectdir/<br />
<br />
==== Windows ====<br />
<br />
On windows computer, you should connect to <code>\\protocol1.deepsense.ca\data</code> or <code>\\protocol2.deepsense.ca\data</code>. To do this the first time, open a file explorer window. <br />
Right-click on This PC, and select "add a network location". In the wizard, click next and then select "Choose a custom network location" (this was the only option I saw). Highlight it, and click next. On the following screen, enter one of the addresses above, and click next. You may now enter a name for this location. Do so, and click next again. On the last screen, you should be able to look over your selections, and then click Finish. The name you chose should now be available under "This PC" in your file explorer. <br />
<br />
You may also have to change a SMB security level setting as follows (this was necessary in Windows 10):<br />
<br />
Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.<br />
<br />
==== File Permissions ====<br />
<br />
Unfortunately, samba won't preserve the proper file permissions. We find it strips the executable bit from any file that has it switched on. You can change an individual file by using <code>chmod ug+x filename</code>. If you want to change many files at once, and are unsure of how to, please send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).<br />
<br />
== Large Transfers ==<br />
<br />
For large transfers (>100Gb), We generally find it is best to put the data on an external drive. To make such arrangements, please email [mailto:support@deepsense.ca support@deepsense.ca]. We can then plug it in directly in our server room, and transfer the data for you.<br />
<br />
== From the World Wide Web ==<br />
<br />
The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].<br />
<br />
<br />
If you have URLs for multiple datasets, you can also use python code (or others) to download the files you need. You can write a script that will look like this:<br />
<br />
<code><br />
import urllib<br />
<br />
urls=[ "url1", "url2"]<br />
for url in urls:<br />
urllib.request.urlretrieve( url, filename=destination)<br />
</code><br />
<br />
Of course, you'll have to properly specify the filename <code>destination</code>.</div>Jnewporthttps://docs.deepsense.ca/index.php?title=How_to_Transfer_Data&diff=562How to Transfer Data2021-06-22T16:01:03Z<p>Jnewport: /* From the World Wide Web */</p>
<hr />
<div>There are different methods for transferring data to and from the DeepSense platform. Which method you use will depend from where you are transferring the data, as well as the size of the data.<br />
<br />
== To and From Your Personal Computer ==<br />
<br />
=== Small Transfers ===<br />
<br />
For small transfers (<5Gb), you can use the two login nodes. Since they are the primary point of access for the platform, they may be in heavy use. We do not want to overload them unnecessarily for data transfer. Please only use this for small amounts of data.<br />
<br />
The most common method for transferring data securely between machines will be <code>scp</code>. This is pretty straightforward to use, however the destination files will have the wrong permissions set. It will remove group permissions, so while you will be able to access the data, no one else in your group will be. This is fine if you are the only one working on the project.<br />
<br />
'''Example''': scp -r /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
One can also use <code>rsync</code> (see the [https://linux.die.net/man/1/rsync man page]). This has more options than <code>scp</code>, and can be used to sync files<br />
between two machines. <br />
<br />
'''Example''': rsync -azvhP /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
The rsync options above are:<br />
* a - archive mode, equal to rlptgoD (recursive, preserve links, times, permissions, group, owner, etc)<br />
* z - use compression when copying<br />
* v - verbose: list files copied<br />
* h - human readable: output numbers in human readable format<br />
* P - same as --partial --progress. Show progress while transferring, and keep partial files.<br />
<br />
'''Note''': We recommend always using the option <code>-p</code> (using <code>-a</code> also invokes <code>-p</code>). This ensures that everyone in your group should have the same permissions to the file as you do.<br />
<br />
=== Medium Size ===<br />
<br />
For medium sized transfers (between 5Gb and 100Gb), you should use the protocol nodes. They (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for data transfers. However, they are only accessible via samba. <br />
<br />
==== Mac OSX ====<br />
<br />
[[File:macSambaConnect.png|thumb|Connect via samba on OSX]]<br />
<br />
On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box (see image), type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem.<br />
<br />
If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/</code>. You can now use rsync to copy files to your project's subdirectory.<br />
<br />
'''Example''': rsync&nbsp;&#8209;rzvh&nbsp;/path/to/files/&nbsp;/Volumes/data/projectdir/<br />
<br />
==== Windows ====<br />
<br />
On windows computer, you should connect to <code>\\protocol1.deepsense.ca\data</code> or <code>\\protocol2.deepsense.ca\data</code>. To do this the first time, open a file explorer window. <br />
Right-click on This PC, and select "add a network location". In the wizard, click next and then select "Choose a custom network location" (this was the only option I saw). Highlight it, and click next. On the following screen, enter one of the addresses above, and click next. You may now enter a name for this location. Do so, and click next again. On the last screen, you should be able to look over your selections, and then click Finish. The name you chose should now be available under "This PC" in your file explorer. <br />
<br />
You may also have to change a SMB security level setting as follows (this was necessary in Windows 10):<br />
<br />
Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.<br />
<br />
==== File Permissions ====<br />
<br />
Unfortunately, samba won't preserve the proper file permissions. We find it strips the executable bit from any file that has it switched on. You can change an individual file by using <code>chmod ug+x filename</code>. If you want to change many files at once, and are unsure of how to, please send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).<br />
<br />
== Large Transfers ==<br />
<br />
For large transfers (>100Gb), We generally find it is best to put the data on an external drive. To make such arrangements, please email [mailto:support@deepsense.ca support@deepsense.ca]. We can then plug it in directly in our server room, and transfer the data for you.<br />
<br />
== From the World Wide Web ==<br />
<br />
The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].<br />
<br />
<br />
If you have URLs for multiple datasets, you can also use python code (or others) to download the files you need. You can write a script that will look like this:<br />
<br />
<code><br />
import urllib<br />
...<br />
urls=[ "url1", "url2", ... ]<br />
for url in urls:<br />
urllib.request.urlretrieve( url, filename=destination)<br />
</code><br />
<br />
Of course, you'll have to properly specify the filename <code>destination</code>.</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Training_Projects&diff=545Training Projects2021-05-31T18:26:45Z<p>Jnewport: /* 1. Object Detection */</p>
<hr />
<div><div class="noautonum"><br />
<br />
DeepSense has compiled a few data sets for students, and others interested in the ocean and AI, so they can have the opportunity to complete AI projects independently. We hope participants can learn about a specific type of ocean related data, and experience an explicit AI project. It is expected that the participants work on the project alone, but we have provided some guidance that includes notebooks, data, outputs and models to try to improve upon. <br />
<br />
We have found that the data cleaning step can take a long time, so our hope is that these datasets will be reasonably clean, allowing the participants to explore ocean AI. <br />
<br />
== 1. Object Detection ==<br />
<br />
We used the google open images database to obtain approximately 650 images of starfish. The images were already separated into train, test and validation sets. The metadata linked below is only for the starfish images, not for the entire dataset. The metadata includes coordinates for bounding boxes around the starfish. <br />
<br />
[https://drive.google.com/drive/folders/19Ti_XlUuj4vyo3hlBUMJ0dwszrzR58lB?usp=sharing Google Drive Directory]<br />
<br />
This contains the files:<br />
<br />
[https://drive.google.com/file/d/1r_2DNOF2WSFXdJLp7mditL5Wp4wbpOyR/view?usp=sharing Starfish Dataset]<br />
<br />
[https://drive.google.com/file/d/1gMCKW9Ih1wCVC9a_XP3yI97OCs1k46Hs/view?usp=sharing Training metadata]<br />
<br />
[https://drive.google.com/file/d/1TwCdyNJT_2Lzn0UBaDUax68oFwsY1WZm/view?usp=sharing Test metadata]<br />
<br />
[https://drive.google.com/file/d/1uI7gxRlLddfQE7VUm1-vGJE_1g1wvPd3/view?usp=sharing Validation metadata]<br />
<br />
If you want to download other categories of images from the open images database, you can do so by following the instructions here:<br />
<br />
[https://drive.google.com/file/d/1q55Q3wHfeRwD_gkuMMrRZcT_Vbtz1qG7/view?usp=sharing Download Instructions]<br />
<br />
After you have the datasets, you can download and install YOLO v4 using the following instructions:<br />
<br />
[https://drive.google.com/file/d/1OYYD7hcid5BNlR-4bvqZahh0arVO4dtG/view?usp=sharing Installation Instructions]<br />
<br />
[https://drive.google.com/file/d/1PQb76ttuHkbVr5TGW_aReCu9mlVwlZq4/view?usp=sharing Configuration Instructions]<br />
<br />
[https://drive.google.com/file/d/1mvYdpKCIvpeLoIBhyd_EPviNBRMTbkH4/view?usp=sharing Metadata Conversion Script]<br />
<br />
If you want to run this on google colab, check out the following wiki: [https://github.com/AlexeyAB/darknet/wiki Darknet Wiki]. At the top there is a link to a colab notebook, and a video tutorial.<br />
<br />
== 2. Regression ==<br />
<br />
The buoy collects environment measurements including wind speed and direction, surface temperature, current speed, wave height, and peak wave period. This wind and wave data are used to decide if conditions allow the safe transfer of pilots and passage of vessels, as they require a minimum depth of water which may not be met if the waves are too large. The current Red Shoal Buoy is under maintenance. Such a duration without accurate environmental measurements would significantly impair the ability to ensure the safe guidance of vessels. In this project, we are trying to predict the environment measurements of the buoy which is under maintenance using the values of other active operational buoy so that the authorities could allow the safe passage of vessels.<br />
<br />
Predicting the values of one buoy using the parameters of another buoy. In this project, we are using the dataset of Mouth of Placentia Bay Buoy, Pilot Boarding Station / Red Island Shoal Buoy, Placentia Bay: Ragged Islands – KLUMI( Land station) which are located in Newfoundland and Labrador. <br />
<br />
'''Datasets'''<br />
<br />
[https://drive.google.com/file/d/1rinZ5XgK_f64NxtV-Vy0ZWCEp-oWajxJ/view?usp=sharing Mouth of Placentia Bay Buoy]<br />
<br />
[https://drive.google.com/file/d/1WgaF7Q_jej-YlrAJEfuGnqhJ5l3xAXbJ/view?usp=sharing Pilot Boarding Station / Red Island Shoal Buoy]<br />
<br />
[https://drive.google.com/file/d/14GB14C2Qls715NKbTkhZocINw5cvJxQG/view?usp=sharing Placentia Bay: Ragged Islands – KLUMI( Land station)]<br />
<br />
The dataset available here is till April 19, 2021. You can get the latest dataset from [https://www.smartatlantic.ca smartatlantic].<br />
<br />
[https://drive.google.com/file/d/1C_Df3p3DPXK8tC3Y6nWYlyq6ZNsJd3_b/view?usp=sharing '''Download Instructions''']<br />
<br />
[https://drive.google.com/file/d/1EpKrdx1FiFezAqwscgj4SSxgsqfvzU9e/view?usp=sharing '''Data Dictionary''']<br />
<br />
[https://drive.google.com/file/d/1FZkyjqiBx51gYXjYjcpne-tG6G5L0lK8/view?usp=sharing '''Visual Representation of data''']<br />
<br />
[https://drive.google.com/file/d/1t1DqNL6LZOiHzbrkxIZbzQlD-PVVaoqG/view?usp=sharing '''Buoy Location''']<br />
<br />
<br />
''You can find the instructions to clean the dataset, merging of files and training the ML models from the below link:''<br />
<br />
[https://drive.google.com/file/d/1sr7QFHAycJ7KRBXuPgxgFDLxvHQvyngz/view?usp=sharing '''Instructions for Cleaning/Merging/Training''']<br />
<br />
<br />
''We have implemented the code on IBM Watson Cloud and encourage you to use this to get the experience of Cloud. Below link will provide you the instructions for using the IBM Watson Cloud. The Lite version of this cloud is free and provide you 25GB storage which is enough for this project.''<br />
<br />
[https://drive.google.com/file/d/1ns7OrUy4JURdv5QwD3vjGrZszU0DQ3wa/view?usp=sharing '''Instructions for using IBM Watson Cloud''']<br />
<br />
<br />
''We have created notebooks with the code for your reference in the below link.''<br />
<br />
[https://drive.google.com/file/d/1WpFhgKkoq19k4XR0dxp1wxgXFsPPnMgx/view?usp=sharing '''Links for Notebooks''']<br />
<br />
<br />
'''REFERENCES'''<br />
<br />
[https://www.thejot.net/article-preview/?show_article_preview=1193 A MACHINE LEARNING REDUNDANCY MODEL FOR THE HERRING COVE SMART BUOY]<br />
<br />
==3. NLP==<br />
NLP project is related to the Sentiment Analysis on Climate change. We have used the dataset available on data.world(Link provided below). We have applied BERT to do Sentiment analysis. BERT has become a new standard for Natural Language Processing (NLP). It achieved a whole new state-of-the-art on eleven NLP task, including text classification, sentiment analysis, sequence labeling, question answering, and many more<br />
<br />
[https://data.world/crowdflower/sentiment-of-climate-change Dataset]<br />
<br />
[https://drive.google.com/file/d/12KeZXZjzeHpFwuuWA3PKW0rUo_feomkI/view?usp=sharing Instruction for using Google Colab and download dataset]<br />
<br />
[https://drive.google.com/file/d/1rZqxIo3pTKY99n9O_RaSiAiF3c2MQnAJ/view?usp=sharing Steps to do Sentiment Analysis]<br />
<br />
[https://colab.research.google.com/drive/18r3qvyJhNJ4gkNAB9dENGUQUBM2L-P4s?usp=sharing Link to Notebook]<br />
<br />
</div> <!-- autonum --></div>Jnewporthttps://docs.deepsense.ca/index.php?title=Training_Projects&diff=544Training Projects2021-05-31T18:24:37Z<p>Jnewport: /* 1. Object Detection */</p>
<hr />
<div><div class="noautonum"><br />
<br />
DeepSense has compiled a few data sets for students, and others interested in the ocean and AI, so they can have the opportunity to complete AI projects independently. We hope participants can learn about a specific type of ocean related data, and experience an explicit AI project. It is expected that the participants work on the project alone, but we have provided some guidance that includes notebooks, data, outputs and models to try to improve upon. <br />
<br />
We have found that the data cleaning step can take a long time, so our hope is that these datasets will be reasonably clean, allowing the participants to explore ocean AI. <br />
<br />
== 1. Object Detection ==<br />
<br />
We used the google open images database to obtain approximately 650 images of starfish. The images were already separated into train, test and validation sets. The metadata linked below is only for the starfish images, not for the entire dataset. The metadata includes coordinates for bounding boxes around the starfish. <br />
<br />
[https://drive.google.com/drive/folders/19Ti_XlUuj4vyo3hlBUMJ0dwszrzR58lB?usp=sharing Google Drive Directory]<br />
<br />
This contains the files:<br />
<br />
[https://drive.google.com/file/d/1r_2DNOF2WSFXdJLp7mditL5Wp4wbpOyR/view?usp=sharing Starfish Dataset]<br />
<br />
[https://drive.google.com/file/d/1gMCKW9Ih1wCVC9a_XP3yI97OCs1k46Hs/view?usp=sharing Training metadata]<br />
<br />
[https://drive.google.com/file/d/1TwCdyNJT_2Lzn0UBaDUax68oFwsY1WZm/view?usp=sharing Test metadata]<br />
<br />
[https://drive.google.com/file/d/1uI7gxRlLddfQE7VUm1-vGJE_1g1wvPd3/view?usp=sharing Validation metadata]<br />
<br />
If you want to download other categories of images from the open images database, you can do so by following the instructions here:<br />
<br />
[https://drive.google.com/file/d/1q55Q3wHfeRwD_gkuMMrRZcT_Vbtz1qG7/view?usp=sharing Download Instructions]<br />
<br />
After you have the datasets, you can download and install YOLO v4 using the following instructions:<br />
<br />
[https://drive.google.com/file/d/1OYYD7hcid5BNlR-4bvqZahh0arVO4dtG/view?usp=sharing Installation Instructions]<br />
<br />
[https://drive.google.com/file/d/1PQb76ttuHkbVr5TGW_aReCu9mlVwlZq4/view?usp=sharing Configuration Instructions]<br />
<br />
[https://drive.google.com/file/d/1mvYdpKCIvpeLoIBhyd_EPviNBRMTbkH4/view?usp=sharing Metadata Conversion Script]<br />
<br />
== 2. Regression ==<br />
<br />
The buoy collects environment measurements including wind speed and direction, surface temperature, current speed, wave height, and peak wave period. This wind and wave data are used to decide if conditions allow the safe transfer of pilots and passage of vessels, as they require a minimum depth of water which may not be met if the waves are too large. The current Red Shoal Buoy is under maintenance. Such a duration without accurate environmental measurements would significantly impair the ability to ensure the safe guidance of vessels. In this project, we are trying to predict the environment measurements of the buoy which is under maintenance using the values of other active operational buoy so that the authorities could allow the safe passage of vessels.<br />
<br />
Predicting the values of one buoy using the parameters of another buoy. In this project, we are using the dataset of Mouth of Placentia Bay Buoy, Pilot Boarding Station / Red Island Shoal Buoy, Placentia Bay: Ragged Islands – KLUMI( Land station) which are located in Newfoundland and Labrador. <br />
<br />
'''Datasets'''<br />
<br />
[https://drive.google.com/file/d/1rinZ5XgK_f64NxtV-Vy0ZWCEp-oWajxJ/view?usp=sharing Mouth of Placentia Bay Buoy]<br />
<br />
[https://drive.google.com/file/d/1WgaF7Q_jej-YlrAJEfuGnqhJ5l3xAXbJ/view?usp=sharing Pilot Boarding Station / Red Island Shoal Buoy]<br />
<br />
[https://drive.google.com/file/d/14GB14C2Qls715NKbTkhZocINw5cvJxQG/view?usp=sharing Placentia Bay: Ragged Islands – KLUMI( Land station)]<br />
<br />
The dataset available here is till April 19, 2021. You can get the latest dataset from [https://www.smartatlantic.ca smartatlantic].<br />
<br />
[https://drive.google.com/file/d/1C_Df3p3DPXK8tC3Y6nWYlyq6ZNsJd3_b/view?usp=sharing '''Download Instructions''']<br />
<br />
[https://drive.google.com/file/d/1EpKrdx1FiFezAqwscgj4SSxgsqfvzU9e/view?usp=sharing '''Data Dictionary''']<br />
<br />
[https://drive.google.com/file/d/1FZkyjqiBx51gYXjYjcpne-tG6G5L0lK8/view?usp=sharing '''Visual Representation of data''']<br />
<br />
[https://drive.google.com/file/d/1t1DqNL6LZOiHzbrkxIZbzQlD-PVVaoqG/view?usp=sharing '''Buoy Location''']<br />
<br />
<br />
''You can find the instructions to clean the dataset, merging of files and training the ML models from the below link:''<br />
<br />
[https://drive.google.com/file/d/1sr7QFHAycJ7KRBXuPgxgFDLxvHQvyngz/view?usp=sharing '''Instructions for Cleaning/Merging/Training''']<br />
<br />
<br />
''We have implemented the code on IBM Watson Cloud and encourage you to use this to get the experience of Cloud. Below link will provide you the instructions for using the IBM Watson Cloud. The Lite version of this cloud is free and provide you 25GB storage which is enough for this project.''<br />
<br />
[https://drive.google.com/file/d/1ns7OrUy4JURdv5QwD3vjGrZszU0DQ3wa/view?usp=sharing '''Instructions for using IBM Watson Cloud''']<br />
<br />
<br />
''We have created notebooks with the code for your reference in the below link.''<br />
<br />
[https://drive.google.com/file/d/1WpFhgKkoq19k4XR0dxp1wxgXFsPPnMgx/view?usp=sharing '''Links for Notebooks''']<br />
<br />
<br />
'''REFERENCES'''<br />
<br />
[https://www.thejot.net/article-preview/?show_article_preview=1193 A MACHINE LEARNING REDUNDANCY MODEL FOR THE HERRING COVE SMART BUOY]<br />
<br />
==3. NLP==<br />
NLP project is related to the Sentiment Analysis on Climate change. We have used the dataset available on data.world(Link provided below). We have applied BERT to do Sentiment analysis. BERT has become a new standard for Natural Language Processing (NLP). It achieved a whole new state-of-the-art on eleven NLP task, including text classification, sentiment analysis, sequence labeling, question answering, and many more<br />
<br />
[https://data.world/crowdflower/sentiment-of-climate-change Dataset]<br />
<br />
[https://drive.google.com/file/d/12KeZXZjzeHpFwuuWA3PKW0rUo_feomkI/view?usp=sharing Instruction for using Google Colab and download dataset]<br />
<br />
[https://drive.google.com/file/d/1rZqxIo3pTKY99n9O_RaSiAiF3c2MQnAJ/view?usp=sharing Steps to do Sentiment Analysis]<br />
<br />
[https://colab.research.google.com/drive/18r3qvyJhNJ4gkNAB9dENGUQUBM2L-P4s?usp=sharing Link to Notebook]<br />
<br />
</div> <!-- autonum --></div>Jnewporthttps://docs.deepsense.ca/index.php?title=Training_Projects&diff=543Training Projects2021-05-31T18:23:05Z<p>Jnewport: /* 1. Object Detection */</p>
<hr />
<div><div class="noautonum"><br />
<br />
DeepSense has compiled a few data sets for students, and others interested in the ocean and AI, so they can have the opportunity to complete AI projects independently. We hope participants can learn about a specific type of ocean related data, and experience an explicit AI project. It is expected that the participants work on the project alone, but we have provided some guidance that includes notebooks, data, outputs and models to try to improve upon. <br />
<br />
We have found that the data cleaning step can take a long time, so our hope is that these datasets will be reasonably clean, allowing the participants to explore ocean AI. <br />
<br />
== 1. Object Detection ==<br />
<br />
We used the google open images database to obtain approximately 650 images of starfish. The images were already separated into train, test and validation sets. The metadata linked below is only for the starfish images, not for the entire dataset. The metadata includes coordinates for bounding boxes around the starfish. <br />
<br />
[https://drive.google.com/drive/folders/19Ti_XlUuj4vyo3hlBUMJ0dwszrzR58lB?usp=sharing Google Drive Directory]<br />
<br />
This contains the files:<br />
<br />
[https://drive.google.com/file/d/1r_2DNOF2WSFXdJLp7mditL5Wp4wbpOyR/view?usp=sharing Starfish Dataset]<br />
<br />
[https://drive.google.com/file/d/1gMCKW9Ih1wCVC9a_XP3yI97OCs1k46Hs/view?usp=sharing Training metadata]<br />
<br />
[https://drive.google.com/file/d/1TwCdyNJT_2Lzn0UBaDUax68oFwsY1WZm/view?usp=sharing Test metadata]<br />
<br />
[https://drive.google.com/file/d/1uI7gxRlLddfQE7VUm1-vGJE_1g1wvPd3/view?usp=sharing Validation metadata]<br />
<br />
If you want to download other categories of images from the open images database, you can do so by following the instructions here:<br />
<br />
[https://drive.google.com/file/d/1q55Q3wHfeRwD_gkuMMrRZcT_Vbtz1qG7/view?usp=sharing Download Instructions]<br />
<br />
[https://drive.google.com/file/d/1OYYD7hcid5BNlR-4bvqZahh0arVO4dtG/view?usp=sharing Installation Instructions]<br />
<br />
[https://drive.google.com/file/d/1PQb76ttuHkbVr5TGW_aReCu9mlVwlZq4/view?usp=sharing Configuration Instructions]<br />
<br />
[https://drive.google.com/file/d/1mvYdpKCIvpeLoIBhyd_EPviNBRMTbkH4/view?usp=sharing Metadata Conversion Script]<br />
<br />
[https://drive.google.com/file/d/1mvYdpKCIvpeLoIBhyd_EPviNBRMTbkH4/view?usp=sharing Conversion script]<br />
<br />
== 2. Regression ==<br />
<br />
The buoy collects environment measurements including wind speed and direction, surface temperature, current speed, wave height, and peak wave period. This wind and wave data are used to decide if conditions allow the safe transfer of pilots and passage of vessels, as they require a minimum depth of water which may not be met if the waves are too large. The current Red Shoal Buoy is under maintenance. Such a duration without accurate environmental measurements would significantly impair the ability to ensure the safe guidance of vessels. In this project, we are trying to predict the environment measurements of the buoy which is under maintenance using the values of other active operational buoy so that the authorities could allow the safe passage of vessels.<br />
<br />
Predicting the values of one buoy using the parameters of another buoy. In this project, we are using the dataset of Mouth of Placentia Bay Buoy, Pilot Boarding Station / Red Island Shoal Buoy, Placentia Bay: Ragged Islands – KLUMI( Land station) which are located in Newfoundland and Labrador. <br />
<br />
'''Datasets'''<br />
<br />
[https://drive.google.com/file/d/1rinZ5XgK_f64NxtV-Vy0ZWCEp-oWajxJ/view?usp=sharing Mouth of Placentia Bay Buoy]<br />
<br />
[https://drive.google.com/file/d/1WgaF7Q_jej-YlrAJEfuGnqhJ5l3xAXbJ/view?usp=sharing Pilot Boarding Station / Red Island Shoal Buoy]<br />
<br />
[https://drive.google.com/file/d/14GB14C2Qls715NKbTkhZocINw5cvJxQG/view?usp=sharing Placentia Bay: Ragged Islands – KLUMI( Land station)]<br />
<br />
The dataset available here is till April 19, 2021. You can get the latest dataset from [https://www.smartatlantic.ca smartatlantic].<br />
<br />
[https://drive.google.com/file/d/1C_Df3p3DPXK8tC3Y6nWYlyq6ZNsJd3_b/view?usp=sharing '''Download Instructions''']<br />
<br />
[https://drive.google.com/file/d/1EpKrdx1FiFezAqwscgj4SSxgsqfvzU9e/view?usp=sharing '''Data Dictionary''']<br />
<br />
[https://drive.google.com/file/d/1FZkyjqiBx51gYXjYjcpne-tG6G5L0lK8/view?usp=sharing '''Visual Representation of data''']<br />
<br />
[https://drive.google.com/file/d/1t1DqNL6LZOiHzbrkxIZbzQlD-PVVaoqG/view?usp=sharing '''Buoy Location''']<br />
<br />
<br />
''You can find the instructions to clean the dataset, merging of files and training the ML models from the below link:''<br />
<br />
[https://drive.google.com/file/d/1sr7QFHAycJ7KRBXuPgxgFDLxvHQvyngz/view?usp=sharing '''Instructions for Cleaning/Merging/Training''']<br />
<br />
<br />
''We have implemented the code on IBM Watson Cloud and encourage you to use this to get the experience of Cloud. Below link will provide you the instructions for using the IBM Watson Cloud. The Lite version of this cloud is free and provide you 25GB storage which is enough for this project.''<br />
<br />
[https://drive.google.com/file/d/1ns7OrUy4JURdv5QwD3vjGrZszU0DQ3wa/view?usp=sharing '''Instructions for using IBM Watson Cloud''']<br />
<br />
<br />
''We have created notebooks with the code for your reference in the below link.''<br />
<br />
[https://drive.google.com/file/d/1WpFhgKkoq19k4XR0dxp1wxgXFsPPnMgx/view?usp=sharing '''Links for Notebooks''']<br />
<br />
<br />
'''REFERENCES'''<br />
<br />
[https://www.thejot.net/article-preview/?show_article_preview=1193 A MACHINE LEARNING REDUNDANCY MODEL FOR THE HERRING COVE SMART BUOY]<br />
<br />
==3. NLP==<br />
NLP project is related to the Sentiment Analysis on Climate change. We have used the dataset available on data.world(Link provided below). We have applied BERT to do Sentiment analysis. BERT has become a new standard for Natural Language Processing (NLP). It achieved a whole new state-of-the-art on eleven NLP task, including text classification, sentiment analysis, sequence labeling, question answering, and many more<br />
<br />
[https://data.world/crowdflower/sentiment-of-climate-change Dataset]<br />
<br />
[https://drive.google.com/file/d/12KeZXZjzeHpFwuuWA3PKW0rUo_feomkI/view?usp=sharing Instruction for using Google Colab and download dataset]<br />
<br />
[https://drive.google.com/file/d/1rZqxIo3pTKY99n9O_RaSiAiF3c2MQnAJ/view?usp=sharing Steps to do Sentiment Analysis]<br />
<br />
[https://colab.research.google.com/drive/18r3qvyJhNJ4gkNAB9dENGUQUBM2L-P4s?usp=sharing Link to Notebook]<br />
<br />
</div> <!-- autonum --></div>Jnewporthttps://docs.deepsense.ca/index.php?title=Acknowledging_DeepSense&diff=529Acknowledging DeepSense2021-05-19T18:52:34Z<p>Jnewport: </p>
<hr />
<div>Please acknowledge DeepSense and your industry partners when publishing results that used DeepSense resources.<br />
<br />
DeepSense resources include computing hardware, computing software, and staff expertise.<br />
<br />
<br />
The exact wording of the acknowledgement may vary, depending on the type of submission. For example, a short paper may only require a short acknowledgement. Here is an example:<br />
<br />
<blockquote>"This research was enabled in part by support provided by '''industry partner (web address)''', and computations were performed on the DeepSense (deepsense.ca) high-performance computing platform."</blockquote><br />
<br />
On the other hand, a thesis is generally longer, and will have a more complete acknowledgement section. Here is an example for a project that was funded through mitacs:<br />
<br />
<blockquote>"This work was supported by Mitacs through the Mitacs Accelerate program. In addition to Mitacs, this research was enabled in part by support provided by '''industry partner (web address)''' in the form of the aforementioned Mitacs Accelerate program. Computations were performed on the DeepSense (deepsense.ca) high-performance computing platform. DeepSense is funded by ACOA, the Province of Nova Scotia, the Centre for Ocean Ventures and Entrepreneurship (COVE), IBM Canada Ltd. and the Ocean Frontier Institute (OFI)."</blockquote><br />
<br />
<br />
We would appreciate references to any published acknowledgments so they can be included in reports on the impact of DeepSense.<br />
<br />
DeepSense can also help advertise your success and collaborate on media releases. Please contact us if you have news to share about research using DeepSense resources.</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Acknowledging_DeepSense&diff=528Acknowledging DeepSense2021-05-19T18:52:13Z<p>Jnewport: </p>
<hr />
<div>Please acknowledge DeepSense and your industry partners when publishing results that used DeepSense resources.<br />
<br />
DeepSense resources include computing hardware, computing software, and staff expertise.<br />
<br />
<br />
The exact wording of the acknowledgement may vary, depending on the type of submission. For example, a short paper may only require a short acknowledgement. Here is an example:<br />
<br />
<blockquote>"This research was enabled in part by support provided by '''industry parter(web address)''', and computations were performed on the DeepSense (deepsense.ca) high-performance computing platform."</blockquote><br />
<br />
On the other hand, a thesis is generally longer, and will have a more complete acknowledgement section. Here is an example for a project that was funded through mitacs:<br />
<br />
<blockquote>"This work was supported by Mitacs through the Mitacs Accelerate program. In addition to Mitacs, this research was enabled in part by support provided by '''industry parter(web address)''' in the form of the aforementioned Mitacs Accelerate program. Computations were performed on the DeepSense (deepsense.ca) high-performance computing platform. DeepSense is funded by ACOA, the Province of Nova Scotia, the Centre for Ocean Ventures and Entrepreneurship (COVE), IBM Canada Ltd. and the Ocean Frontier Institute (OFI)."</blockquote><br />
<br />
<br />
We would appreciate references to any published acknowledgments so they can be included in reports on the impact of DeepSense.<br />
<br />
DeepSense can also help advertise your success and collaborate on media releases. Please contact us if you have news to share about research using DeepSense resources.</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Acknowledging_DeepSense&diff=527Acknowledging DeepSense2021-05-17T18:20:07Z<p>Jnewport: </p>
<hr />
<div>Please acknowledge DeepSense and your industry partners when publishing results that used DeepSense resources.<br />
<br />
DeepSense resources include computing hardware, computing software, and staff expertise.<br />
<br />
<br />
The exact wording of the acknowledgement may vary, depending on the type of submission. For example, a short paper may only require a short acknowledgement. Here is an example:<br />
<br />
<blockquote>"This research was enabled in part by support provided by '''(industry parter)(web address)''', and computations were performed on the DeepSense (deepsense.ca) high-performance computing platform."</blockquote><br />
<br />
On the other hand, a thesis is generally longer, and will have a more complete acknowledgement section. Here is an example for a project that was funded through mitacs:<br />
<br />
<blockquote>"This work was supported by Mitacs through the Mitacs Accelerate program. In addition to Mitacs, this research was enabled in part by support provided by '''(industry parter)(web address)''' in the form of the aforementioned Mitacs Accelerate program. Computations were performed on the DeepSense (deepsense.ca) high-performance computing platform. DeepSense is funded by ACOA, the Province of Nova Scotia, the Centre for Ocean Ventures and Entrepreneurship (COVE), IBM Canada Ltd. and the Ocean Frontier Institute (OFI)."</blockquote><br />
<br />
<br />
We would appreciate references to any published acknowledgments so they can be included in reports on the impact of DeepSense.<br />
<br />
DeepSense can also help advertise your success and collaborate on media releases. Please contact us if you have news to share about research using DeepSense resources.</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Acknowledging_DeepSense&diff=526Acknowledging DeepSense2021-05-17T18:19:29Z<p>Jnewport: </p>
<hr />
<div>Please acknowledge DeepSense and your industry partners when publishing results that used DeepSense resources.<br />
<br />
DeepSense resources include computing hardware, computing software, and staff expertise.<br />
<br />
<br />
The exact wording of the acknowledgement may vary, depending on the type of submission. For example, a short paper may only require a short acknowledgement. Here is an example:<br />
<br />
<blockquote>"This research was enabled in part by support provided by (industry parter)(web address), and computations were performed on the DeepSense (deepsense.ca) high-performance computing platform."</blockquote><br />
<br />
On the other hand, a thesis is generally longer, and will have a more complete acknowledgement section. Here is an example for a project that was funded through mitacs:<br />
<br />
<blockquote>"This work was supported by Mitacs through the Mitacs Accelerate program. In addition to Mitacs, this research was enabled in part by support provided by '''COMPANY''' in the form of the aforementioned Mitacs Accelerate program. Computations were performed on the DeepSense (deepsense.ca) high-performance computing platform. DeepSense is funded by ACOA, the Province of Nova Scotia, the Centre for Ocean Ventures and Entrepreneurship (COVE), IBM Canada Ltd. and the Ocean Frontier Institute (OFI)."</blockquote><br />
<br />
<br />
We would appreciate references to any published acknowledgments so they can be included in reports on the impact of DeepSense.<br />
<br />
DeepSense can also help advertise your success and collaborate on media releases. Please contact us if you have news to share about research using DeepSense resources.</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Acknowledging_DeepSense&diff=525Acknowledging DeepSense2021-05-17T18:18:17Z<p>Jnewport: </p>
<hr />
<div>Please acknowledge DeepSense and your industry partners when publishing results that used DeepSense resources.<br />
<br />
DeepSense resources include computing hardware, computing software, and staff expertise.<br />
<br />
<br />
The exact wording of the acknowledgement may vary, depending on the type of submission. For example, a short paper may only require a short acknowledgement. Here is an example:<br />
<br />
"This research was enabled in part by support provided by (industry parter)(web address), and computations were performed on the DeepSense (deepsense.ca) high-performance computing platform."<br />
<br />
On the other hand, a thesis is generally longer, and will have a more complete acknowledgement section. Here is an example for a project that was funded through mitacs:<br />
<br />
"This work was supported by Mitacs through the Mitacs Accelerate program. In addition to Mitacs, this research was enabled in part by support provided by '''COMPANY''' in the form of the aforementioned Mitacs Accelerate program. Computations were performed on the DeepSense (deepsense.ca) high-performance computing platform. DeepSense is funded by ACOA, the Province of Nova Scotia, the Centre for Ocean Ventures and Entrepreneurship (COVE), IBM Canada Ltd. and the Ocean Frontier Institute (OFI)."<br />
<br />
<br />
We would appreciate references to any published acknowledgments so they can be included in reports on the impact of DeepSense.<br />
<br />
DeepSense can also help advertise your success and collaborate on media releases. Please contact us if you have news to share about research using DeepSense resources.</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Acknowledging_DeepSense&diff=524Acknowledging DeepSense2021-05-17T18:17:03Z<p>Jnewport: </p>
<hr />
<div>Please acknowledge DeepSense and your industry partners when publishing results that used DeepSense resources.<br />
<br />
DeepSense resources include computing hardware, computing software, and staff expertise.<br />
<br />
<br />
The exact wording of the acknowledgement may vary, depending on the type of submission. For example, a short paper may only require a short acknowledgement. Here is an example:<br />
<br />
"This research was enabled in part by support provided by (industry parter)(web address) and DeepSense (www.deepsense.ca)."<br />
<br />
On the other hand, a thesis is generally longer, and will have a more complete acknowledgement section. Here is an example for a project that was funded through mitacs:<br />
<br />
"This work was supported by Mitacs through the Mitacs Accelerate program. In addition to Mitacs, this research was enabled in part by support provided by '''COMPANY''' in the form of the aforementioned Mitacs Accelerate program. Computations were performed on the DeepSense (deepsense.ca) high-performance computing platform. DeepSense is funded by ACOA, the Province of Nova Scotia, the Centre for Ocean Ventures and Entrepreneurship (COVE), IBM Canada Ltd. and the Ocean Frontier Institute (OFI)."<br />
<br />
<br />
We would appreciate references to any published acknowledgments so they can be included in reports on the impact of DeepSense.<br />
<br />
DeepSense can also help advertise your success and collaborate on media releases. Please contact us if you have news to share about research using DeepSense resources.</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Training_Projects&diff=511Training Projects2021-04-22T17:40:39Z<p>Jnewport: /* Object Detection */</p>
<hr />
<div><br />
DeepSense has compiled a few data sets for students, and others interested in the ocean and AI, so they can have the opportunity to complete AI projects independently. We hope participants can learn about a specific type of ocean related data, and experience an explicit AI project. It is expected that the participants work on the project alone, but we have provided some guidance that includes notebooks, data, outputs and models to try to improve upon. <br />
<br />
We have found that the data cleaning step can take a long time, so our hope is that these datasets will be reasonably clean, allowing the participants to explore ocean AI. <br />
<br />
== Object Detection ==<br />
<br />
We used the google open images database to obtain approximately 650 images of starfish. The images were already separated into train, test and validation sets. The metadata linked below is only for the starfish images, not for the entire dataset. The metadata includes coordinates for bounding boxes around the starfish. <br />
<br />
[https://drive.google.com/file/d/1r_2DNOF2WSFXdJLp7mditL5Wp4wbpOyR/view?usp=sharing Starfish Dataset]<br />
<br />
[https://drive.google.com/file/d/1gMCKW9Ih1wCVC9a_XP3yI97OCs1k46Hs/view?usp=sharing Training metadata]<br />
<br />
[https://drive.google.com/file/d/1TwCdyNJT_2Lzn0UBaDUax68oFwsY1WZm/view?usp=sharing Test metadata]<br />
<br />
[https://drive.google.com/file/d/1uI7gxRlLddfQE7VUm1-vGJE_1g1wvPd3/view?usp=sharing Validation metadata]<br />
<br />
If you want to download other categories of images from the open images database, you can do so by following the instructions here:<br />
<br />
[https://drive.google.com/file/d/1q55Q3wHfeRwD_gkuMMrRZcT_Vbtz1qG7/view?usp=sharing Download Instructions]<br />
<br />
== Regression ==<br />
<br />
Predicting the values of one buoy using the parameters of another buoy. In this project, we are using the dataset of Mouth of Placentia Bay Buoy, Pilot Boarding Station / Red Island Shoal Buoy, Placentia Bay: Ragged Islands – KLUMI( Land station) which are located in Newfoundland and Labrador. <br />
<br />
'''Datasets'''<br />
<br />
[https://drive.google.com/file/d/1rinZ5XgK_f64NxtV-Vy0ZWCEp-oWajxJ/view?usp=sharing Mouth of Placentia Bay Buoy]<br />
<br />
[https://drive.google.com/file/d/1WgaF7Q_jej-YlrAJEfuGnqhJ5l3xAXbJ/view?usp=sharing Pilot Boarding Station / Red Island Shoal Buoy]<br />
<br />
[https://drive.google.com/file/d/14GB14C2Qls715NKbTkhZocINw5cvJxQG/view?usp=sharing Placentia Bay: Ragged Islands – KLUMI( Land station)]<br />
<br />
The dataset available here is till April 19, 2021. You can get the latest dataset from [https://www.smartatlantic.ca smartatlantic].<br />
<br />
[https://drive.google.com/file/d/1C_Df3p3DPXK8tC3Y6nWYlyq6ZNsJd3_b/view?usp=sharing '''Download Instructions''']<br />
<br />
[https://drive.google.com/file/d/1EpKrdx1FiFezAqwscgj4SSxgsqfvzU9e/view?usp=sharing '''Data Dictionary''']<br />
<br />
[https://drive.google.com/file/d/1FZkyjqiBx51gYXjYjcpne-tG6G5L0lK8/view?usp=sharing '''Visual Representation of data''']<br />
<br />
[https://drive.google.com/file/d/1t1DqNL6LZOiHzbrkxIZbzQlD-PVVaoqG/view?usp=sharing '''Buoy Location''']<br />
<br />
<br />
<br />
'''REFERENCES'''<br />
<br />
[https://www.thejot.net/article-preview/?show_article_preview=1193 A MACHINE LEARNING REDUNDANCY MODEL FOR THE HERRING COVE SMART BUOY]</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Training_Projects&diff=499Training Projects2021-04-19T19:14:28Z<p>Jnewport: /* Object Detection */</p>
<hr />
<div>DeepSense has compiled a few data sets for students, and others interested in the ocean and AI, so they can have the opportunity to complete AI projects independently. We hope participants can learn about a specific type of ocean related data, and experience an explicit AI project. It is expected that the participants work on the project alone, but we have provided some guidance that includes notebooks, data, outputs and models to try to improve upon. <br />
<br />
We have found that the data cleaning step can take a long time, so our hope is that these datasets will be reasonably clean, allowing the participants to explore ocean AI. <br />
<br />
== Object Detection ==<br />
<br />
We used the google open images database to obtain approximately 650 images of starfish. The metadata includes coordinates for bounding boxes around the starfish. <br />
<br />
[https://drive.google.com/file/d/1r_2DNOF2WSFXdJLp7mditL5Wp4wbpOyR/view?usp=sharing Dataset]<br />
<br />
[https://drive.google.com/file/d/1q55Q3wHfeRwD_gkuMMrRZcT_Vbtz1qG7/view?usp=sharing Download Instructions]</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Training_Projects&diff=498Training Projects2021-04-19T19:14:18Z<p>Jnewport: /* Object Detection */</p>
<hr />
<div>DeepSense has compiled a few data sets for students, and others interested in the ocean and AI, so they can have the opportunity to complete AI projects independently. We hope participants can learn about a specific type of ocean related data, and experience an explicit AI project. It is expected that the participants work on the project alone, but we have provided some guidance that includes notebooks, data, outputs and models to try to improve upon. <br />
<br />
We have found that the data cleaning step can take a long time, so our hope is that these datasets will be reasonably clean, allowing the participants to explore ocean AI. <br />
<br />
== Object Detection ==<br />
<br />
We used the google open images database to obtain approximately 650 images of starfish. The metadata includes coordinates for bounding boxes around the starfish. <br />
<br />
[https://drive.google.com/file/d/1r_2DNOF2WSFXdJLp7mditL5Wp4wbpOyR/view?usp=sharing Dataset]<br />
[https://drive.google.com/file/d/1q55Q3wHfeRwD_gkuMMrRZcT_Vbtz1qG7/view?usp=sharing Download Instructions]</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Training_Projects&diff=497Training Projects2021-04-19T19:12:40Z<p>Jnewport: Created, added content</p>
<hr />
<div>DeepSense has compiled a few data sets for students, and others interested in the ocean and AI, so they can have the opportunity to complete AI projects independently. We hope participants can learn about a specific type of ocean related data, and experience an explicit AI project. It is expected that the participants work on the project alone, but we have provided some guidance that includes notebooks, data, outputs and models to try to improve upon. <br />
<br />
We have found that the data cleaning step can take a long time, so our hope is that these datasets will be reasonably clean, allowing the participants to explore ocean AI. <br />
<br />
== Object Detection ==<br />
<br />
We used the google open images database to obtain approximately 650 images of starfish. The metadata includes coordinates for bounding boxes around the starfish. <br />
<br />
[[ https://drive.google.com/file/d/1r_2DNOF2WSFXdJLp7mditL5Wp4wbpOyR/view?usp=sharing Dataset]]</div>Jnewporthttps://docs.deepsense.ca/index.php?title=External_Data_Sources&diff=496External Data Sources2021-04-06T23:30:06Z<p>Jnewport: /* Oceans Datasets */</p>
<hr />
<div>Students working on a DeepSense project will generally be given data by the industry partner. However, if you want to test out our system and don't have data yet, there are many resources available. Additionally, you may want to add other data sources to expand your data. This is a list of external data sources.<br />
<br />
== Oceans Datasets ==<br />
<br />
# [https://cioosatlantic.ca/ CIOOS ] - Canadian Integrated Ocean Observing System. This is a newly created Canadian entity. Data that is collected by others, such as academic researchers, is being collated there with meta data available for users.<br />
# [https://ooinet.oceanobservatories.org/data_access/ The Ocean Observatories Initiative ]<br />
# [https://www.dfo-mpo.gc.ca/science/data-donnees/index-eng.html DFO databases] <br />
# Bedford Institute of Oceanography (BIO) – This has many different databases. You have to request access, and can’t download the datasets, as they are too big. You have to formulate a query, and will be e-mailed the results. But, they do have a lot of historical data, some dating back 100 years.<br />
#* [https://www.dfo-mpo.gc.ca/science/data-donnees/biochem/index-eng.html BioChem ] is a national Department of Fisheries and Oceans (DFO) archive for plankton and chemical data.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/climate-climat-en.php Hydrographic ] Over 850,000 temperature-salinity profiles for the Northwest Atlantic from 1920 to January 2010. Requests for more up-to-date data may be made online by completing the Data Request Form.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/cts-en.php Coastal Time Series ] Daily temperatures from over 3000 inshore moored thermographs from the East Coast of Canada.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/ocdb-en.php Ocean Colour Database] Satellite derived (SeaWiFS) ocean colour for the Northwest Atlantic from 1997 to present.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/odi-en.php Ocean Data Inventory ] Inventory of moored current meters, thermographs and tide gauges from the East Coast of Canada, 1960 to present.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/sst-en.php Sea-surface Temperature] Satellite derived (National Oceanic and Atmospheric Administration / Advanced Very High Resolution Radiometer from Jet Propulsion Laboratory) sea-surface temperature for the Northwest Atlantic from 1982 to present.<br />
# [https://datasetsearch.research.google.com/search?query=ocean&docid=%2FVGPxzi0hEY%2B0DVvAAAAAA%3D%3D Google Datasets ]<br />
# [https://data.novascotia.ca/browse?category=Fishing+and+Aquaculture Nova Scotia Open Data ] - lots of non-ocean data sets, as well<br />
# [https://www.smartatlantic.ca/ Smart Atlantic ] - This program operates several buoys throughout Nova Scotia, Newfoundland and New Brunswick. There is range of data available from each buoy. Some buoys are more reliable than others. If you have an interest in exploring this data, talk to us. We may help brainstorm some areas of focus.<br />
# [https://www.noaa.gov/ NOAA] - The national ocean and atmospheric association has a range of areas to explore, from weather, to marine and aviation to fisheries. You may have to dig a little.<br />
# [https://scihub.copernicus.eu/ ESA ] - The European Space Agency has several satellites in orbit. Here is the most useful one.<br />
<br />
== Non-Oceans Datasets ==<br />
<br />
# [https://cloud.google.com/bigquery/public-data/ BigQuery public datasets]<br />
# [https://archive.ics.uci.edu/ml/index.php UCI ML repository]<br />
# [https://data.nasa.gov/browse NASA Open Data ]<br />
# [https://open.canada.ca/en/open-data Canadian Government ] Open Data<br />
# [https://data.novascotia.ca/browse Nova Scotia Open Data ] - lots of ocean data sets, as well<br />
# [https://www.re3data.org/ Registry of Research Data Repositories ]<br />
# [https://www.kdnuggets.com/datasets/index.html KD Nuggets ] - Links to lots of datasets</div>Jnewporthttps://docs.deepsense.ca/index.php?title=External_Data_Sources&diff=494External Data Sources2021-04-05T15:11:16Z<p>Jnewport: /* Oceans Datasets */</p>
<hr />
<div>Students working on a DeepSense project will generally be given data by the industry partner. However, if you want to test out our system and don't have data yet, there are many resources available. Additionally, you may want to add other data sources to expand your data. This is a list of external data sources.<br />
<br />
== Oceans Datasets ==<br />
<br />
# [https://cioosatlantic.ca/ CIOOS ] - Canadian Integrated Ocean Observing System. This is a newly created Canadian entity. Data that is collected by others, such as academic researchers, is being collated there with meta data available for users.<br />
# [https://ooinet.oceanobservatories.org/data_access/ The Ocean Observatories Initiative ]<br />
# Bedford Institute of Oceanography (BIO) – This has many different databases. You have to request access, and can’t download the datasets, as they are too big. You have to formulate a query, and will be e-mailed the results. But, they do have a lot of historical data, some dating back 100 years.<br />
#* [https://www.dfo-mpo.gc.ca/science/data-donnees/biochem/index-eng.html BioChem ] is a national Department of Fisheries and Oceans (DFO) archive for plankton and chemical data.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/climate-climat-en.php Hydrographic ] Over 850,000 temperature-salinity profiles for the Northwest Atlantic from 1920 to January 2010. Requests for more up-to-date data may be made online by completing the Data Request Form.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/cts-en.php Coastal Time Series ] Daily temperatures from over 3000 inshore moored thermographs from the East Coast of Canada.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/ocdb-en.php Ocean Colour Database] Satellite derived (SeaWiFS) ocean colour for the Northwest Atlantic from 1997 to present.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/odi-en.php Ocean Data Inventory ] Inventory of moored current meters, thermographs and tide gauges from the East Coast of Canada, 1960 to present.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/sst-en.php Sea-surface Temperature] Satellite derived (National Oceanic and Atmospheric Administration / Advanced Very High Resolution Radiometer from Jet Propulsion Laboratory) sea-surface temperature for the Northwest Atlantic from 1982 to present.<br />
# [https://datasetsearch.research.google.com/search?query=ocean&docid=%2FVGPxzi0hEY%2B0DVvAAAAAA%3D%3D Google Datasets ]<br />
# [https://data.novascotia.ca/browse?category=Fishing+and+Aquaculture Nova Scotia Open Data ] - lots of non-ocean data sets, as well<br />
# [https://www.smartatlantic.ca/ Smart Atlantic ] - This program operates several buoys throughout Nova Scotia, Newfoundland and New Brunswick. There is range of data available from each buoy. Some buoys are more reliable than others. If you have an interest in exploring this data, talk to us. We may help brainstorm some areas of focus.<br />
# [https://www.noaa.gov/ NOAA] - The national ocean and atmospheric association has a range of areas to explore, from weather, to marine and aviation to fisheries. You may have to dig a little.<br />
# [https://scihub.copernicus.eu/ ESA ] - The European Space Agency has several satellites in orbit. Here is the most useful one.<br />
<br />
== Non-Oceans Datasets ==<br />
<br />
# [https://cloud.google.com/bigquery/public-data/ BigQuery public datasets]<br />
# [https://archive.ics.uci.edu/ml/index.php UCI ML repository]<br />
# [https://data.nasa.gov/browse NASA Open Data ]<br />
# [https://open.canada.ca/en/open-data Canadian Government ] Open Data<br />
# [https://data.novascotia.ca/browse Nova Scotia Open Data ] - lots of ocean data sets, as well<br />
# [https://www.re3data.org/ Registry of Research Data Repositories ]<br />
# [https://www.kdnuggets.com/datasets/index.html KD Nuggets ] - Links to lots of datasets</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Paper_Prep&diff=493Paper Prep2021-03-01T19:21:17Z<p>Jnewport: </p>
<hr />
<div>Typically, the work a student does on a project is suitable for publication. The company and/or funding agency usually requires a report of the work you’ve done, so the first step is to start with that report and expand upon it as publications will typically have more technical details than your reports. <br />
Before writing the research paper, you should keep the following points in mind:<br />
<br />
'''· Once you have an idea of conferences or journal submission, notify your company partner. Ensure they have enough notice to prepare for and review your submissions.'''<br />
<br />
'''· Throughout your project consider any information you may want to include in your papers or thesis. Some information is proprietary to the company or their customers. There is a risk that the''' <br />
'''information you include could be in breach of your NDA or contracts they have signed with others.'''<br />
<br />
'''· Companies must approve the inclusion of information, especially if specific details are provided about datasets, data dictionary, source of the dataset, images of the company’s property.'''<br />
<br />
<br />
If you are inexperienced in publishing papers, the DeepSense team is here to support you. The major sections of the research paper are <br />
<br />
<br />
==Title of the Paper==<br />
The title of the paper should be centrally aligned and at the top of the paper. The format of the title should be according to the requirement of the Journal for which you are going to submit your paper. <br />
<br />
==Abstract==<br />
<br />
An abstract is a short summary of your research paper, usually about a paragraph (6-7 sentences, 150-250 words) long depends upon the Journal you are targeting. It always appears at the beginning of a manuscript acting as the point-of-entry for any given academic paper. It helps the reader to understand the essence of the quickly and prepare them to go through the detailed information, analyses, and arguments in the full paper. Content to be considered while writing the abstract <br />
<br />
*Background information for your research <br />
<br />
*Problem statement you are addressing <br />
<br />
*Why is it important to address these issues/problems? <br />
<br />
*What previous research has done so far? <br />
<br />
*Methodology you have used for the analysis <br />
<br />
*Main findings, results, or arguments <br />
<br />
*Significance of your findings <br />
<br />
==Introduction==<br />
It is an overview of the problem you are examining – including your main argument (thesis statement). It also offers a short justification regarding the importance of your problem. It also contains a brief explanation of the paper’s scope and planned method to be used in examining or solving the issue. While addressing an issue, don’t forget to include the story behind the issue, the impact of this issue on society, possible solutions to be explored in your study, and how you organize your paper. <br />
<br />
==Literature Review==<br />
This section is related to the description of the related theories that were used to explain the issue, a summary of the methodology, any major findings from the study, limitations raised regarding findings, describe a method that suits best for your own research based on what you have studied so far. <br />
<br />
==Methods==<br />
This section is related to the description of the methodology that will be used to solve the problem selected by you. The methodology may include the data collection process, data pre-processing, features extraction method, various algorithms you are using. <br />
<br />
==Results and Discussion==<br />
This section contains the major findings. You can use tables, charts, and graphical illustrations to explain the findings. You can also discuss if anything amazes you, compare your findings with previous studies, and express any limitations if your model has so that other researchers could use these findings in their research to get better results. <br />
<br />
==Conclusion and Recommendations==<br />
This section is a brief recap of the issue examined, the method used and major finding(s), briefly remind readers about the original goal of this study and what you accomplished in your research work and describe how future researchers can expand or build on your work. <br />
<br />
==Acknowledgement==<br />
Finally, don’t forget to acknowledge those who provided support to you in terms of any funding, technology or any kind of guidance.<br />
<br />
We do ask that you [[Acknowledging DeepSense | acknowledge DeepSense]] in any publications, as well as seminars or conference talks on your work.<br />
<br />
==References==<br />
Apply correct citation and formatting. The most used are <br />
<br />
*MLA Modern Language Association) style <br />
<br />
*APA (American Psychological Association) style <br />
<br />
*Harvard <br />
<br />
*Chicago <br />
<br />
Examples <br />
<br />
'''MLA''' <br />
<br />
<code>Changizi, R., et al. "Species identification reveals mislabeling of important fish products in Iran by DNA barcoding." Iranian Journal of Fisheries Sciences 12.4 (2013): 783-791.</code> <br />
<br />
'''APA''' <br />
<br />
<code>Changizi, R., Farahmand, H., Soltani, M., Asareh, R., & Ghiasvand, Z. (2013). Species identification reveals mislabeling of important fish products in Iran by DNA barcoding. Iranian Journal of Fisheries Sciences, 12(4), 783-791.</code> <br />
<br />
'''Chicago''' <br />
<br />
<code>Changizi, R., H. Farahmand, M. Soltani, R. Asareh, and Z. Ghiasvand. "Species identification reveals mislabeling of important fish products in Iran by DNA barcoding." Iranian Journal of Fisheries Sciences 12, no. 4 (2013): 783-791. </code><br />
<br />
'''Harvard''' <br />
<br />
<code>Changizi, R., Farahmand, H., Soltani, M., Asareh, R. and Ghiasvand, Z., 2013. Species identification reveals mislabeling of important fish products in Iran by DNA barcoding. Iranian Journal of Fisheries Sciences, 12(4), pp.783-791. </code><br />
<br />
<br />
<br />
==Revision of the Paper==<br />
<br />
After writing the paper, you need to revise it as it upgrades your paper by removing the unwanted errors.Following is the checklist that need to remember during the revision of the paper after writing the first draft <br />
<br />
'''Checklist:''' <br />
<br />
*Is my problem statement concise and clear? <br />
<br />
*Did I follow my outline or miss anything? <br />
<br />
*Is my paper organized in a logical way that is easy to understand? <br />
<br />
*Are all sources properly cited to ensure that I am not plagiarizing? Are all my citations accurate and in correct format? <br />
<br />
*Have I proved my thesis with strong supporting arguments? <br />
<br />
*Is there any unfinished sentences, unnecessary or repetitious words, spelling or grammatical errors? <br />
<br />
*Did I avoid using contractions? Use “cannot” instead of “can’t”, “do not” instead of “don’t”? <br />
<br />
*Avoid using phrases such as “I think”, “I guess”, “I suppose” <br />
<br />
*Did I leave a sense of completion for my readers at the end of the paper? <br />
<br />
*Did I plagiarize my paper? <br />
<br />
<br />
Re-read for grammatical errors </br><br />
<br />
<br />
==Tools==<br />
<br />
Various tools can be used to improve the quality of research paper. <br />
<br />
=== Grammar check tool ===<br />
<br />
This can be used to find and correct grammatical errors or spelling mistakes that would be helpful in producing good quality paper. <br />
<br />
===Plagiarism check tool===<br />
<br />
This can be used to ensure if there is any copied content in your paper and remind you about the citations required. <br />
<br />
===Citation generators tool=== <br />
<br />
This can be used to write the citations considering required conventions. <br />
<br />
<br />
<br />
'''Some of the examples of research paper''' <br />
<br />
*[https://www.sciencedirect.com/science/article/abs/pii/S0141113620306553 Pechenik, J. A., Chaparro, O. R., Lazarus, Z. M., Tellado, G. V., Ostapovich, E. M., & Clark, D. (2020). Impact of short-term elevated temperature stress on winter-acclimated individuals of the marine gastropod Crepidula fornicata. Marine Environmental Research, 105180] <br />
<br />
*[https://www.sciencedirect.com/science/article/abs/pii/S0141113620304608 Sun, Y., Song, Z., Zhang, H., Liu, P., & Hu, X. (2020). Seagrass vegetation affect the vertical organization of microbial communities in sediment. Marine Environmental Research, 105174.] <br />
<br />
*[https://www.sciencedirect.com/science/article/abs/pii/S096456911630134X La Manna, G., Ronchetti, F., & Sarà, G. (2016). Predicting common bottlenose dolphin habitat preference to dynamically adapt management measures from a Marine Spatial Planning perspective. Ocean & coastal management, 130, 317-327.]</div>Jnewporthttps://docs.deepsense.ca/index.php?title=External_Data_Sources&diff=491External Data Sources2021-01-21T15:10:40Z<p>Jnewport: /* Non-Oceans Datasets */</p>
<hr />
<div>Students working on a DeepSense project will generally be given data by the industry partner. However, if you want to test out our system and don't have data yet, there are many resources available. Additionally, you may want to add other data sources to expand your data. This is a list of external data sources.<br />
<br />
== Oceans Datasets ==<br />
<br />
# [https://cioosatlantic.ca/ CIOOS ] - Canadian Integrated Ocean Observing System. This is a newly created Canadian entity. Data that is collected by others, such as academic researchers, is being collated there with meta data available for users.<br />
# [https://ooinet.oceanobservatories.org/data_access/ The Ocean Observatories Initiative ]<br />
# Bedford Institute of Oceanography (BIO) – This has many different databases. You have to request access, and can’t download the datasets, as they are too big. You have to formulate a query, and will be e-mailed the results. But, they do have a lot of historical data, some dating back 100 years.<br />
#* [https://www.dfo-mpo.gc.ca/science/data-donnees/biochem/index-eng.html BioChem ] is a national Department of Fisheries and Oceans (DFO) archive for plankton and chemical data.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/climate-climat-en.php Hydrographic ] Over 850,000 temperature-salinity profiles for the Northwest Atlantic from 1920 to January 2010. Requests for more up-to-date data may be made online by completing the Data Request Form.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/cts-en.php Coastal Time Series ] Daily temperatures from over 3000 inshore moored thermographs from the East Coast of Canada.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/ocdb-en.php Ocean Colour Database] Satellite derived (SeaWiFS) ocean colour for the Northwest Atlantic from 1997 to present.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/odi-en.php Ocean Data Inventory ] Inventory of moored current meters, thermographs and tide gauges from the East Coast of Canada, 1960 to present.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/sst-en.php Sea-surface Temperature] Satellite derived (National Oceanic and Atmospheric Administration / Advanced Very High Resolution Radiometer from Jet Propulsion Laboratory) sea-surface temperature for the Northwest Atlantic from 1982 to present.<br />
# [https://datasetsearch.research.google.com/search?query=ocean&docid=%2FVGPxzi0hEY%2B0DVvAAAAAA%3D%3D Google Datasets ]<br />
# [https://data.novascotia.ca/browse?category=Fishing+and+Aquaculture Nova Scotia Open Data ] - lots of non-ocean data sets, as well<br />
# [https://www.smartatlantic.ca/ Smart Atlantic ] - This program operates several buoys throughout Nova Scotia, Newfoundland and New Brunswick. There is range of data available from each buoy. Some buoys are more reliable than others. If you have an interest in exploring this data, talk to us. We may help brainstorm some areas of focus.<br />
# [https://www.noaa.gov/ NOAA] - The national ocean and atmospheric association has a range of areas to explore, from weather, to marine and aviation to fisheries. You may have to dig a little.<br />
# [https://scihub.copernicus.eu/ ESA ] - The European Space Agency has several satellites in orbit. Here is the most useful one.<br />
# [https://www.sciencenewsforstudents.org/article/explainer-what-are-lidar-radar-and-sonar Lidar, Sonar and Radar ]<br />
<br />
== Non-Oceans Datasets ==<br />
<br />
# [https://cloud.google.com/bigquery/public-data/ BigQuery public datasets]<br />
# [https://archive.ics.uci.edu/ml/index.php UCI ML repository]<br />
# [https://data.nasa.gov/browse NASA Open Data ]<br />
# [https://open.canada.ca/en/open-data Canadian Government ] Open Data<br />
# [https://data.novascotia.ca/browse Nova Scotia Open Data ] - lots of ocean data sets, as well<br />
# [https://www.re3data.org/ Registry of Research Data Repositories ]<br />
# [https://www.kdnuggets.com/datasets/index.html KD Nuggets ] - Links to lots of datasets</div>Jnewporthttps://docs.deepsense.ca/index.php?title=External_Data_Sources&diff=490External Data Sources2021-01-19T16:46:52Z<p>Jnewport: /* Non-Oceans Datasets */</p>
<hr />
<div>Students working on a DeepSense project will generally be given data by the industry partner. However, if you want to test out our system and don't have data yet, there are many resources available. Additionally, you may want to add other data sources to expand your data. This is a list of external data sources.<br />
<br />
== Oceans Datasets ==<br />
<br />
# [https://cioosatlantic.ca/ CIOOS ] - Canadian Integrated Ocean Observing System. This is a newly created Canadian entity. Data that is collected by others, such as academic researchers, is being collated there with meta data available for users.<br />
# [https://ooinet.oceanobservatories.org/data_access/ The Ocean Observatories Initiative ]<br />
# Bedford Institute of Oceanography (BIO) – This has many different databases. You have to request access, and can’t download the datasets, as they are too big. You have to formulate a query, and will be e-mailed the results. But, they do have a lot of historical data, some dating back 100 years.<br />
#* [https://www.dfo-mpo.gc.ca/science/data-donnees/biochem/index-eng.html BioChem ] is a national Department of Fisheries and Oceans (DFO) archive for plankton and chemical data.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/climate-climat-en.php Hydrographic ] Over 850,000 temperature-salinity profiles for the Northwest Atlantic from 1920 to January 2010. Requests for more up-to-date data may be made online by completing the Data Request Form.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/cts-en.php Coastal Time Series ] Daily temperatures from over 3000 inshore moored thermographs from the East Coast of Canada.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/ocdb-en.php Ocean Colour Database] Satellite derived (SeaWiFS) ocean colour for the Northwest Atlantic from 1997 to present.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/odi-en.php Ocean Data Inventory ] Inventory of moored current meters, thermographs and tide gauges from the East Coast of Canada, 1960 to present.<br />
#* [https://www.bio.gc.ca/science/data-donnees/base/data-donnees/sst-en.php Sea-surface Temperature] Satellite derived (National Oceanic and Atmospheric Administration / Advanced Very High Resolution Radiometer from Jet Propulsion Laboratory) sea-surface temperature for the Northwest Atlantic from 1982 to present.<br />
# [https://datasetsearch.research.google.com/search?query=ocean&docid=%2FVGPxzi0hEY%2B0DVvAAAAAA%3D%3D Google Datasets ]<br />
# [https://data.novascotia.ca/browse?category=Fishing+and+Aquaculture Nova Scotia Open Data ] - lots of non-ocean data sets, as well<br />
# [https://www.smartatlantic.ca/ Smart Atlantic ] - This program operates several buoys throughout Nova Scotia, Newfoundland and New Brunswick. There is range of data available from each buoy. Some buoys are more reliable than others. If you have an interest in exploring this data, talk to us. We may help brainstorm some areas of focus.<br />
# [https://www.noaa.gov/ NOAA] - The national ocean and atmospheric association has a range of areas to explore, from weather, to marine and aviation to fisheries. You may have to dig a little.<br />
# [https://scihub.copernicus.eu/ ESA ] - The European Space Agency has several satellites in orbit. Here is the most useful one.<br />
# [https://www.sciencenewsforstudents.org/article/explainer-what-are-lidar-radar-and-sonar Lidar, Sonar and Radar ]<br />
<br />
== Non-Oceans Datasets ==<br />
<br />
# [https://cloud.google.com/bigquery/public-data/ BigQuery public datasets]<br />
# [https://data.nasa.gov/browse NASA Open Data ]<br />
# [https://open.canada.ca/en/open-data Canadian Government ] Open Data<br />
# [https://data.novascotia.ca/browse Nova Scotia Open Data ] - lots of ocean data sets, as well<br />
# [https://www.re3data.org/ Registry of Research Data Repositories ]<br />
# [https://www.kdnuggets.com/datasets/index.html KD Nuggets ] - Links to lots of datasets</div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Sidebar&diff=480MediaWiki:Sidebar2020-12-18T16:57:02Z<p>Jnewport: </p>
<hr />
<div><br />
* navigation<br />
** mainpage|mainpage-description<br />
* DeepSense HPC Platform<br />
** mainpage | Cluster Status<br />
** Resources | Resources<br />
* Getting Support<br />
** Contact information | Contact - Support email<br />
* Getting Started <br />
** Requesting access | Requesting access<br />
** Video Tutorials | Video Tutorials<br />
** Accessing Systems | Accessing Systems<br />
*** VPN Setup | VPN Setup<br />
*** SSH client setup | SSH client setup<br />
*** Basic Linux | Basic Linux<br />
*** Glossary | Glossary for Clusters<br />
*** Info for first time cluster users | Intro to Clusters<br />
** LSF | Basic LSF Jobs<br />
** CWS | Conductor with Spark<br />
** Visualization | Visualization<br />
* Machine Learning On DeepSense<br />
** Deep Learning Frameworks | ML/DL Frameworks <br />
** Software | Software <br />
*** Available software | Available Software<br />
*** Installing Software | Installing Software<br />
*** Getting started with Deep Learning | Using Software<br />
** Running ML Jobs | Running ML jobs<br />
***Submitting Jobs | Submitting Jobs<br />
***Checking Job Status | Checking Job Status<br />
***Writing Script| Writing Script <br />
** Getting started with Jupyter Notebook | Using Jupyter Notebook<br />
** Deep Learning Tutorials | ML/DL Tutorials<br />
* Storage System<br />
** Overview of your Storage on DeepSense | Storage Overview<br />
** How to Transfer Data | How to Transfer Data<br />
** Backup Policies | Backup Policies<br />
** Quota Information and Management | Storage Quotas<br />
* FAQ<br />
** Restrictions | Restrictions<br />
** Workarounds | Workarounds<br />
** Best Practices | Best Practices<br />
*** Your Accounts | Your Accounts<br />
*** Data Storage | Data Storage<br />
*** LSF Jobs | LSF Jobs<br />
* Writing Tips<br />
** Mitacs Accelerate Proposals | Mitacs Accelerate Proposals<br />
** Paper Prep | Paper Prep<br />
* DeepSense<br />
** https://deepsense.ca | DeepSense home page<br />
** Acknowledging DeepSense | Acknowledging DeepSense<br />
** Terms of Use | Terms of use<br />
* Additional Resources<br />
** Related Links | Related Links<br />
** External Links | External Links<br />
** External Data Sources | External Data Sources</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Data_Storage&diff=479Data Storage2020-12-18T15:50:33Z<p>Jnewport: </p>
<hr />
<div><br />
Here we outline some best practices for data storage on the DeepSense platform.<br />
<br />
== Proper use of Filesystems ==<br />
<br />
Storing your data in the proper location is important as it will help everyone on your project locate and access the data. <br />
<br />
Your home directory should be used for your personal scripts, or test code. Since only you have access to your home directory, it is not a good place to store your data. Instead, store your data in the data filesystem where everyone in your project has access to it.<br />
<br />
The scratch filesystem is meant for temporary data only. It is not backed up. Again, anyone in your project group will have access to this filesystem.<br />
<br />
== Best Practices ==<br />
<br />
* Check your space usage regularly to make sure you are below your quota<br />
* Small amounts of data may be transferred via the login nodes, using ''scp'' or ''rsync''<br />
* Medium amounts of data '''must''' be transferred using the protocol nodes<br />
* Large amounts of data '''can''' be transferred via the protocol nodes, using samba. Alternatively, you can bring an external hard drive to us and we can directly transfer your data<br />
* Regularly remove temporary data stored in the scratch filesystem, as this will be used for vast amounts of temporary data<br />
* Data no longer used should be removed, as DeepSense is not intended for long term storage<br />
* When your project/account are closed, please remove all data<br />
* To help us avoid restricting the storage policies, please use the space conscientiously</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Video_Tutorials&diff=478Video Tutorials2020-12-18T15:37:33Z<p>Jnewport: /* Installing Software */</p>
<hr />
<div>DeepSense has created a series of videos to help guide our users through some of the technical onboarding activities and help address some challenges they may face interfacing with the computing cluster throughout their project. As always, feel free to send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]) if you need any help.<br />
<br />
== Getting Started on a Mac ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. <br />
<br />
=== Installing VPN ===<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
{{#ev:youtube|_D-vJLmKogY}}<br />
<br />
We have confirmed this works for OSX 12 (Sierra) and newer. If you have an older operating system, it may not work.<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|I8BHxJDjdIk}}<br />
<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to [[How to Transfer Data| transfer some data]]. This video will demonstrate how.<br />
<br />
{{#ev:youtube|A_-93pxhQbY}} <br />
<br />
<br />
== Getting Started using Windows ==<br />
<br />
Here are a few introductory videos for users who want to connect using a Windows computer. <br />
<br />
=== Installing VPN ===<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN. <br />
{{#ev:youtube|aKzrkuDo5o4}}<br />
<br />
We have verified this works with Windows 10. If you have an older operating system, it may not work.<br />
<br />
<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|GQCOx4nXN5k}}<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to [[How to Transfer Data| transfer some data]]. This video will demonstrate how.<br />
<br />
{{#ev:youtube|jjzkh77UskU}}<br />
<br />
<br />
== Introductory Videos ==<br />
<br />
We have created a few introductory video tutorials for general HPC systems and linux. <br />
<br />
=== Intro to HPC and DeepSense ===<br />
<br />
{{#ev:youtube|NgsAOKfWAN0}} <br />
<br />
=== Intro to Linux ===<br />
<br />
{{#ev:youtube|bxOjyCLSneI}}<br />
<br />
=== Intro to LSF and Conductor with Spark ===<br />
<br />
At DeepSense we use two different job schedulers. They are called LSF (Load Sharing Facility) and CwS (Conductor with Spark). You can also check out our pages on [[LSF]] and [[CWS]].<br />
<br />
{{#ev:youtube|X87TmO46iy4}} <br />
<br />
== Installing Software ==<br />
<br />
Now that you know how to get started at DeepSense, and are familiar with our systems, you will want to start coding. <br />
<br />
=== Installing Software ===<br />
<br />
Here is a video that will demonstrate what software is available on Deepsense platform, and how to install them. You can also check out our pages on [[Available software]] and [[Installing Software]].<br />
<br />
{{#ev:youtube|_-bSrdmFXX0}}<br />
<br />
=== Using Jupyter Notebook ===<br />
<br />
Many of our users use Jupyter Notebooks to test their code. We have created this video that will demonstrate how to work with Jupyter Notebooks. You can also check out our page on [[Getting started with Jupyter Notebooks]].<br />
<br />
{{#ev:youtube|rBlzi3ImgIY}}</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Getting_started_with_Jupyter_Notebooks&diff=476Getting started with Jupyter Notebooks2020-12-18T15:36:14Z<p>Jnewport: Jnewport moved page Getting started with Jupyter Notebook to Getting started with Jupyter Notebooks: pluralized notebooks</p>
<hr />
<div><div class="noautonum"><br />
<br />
<br />
<br />
<br />
== 1. Request an interactive session on a GPU compute node ==<br />
<br />
<!-- TODO: still need to set up queues to fairly share GPUs --><br />
<!-- TODO: write instructions doing this with regular LSF without an interactive session. We don't want to encourage everyone to use interactive sessions --><br />
<code>bsub -Is -gpu - bash</code><br />
<br />
== 2. Start a python2 Jupyter notebook ==<br />
<br />
=== Start the notebook ===<br />
<br />
<code>jupyter notebook --no-browser --ip=0.0.0.0</code><br />
<br />
=== Sample output ===<br />
<pre>[I 13:32:23.937 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).<br />
[C 13:32:23.937 NotebookApp] <br />
<br />
Copy/paste this URL into your browser when you connect for the first time,<br />
to login with a token:<br />
http://ds-cmgpu-04:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba<br />
</pre><br />
<br />
=== Copy the URL, host, and port ===<br />
<br />
Copy the URL but don’t paste it in your browser yet.<br />
<br />
Make a note of which compute host and port the notebook is running on (e.g. host ds-cmgpu-04 and port 8888 in this case)<br />
<br />
== 3. Port Forwarding ==<br />
<br />
In a separate terminal window from your local computer, forward your local port to the remote host.<br />
<br />
=== ssh command port forwarding ===<br />
<br />
<code> ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port></code><br />
<br />
for example, <code>ssh -l user1 login1.deepsense.ca -L 8888:ds-cmgpu-04:8888</code><br />
<br />
Note that you may need to use a different <local_port> than 8888 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8888 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8889 or another unused port.<br />
<br />
=== PuTTY port forwarding on Windows ===<br />
<br />
If you are using a PuTTY terminal from a Windows computer to access DeepSense then you can still forward ports.<br />
<br />
Before starting your session, scroll down to the option <code>Connection->SSH->Tunnels</code> in the Category pane.<br />
<br />
Enter the <code>local_port</code> in the <code>Source port</code> field. For example, <code>8888</code>.<br />
<br />
Enter <code><remote_host>:<remote_port></code> in the Destination field. For example, <code>ds-cmgpu-04:8888</code>.<br />
<br />
Press the <code>Add</code> button to add the port forwarding rule to your PuTTY session.<br />
<br />
Finally, open the session as usual.<br />
<br />
== 4. Open the desired sample notebook ==<br />
<br />
Enter the copied URL in your web browser but change the remote host name to “localhost” before pressing enter.<br />
<br />
e.g <code>http://localhost:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba</code><br />
<br />
'''Note''': On our macs, this worked in Chrome, but not in Safari. Unfortunately, there was no error reported, it simply could not connect.<br />
<br />
== 5. How to submit Jupyter Notebook jobs in batch mode==<br />
<br />
Users usually would like to submit Jupyter Notebook jobs in interactive mode using LSF. By submitting LSF jobs in interactive mode, users are assigned interactive sessions for them to open Jupyter Notebook to edit, compile, and execute their scripts. Using the features of Jupyter Notebook, users find it is very convenient to monitor the execution of their scripts for further debugging.<br />
However, users have to manually exit the interactive session after their jobs are done on the compute nodes. If users forget to exit the session, the session would be open forever and the computing resources are hold by the users.<br />
<br />
Therefore, we introduce users a way to submit Jupyter Notebook jobs in batch mode. First, users need to understand when they should submit Jupyter Notebook jobs in batch mode. If users are still at the stage of editing, debugging, or testing their Jupyter Notebook scripts, they should use the interactive mode. However, if users have finished the editing, debugging, and testing, they can submit the jobs in batch mode.<br />
<br />
For example, a user's job would need to run for a few days after he/she changes some hyperparameters, e.g. the learning rate. The user may not want to submit an interactive job and keep the Jupyter Notebook open all the time while the job is running on a compute node. The user just wants to let the compute node do everything for him/her without worrying about the opened Jupyter Notebook session. <br />
<br />
The user can submit the job using the following command:<br />
<br />
(py36-torch-env-install) [luy@ds-lg-01 ~]$ bsub -gpu - jupyter nbconvert --to notebook --execute ./gpu.conda3.py36.ipynb --output gpu.conda3.py36.output.2.ipynb<br />
Job <5743> is submitted to queue <gpu>.<br />
<br />
You would not be directed to a session on a compute node like you submit interactive jobs. You would get a message confirming you have successfully submitted the job with its jobid, 5743 in the above example.<br />
<br />
In the above command, the script name is "gpu.conda3.py36.ipynb" which is a Jupyter Notebook script. The parameter "--output gpu.conda3.py36.output.2.ipynb" would direct the output of the Jupyter Notebook script written to a new Jupyter Notebook file with name "gpu.conda3.py36.output.2.ipynb". After the job is finished, you would be able to see all the code in your original script and also the corresponding output of the cells in the output file.<br />
<br />
Remember you would need to activate your anaconda environment before submitting the job.<br />
<br />
After the job is submitted, you can check the jobs status using:<br />
bjobs -l -gpu 5743<br />
<br />
'''Enjoy Deep Learning on DeepSense!''' <br />
<br />
<br />
</div> <!-- autonum --></div>Jnewporthttps://docs.deepsense.ca/index.php?title=Getting_started_with_Jupyter_Notebook&diff=477Getting started with Jupyter Notebook2020-12-18T15:36:14Z<p>Jnewport: Jnewport moved page Getting started with Jupyter Notebook to Getting started with Jupyter Notebooks: pluralized notebooks</p>
<hr />
<div>#REDIRECT [[Getting started with Jupyter Notebooks]]</div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Sidebar&diff=475MediaWiki:Sidebar2020-12-18T15:33:14Z<p>Jnewport: </p>
<hr />
<div><br />
* navigation<br />
** mainpage|mainpage-description<br />
* DeepSense HPC Platform<br />
** mainpage | Cluster Status<br />
** Resources | Resources<br />
* Getting Support<br />
** Contact information | Contact - Support email<br />
** Video Tutorials | Video Tutorials<br />
* Getting Started <br />
** Requesting access | Requesting access<br />
** Accessing Systems | Accessing Systems<br />
*** VPN Setup | VPN Setup<br />
*** SSH client setup | SSH client setup<br />
*** Basic Linux | Basic Linux<br />
*** Glossary | Glossary for Clusters<br />
*** Info for first time cluster users | Intro to Clusters<br />
** LSF | Basic LSF Jobs<br />
** CWS | Conductor with Spark<br />
** Visualization | Visualization<br />
* Machine Learning On DeepSense<br />
** Deep Learning Frameworks | ML/DL Frameworks <br />
** Software | Software <br />
*** Available software | Available Software<br />
*** Installing Software | Installing Software<br />
*** Getting started with Deep Learning | Using Software<br />
** Running ML Jobs | Running ML jobs<br />
***Submitting Jobs | Submitting Jobs<br />
***Checking Job Status | Checking Job Status<br />
***Writing Script| Writing Script <br />
** Getting started with Jupyter Notebook | Using Jupyter Notebook<br />
** Deep Learning Tutorials | ML/DL Tutorials<br />
* Storage System<br />
** Overview of your Storage on DeepSense | Storage Overview<br />
** How to Transfer Data | How to Transfer Data<br />
** Backup Policies | Backup Policies<br />
** Quota Information and Management | Storage Quotas<br />
* FAQ<br />
** Restrictions | Restrictions<br />
** Workarounds | Workarounds<br />
** Best Practices | Best Practices<br />
*** Your Accounts | Your Accounts<br />
*** Data Storage | Data Storage<br />
*** LSF Jobs | LSF Jobs<br />
* Writing Tips<br />
** Mitacs Accelerate Proposals | Mitacs Accelerate Proposals<br />
** Paper Prep | Paper Prep<br />
* DeepSense<br />
** https://deepsense.ca | DeepSense home page<br />
** Acknowledging DeepSense | Acknowledging DeepSense<br />
** Terms of Use | Terms of use<br />
* Additional Resources<br />
** Related Links | Related Links<br />
** External Links | External Links<br />
** External Data Sources | External Data Sources</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Video_Tutorials&diff=474Video Tutorials2020-12-18T15:31:31Z<p>Jnewport: </p>
<hr />
<div>DeepSense has created a series of videos to help guide our users through some of the technical onboarding activities and help address some challenges they may face interfacing with the computing cluster throughout their project. As always, feel free to send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]) if you need any help.<br />
<br />
== Getting Started on a Mac ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. <br />
<br />
=== Installing VPN ===<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
{{#ev:youtube|_D-vJLmKogY}}<br />
<br />
We have confirmed this works for OSX 12 (Sierra) and newer. If you have an older operating system, it may not work.<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|I8BHxJDjdIk}}<br />
<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to [[How to Transfer Data| transfer some data]]. This video will demonstrate how.<br />
<br />
{{#ev:youtube|A_-93pxhQbY}} <br />
<br />
<br />
== Getting Started using Windows ==<br />
<br />
Here are a few introductory videos for users who want to connect using a Windows computer. <br />
<br />
=== Installing VPN ===<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN. <br />
{{#ev:youtube|aKzrkuDo5o4}}<br />
<br />
We have verified this works with Windows 10. If you have an older operating system, it may not work.<br />
<br />
<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|GQCOx4nXN5k}}<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to [[How to Transfer Data| transfer some data]]. This video will demonstrate how.<br />
<br />
{{#ev:youtube|jjzkh77UskU}}<br />
<br />
<br />
== Introductory Videos ==<br />
<br />
We have created a few introductory video tutorials for general HPC systems and linux. <br />
<br />
=== Intro to HPC and DeepSense ===<br />
<br />
{{#ev:youtube|NgsAOKfWAN0}} <br />
<br />
=== Intro to Linux ===<br />
<br />
{{#ev:youtube|bxOjyCLSneI}}<br />
<br />
=== Intro to LSF and Conductor with Spark ===<br />
<br />
At DeepSense we use two different job schedulers. They are called LSF (Load Sharing Facility) and CwS (Conductor with Spark). You can also check out our pages on [[LSF]] and [[CWS]].<br />
<br />
{{#ev:youtube|X87TmO46iy4}} <br />
<br />
== Installing Software ==<br />
<br />
Now that you know how to get started at DeepSense, and are familiar with our systems, you will want to start coding. <br />
<br />
=== Installing Software ===<br />
<br />
Here is a video that will demonstrate what software is available on Deepsense platform, and how to install them. You can also check out our pages on [[Available Software]] and [[Installing Software]].<br />
<br />
{{#ev:youtube|_-bSrdmFXX0}} <br />
<br />
=== Using Jupyter Notebook ===<br />
<br />
Many of our users use Jupyter Notebooks to test their code. We have created this video that will demonstrate how to work with Jupyter Notebooks. You can also check out our page on [[Getting started with Jupyter Notebooks]].<br />
<br />
{{#ev:youtube|rBlzi3ImgIY}}</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Video_Tutorials&diff=473Video Tutorials2020-12-18T15:27:55Z<p>Jnewport: </p>
<hr />
<div>DeepSense has created a series of videos to help guide our users through some of the technical onboarding activities and help address some challenges they may face interfacing with the computing cluster throughout their project. As always, feel free to send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]) if you need any help.<br />
<br />
== Getting Started on a Mac ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. <br />
<br />
=== Installing VPN ===<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
{{#ev:youtube|_D-vJLmKogY}}<br />
<br />
We have confirmed this works for OSX 12 (Sierra) and newer. If you have an older operating system, it may not work.<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|I8BHxJDjdIk}}<br />
<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to [[How to Transfer Data| transfer data]]. This video will demonstrate how.<br />
<br />
{{#ev:youtube|A_-93pxhQbY}} <br />
<br />
<br />
== Getting Started using Windows ==<br />
<br />
Here are a few introductory videos for users who want to connect using a Windows computer. <br />
<br />
=== Installing VPN ===<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN. <br />
{{#ev:youtube|aKzrkuDo5o4}}<br />
<br />
We have verified this works with Windows 10. If you have an older operating system, it may not work.<br />
<br />
<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|GQCOx4nXN5k}}<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
{{#ev:youtube|jjzkh77UskU}}<br />
<br />
<br />
== Introductory Videos ==<br />
<br />
We have created a few introductory video tutorials for general HPC systems and linux. <br />
<br />
=== Intro to HPC and DeepSense ===<br />
<br />
{{#ev:youtube|NgsAOKfWAN0}} <br />
<br />
=== Intro to Linux ===<br />
<br />
{{#ev:youtube|bxOjyCLSneI}}<br />
<br />
=== Intro to LSF and Conductor with Spark ===<br />
<br />
At DeepSense we use two different job schedulers. They are called LSF (Load Sharing Facility) and CwS (Conductor with Spark). <br />
<br />
{{#ev:youtube|X87TmO46iy4}} <br />
<br />
== Installing Software ==<br />
<br />
Now that you know how to get started at DeepSense, and are familiar with our systems, you will want to start coding. Here is a video that will demonstrate what software is available on Deepsense platform, and how to install them.<br />
<br />
=== Installing Software ===<br />
<br />
{{#ev:youtube|_-bSrdmFXX0}} <br />
<br />
=== Using Jupyter Notebook ===<br />
<br />
Many of our users use Jupyter Notebooks to test their code. We have created this video that will demonstrate how to work with Jupyter Notebooks.<br />
<br />
{{#ev:youtube|rBlzi3ImgIY}}</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Video_Tutorials&diff=472Video Tutorials2020-12-18T15:25:44Z<p>Jnewport: </p>
<hr />
<div>DeepSense has created a series of videos to help guide our users through some of the technical onboarding activities and help address some challenges they may face interfacing with the computing cluster throughout their project. As always, feel free to send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]) if you need any help.<br />
<br />
== Getting Started on a Mac ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. <br />
<br />
=== Installing VPN ===<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
{{#ev:youtube|_D-vJLmKogY}}<br />
<br />
We have confirmed this works for OSX 12 (Sierra) and newer. If you have an older operating system, it may not work.<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|I8BHxJDjdIk}}<br />
<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
{{#ev:youtube|A_-93pxhQbY}} <br />
<br />
<br />
== Getting Started using Windows ==<br />
<br />
Here are a few introductory videos for users who want to connect using a Windows computer. <br />
<br />
=== Installing VPN ===<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN. <br />
{{#ev:youtube|aKzrkuDo5o4}}<br />
<br />
We have verified this works with Windows 10. If you have an older operating system, it may not work.<br />
<br />
<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|GQCOx4nXN5k}}<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
{{#ev:youtube|jjzkh77UskU}}<br />
<br />
<br />
== Introductory Videos ==<br />
<br />
We have created a few introductory video tutorials for general HPC systems and linux. <br />
<br />
=== Intro to HPC and DeepSense ===<br />
<br />
{{#ev:youtube|NgsAOKfWAN0}} <br />
<br />
=== Intro to Linux ===<br />
<br />
{{#ev:youtube|bxOjyCLSneI}}<br />
<br />
=== Intro to LSF and Conductor with Spark ===<br />
<br />
At DeepSense we use two different job schedulers. They are called LSF (Load Sharing Facility) and CwS (Conductor with Spark). <br />
<br />
{{#ev:youtube|X87TmO46iy4}} <br />
<br />
== Installing Software ==<br />
<br />
Now that you know how to get started at DeepSense, and are familiar with our systems, you will want to start coding. Here is a video that will demonstrate what software is available on Deepsense platform, and how to install them.<br />
<br />
=== Installing Software ===<br />
<br />
{{#ev:youtube|_-bSrdmFXX0}} <br />
<br />
=== Using Jupyter Notebook ===<br />
<br />
Many of our users use Jupyter Notebooks to test their code. We have created this video that will demonstrate how to work with Jupyter Notebooks.<br />
<br />
{{#ev:youtube|rBlzi3ImgIY}}</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Video_Tutorials&diff=471Video Tutorials2020-12-18T15:25:22Z<p>Jnewport: </p>
<hr />
<div>DeepSense has created a series of videos to help guide our users through some of the technical onboarding activities and help address some challenges they may face interfacing with the computing cluster throughout their project. As always, feel free to send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]) if you need any help.<br />
<br />
== Getting Started on a Mac ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. <br />
<br />
=== Installing VPN<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
{{#ev:youtube|_D-vJLmKogY}}<br />
<br />
We have confirmed this works for OSX 12 (Sierra) and newer. If you have an older operating system, it may not work.<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|I8BHxJDjdIk}}<br />
<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
{{#ev:youtube|A_-93pxhQbY}} <br />
<br />
<br />
== Getting Started using Windows ==<br />
<br />
Here are a few introductory videos for users who want to connect using a Windows computer. <br />
<br />
=== Installing VPN ===<br />
<br />
In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN. <br />
{{#ev:youtube|aKzrkuDo5o4}}<br />
<br />
We have verified this works with Windows 10. If you have an older operating system, it may not work.<br />
<br />
<br />
<br />
=== How to Connect ===<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|GQCOx4nXN5k}}<br />
<br />
=== Transferring Data ===<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
{{#ev:youtube|jjzkh77UskU}}<br />
<br />
<br />
== Introductory Videos ==<br />
<br />
We have created a few introductory video tutorials for general HPC systems and linux. <br />
<br />
=== Intro to HPC and DeepSense ===<br />
<br />
{{#ev:youtube|NgsAOKfWAN0}} <br />
<br />
=== Intro to Linux ===<br />
<br />
{{#ev:youtube|bxOjyCLSneI}}<br />
<br />
=== Intro to LSF and Conductor with Spark ===<br />
<br />
At DeepSense we use two different job schedulers. They are called LSF (Load Sharing Facility) and CwS (Conductor with Spark). <br />
<br />
{{#ev:youtube|X87TmO46iy4}} <br />
<br />
== Installing Software ==<br />
<br />
Now that you know how to get started at DeepSense, and are familiar with our systems, you will want to start coding. Here is a video that will demonstrate what software is available on Deepsense platform, and how to install them.<br />
<br />
=== Installing Software ===<br />
<br />
{{#ev:youtube|_-bSrdmFXX0}} <br />
<br />
=== Using Jupyter Notebook ===<br />
<br />
Many of our users use Jupyter Notebooks to test their code. We have created this video that will demonstrate how to work with Jupyter Notebooks.<br />
<br />
{{#ev:youtube|rBlzi3ImgIY}}</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Video_Tutorials&diff=470Video Tutorials2020-12-18T15:18:02Z<p>Jnewport: </p>
<hr />
<div>DeepSense has created a series of videos to help guide our users through some of the technical onboarding activities and help address some challenges they may face interfacing with the computing cluster throughout their project. As always, feel free to send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]) if you need any help.<br />
<br />
== Getting Started on a Mac ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
=== Installing VPN<br />
{{#ev:youtube|_D-vJLmKogY}}<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
=== How to Connect ===<br />
{{#ev:youtube|I8BHxJDjdIk}}<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
=== Transferring Data ===<br />
{{#ev:youtube|A_-93pxhQbY}} <br />
<br />
<br />
== Getting Started using Windows ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
=== Installing VPN ===<br />
{{#ev:youtube|aKzrkuDo5o4}}<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
=== How to Connect ===<br />
{{#ev:youtube|GQCOx4nXN5k}}<br />
<br />
=== Transferring Data ===<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
{{#ev:youtube|jjzkh77UskU}}<br />
<br />
<br />
== Introductory Videos ==<br />
<br />
=== Intro to HPC and DeepSense ===<br />
<br />
{{#ev:youtube|NgsAOKfWAN0}} <br />
<br />
=== Intro to Linux ===<br />
<br />
{{#ev:youtube|bxOjyCLSneI}}<br />
<br />
=== Intro to LSF and Conductor with Spark ===<br />
<br />
{{#ev:youtube|X87TmO46iy4}} <br />
<br />
== Installing Software ==<br />
This video will demonstrate how to install Softwares available on Deepsense platform.<br />
<br />
=== Installing Software ===<br />
<br />
{{#ev:youtube|_-bSrdmFXX0}} <br />
<br />
=== Using Jupyter Notebook ===<br />
This video will demonstrate how to work with Jupyter Notebook.<br />
<br />
{{#ev:youtube|rBlzi3ImgIY}} <br />
Using Jupyter Notebook Video</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Video_Tutorials&diff=467Video Tutorials2020-12-18T15:14:46Z<p>Jnewport: </p>
<hr />
<div>DeepSense has created a series of videos to help guide our users through some of the technical onboarding activities and help address some challenges they may face interfacing with the computing cluster throughout their project. As always, feel free to send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]) if you need any help.<br />
<br />
== Getting Started on a Mac ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
=== Installing VPN<br />
{{#ev:youtube|_D-vJLmKogY}}<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
=== How to Connect ===<br />
{{#ev:youtube|I8BHxJDjdIk}}<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
=== Transferring Data ===<br />
{{#ev:youtube|A_-93pxhQbY}} <br />
<br />
<br />
== Getting Started using Windows ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
=== Installing VPN ===<br />
{{#ev:youtube|aKzrkuDo5o4}}<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
=== How to Connect ===<br />
{{#ev:youtube|GQCOx4nXN5k}}<br />
<br />
=== Transferring Data ===<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
{{#ev:youtube|jjzkh77UskU}}<br />
<br />
== Installing Software ==<br />
This video will demonstrate how to install Softwares available on Deepsense platform.<br />
<br />
Installing Software Video<br />
<br />
== Using Jupyter Notebook ==<br />
This video will demonstrate how to work with Jupyter Notebook.<br />
<br />
Using Jupyter Notebook Video</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Video_Tutorials&diff=466Video Tutorials2020-12-18T15:13:23Z<p>Jnewport: </p>
<hr />
<div>DeepSense has created a series of videos to help guide our users through some of the technical onboarding activities and help address some challenges they may face interfacing with the computing cluster throughout their project. As always, feel free to send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]) if you need any help.<br />
<br />
== Getting Started on a Mac ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
<br />
{{#ev:youtube|_D-vJLmKogY}}<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|I8BHxJDjdIk}}<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
{{#ev:youtube|A_-93pxhQbY}} <br />
<br />
<br />
== Getting Started using Windows ==<br />
<br />
Here are a few introductory videos for users who want to connect with Mac OSX. In order to connect to the DeepSense platform you need to be on the Dalhousie Campus. If you need to connect remotely, you will need to use a VPN. One can use the Dalhousie VPN, but we find it can be slow. Here are the instructions for installing the DeepSense VPN.<br />
<br />
{{#ev:youtube|aKzrkuDo5o4}}<br />
<br />
Once you have the VPN installed, you can connect to the DeepSense platform. This short video will demonstrate.<br />
<br />
{{#ev:youtube|GQCOx4nXN5k}}<br />
<br />
Once you can connect to the DeepSense platform, you'll probably want to transfer some data. <br />
<br />
{{#ev:youtube|jjzkh77UskU}}<br />
<br />
== Installing Software ==<br />
This video will demonstrate how to install Softwares available on Deepsense platform.<br />
<br />
Installing Software Video<br />
<br />
== Using Jupyter Notebook ==<br />
This video will demonstrate how to work with Jupyter Notebook.<br />
<br />
Using Jupyter Notebook Video</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Video_Tutorials&diff=463Video Tutorials2020-12-18T14:56:50Z<p>Jnewport: /* Mac */</p>
<hr />
<div>DeepSense has created a series of videos to help guide our users through some of the technical onboarding activities and help address some challenges they may face interfacing with the computing cluster throughout their project.<br />
<br />
== VPN Setup ==<br />
This video shows you how to setup your VPN to connect to the DeepSense cluster.<br />
<br />
=== Mac ===<br />
Mac Video<br />
{{#ev:youtube|_D-vJLmKogY}}<br />
<br />
=== Windows ===<br />
Windows Video<br />
<br />
== Logging into the Cluster ==<br />
This video will demonstrate how to login to the DeepSense cluster.<br />
<br />
=== Mac ===<br />
Mac Video<br />
<br />
=== Windows ===<br />
Windows Video<br />
<br />
== Installing Software ==<br />
This video will demonstrate how to install Softwares available on Deepsense platform.<br />
<br />
Installing Software Video<br />
<br />
== Using Jupyter Notebook ==<br />
This video will demonstrate how to work with Jupyter Notebook.<br />
<br />
Using Jupyter Notebook Video</div>Jnewporthttps://docs.deepsense.ca/index.php?title=How_to_Transfer_Data&diff=444How to Transfer Data2020-12-15T20:17:49Z<p>Jnewport: /* Large Transfers */</p>
<hr />
<div>There are different methods for transferring data to and from the DeepSense platform. Which method you use will depend from where you are transferring the data, as well as the size of the data.<br />
<br />
== To and From Your Personal Computer ==<br />
<br />
=== Small Transfers ===<br />
<br />
For small transfers (<5Gb), you can use the two login nodes. Since they are the primary point of access for the platform, they may be in heavy use. We do not want to overload them unnecessarily for data transfer. Please only use this for small amounts of data.<br />
<br />
The most common method for transferring data securely between machines will be <code>scp</code>. This is pretty straightforward to use, however the destination files will have the wrong permissions set. It will remove group permissions, so while you will be able to access the data, no one else in your group will be. This is fine if you are the only one working on the project.<br />
<br />
'''Example''': scp -r /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
One can also use <code>rsync</code> (see the [https://linux.die.net/man/1/rsync man page]). This has more options than <code>scp</code>, and can be used to sync files<br />
between two machines. <br />
<br />
'''Example''': rsync -azvhP /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
The rsync options above are:<br />
* a - archive mode, equal to rlptgoD (recursive, preserve links, times, permissions, group, owner, etc)<br />
* z - use compression when copying<br />
* v - verbose: list files copied<br />
* h - human readable: output numbers in human readable format<br />
* P - same as --partial --progress. Show progress while transferring, and keep partial files.<br />
<br />
'''Note''': We recommend always using the option <code>-p</code> (using <code>-a</code> also invokes <code>-p</code>). This ensures that everyone in your group should have the same permissions to the file as you do.<br />
<br />
=== Medium Size ===<br />
<br />
For medium sized transfers (between 5Gb and 100Gb), you should use the protocol nodes. They (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for data transfers. However, they are only accessible via samba. <br />
<br />
==== Mac OSX ====<br />
<br />
[[File:macSambaConnect.png|thumb|Connect via samba on OSX]]<br />
<br />
On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box (see image), type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem.<br />
<br />
If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/</code>. You can now use rsync to copy files to your project's subdirectory.<br />
<br />
'''Example''': rsync&nbsp;&#8209;rzvh&nbsp;/path/to/files/&nbsp;/Volumes/data/projectdir/<br />
<br />
==== Windows ====<br />
<br />
On windows computer, you should connect to <code>\\protocol1.deepsense.ca\data</code> or <code>\\protocol2.deepsense.ca\data</code>. To do this the first time, open a file explorer window. <br />
Right-click on This PC, and select "add a network location". In the wizard, click next and then select "Choose a custom network location" (this was the only option I saw). Highlight it, and click next. On the following screen, enter one of the addresses above, and click next. You may now enter a name for this location. Do so, and click next again. On the last screen, you should be able to look over your selections, and then click Finish. The name you chose should now be available under "This PC" in your file explorer. <br />
<br />
You may also have to change a SMB security level setting as follows (this was necessary in Windows 10):<br />
<br />
Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.<br />
<br />
==== File Permissions ====<br />
<br />
Unfortunately, samba won't preserve the proper file permissions. We find it strips the executable bit from any file that has it switched on. You can change an individual file by using <code>chmod ug+x filename</code>. If you want to change many files at once, and are unsure of how to, please send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).<br />
<br />
== Large Transfers ==<br />
<br />
For large transfers (>100Gb), We generally find it is best to put the data on an external drive. To make such arrangements, please email [mailto:support@deepsense.ca support@deepsense.ca]. We can then plug it in directly in our server room, and transfer the data for you.<br />
<br />
== From the World Wide Web ==<br />
<br />
The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].</div>Jnewporthttps://docs.deepsense.ca/index.php?title=How_to_Transfer_Data&diff=443How to Transfer Data2020-12-15T20:15:35Z<p>Jnewport: </p>
<hr />
<div>There are different methods for transferring data to and from the DeepSense platform. Which method you use will depend from where you are transferring the data, as well as the size of the data.<br />
<br />
== To and From Your Personal Computer ==<br />
<br />
=== Small Transfers ===<br />
<br />
For small transfers (<5Gb), you can use the two login nodes. Since they are the primary point of access for the platform, they may be in heavy use. We do not want to overload them unnecessarily for data transfer. Please only use this for small amounts of data.<br />
<br />
The most common method for transferring data securely between machines will be <code>scp</code>. This is pretty straightforward to use, however the destination files will have the wrong permissions set. It will remove group permissions, so while you will be able to access the data, no one else in your group will be. This is fine if you are the only one working on the project.<br />
<br />
'''Example''': scp -r /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
One can also use <code>rsync</code> (see the [https://linux.die.net/man/1/rsync man page]). This has more options than <code>scp</code>, and can be used to sync files<br />
between two machines. <br />
<br />
'''Example''': rsync -azvhP /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
The rsync options above are:<br />
* a - archive mode, equal to rlptgoD (recursive, preserve links, times, permissions, group, owner, etc)<br />
* z - use compression when copying<br />
* v - verbose: list files copied<br />
* h - human readable: output numbers in human readable format<br />
* P - same as --partial --progress. Show progress while transferring, and keep partial files.<br />
<br />
'''Note''': We recommend always using the option <code>-p</code> (using <code>-a</code> also invokes <code>-p</code>). This ensures that everyone in your group should have the same permissions to the file as you do.<br />
<br />
=== Medium Size ===<br />
<br />
For medium sized transfers (between 5Gb and 100Gb), you should use the protocol nodes. They (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for data transfers. However, they are only accessible via samba. <br />
<br />
==== Mac OSX ====<br />
<br />
[[File:macSambaConnect.png|thumb|Connect via samba on OSX]]<br />
<br />
On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box (see image), type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem.<br />
<br />
If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/</code>. You can now use rsync to copy files to your project's subdirectory.<br />
<br />
'''Example''': rsync&nbsp;&#8209;rzvh&nbsp;/path/to/files/&nbsp;/Volumes/data/projectdir/<br />
<br />
==== Windows ====<br />
<br />
On windows computer, you should connect to <code>\\protocol1.deepsense.ca\data</code> or <code>\\protocol2.deepsense.ca\data</code>. To do this the first time, open a file explorer window. <br />
Right-click on This PC, and select "add a network location". In the wizard, click next and then select "Choose a custom network location" (this was the only option I saw). Highlight it, and click next. On the following screen, enter one of the addresses above, and click next. You may now enter a name for this location. Do so, and click next again. On the last screen, you should be able to look over your selections, and then click Finish. The name you chose should now be available under "This PC" in your file explorer. <br />
<br />
You may also have to change a SMB security level setting as follows (this was necessary in Windows 10):<br />
<br />
Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.<br />
<br />
==== File Permissions ====<br />
<br />
Unfortunately, samba won't preserve the proper file permissions. We find it strips the executable bit from any file that has it switched on. You can change an individual file by using <code>chmod ug+x filename</code>. If you want to change many files at once, and are unsure of how to, please send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).<br />
<br />
== Large Transfers ==<br />
<br />
For large transfers (>100Gb), we suggest you email [mailto:support@deepsense.ca support@deepsense.ca] for assistance. If you are local, you may be able to provide an external hard drive which we can plug in directly in our server room. We also have special software that can be used to speed up large transfers.<br />
<br />
== From the World Wide Web ==<br />
<br />
The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].</div>Jnewporthttps://docs.deepsense.ca/index.php?title=How_to_Transfer_Data&diff=442How to Transfer Data2020-12-15T20:06:12Z<p>Jnewport: /* Small Transfers */</p>
<hr />
<div>There are different methods for transferring data to and from the DeepSense platform. Which method you use will depend from where you are transferring the data, as well as the size of the data.<br />
<br />
== To and From Your Personal Computer ==<br />
<br />
=== Small Transfers ===<br />
<br />
For small transfers (<5Gb), you can use the two login nodes. Since they are the primary point of access for the platform, they may be in heavy use. We do not want to overload them unnecessarily for data transfer. Please only use this for small amounts of data.<br />
<br />
The most common method for transferring data securely between machines will be <code>scp</code>. This is pretty straightforward to use, however the destination files will have the wrong permissions set. It will remove group permissions, so while you will be able to access the data, no one else in your group will be. This is fine if you are the only one working on the project.<br />
<br />
'''Example''': scp -r /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
One can also use <code>rsync</code> (see the [https://linux.die.net/man/1/rsync man page]). This has more options than <code>scp</code>, and can be used to sync files<br />
between two machines. <br />
<br />
'''Example''': rsync -azvhP /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
The rsync options above are:<br />
* a - archive mode, equal to rlptgoD (recursive, preserve links, times, permissions, group, owner, etc)<br />
* z - use compression when copying<br />
* v - verbose: list files copied<br />
* h - human readable: output numbers in human readable format<br />
* P - same as --partial --progress. Show progress while transferring, and keep partial files.<br />
<br />
'''Note''': We recommend always using the option <code>-p</code> (using <code>-a</code> also invokes <code>-p</code>). This ensures that everyone in your group should have the same permissions to the file as you do.<br />
<br />
=== Medium Size ===<br />
<br />
For medium sized transfers (between 5Gb and 100Gb), you should use the protocol nodes. They (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for data transfers. However, they are only accessible via samba. <br />
<br />
==== Mac OSX ====<br />
<br />
[[File:macSambaConnect.png|thumb|Connect via samba on OSX]]<br />
<br />
On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box (see image), type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem.<br />
<br />
If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/</code>. You can now use rsync to copy files to your project's subdirectory.<br />
<br />
'''Example''': rsync&nbsp;&#8209;rzvh&nbsp;/path/to/files/&nbsp;/Volumes/data/projectdir/<br />
<br />
==== Windows ====<br />
<br />
On windows computer, you should connect to <code>\\protocol1.deepsense.ca\data</code> or <code>\\protocol2.deepsense.ca\data</code>. To do this the first time, open a file explorer window. <br />
Right-click on This PC, and select "add a network location". In the wizard, click next and then select "Choose a custom network location" (this was the only option I saw). Highlight it, and click next. On the following screen, enter one of the addresses above, and click next. You may now enter a name for this location. Do so, and click next again. On the last screen, you should be able to look over your selections, and then click Finish. The name you chose should now be available under "This PC" in your file explorer. <br />
<br />
You may also have to change a SMB security level setting as follows (this was necessary in Windows 10):<br />
<br />
Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.<br />
<br />
=== Large Transfers ===<br />
<br />
For large transfers (>100Gb), we suggest you email [mailto:support@deepsense.ca support@deepsense.ca] for assistance. If you are local, you may be able to provide an external hard drive which we can plug in directly in our server room. We also have special software that can be used to speed up large transfers.<br />
<br />
== From the World Wide Web ==<br />
<br />
The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].</div>Jnewporthttps://docs.deepsense.ca/index.php?title=How_to_Transfer_Data&diff=441How to Transfer Data2020-12-15T20:05:57Z<p>Jnewport: /* Small Transfers */</p>
<hr />
<div>There are different methods for transferring data to and from the DeepSense platform. Which method you use will depend from where you are transferring the data, as well as the size of the data.<br />
<br />
== To and From Your Personal Computer ==<br />
<br />
=== Small Transfers ===<br />
<br />
For small transfers (<5Gb), you can use the two login nodes. Since they are the primary point of access for the platform, they may be in heavy use. We do not want to overload them unnecessarily for data transfer. Please only use this for small amounts of data.<br />
<br />
The most common method for transferring data securely between machines will be <code>scp</code>. This is pretty straightforward to use, however the destination files will have the wrong permissions set. It will remove group permissions, so while you will be able to access the data, no one else in your group will be. This is fine if you are the only one working on the project.<br />
<br />
'''Example''': scp -r /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
One can also use <code>rsync</code> (see the [https://linux.die.net/man/1/rsync man page]). This has more options than <code>scp</code>, and can be used to sync files<br />
between two machines. <br />
<br />
'''Example''': rsync -azvhP /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
The rsync options above are:<br />
* a - archive mode, equal to rlptgoD (recursive, preserve links, times, permissions, group, owner, etc)<br />
* z - use compression when copying<br />
* v - verbose: list files copied<br />
* h - human readable: output numbers in human readable format<br />
* P - same as --partial --progress. Show progress while transferring, and keep partial files.<br />
<br />
'''Note''': We recommend always using the option <code>-p</code> (Using <code>-a</code> also invokes <code>-p</code>). This ensures that everyone in your group should have the same permissions to the file as you do.<br />
<br />
=== Medium Size ===<br />
<br />
For medium sized transfers (between 5Gb and 100Gb), you should use the protocol nodes. They (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for data transfers. However, they are only accessible via samba. <br />
<br />
==== Mac OSX ====<br />
<br />
[[File:macSambaConnect.png|thumb|Connect via samba on OSX]]<br />
<br />
On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box (see image), type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem.<br />
<br />
If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/</code>. You can now use rsync to copy files to your project's subdirectory.<br />
<br />
'''Example''': rsync&nbsp;&#8209;rzvh&nbsp;/path/to/files/&nbsp;/Volumes/data/projectdir/<br />
<br />
==== Windows ====<br />
<br />
On windows computer, you should connect to <code>\\protocol1.deepsense.ca\data</code> or <code>\\protocol2.deepsense.ca\data</code>. To do this the first time, open a file explorer window. <br />
Right-click on This PC, and select "add a network location". In the wizard, click next and then select "Choose a custom network location" (this was the only option I saw). Highlight it, and click next. On the following screen, enter one of the addresses above, and click next. You may now enter a name for this location. Do so, and click next again. On the last screen, you should be able to look over your selections, and then click Finish. The name you chose should now be available under "This PC" in your file explorer. <br />
<br />
You may also have to change a SMB security level setting as follows (this was necessary in Windows 10):<br />
<br />
Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.<br />
<br />
=== Large Transfers ===<br />
<br />
For large transfers (>100Gb), we suggest you email [mailto:support@deepsense.ca support@deepsense.ca] for assistance. If you are local, you may be able to provide an external hard drive which we can plug in directly in our server room. We also have special software that can be used to speed up large transfers.<br />
<br />
== From the World Wide Web ==<br />
<br />
The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].</div>Jnewporthttps://docs.deepsense.ca/index.php?title=How_to_Transfer_Data&diff=440How to Transfer Data2020-12-15T20:05:07Z<p>Jnewport: </p>
<hr />
<div>There are different methods for transferring data to and from the DeepSense platform. Which method you use will depend from where you are transferring the data, as well as the size of the data.<br />
<br />
== To and From Your Personal Computer ==<br />
<br />
=== Small Transfers ===<br />
<br />
For small transfers (<5Gb), you can use the two login nodes. Since they are the primary point of access for the platform, they may be in heavy use. We do not want to overload them unnecessarily for data transfer. Please only use this for small amounts of data.<br />
<br />
The most common method for transferring data securely between machines will be <code>scp</code>. This is pretty straightforward to use, however the destination files will have the wrong permissions set. It will remove group permissions, so while you will be able to access the data, no one else in your group will be. This is fine if you are the only one working on the project.<br />
<br />
'''Example''': scp -r /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
One can also use <code>rsync</code> (see the [https://linux.die.net/man/1/rsync man page]). This has more options than <code>scp</code>, and can be used to sync files<br />
between two machines. <br />
<br />
'''Example''': rsync -azvhP /path/to/files/ username@login2.deepsense.ca:/data/projectdir/<br />
<br />
The rsync options above are:<br />
* a - archive mode, equal to rlptgoD (recursive, preserve links, times, permissions, group, owner, etc)<br />
* z - use compression when copying<br />
* v - verbose: list files copied<br />
* h - human readable: output numbers in human readable format<br />
* P - same as --partial --progress. Show progress while transferring, and keep partial files.<br />
<br />
'''Note''': We recommend always using the option <code>-p</code>. Using <code>-a</code> also invokes <code>-p</code>. This ensures that everyone in your group should have the same permissions to the file as you do.<br />
<br />
=== Medium Size ===<br />
<br />
For medium sized transfers (between 5Gb and 100Gb), you should use the protocol nodes. They (<code>protocol1.deepsense.ca</code>, <code>protocol2.deepsense.ca</code>) are specifically meant for data transfers. However, they are only accessible via samba. <br />
<br />
==== Mac OSX ====<br />
<br />
[[File:macSambaConnect.png|thumb|Connect via samba on OSX]]<br />
<br />
On a Mac, open finder and hit ⌘-K, or use the menu ''Go -> Connect to Server''. In the dialog box (see image), type the address for either protocol node, and you can login. This will connect you to the <code>/data</code> filesystem.<br />
<br />
If you want to use <code>rsync</code> to transfer data via the protocol nodes, you have to mount one. On a Mac, the easiest way is to connect to the protocol node as in the previous paragraph. This will mount it at <code>/Volumes/data/</code>. You can now use rsync to copy files to your project's subdirectory.<br />
<br />
'''Example''': rsync&nbsp;&#8209;rzvh&nbsp;/path/to/files/&nbsp;/Volumes/data/projectdir/<br />
<br />
==== Windows ====<br />
<br />
On windows computer, you should connect to <code>\\protocol1.deepsense.ca\data</code> or <code>\\protocol2.deepsense.ca\data</code>. To do this the first time, open a file explorer window. <br />
Right-click on This PC, and select "add a network location". In the wizard, click next and then select "Choose a custom network location" (this was the only option I saw). Highlight it, and click next. On the following screen, enter one of the addresses above, and click next. You may now enter a name for this location. Do so, and click next again. On the last screen, you should be able to look over your selections, and then click Finish. The name you chose should now be available under "This PC" in your file explorer. <br />
<br />
You may also have to change a SMB security level setting as follows (this was necessary in Windows 10):<br />
<br />
Control Panel&nbsp;>&nbsp;System and Security&nbsp;>&nbsp;Administrative tools&nbsp;>&nbsp;Local Security Policy&nbsp;>&nbsp;expand Local Policies&nbsp;>&nbsp;Security options&nbsp;>&nbsp;click on Network security: Lan Manager authentication level&nbsp;>&nbsp;Then in the field choose&nbsp;>&nbsp;Send NTLMv2 responses only&nbsp;>&nbsp;click on Apply, then ok and close all.<br />
<br />
=== Large Transfers ===<br />
<br />
For large transfers (>100Gb), we suggest you email [mailto:support@deepsense.ca support@deepsense.ca] for assistance. If you are local, you may be able to provide an external hard drive which we can plug in directly in our server room. We also have special software that can be used to speed up large transfers.<br />
<br />
== From the World Wide Web ==<br />
<br />
The standard tool for downloading data from websites is [https://en.wikipedia.org/wiki/Wget wget]. Also available is [https://curl.haxx.se/ curl]. The two are compared in this [https://unix.stackexchange.com/questions/47434/what-is-the-difference-between-curl-and-wget StackExchange article].</div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Common.css&diff=439MediaWiki:Common.css2020-12-12T02:09:26Z<p>Jnewport: </p>
<hr />
<div>/* CSS placed here will be applied to all skins */<br />
<br />
<br />
/* use a <div class="noautonum"> around a page to suppress autonumbering of sections. */<br />
<br />
#n-VPN-Setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Basic-Linux{<br />
margin-left: 20px!important;<br />
}<br />
#n-SSH-client-setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Glossary-for-Clusters{<br />
margin-left: 20px!important;<br />
}<br />
#n-Intro-to-Clusters{<br />
margin-left: 20px!important;<br />
}<br />
#n-Available-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Installing-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Using-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Submitting-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
#n-Checking-Job-Status{<br />
margin-left: 20px!important;<br />
}<br />
#n-Writing-Script{<br />
margin-left: 20px!important;<br />
}<br />
<br />
#n-Your-Accounts{<br />
margin-left: 20px!important;<br />
}<br />
#n-Data-Storage{<br />
margin-left: 20px!important;<br />
}<br />
#n-LSF-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
<br />
.noautonum .tocnumber { display: none; }<br />
.noautonum .mw-headline-number { display: none; }</div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Common.css&diff=438MediaWiki:Common.css2020-12-12T02:09:02Z<p>Jnewport: </p>
<hr />
<div>/* CSS placed here will be applied to all skins */<br />
<br />
<br />
/* use a <div class="noautonum"> around a page to suppress autonumbering of sections. */<br />
<br />
#n-VPN-Setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Basic-Linux{<br />
margin-left: 20px!important;<br />
}<br />
#n-SSH-client-setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Glossary-for-Clusters{<br />
margin-left: 20px!important;<br />
}<br />
#n-Intro-for-Clusters{<br />
margin-left: 20px!important;<br />
}<br />
#n-Available-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Installing-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Using-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Submitting-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
#n-Checking-Job-Status{<br />
margin-left: 20px!important;<br />
}<br />
#n-Writing-Script{<br />
margin-left: 20px!important;<br />
}<br />
<br />
#n-Your-Accounts{<br />
margin-left: 20px!important;<br />
}<br />
#n-Data-Storage{<br />
margin-left: 20px!important;<br />
}<br />
#n-LSF-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
<br />
.noautonum .tocnumber { display: none; }<br />
.noautonum .mw-headline-number { display: none; }</div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Common.css&diff=437MediaWiki:Common.css2020-12-12T02:08:34Z<p>Jnewport: </p>
<hr />
<div>/* CSS placed here will be applied to all skins */<br />
<br />
<br />
/* use a <div class="noautonum"> around a page to suppress autonumbering of sections. */<br />
<br />
#n-VPN-Setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Basic-Linux{<br />
margin-left: 20px!important;<br />
}<br />
#n-SSH-client-setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Glossary-for-Clusters{<br />
margin-left: 20px!important;<br />
}<br />
#n-Info-for-Clusters{<br />
margin-left: 20px!important;<br />
}<br />
#n-Available-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Installing-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Using-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Submitting-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
#n-Checking-Job-Status{<br />
margin-left: 20px!important;<br />
}<br />
#n-Writing-Script{<br />
margin-left: 20px!important;<br />
}<br />
<br />
#n-Your-Accounts{<br />
margin-left: 20px!important;<br />
}<br />
#n-Data-Storage{<br />
margin-left: 20px!important;<br />
}<br />
#n-LSF-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
<br />
.noautonum .tocnumber { display: none; }<br />
.noautonum .mw-headline-number { display: none; }</div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Common.css&diff=436MediaWiki:Common.css2020-12-12T02:07:03Z<p>Jnewport: </p>
<hr />
<div>/* CSS placed here will be applied to all skins */<br />
<br />
<br />
/* use a <div class="noautonum"> around a page to suppress autonumbering of sections. */<br />
<br />
#n-VPN-Setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Basic-Linux{<br />
margin-left: 20px!important;<br />
}<br />
#n-SSH-client-setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Glossary{<br />
margin-left: 20px!important;<br />
}<br />
#n-Info-for-first-time-cluster-users{<br />
margin-left: 20px!important;<br />
}<br />
#n-Available-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Installing-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Using-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Submitting-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
#n-Checking-Job-Status{<br />
margin-left: 20px!important;<br />
}<br />
#n-Writing-Script{<br />
margin-left: 20px!important;<br />
}<br />
<br />
#n-Your-Accounts{<br />
margin-left: 20px!important;<br />
}<br />
#n-Data-Storage{<br />
margin-left: 20px!important;<br />
}<br />
#n-LSF-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
<br />
.noautonum .tocnumber { display: none; }<br />
.noautonum .mw-headline-number { display: none; }</div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Common.css&diff=435MediaWiki:Common.css2020-12-12T02:06:24Z<p>Jnewport: </p>
<hr />
<div>/* CSS placed here will be applied to all skins */<br />
<br />
<br />
/* use a <div class="noautonum"> around a page to suppress autonumbering of sections. */<br />
<br />
#n-VPN-Setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Basic-Linux{<br />
margin-left: 20px!important;<br />
}<br />
#n-SSH-client-setup{<br />
margin-left: 20px!important;<br />
}<br />
}<br />
#n-Glossary{<br />
margin-left: 20px!important;<br />
}<br />
}<br />
#n-Info-for-first-time-cluster-users{<br />
margin-left: 20px!important;<br />
}<br />
<br />
<br />
#n-Available-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Installing-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Using-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Submitting-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
#n-Checking-Job-Status{<br />
margin-left: 20px!important;<br />
}<br />
#n-Writing-Script{<br />
margin-left: 20px!important;<br />
}<br />
<br />
#n-Your-Accounts{<br />
margin-left: 20px!important;<br />
}<br />
#n-Data-Storage{<br />
margin-left: 20px!important;<br />
}<br />
#n-LSF-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
<br />
.noautonum .tocnumber { display: none; }<br />
.noautonum .mw-headline-number { display: none; }</div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Sidebar&diff=434MediaWiki:Sidebar2020-12-12T02:04:36Z<p>Jnewport: </p>
<hr />
<div><br />
* navigation<br />
** mainpage|mainpage-description<br />
* DeepSense HPC Platform<br />
** mainpage | Cluster Status<br />
** Resources | Resources<br />
* Getting Support<br />
** Contact information | Contact - Support email<br />
* Getting Started <br />
** Requesting access | Requesting access<br />
** Accessing Systems | Accessing Systems<br />
*** VPN Setup | VPN Setup<br />
*** SSH client setup | SSH client setup<br />
*** Basic Linux | Basic Linux<br />
*** Glossary | Glossary for Clusters<br />
*** Info for first time cluster users | Intro to Clusters<br />
** LSF | Basic LSF Jobs<br />
** CWS | Conductor with Spark<br />
* Machine Learning On DeepSense<br />
** Deep Learning Frameworks | ML/DL Frameworks <br />
** Software | Software <br />
*** Available software | Available Software<br />
*** Installing Software | Installing Software<br />
*** Getting started with Deep Learning | Using Software<br />
** Running ML Jobs | Running ML jobs<br />
***Submitting Jobs | Submitting Jobs<br />
***Checking Job Status | Checking Job Status<br />
***Writing Script| Writing Script <br />
** Getting started with Jupyter Notebook | Using Jupyter Notebook<br />
** Deep Learning Tutorials | ML/DL Tutorials<br />
* Storage System<br />
** Overview of your Storage on DeepSense | Storage Overview<br />
** How to Transfer Data | How to Transfer Data<br />
** Backup Policies | Backup Policies<br />
** Quota Information and Management | Storage Quotas<br />
* FAQ<br />
** Restrictions | Restrictions<br />
** Workarounds | Workarounds<br />
** Best Practices | Best Practices<br />
*** Your Accounts | Your Accounts<br />
*** Data Storage | Data Storage<br />
*** LSF Jobs | LSF Jobs<br />
* Writing Tips<br />
** Mitacs Accelerate Proposals | Mitacs Accelerate Proposals<br />
** Paper Prep | Paper Prep<br />
* DeepSense<br />
** https://deepsense.ca | DeepSense home page<br />
** Acknowledging DeepSense | Acknowledging DeepSense<br />
** Terms of Use | Terms of use<br />
* Additional Resources<br />
** Related Links | Related Links<br />
** External Links | External Links<br />
** External Data Sources | External Data Sources</div>Jnewporthttps://docs.deepsense.ca/index.php?title=Paper_Prep&diff=420Paper Prep2020-12-09T17:24:18Z<p>Jnewport: /* Acknowledgement */</p>
<hr />
<div>Typically, the work a student does on a project is suitable for publication. The company and/or funding agency usually requires a report of the work you’ve done, so the first step is to start with that report and expand upon it as publications will typically have more technical details than your reports. If you are inexperienced publishing papers, the DeepSense team is here to support you. The major sections of the research paper are <br />
<br />
<br />
==Title of the Paper==<br />
The title of the paper should be centrally aligned and at the top of the paper. The format of the title should be according to the requirement of the Journal for which you are going to submit your paper. <br />
<br />
==Abstract==<br />
<br />
An abstract is a short summary of your research paper, usually about a paragraph (6-7 sentences, 150-250 words) long depends upon the Journal you are targeting. It always appears at the beginning of a manuscript acting as the point-of-entry for any given academic paper. It helps the reader to understand the essence of the quickly and prepare them to go through the detailed information, analyses, and arguments in the full paper. Content to be considered while writing the abstract <br />
<br />
*Background information for your research <br />
<br />
*Problem statement you are addressing <br />
<br />
*Why is it important to address these issues/problems? <br />
<br />
*What previous research has done so far? <br />
<br />
*Methodology you have used for the analysis <br />
<br />
*Main findings, results, or arguments <br />
<br />
*Significance of your findings <br />
<br />
==Introduction==<br />
It is an overview of the problem you are examining – including your main argument (thesis statement). It also offers a short justification regarding the importance of your problem. It also contains a brief explanation of the paper’s scope and planned method to be used in examining or solving the issue. While addressing an issue, don’t forget to include the story behind the issue, the impact of this issue on society, possible solutions to be explored in your study, and how you organize your paper. <br />
<br />
==Literature Review==<br />
This section is related to the description of the related theories that were used to explain the issue, a summary of the methodology, any major findings from the study, limitations raised regarding findings, describe a method that suits best for your own research based on what you have studied so far. <br />
<br />
==Methods==<br />
This section is related to the description of the methodology that will be used to solve the problem selected by you. The methodology may include the data collection process, data pre-processing, features extraction method, various algorithms you are using. <br />
<br />
==Results and Discussion==<br />
This section contains the major findings. You can use tables, charts, and graphical illustrations to explain the findings. You can also discuss if anything amazes you, compare your findings with previous studies, and express any limitations if your model has so that other researchers could use these findings in their research to get better results. <br />
<br />
==Conclusion and Recommendations==<br />
This section is a brief recap of the issue examined, the method used and major finding(s), briefly remind readers about the original goal of this study and what you accomplished in your research work and describe how future researchers can expand or build on your work. <br />
<br />
==Acknowledgement==<br />
Finally, don’t forget to acknowledge those who provided support to you in terms of any funding, technology or any kind of guidance.<br />
<br />
We do ask that you [[Acknowledging DeepSense | acknowledge DeepSense]] in any publications, as well as seminars or conference talks on your work.<br />
<br />
==References==<br />
Apply correct citation and formatting. The most used are <br />
<br />
*MLA Modern Language Association) style <br />
<br />
*APA (American Psychological Association) style <br />
<br />
*Harvard <br />
<br />
*Chicago <br />
<br />
Examples <br />
<br />
'''MLA''' <br />
<br />
<code>Changizi, R., et al. "Species identification reveals mislabeling of important fish products in Iran by DNA barcoding." Iranian Journal of Fisheries Sciences 12.4 (2013): 783-791.</code> <br />
<br />
'''APA''' <br />
<br />
<code>Changizi, R., Farahmand, H., Soltani, M., Asareh, R., & Ghiasvand, Z. (2013). Species identification reveals mislabeling of important fish products in Iran by DNA barcoding. Iranian Journal of Fisheries Sciences, 12(4), 783-791.</code> <br />
<br />
'''Chicago''' <br />
<br />
<code>Changizi, R., H. Farahmand, M. Soltani, R. Asareh, and Z. Ghiasvand. "Species identification reveals mislabeling of important fish products in Iran by DNA barcoding." Iranian Journal of Fisheries Sciences 12, no. 4 (2013): 783-791. </code><br />
<br />
'''Harvard''' <br />
<br />
<code>Changizi, R., Farahmand, H., Soltani, M., Asareh, R. and Ghiasvand, Z., 2013. Species identification reveals mislabeling of important fish products in Iran by DNA barcoding. Iranian Journal of Fisheries Sciences, 12(4), pp.783-791. </code><br />
<br />
<br />
<br />
==Revision of the Paper==<br />
<br />
After writing the paper, you need to revise it as it upgrades your paper by removing the unwanted errors.Following is the checklist that need to remember during the revision of the paper after writing the first draft <br />
<br />
'''Checklist:''' <br />
<br />
*Is my problem statement concise and clear? <br />
<br />
*Did I follow my outline or miss anything? <br />
<br />
*Is my paper organized in a logical way that is easy to understand? <br />
<br />
*Are all sources properly cited to ensure that I am not plagiarizing? Are all my citations accurate and in correct format? <br />
<br />
*Have I proved my thesis with strong supporting arguments? <br />
<br />
*Is there any unfinished sentences, unnecessary or repetitious words, spelling or grammatical errors? <br />
<br />
*Did I avoid using contractions? Use “cannot” instead of “can’t”, “do not” instead of “don’t”? <br />
<br />
*Avoid using phrases such as “I think”, “I guess”, “I suppose” <br />
<br />
*Did I leave a sense of completion for my readers at the end of the paper? <br />
<br />
*Did I plagiarize my paper? <br />
<br />
<br />
Re-read for grammatical errors </br><br />
<br />
<br />
==Tools==<br />
<br />
Various tools can be used to improve the quality of research paper. <br />
<br />
=== Grammar check tool ===<br />
<br />
This can be used to find and correct grammatical errors or spelling mistakes that would be helpful in producing good quality paper. <br />
<br />
===Plagiarism check tool===<br />
<br />
This can be used to ensure if there is any copied content in your paper and remind you about the citations required. <br />
<br />
===Citation generators tool=== <br />
<br />
This can be used to write the citations considering required conventions. <br />
<br />
<br />
<br />
'''Some of the examples of research paper''' <br />
<br />
*[https://www.sciencedirect.com/science/article/abs/pii/S0141113620306553 Pechenik, J. A., Chaparro, O. R., Lazarus, Z. M., Tellado, G. V., Ostapovich, E. M., & Clark, D. (2020). Impact of short-term elevated temperature stress on winter-acclimated individuals of the marine gastropod Crepidula fornicata. Marine Environmental Research, 105180] <br />
<br />
*[https://www.sciencedirect.com/science/article/abs/pii/S0141113620304608 Sun, Y., Song, Z., Zhang, H., Liu, P., & Hu, X. (2020). Seagrass vegetation affect the vertical organization of microbial communities in sediment. Marine Environmental Research, 105174.] <br />
<br />
*[https://www.sciencedirect.com/science/article/abs/pii/S096456911630134X La Manna, G., Ronchetti, F., & Sarà, G. (2016). Predicting common bottlenose dolphin habitat preference to dynamically adapt management measures from a Marine Spatial Planning perspective. Ocean & coastal management, 130, 317-327.]</div>Jnewporthttps://docs.deepsense.ca/index.php?title=DeepSense_Documentation&diff=417DeepSense Documentation2020-12-07T13:34:24Z<p>Jnewport: /* Note */</p>
<hr />
<div>'''Welcome to the DeepSense technical documentation wiki'''. This is the primary source for users with questions on the DeepSense equipment and services. You'll now find all of our content on the sidebar. Just below you can see the cluster status, and information about any planned outages we may have. <br />
<br />
We routinely make changes and update the content. If you see anything missing, or have any suggestions for content, we would appreciate hearing from you. You can send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).<br />
<br />
== Cluster Status ==<br />
<br />
'''<span style="font-size:120%>Cluster status</span>'''<br />
{|class="wikitable" style="text-align: center; color: black; font-style:bold"<br />
|'''Status'''<br />
|style="width:20% | '''Planned Outage'''<br />
|style="width:70% | '''Notes'''<br />
|-<br />
|style="Color:green" | Online<br />
|<br />
|<br />
|}<br />
Legend:<br/><br />
<span style="color:green">Online</span>: cluster is running normally<br/><br />
<span style="color:orange">Partially Online</span>: cluster has some problems and is partially available<br/><br />
<span style="color:red">Offline</span>: cluster is offine and users are not able to log in<br/></div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Common.css&diff=413MediaWiki:Common.css2020-12-04T19:08:00Z<p>Jnewport: </p>
<hr />
<div>/* CSS placed here will be applied to all skins */<br />
<br />
<br />
/* use a <div class="noautonum"> around a page to suppress autonumbering of sections. */<br />
<br />
#n-VPN-Setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Basic-Linux{<br />
margin-left: 20px!important;<br />
}<br />
#n-SSH-client-setup{<br />
margin-left: 20px!important;<br />
}<br />
#n-Available-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Installing-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Using-Software{<br />
margin-left: 20px!important;<br />
}<br />
#n-Submitting-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
#n-Checking-Job-Status{<br />
margin-left: 20px!important;<br />
}<br />
#n-Writing-Script{<br />
margin-left: 20px!important;<br />
}<br />
<br />
#n-Your-Accounts{<br />
margin-left: 20px!important;<br />
}<br />
#n-Data-Storage{<br />
margin-left: 20px!important;<br />
}<br />
#n-LSF-Jobs{<br />
margin-left: 20px!important;<br />
}<br />
<br />
.noautonum .tocnumber { display: none; }<br />
.noautonum .mw-headline-number { display: none; }</div>Jnewporthttps://docs.deepsense.ca/index.php?title=MediaWiki:Sidebar&diff=412MediaWiki:Sidebar2020-12-04T19:07:14Z<p>Jnewport: </p>
<hr />
<div><br />
* navigation<br />
** mainpage|mainpage-description<br />
* DeepSense HPC Platform<br />
** mainpage | Cluster Status<br />
** Resources | Resources<br />
* Getting Support<br />
** Contact information | Contact - Support email<br />
* Getting Started <br />
** Requesting access | Requesting access<br />
** Accessing Systems | Accessing Systems<br />
*** VPN Setup | VPN Setup<br />
*** SSH client setup | SSH client setup<br />
*** Basic Linux | Basic Linux<br />
** LSF | Running jobs<br />
* Machine Learning On DeepSense<br />
** Deep Learning Frameworks | ML/DL Frameworks <br />
** Software | Software <br />
*** Available software | Available Software<br />
*** Installing Software | Installing Software<br />
*** Getting started with Deep Learning | Using Software<br />
** Running ML Jobs | Running ML jobs<br />
***Submitting Jobs | Submitting Jobs<br />
***Checking Job Status | Checking Job Status<br />
***Writing Script| Writing Script <br />
** Getting started with Jupyter Notebook | Using Jupyter Notebook<br />
** Deep Learning Tutorials | ML/DL Tutorials<br />
* Storage System<br />
** Overview of your Storage on DeepSense | Storage Overview<br />
** How to Transfer Data | How to Transfer Data<br />
** Backup Policies | Backup Policies<br />
** Quota Information and Management | Storage Quotas<br />
* FAQ<br />
** Restrictions | Restrictions<br />
** Workarounds | Workarounds<br />
** Best Practices | Best Practices<br />
*** Your Accounts | Your Accounts<br />
*** Data Storage | Data Storage<br />
*** LSF Jobs | LSF Jobs<br />
* Writing Tips<br />
** Mitacs Accelerate Proposals | Mitacs Accelerate Proposals<br />
** Paper Prep | Paper Prep<br />
* DeepSense<br />
** https://deepsense.ca | DeepSense home page<br />
** Acknowledging DeepSense | Acknowledging DeepSense<br />
** Terms of Use | Terms of use<br />
* Additional Resources<br />
** Related Links | Related Links<br />
** External Links | External Links<br />
** External Data Sources | External Data Sources</div>Jnewport