DeepSense Docs - User contributions [en-ca]

Deep Learning Tutorials

2020-07-06T13:21:28Z

Cwhidden:

<div class="noautonum">

This is a collection of deep learning tutorials that were explored as part of the training of the 2019 DeepSense fellowships. These were explored over multiple sessions with 1-3 notebooks explored per 1-1.5 hour session. Both Tensorflow and pytorch are explored.

== 1. Get started with DeepSense ==

Follow all the steps from [[Getting started]] and [[Getting started with Deep Learning]]. This tutorial assumes you can log on to the DeepSense compute platform and have a version of Anaconda python on your path with Tensorflow and Pytorch installed in an anaconda environment.

== 2. Tensorflow Preparation ==

=== Download the example notebooks ===

git clone https://github.com/aymericdamien/TensorFlow-Examples.git

=== request a gpu session ===

bsub -Is -q gpu bash

=== activate your anaconda environment ===

conda activate tensorflow

Note: this assumes you've followed the getting started instructions and have created a python environment called <code>tensorflow</code> with the <code>tensorflow-gpu</code> package installed from the IBM-AI repository. If not then please follow those instructions.

=== start a jupyter notebook ===
jupyter notebook --no-browser --ip=0.0.0.0

=== open an SSH tunnel to access the notebook ===
just as we did in the [[Getting started with Deep Learning]] tutorial, open an SSH tunnel in another window
ssh -l <user> login1.deepsense.ca -L <port>:ds-cmgpu-<num>:<port>

=== open the notebook in your browser ===
In a web browser navigate to the page listed in the jupyter notebook output. Remember to replace the node name with <code>localhost</code>

== 3. Tensorflow Example Notebooks ==

=== 1. helloworld.ipynb ===

TensorFlow-Examples/notebooks/1_Introduction/helloworld.ipynb

This is an introduction to a python jupyter notebook.

Select a box with code. Press the shift and enter keys together to run the code in that box. You'll notice a star appear beside running code and a number in brackets appear beside finished code to indicate the order in which code boxes were run.

Often in an example notebook you will see code that already has output cached. You still need to run all previous code boxes and may want to use the menu to clear all output. If there is an error then you can modify the code or fix the error (e.g. download a dependency) and try again.

If you are missing a required dependency in later notebooks then you can install that package into your anaconda environment in a terminal window and it will be immediately accessible from the notebook. You do not need to close and restart the notebook or SSH tunnel.

You can also run the entire notebook using the menu.

When you are finished with a notebook you should use the menu to halt the kernel before closing the notebook. This clears resources such as GPU memory.

=== 2. kmeans.ipynb ===

TensorFlow-Examples/notebooks/2_BasicModels/kmeans.ipynb

This is an introduction to a basic machine learning model, kmeans. In this tutorial the kmeans algorithm is used to classify handwritten digits.

kmeans works by clustering different training examples and comparing each new input to the mean of each cluster.

Run the notebook and learn more about kmeans.

Observe that machine learning methods often have parameters that must be chosen. You can use default parameters but optimizing these parameters can greatly improve the accuracy of a model.

For a simple example, try increasing the length of training by changing the <code>num_steps</code> variable from 50 to 100 and running it again.

What happens if you modify other parameters and run the notebook again? How do these parameters change the training time and accuracy of the model?

Note: depending on your version of tensorflow you may need to modify the notebook. In some versions of tensorflow the kmeans.training_graph() function returns a different number of variables than the notebook expects (such as the cluster_centers_vars variable which may need to be removed from the code).

=== 3. random_forest.ipynb ===

TensorFlow-Examples/notebooks/2_BasicModels/random_forest.ipynb

This notebook uses a different model, random forests, to classify the handwritten digits. A random forest is a set of decision trees, each of which are trained to learn part of the problem.

Random forests are one of the most commonly used machine learning models because they are quick to train and give good performance on many tasks.

Run the notebook and observe that the random forest provides better accuracy than the kmeans model you trained previously.

Try changing parameters such as increasing the number of decision trees or the training time.

If you increase the number of trees or training time by too much then you will see that you achieve worse performance on the test set than on the training set. This is called overfitting and means that your model is learning specific details from the training set that do not generalize to the test set. This is a common problem in machine learning and needs to be considered whenever you train a model. You may need to simplify your model, give it more training examples by collecting more data or use data augmentation.

=== 4. convolutional_network.ipynb ===

TensorFlow-Examples/notebooks/3_NeuralNetworks/convolutional_network.ipynb

This is our first example of a commonly applied kind of neural network called a convolutional neural network (CNN), also applied to handwriting classification.

The notebook provides a good description of this kind of network which gradually reduces the size of the input using convolutional layers. This forces the network to learn information about multiple input variables and provides good accuracy.

Run the notebook and observe that this simple CNN greatly outperforms kmeans and random forests for handwriting recognition.

=== 5. linear_regression.ipynb ===

notebooks/TensorFlow-Examples/notebooks/2_BasicModels/linear_regression.ipynb

This notebook is an example of a different problem, called regression. Regression attempts to predict a value, unlike classification which determines which class an input example belongs to. This is a simple example that attempts to fit a line to best match a set of data points with x and y values.

Note: you may need to add backets to the print statements depending on your version of python and the version of this example notebook

=== 5. autoencoder.ipynb ===

notebooks/TensorFlow-Examples/notebooks/3_NeuralNetworks/autoencoder.ipynb

An autoencoder trains two seperate networks, an encoder and a decoder. This has many applications using the intermediate "latent representation". This includes compression (by storing just the latent representation), translation (by training an encoder with multiple different languages and then training different decoders for each language), and modifying the style of the input (by similarly leveraging different encoder or decoder frameworks).

=== 6. gan.ipynb ===

TensorFlow-Examples/notebooks/3_NeuralNetworks/gan.ipynb

A generative adversarial network (GAN) can be used to generate new data that looks similar to training data. This kind of network is how so called "deepfakes" are made that alter teh style of images or video.

The basic idea is to traing two opposing networks. The generator tries to generate examples and the discriminator tries to determine if examples are real or fake. Both networks learn and improve together which greatly improves performance over using a single network for either purpose.

=== 7. recurrent_network.ipynb ===

TensorFlow-Examples/notebooks/3_NeuralNetworks/recurrent_network.ipynb

A recurrent neural network is often applied to text processing or other problems that consider a sequence of events or letters. This type of network retains information at each step and thus has a type of memory that learns both from training examples as well as from parts of a specific input instance that have already been processed.

== 4. Pytorch ==

There are a variety of example notebooks for the pytorch framework at https://pytorch.org/tutorials/ . For example, there is a series of three notebooks on text processing and translation using recurrent neural networks:

=== 1. classifying names by country of origin ===

https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html

=== 2. generating names similar to those from those countries of origin ===

https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html

=== 3. text translation using seq2seq latent encoding of text with different decoders ===

https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

== 5. Tensor Processing Units ==

Google has developed dedicated deep learning processors called tensor processing units (TPUs). The contest web site kaggle has an online notebook where you can try out these TPUs to classify images of flowers by species.

https://www.kaggle.com/c/flower-classification-with-tpus

Try out the notebook and compare the performance of CPU, GPU, and TPU computing. You will need to modify the parameters of the notebook heavily to see any results at all with CPUs, while you will see an obvious speed improvement from using the TPUs over the GPUs.

Deep Learning Tutorials

2020-07-02T22:34:12Z

Cwhidden:

<div class="noautonum">

== 1. Get started with DeepSense ==

Follow all the steps from [[Getting started]] and [[Getting started with Deep Learning]]. This tutorial assumes you can log on to the DeepSense compute platform and have a version of Anaconda python on your path with Tensorflow and Pytorch installed in an anaconda environment.

== 2. Tensorflow Preparation ==

=== Download the example notebooks ===

git clone https://github.com/aymericdamien/TensorFlow-Examples.git

=== request a gpu session ===

bsub -Is -q gpu bash

=== activate your anaconda environment ===

conda activate tensorflow

Note: this assumes you've followed the getting started instructions and have created a python environment called <code>tensorflow</code> with the <code>tensorflow-gpu</code> package installed from the IBM-AI repository. If not then please follow those instructions.

=== start a jupyter notebook ===
jupyter notebook --no-browser --ip=0.0.0.0

=== open an SSH tunnel to access the notebook ===
just as we did in the [[Getting started with Deep Learning]] tutorial, open an SSH tunnel in another window
ssh -l <user> login1.deepsense.ca -L <port>:ds-cmgpu-<num>:<port>

=== open the notebook in your browser ===
In a web browser navigate to the page listed in the jupyter notebook output. Remember to replace the node name with <code>localhost</code>

== 3. Tensorflow Example Notebooks ==

=== 1. helloworld.ipynb ===

TensorFlow-Examples/notebooks/1_Introduction/helloworld.ipynb

This is an introduction to a python jupyter notebook.

Select a box with code. Press the shift and enter keys together to run the code in that box. You'll notice a star appear beside running code and a number in brackets appear beside finished code to indicate the order in which code boxes were run.

Often in an example notebook you will see code that already has output cached. You still need to run all previous code boxes and may want to use the menu to clear all output. If there is an error then you can modify the code or fix the error (e.g. download a dependency) and try again.

If you are missing a required dependency in later notebooks then you can install that package into your anaconda environment in a terminal window and it will be immediately accessible from the notebook. You do not need to close and restart the notebook or SSH tunnel.

You can also run the entire notebook using the menu.

When you are finished with a notebook you should use the menu to halt the kernel before closing the notebook. This clears resources such as GPU memory.

=== 2. kmeans.ipynb ===

TensorFlow-Examples/notebooks/2_BasicModels/kmeans.ipynb

This is an introduction to a basic machine learning model, kmeans. In this tutorial the kmeans algorithm is used to classify handwritten digits.

kmeans works by clustering different training examples and comparing each new input to the mean of each cluster.

Run the notebook and learn more about kmeans.

Observe that machine learning methods often have parameters that must be chosen. You can use default parameters but optimizing these parameters can greatly improve the accuracy of a model.

For a simple example, try increasing the length of training by changing the <code>num_steps</code> variable from 50 to 100 and running it again.

What happens if you modify other parameters and run the notebook again? How do these parameters change the training time and accuracy of the model?

Note: depending on your version of tensorflow you may need to modify the notebook. In some versions of tensorflow the kmeans.training_graph() function returns a different number of variables than the notebook expects (such as the cluster_centers_vars variable which may need to be removed from the code).

=== 3. random_forest.ipynb ===

TensorFlow-Examples/notebooks/2_BasicModels/random_forest.ipynb

This notebook uses a different model, random forests, to classify the handwritten digits. A random forest is a set of decision trees, each of which are trained to learn part of the problem.

Random forests are one of the most commonly used machine learning models because they are quick to train and give good performance on many tasks.

Run the notebook and observe that the random forest provides better accuracy than the kmeans model you trained previously.

Try changing parameters such as increasing the number of decision trees or the training time.

If you increase the number of trees or training time by too much then you will see that you achieve worse performance on the test set than on the training set. This is called overfitting and means that your model is learning specific details from the training set that do not generalize to the test set. This is a common problem in machine learning and needs to be considered whenever you train a model. You may need to simplify your model, give it more training examples by collecting more data or use data augmentation.

=== 4. convolutional_network.ipynb ===

TensorFlow-Examples/notebooks/3_NeuralNetworks/convolutional_network.ipynb

This is our first example of a commonly applied kind of neural network called a convolutional neural network (CNN), also applied to handwriting classification.

The notebook provides a good description of this kind of network which gradually reduces the size of the input using convolutional layers. This forces the network to learn information about multiple input variables and provides good accuracy.

Run the notebook and observe that this simple CNN greatly outperforms kmeans and random forests for handwriting recognition.

=== 5. linear_regression.ipynb ===

notebooks/TensorFlow-Examples/notebooks/2_BasicModels/linear_regression.ipynb

This notebook is an example of a different problem, called regression. Regression attempts to predict a value, unlike classification which determines which class an input example belongs to. This is a simple example that attempts to fit a line to best match a set of data points with x and y values.

Note: you may need to add backets to the print statements depending on your version of python and the version of this example notebook

=== 5. autoencoder.ipynb ===

notebooks/TensorFlow-Examples/notebooks/3_NeuralNetworks/autoencoder.ipynb

An autoencoder trains two seperate networks, an encoder and a decoder. This has many applications using the intermediate "latent representation". This includes compression (by storing just the latent representation), translation (by training an encoder with multiple different languages and then training different decoders for each language), and modifying the style of the input (by similarly leveraging different encoder or decoder frameworks).

=== 6. gan.ipynb ===

TensorFlow-Examples/notebooks/3_NeuralNetworks/gan.ipynb

A generative adversarial network (GAN) can be used to generate new data that looks similar to training data. This kind of network is how so called "deepfakes" are made that alter teh style of images or video.

The basic idea is to traing two opposing networks. The generator tries to generate examples and the discriminator tries to determine if examples are real or fake. Both networks learn and improve together which greatly improves performance over using a single network for either purpose.

=== 7. recurrent_network.ipynb ===

TensorFlow-Examples/notebooks/3_NeuralNetworks/recurrent_network.ipynb

A recurrent neural network is often applied to text processing or other problems that consider a sequence of events or letters. This type of network retains information at each step and thus has a type of memory that learns both from training examples as well as from parts of a specific input instance that have already been processed.

== 4. Pytorch ==

There are a variety of example notebooks for the pytorch framework at https://pytorch.org/tutorials/ . For example, there is a series of three notebooks on text processing and translation using recurrent neural networks:

=== 1. classifying names by country of origin ===

https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html

=== 2. generating names similar to those from those countries of origin ===

https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html

=== 3. text translation using seq2seq latent encoding of text with different decoders ===

https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

== 5. Tensor Processing Units ==

Google has developed dedicated deep learning processors called tensor processing units (TPUs). The contest web site kaggle has an online notebook where you can try out these TPUs to classify images of flowers by species.

https://www.kaggle.com/c/flower-classification-with-tpus

Try out the notebook and compare the performance of CPU, GPU, and TPU computing. You will need to modify the parameters of the notebook heavily to see any results at all with CPUs, while you will see an obvious speed improvement from using the TPUs over the GPUs.

Deep Learning Tutorials

2020-07-02T22:26:03Z

Cwhidden:

Deep Learning Tutorials

2020-07-02T18:04:42Z

Cwhidden: Created page with "<div class="noautonum"> == 1. Get started with DeepSense == Follow all the steps from Getting started and Getting started with Deep Learning. This tutorial assumes y..."

DeepSense Documentation

2020-07-02T17:43:44Z

Cwhidden: /* Guides */

Mitacs Accelerate Proposals

2020-06-26T19:56:23Z

Cwhidden: Created page with "The Mitacs Accelerate program is a common funding method for DeepSense projects. This guide will help you through the process of writing a funding proposal for submission to M..."

The Mitacs Accelerate program is a common funding method for DeepSense projects. This guide will help you through the process of writing a funding proposal for submission to Mitacs.

Mitacs and a company each pay for half of the project. Mitacs funding is allocated in *units* of 4-6 months with each unit paying $15000 to the student.

See the [https://www.mitacs.ca/en/programs/accelerate/proposal|The Mitacs web site] for the proposal template and guide.

== Planning Meeting ==
As with any project, the first step is to have a planning meeting with the student, company, supervisor, and possibly with a DeepSense staff member. If you are reading this guide you probably already had at least one meeting with the company but there are a few items you need to be clear on. If necessary, have another meeting or discuss these items by email or phone with your supervisor, DeepSense staff, and/or the company:

# Number of students and duration of the project
# Overall goal of the project
# Deliverables
# A general idea of how to approach the project and belief it is feasible

== General advice ==

* There are a lot of sections in the proposal so write the proposal in stages instead of trying to do it all at once or in order.

* Be iterative: write an outline of what you intend to do, expand the outline to sentences and paragraphs, and then edit for clarity and content.

* Get feedback often. Involve your supervisor, other students on the project or related projects, and DeepSense staff. It will take several rounds of feedback and editing to write a strong proposal.

* Be mindful of both the science and the benefit to the company. A DeepSense project is a mix of both. There needs to be specified deliverables for the company and an explanation of how those deliverables solve a problem for the company. There also needs to be a research question to solve or novel advance that doesn't currently exist. This can range from applying a known state of the art technique to a novel dataset or situation all the way to developing a brand new solution.

* Use topic sentences and strong closing sentences. A topic sentence is a sentence that explains the purpose of a paragraph and is typically at the beginning of the paragraph. In most cases you should be able to read just the first sentence of each paragraph and still understand the main ideas. A strong closing sentence, especially at the end of each section, will encourage the reader

* Ask your supervisor or DeepSense staff if you can see an example of a successfully funded Mitacs proposal. This will help you understand the scope and detail required in each section.

== First Steps ==

The first steps are to do some research and write down the general idea of the project

=== 2.1 and 7.1: Title ===
Write down a draft title. You may want to change this later on so don't worry too much about writing a catchy title yet.

=== 2.2 Abstract ===
Approximately 200 words. The abstract summarizes the entire project. Write out the problem to be solved, why it is important, the current state of the art and why that is insufficient, your proposed solution, and finally the steps and deliverables of the project.

The abstract is important but does not need to be polished at this stage. Just write out each of these facets and if you can't then you need to do some research or discuss the project further with your supervisor, the company, and/or DeepSense staff.

=== 2.3 Background ===
Minimum 500 words. This is where you explain the current state of the problem in detail and cite relevant research.

First give a general overview of the problem and the broad research area. Cite some relevant survey papers or important guiding works.

Then summarize some of the more specific topics and research papers. Typically you will find papers that partially but do not completely solve your problem. Summarize their results as well as their pros and cons. It should be clear why you have selected these specific results and how this knowledge will contribute to completing the project.

Finally, explain how some of these results together can be used to solve your specific problem. For example, you may be using a deep learning model from one paper, adding a technique from a second paper, and then using a type of analysis from a third paper. Moreover, highlight any gaps that the current literature can't solve for your problem. It is important to explain how these methods fit together and show that your project is novel research.

=== 2.9 References ===

Put your references in this section. Mitacs does not specify a reference format so use any consistent format. We've had good success with the APA format.

This section is just as important as it is in a scientific paper so be sure to check your reference details and order them consistently.

== Project Detail ==

Now that you have a general idea of the project and have done some research it's time to get more specific with the deliverables and methods. Consider getting feedback on the sections you have already completed while you start on the next sections.

=== 2.4 General Objective ===

Write out the main objective of the project. Split this up into sub-objectives. Very briefly summarize any information needed to explain a sub-objective such as how it will be tested or what data will be used.

=== 2.5 Details of internships or subprojects ===

Although numbered like a regular section, each subsection of Details requires a significant explanation and should be treated as a full section in its own right. We'll complete some of these now but go back and do the other later.

For a very large project funded by a single mitacs submission you may need to break the project up into different subprojects. If so then the student(s) working on each subproject should fill out their subprojects information.

=== 2.5.a Name of Intern ===

Write out the name of the student or students working on a shared goal.

=== 2.5.b Specific Objectives ===

Write out the specific objectives that each student listed will work on. These should be matched with the general objective and sub-objectives as well as explaining what each individual student will do.

=== 2.5.c Methodologies ===

In this section you write out in detail how you will accomplish this project.

This is the meat of the proposal and must includes the datasets, tools, and methods you will use in enough detail for an expert external reviewer to determine if the project is feasible and will accomplish your objectives. Consider this like the methods section of a scientific paper.

Explain the analysis you will complete and how you will show that you have met each sub-objective.

=== 2.5.e Deliverables ===

Explain the deliverables of the project such as software, reports, publications, etc. What will the project create and how will it meet the sub-objectives?

=== 2.5.f Benefit to the intern ===

Explain what you will gain from this project. Examples include:
* exposure to an ocean company and/or novel data
* learning a new analysis method
* opportunity to contribute to an industry problem or a solution that will be used by the partner to do X
* opportunity to publish in peer-reviewed venues and/or write a thesis

== Timeline ==

Now that you have a good idea of the specifics of the project it's time to plan. Revise and edit the previous sections and seek feedback.

Then, take the steps your propose to do in your methodology and create a *gannt chart* showing how long each of those steps will take:

=== 2.5.d Timeline ===

First, list out the major steps of the project, goals, and deliverables such as:
* obtain data
* preprocess the data
* design the machine learning model
* train the model
* test the model
* improve the model
* train and test again
* analysis
* write a report

Then think about how long each step of the project will take in weeks, half months, or (for a long project) months. Break the project up into 4 month sections and create a gannt chart for each. You will probably want to do this in a spreadsheet program such as Excel for easy editing and then copy the chart into the mitacs proposal later.

== Basic Information ==

There are some informational sections you need to complete. You will need to obtain some information from your supervisor:

* 2.8 Relationship (if any) to past/other Mitacs projects:

* 3 Declarations

* 4.1 Lead academic supervisor in Canada

* 4.3 Interns

* 4.4 Interns to be determined (TBD)

== Company information ==

When your proposal is nearly ready you will need to send it to your partner company for their feedback. Do this in conjunction with your supervisor and/or DeepSense staff.

Be sure you make it clear to the company that *Section 7.2 Public project overview* will be publicly available on the Mitacs web site. They must review this section carefully.

Section of the proposal that should be completed by or with the help of the partner organization are:

* 2.5.g Partner interaction

* 2.6 Relevance to the partner organization and to Canada

* 4.2 Partner organization in Canada

* 7.2 Public project overview

== Remaining steps ==

There are a few other sections that must be completed

=== 5 Budget and invoicing ===

Your supervisor or DeepSense staff can help with the budget

=== 6 Suggested reviewers ===

You need to suggest 6 reviewers. They cannot be from your university and you cannot have published with them or have plans to collaborate with them in the near future.

Moreover, each reviewer must be from a different university or organization than the others.

We recommend selecting subject experts that are knowledgeable of the research area but also likely to be generally favourable of the research and provide constructive criticism and suggestions. Contact your supervisor and/or DeepSense staff if you need help selecting reviewers.

== Scientific Committee Review ==

The proposal must be reviewed by the DeepSense scientific committee.

== Signatures ==

After the scientific committee has approved the project and the company has agreed and signed the necessary sections, you and your supervisor will need to sign:

* 7.3 Signatures
* Appendix A - Intern consent form
* The separate mitacs IP agreement

== Review Process ==

Proposal review will take approximately 4-8 weeks so it is important to submit the proposal early before the start of a project. Most well-written proposals are funded but you may need to provide extra information to address reviewer comments.

DeepSense Documentation

2020-06-26T18:12:37Z

Cwhidden: /* Guides */ Added page for writing a mitacs

Known problems

2020-06-17T19:28:07Z

Cwhidden: Update with information about IBM WMLA conda channel

== Where did Tensorflow go? ==

On June 26 we will update the GPU compute nodes to a new version of IBM Watson Machine Learning Accelerator. This will change the way you access deep learning packages like Tensorflow and Pytorch. Instead of "activating" these packages, you will be able to install new versions directly in your anaconda environment.

See [[Installing local software]] and [[Getting started with Deep Learning]] for more information.

We are actively updating the wiki documentation to explain the new method of accessing deep learning packages. Please bear with us during these updates as some documentation may still refer to the old method of "activating" deep learning packages

== The bhist command is not working ==

We are aware of a problem causing the bhist command to not find the job log file necessary to print information about completed jobs. Instead, the output is always "no matching job found" even if the user specifies the "-a" option to print information about all running, completed, and failed jobs.

While we work on solving this issue you can manually specify the location of the log file using the -f option. For example, the following command will print all running, completed, and failed jobs for the current user:

bhist -f /lsfshare/lsfswg/lsf/work/DeepSenseLSFCluster/logdir/lsb.events -a

== Where is my job output file? ==
Output may not be written to the specified file immediately when using the <code>-o <filename></code> or <code>-oo <filename></code> options. There are two workarounds for this problem:

# You can use the bpeek <jobid> command to view the output of a currently running job.
# You can send your output to a file with the typical unix output specifications such as <code>> <filename></code> with your executed programs or by specifying output files in programs that support such options.

== Jupyter notebooks or other programs fail trying to access a /run directory ==

The default login shell is BASH. Make sure the following parameter is in your .bashrc file in your home directory, as it prevents a problem where some types of jobs fail when run through the LSF queue. This should be done automatically the first time you log onto DeepSense.

<code>echo 'unset XDG_RUNTIME_DIR' >> ~/.bashrc</code>

This line has been added to the default .bashrc file for new users but older user accounts may need this step to be done manually.

== Browser fails to connect to Jupyter Notebooks ==

On our MacBook Pros, Jupyter notebooks work in Chrome, but don't work in safari. Unfortunately, no error is given. Safari just fails to connect. Please let us know if you have issues with any other browsers, and we can add that info here.

== Cannot Install PyTorch dependencies ==

UnsatisfiableError: The following specifications were found to be in conflict:
- powerai-pytorch-prereqs=0.4.1_12295.5cb3523

You may see this error when attempting to install the pytorch dependencies in a local anaconda environment. This error indicates that some of your installed python packages are not compatible with the pytorch prequisites. In particular, we see this error when conda has been updated to version 4.6 (which may sometimes happen when installing the tensorflow dependencies first).

To resolve this problem, create a new environment with a 4.5.x conda version and then install the pytorch dependencies in that environment.

== Cannot use Caffe on login node or compute nodes without GPUs ==

<nowiki>Cuda number of devices: -579579216
Current device id: -579579216
Current device name:
[==========] Running 2207 tests from 293 test cases.
[----------] Global test environment set-up.
[----------] 9 tests from AccuracyLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN ] AccuracyLayerTest/0.TestSetup
E0206 15:59:26.604874 7990 common.cpp:121] Cannot create Cublas handle. Cublas won't be available.
E0206 15:59:26.611477 7990 common.cpp:128] Cannot create Curand generator. Curand won't be available.
F0206 15:59:26.611616 7990 syncedmem.cpp:500] Check failed: error == cudaSuccess (30 vs. 0) unknown error
*** Check failure stack trace: ***
</nowiki>

You may see this error when attempting to use Caffe on a node without GPUs or a GPU node without specifically requesting a GPU.

To resolve this problem, use a GPU node and request a GPU. Caffe cannot run without an available GPU.

== Cannot see GPUs in an LSF job ==

$ nvidia-smi
No devices were found

GPUs must be requested with the <code>-gpu -</code> option to bsub. See [[LSF#GPU_Computation]] for more information.

== Nested anaconda environments may cause strange behaviour ==

Some users have experienced strange behaviour when activating an anaconda environment within another environment. This may include permission errors, loading incorrect versions of software, or strange conflicts when attempting to install packages. If you encounter problems with a nested anaconda environment then first try deactivating all anaconda environments and activating just the desired environment.

Getting started with Deep Learning

2020-06-17T19:25:35Z

Cwhidden: Update with information about IBM WMLA conda channel

<div class="noautonum">

== Note ==
On June 26 we will update the GPU compute nodes to a new version of IBM Watson Machine Learning Accelerator. This will change the way you access deep learning packages like Tensorflow and Pytorch. Instead of "activating" these packages, you will be able to install new versions directly in your anaconda environment.

We are actively updating the wiki documentation to explain the new method of accessing deep learning packages. Please bear with us during these updates as some documentation may still refer to the old method of "activating" deep learning packages

== 1. Get started with DeepSense ==

Follow all the steps from [[Getting started]]. This tutorial assumes you can log on to the DeepSense compute platform and have a version of Anaconda python on your path. We recommend installing Anaconda in your home directory before starting this tutorial (See [[Installing local software]]).

== 2. Prepare Caffe and download Caffe samples to your home directory ==

(New method)

Activate your anaconda environment. See [[Installing local software]] for how to create an environment. We will assume you have created one called "caffe".
conda activate caffe
Add the IBM-AI anaconda channel if you have not done so already
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
Install caffe if you have not done so already
conda install caffe
Install the caffe samples
caffe-install-samples

(Old method)

<code>/opt/DL/caffe/bin/caffe-install-samples</code>

== 3. Request an interactive session on a GPU compute node ==



<code>bsub -Is -gpu - bash</code>

== 4. Start a python2 Jupyter notebook ==

=== (Old method only) Source the Caffe deep learning toolkit ===

<code>source /opt/DL/caffe/bin/caffe-activate</code>

=== Start the notebook ===

<code>jupyter notebook --no-browser --ip=0.0.0.0</code>

=== Sample output ===
<pre>[I 13:32:23.937 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:32:23.937 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://ds-cmgpu-04:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba
</pre>

=== Copy the URL, host, and port ===

Copy the URL but don’t paste it in your browser yet.

Make a note of which compute host and port the notebook is running on (e.g. host ds-cmgpu-04 and port 8888 in this case)

== 5. Port Forwarding ==

In a separate terminal window from your local computer, forward your local port to the remote host.

=== ssh command port forwarding ===

<code> ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port></code>

for example, <code>ssh -l user1 login1.deepsense.ca -L 8888:ds-cmgpu-04:8888</code>

Note that you may need to use a different <local_port> than 8888 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8888 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8889 or another unused port.

=== PuTTY port forwarding on Windows ===

If you are using a PuTTY terminal from a Windows computer to access DeepSense then you can still forward ports.

Before starting your session, scroll down to the option <code>Connection->SSH->Tunnels</code> in the Category pane.

Enter the <code>local_port</code> in the <code>Source port</code> field. For example, <code>8888</code>.

Enter <code><remote_host>:<remote_port></code> in the Destination field. For example, <code>ds-cmgpu-04:8888</code>.

Press the <code>Add</code> button to add the port forwarding rule to your PuTTY session.

Finally, open the session as usual.

== 6. Open the desired sample notebook ==

Enter the copied URL in your web browser but change the remote host name to “localhost” before pressing enter.

e.g <code>http://localhost:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba</code>

'''Note''': On our macs, this worked in Chrome, but not in Safari. Unfortunately, there was no error reported, it simply could not connect.

Be sure to enter the location of the “caffe-samples” directory in your home directory as your caffe-root in the Caffe example notebooks.

== 7. Enjoy Deep Learning on DeepSense! ==

== 8. More information ==

Go to Caffe's [http://caffe.berkeleyvision.org/ website] for tutorials and example programs that you can run to get started.
See the following links to a couple of the example programs:

[http://caffe.berkeleyvision.org/gathered/examples/mnist.html LeNet MNIST Tutorial] - Train a neural network to understand handwritten digits.

[http://caffe.berkeleyvision.org/gathered/examples/cifar10.html CIFAR-10 tutorial] - Train a convolutional neural network to classify small images.

== 9. Using another deep learning toolkit such as Tensorflow ==

(New Method)
* Ensure any Anaconda dependencies are installed
** for tensorflow, create a new environment and <code>conda install tensorflow</conda>
* Download example notebooks for the deep learning toolkit to your home directory,
** e.g. <code> git clone https://github.com/aymericdamien/TensorFlow-Examples.git</code>

(Old Method)
* Ensure any Anaconda dependencies are installed
** for tensorflow, run <code>/opt/DL/tensorflow/bin/install_dependencies</code>
* Source the appropriate toolkit instead of caffe-activate
** e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>
* Download example notebooks for the deep learning toolkit to your home directory,
** e.g. <code> git clone https://github.com/aymericdamien/TensorFlow-Examples.git</code>

The TensorFlow [https://www.tensorflow.org/ home page] has various information, including Tutorials, How-To documents, and a Getting Started guide.

Additional tutorials and examples are available from the community, for example:

https://github.com/nlintz/TensorFlow-Tutorials

https://github.com/aymericdamien/TensorFlow-Examples

</div>

Getting started

2020-06-17T19:01:32Z

Cwhidden: Update with information about IBM WMLA conda channel

Getting Started with DeepSense

<div class="noautonum">

== Note ==

On June 26 we will update the GPU compute nodes to a new version of IBM Watson Machine Learning Accelerator. This will change the way you access deep learning packages like Tensorflow and Pytorch. Instead of "activating" these packages, you will be able to install new versions directly in your anaconda environment.

We are actively updating the wiki documentation to explain the new method of accessing deep learning packages. Please bear with us during these updates as some documentation may still refer to the old method of "activating" deep learning packages

== 1. Logging on ==

DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus.

For example, if your userid is <code>user1</code>, you can connect to deepsense by typing <code>ssh user1@login1.deepsense.ca</code> just like logging on to any other network computer.

'''Note''': The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes. Keep reading for instructions on how to submit compute jobs to dedicated compute nodes.

== 1.1 VPN ==

To connect to the DeepSense platform from outside of the Dalhousie Campus, you'll need to use a VPN.
If you are are student, staff or faculty, you can use the Dalhousie VPN (https://wireless.dal.ca/vpnsoftware.php).

If you are not a Dalhousie staff, student, or faculty but require offsite access and cannot use the Dalhousie VPN then contact your project leader or support@deepsense.ca to make different arrangements.

== 2. Transfer data ==

For more information, see [[Transferring Data]].

Deepsense has two protocol nodes, protocol1.deepsense.ca and protocol2.deepsense.ca . You can connect to these using the SAMBA transfer protocol, e.g. smb://protocol1.deepsense.ca with your username and password. Please contact your project leader or support@deepsense.ca if you need help transferring large amounts of data.

Data transferred through the protocol nodes will be located in the shared /data directory .

See [[Storage policies]] for more information about the available shared file systems, storage policies, and backup policies.

== 3. Configure your environment ==

DeepSense compute and management nodes are IBM Power8 computers (ppc64le) running Redhat Enterprise Linux. See [[Resources]] for more details on the available nodes.

=== 3.1 Loading a python environment ===

You have two options for using python on DeepSense. You can use the systemwide python install, managed by DeepSense administrators. This is recommended for users new to Linux. You will need to contact DeepSense support to have additional software packages installed in the systemwide python.

Alternatively, you can install an Anaconda python environment or other software in your home directory. This allows you to install or update packages or software without requesting and waiting for DeepSense staff.

==== Systemwide python (managed by DeepSense) ====

DeepSense nodes have anaconda2 python installed in /opt/anaconda2. To use this systemwide python add a parameter to your .bashrc file in your home directory:

<code>echo ". /opt/anaconda2/etc/profile.d/conda.sh" >> ~/.bashrc</code>

Then source your .bashrc file:
<code>source ~/.bashrc</code>

To load the python2 environment run <code>conda activate</code>

To use python3 you can activate the py36 environment:
<code>conda activate py36</code>

You can add either line to your .bashrc file to automatically load the desired environment when you log in.

==== Local python install (managed by individual user) ====

See [[Installing local software]] for more information.

== 4. Running compute jobs ==

DeepSense has two different methods of submitting compute jobs.

=== 4.1 Load Sharing Facility (LSF) ===

LSF is a set of command line tools for submitting compute jobs. You may be familiar with other similar software such as Sun Grid Engine or SLURM.

LSF jobs are submitted using the <code>bsub</code> command.

You can examine the progress of your currently running jobs with the <code>bjobs</code> command.

You can examine the available compute nodes and their available resources with the <code>bhosts</code> command.

For more information about using LSF see [[LSF]].

=== 4.2 Conductor with Spark (CWS) ===

CWS is an IBM web-based graphical interface for creating and running Apache Spark compute jobs.

To use CWS, connect to the IBM Spectrum Computing Cluster Management Console at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Log in with your username and password.

Note that currently you need to accept a self-signed web certificate. In the future this will be fixed.

For more information about using CWS see [[CWS]].

== 5. Deep Learning packages and other available software ==

DeepSense has a variety of Deep Learning packages available as part of IBM Watson Machine Learning Accelerator including Tensorflow, Caffe, and PyTorch. These packages can be installed from the anaconda repository https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

These packages were formerly installed in /opt/DL/ on each compute node and used to need to be activated before using them, e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>.

Deep Learning packages are typically used on the GPU nodes but some deep learning packages can also be used on the login nodes and CPU-only nodes. This can be useful for testing your code or running CPU-bound workloads. Note that some deep learning packages may fail if run without a GPU, e.g. Caffe currently requires a GPU.

For a brief tutorial including running Caffe and Tensorflow in a Jupyter notebook see [[Getting started with Deep Learning]].

See [[Available software]] for the current list of installed software. If you require additional software you are welcome to install it locally in your home directory or contact DeepSense support.

== 6. Technical and research support ==

DeepSense has a dedicated support team of research scientists ready to help you with technical questions, installing software, or even research questions.

If you can't find the answer to your question on this wiki or need more extensive help then send an email to support@deepsense.ca .

See [[Technical support]] for more information about the support available.

</div>

DeepSense Documentation

2020-06-17T18:57:52Z

Cwhidden: Update with information about IBM WMLA conda channel

Installing local software

2020-06-17T17:59:18Z

Cwhidden: Update with information about IBM WMLA conda channel

== Note ==

On June 26 we will update the GPU compute nodes to a new version of IBM Watson Machine Learning Accelerator. This will change the way you access deep learning packages like Tensorflow and Pytorch. Instead of "activating" these packages, you will be able to install new versions directly in your anaconda environment. See below for more information.

We are actively updating the wiki documentation to explain the new method of accessing deep learning packages. Please bear with us during these updates as some documentation may still refer to the old method of "activating" deep learning packages.

== Introduction ==

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact [[Contact_Information|Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation.

If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Contact_Information|Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le).

If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Contact_Information|Technical Support]]. We may install and support it systemwide if there is sufficient interest.

== Installing Anaconda Python in your home directory ==

=== Stop using systemwide anaconda ===

If you added the system anaconda environment to your <code>.bashrc</code> file then remove the line:
. /opt/anaconda2/etc/profile.d/conda.sh

=== Installing Anaconda with a python3 base ===

From your home directory run:
wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-ppc64le.sh
bash Anaconda3-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
. ~/anaconda3/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:
source ~/.bashrc

=== Adding a python2 environment ===
The previous instruction creates a python3 base environment. To add a python2 environment:
conda create -n py27 python=2.7

Activate this environment to use python3:
conda activate py27

note: if you receive an error message then you may need to deactivate the base conda environment first:
conda deactivate
conda activate py27

=== Adding a python3 environment ===
We recommend creating a separate python3 environment from the base environment. This makes it easier to install the specific packages required for IBM PowerAI.
conda create -n py36 python=3.6

Activate this environment to use python3:
conda activate py36

=== (New Method) IBM-AI Deep Learning Anaconda Channel ===

To use deep learning packages like Tensorflow on DeepSense you need to add the IBM-AI anaconda channel to your list of available software channels.
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

We suggest creating a new environment for each deep learning package you want to use. For example for Tensorflow:
conda create -n py36_tensorflow python=3.6
conda activate py36_tensorflow

Then install the anaconda package for the software you need. Again, with Tensorflow as an example:
conda install tensorflow

You can then use tensorflow or other deep learning packages as needed by simply activating that anaconda environment. Unlike the old method, you do not need to specifically activate tensorflow or other deep learning methods.

You can directly visit the IBM-AI anaconda channel URL to see a list of available software (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/)

=== (Old Method) Install PowerAI dependencies ===

Warning: these scripts will install, update, and downgrade some packages to the recommended packages for the current version of PowerAI. You may want to create a separate python environment to use different versions of those packages with other software.

To use Tensorflow first install the Tensorflow dependencies:
/opt/DL/tensorflow/bin/install_dependencies

To use PyTorch first install the PyTorch dependencies:
/opt/DL/pytorch/bin/install_dependencies

The dependencies must be installed in whichever python environment you intend to use. We've encountered some problems installing the PyTorch dependencies directly in the base environment if the base conda environment has been updated to conda version 4.6.2. If you want to use PyTorch, be sure to use a conda environment with a lower version of conda.

=== Install other dependencies ===

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the <code>scikit-learn</code> package in your python3 environment.

First you need to activate the environment:
conda activate py36

Then you install the package
conda install scikit-learn

A list of recommended packages follows in the next section.

=== Recommended packages ===

==== Jupyter Notebooks for deep learning ====
conda install jupyter

=== (Old Method) Testing Deep Learning packages on the login nodes or non-GPU nodes ===

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:
source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

== Compiling Software for DeepSense ==

DeepSense uses IBM Power8 systems running RedHat Enterprise Linux. Code must be compiled for <code>ppc64le</code> which is PowerPC 64 bit Little Endian.

Some software may not have binaries available for <code>ppc64le</code> even if it does for other systems. If this happens then you (or [[Contact_Information|DeepSense support]]) will need to compile the software to run on DeepSense. Visit the web page for the software and see if the source code is available (e.g. through github). If so then follow the compilation instructions to run the software.

You may encounter errors when attempting to compile software for <code>ppc64le</code>. Often this occurs because of differences between <code>ppc64le</code> and other common architectures such as x86 and x86_64.

For example, one DeepSense user attempted to compile the rdkit software package from https://www.rdkit.org/ . This compilation failed when it attempted to use the gcc x86 optimization <code>-mpopcnt</code>. After replacing the optimization with the <code>ppc64le</code> equivalent <code>-mpopcntb</code> the software compiled successfully.

Getting started with Deep Learning

2020-04-28T18:29:08Z

Cwhidden: /* ssh command port forwarding */

<div class="noautonum">

== 1. Get started with DeepSense ==

Follow all the steps from [[Getting started]]. This tutorial assumes you can log on to the DeepSense compute platform and have a version of Anaconda python on your path. We recommend installing Anaconda in your home directory before starting this tutorial (See [[Installing local software]]).

== 2. Download Caffe samples to your home directory ==

<code>/opt/DL/caffe/bin/caffe-install-samples</code>

== 3. Request an interactive session on a GPU compute node ==



<code>bsub -Is -gpu - bash</code>

== 4. Start a python2 Jupyter notebook ==

=== Source the Caffe deep learning toolkit ===

<code>source /opt/DL/caffe/bin/caffe-activate</code>

=== Start the notebook ===

<code>jupyter notebook --no-browser --ip=0.0.0.0</code>

=== Sample output ===
<pre>[I 13:32:23.937 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:32:23.937 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://ds-cmgpu-04:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba
</pre>

=== Copy the URL, host, and port ===

Copy the URL but don’t paste it in your browser yet.

Make a note of which compute host and port the notebook is running on (e.g. host ds-cmgpu-04 and port 8888 in this case)

== 5. Port Forwarding ==

In a separate terminal window from your local computer, forward your local port to the remote host.

=== ssh command port forwarding ===

<code> ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port></code>

for example, <code>ssh -l user1 login1.deepsense.ca -L 8888:ds-cmgpu-04:8888</code>

Note that you may need to use a different <local_port> than 8888 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8888 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8889 or another unused port.

=== PuTTY port forwarding on Windows ===

If you are using a PuTTY terminal from a Windows computer to access DeepSense then you can still forward ports.

Before starting your session, scroll down to the option <code>Connection->SSH->Tunnels</code> in the Category pane.

Enter the <code>local_port</code> in the <code>Source port</code> field. For example, <code>8888</code>.

Enter <code><remote_host>:<remote_port></code> in the Destination field. For example, <code>ds-cmgpu-04:8888</code>.

Press the <code>Add</code> button to add the port forwarding rule to your PuTTY session.

Finally, open the session as usual.

== 6. Open the desired sample notebook ==

Enter the copied URL in your web browser but change the remote host name to “localhost” before pressing enter.

e.g <code>http://localhost:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba</code>

'''Note''': On our macs, this worked in Chrome, but not in Safari. Unfortunately, there was no error reported, it simply could not connect.

Be sure to enter the location of the “caffe-samples” directory in your home directory as your caffe-root in the Caffe example notebooks.

== 7. Enjoy Deep Learning on DeepSense! ==

== 8. More information ==

Go to Caffe's [http://caffe.berkeleyvision.org/ website] for tutorials and example programs that you can run to get started.
See the following links to a couple of the example programs:

[http://caffe.berkeleyvision.org/gathered/examples/mnist.html LeNet MNIST Tutorial] - Train a neural network to understand handwritten digits.

[http://caffe.berkeleyvision.org/gathered/examples/cifar10.html CIFAR-10 tutorial] - Train a convolutional neural network to classify small images.

== 9. Using another deep learning toolkit such as Tensorflow ==

* Ensure any Anaconda dependencies are installed
** for tensorflow, run <code>/opt/DL/tensorflow/bin/install_dependencies</code>
* Source the appropriate toolkit instead of caffe-activate
** e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>
* Download example notebooks for the deep learning toolkit to your home directory,
** e.g. <code> git clone https://github.com/aymericdamien/TensorFlow-Examples.git</code>

The TensorFlow [https://www.tensorflow.org/ home page] has various information, including Tutorials, How-To documents, and a Getting Started guide.

Additional tutorials and examples are available from the community, for example:

https://github.com/nlintz/TensorFlow-Tutorials

https://github.com/aymericdamien/TensorFlow-Examples

</div>

Known problems

2020-02-20T19:48:32Z

Cwhidden:

== The bhist command is not working ==

We are aware of a problem causing the bhist command to not find the job log file necessary to print information about completed jobs. Instead, the output is always "no matching job found" even if the user specifies the "-a" option to print information about all running, completed, and failed jobs.

While we work on solving this issue you can manually specify the location of the log file using the -f option. For example, the following command will print all running, completed, and failed jobs for the current user:

bhist -f /lsfshare/lsfswg/lsf/work/DeepSenseLSFCluster/logdir/lsb.events -a

== Where is my job output file? ==
Output may not be written to the specified file immediately when using the <code>-o <filename></code> or <code>-oo <filename></code> options. There are two workarounds for this problem:

# You can use the bpeek <jobid> command to view the output of a currently running job.
# You can send your output to a file with the typical unix output specifications such as <code>> <filename></code> with your executed programs or by specifying output files in programs that support such options.

== Jupyter notebooks or other programs fail trying to access a /run directory ==

The default login shell is BASH. Make sure the following parameter is in your .bashrc file in your home directory, as it prevents a problem where some types of jobs fail when run through the LSF queue. This should be done automatically the first time you log onto DeepSense.

<code>echo 'unset XDG_RUNTIME_DIR' >> ~/.bashrc</code>

This line has been added to the default .bashrc file for new users but older user accounts may need this step to be done manually.

== Browser fails to connect to Jupyter Notebooks ==

On our MacBook Pros, Jupyter notebooks work in Chrome, but don't work in safari. Unfortunately, no error is given. Safari just fails to connect. Please let us know if you have issues with any other browsers, and we can add that info here.

== Cannot Install PyTorch dependencies ==

UnsatisfiableError: The following specifications were found to be in conflict:
- powerai-pytorch-prereqs=0.4.1_12295.5cb3523

You may see this error when attempting to install the pytorch dependencies in a local anaconda environment. This error indicates that some of your installed python packages are not compatible with the pytorch prequisites. In particular, we see this error when conda has been updated to version 4.6 (which may sometimes happen when installing the tensorflow dependencies first).

To resolve this problem, create a new environment with a 4.5.x conda version and then install the pytorch dependencies in that environment.

== Cannot use Caffe on login node or compute nodes without GPUs ==

<nowiki>Cuda number of devices: -579579216
Current device id: -579579216
Current device name:
[==========] Running 2207 tests from 293 test cases.
[----------] Global test environment set-up.
[----------] 9 tests from AccuracyLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN ] AccuracyLayerTest/0.TestSetup
E0206 15:59:26.604874 7990 common.cpp:121] Cannot create Cublas handle. Cublas won't be available.
E0206 15:59:26.611477 7990 common.cpp:128] Cannot create Curand generator. Curand won't be available.
F0206 15:59:26.611616 7990 syncedmem.cpp:500] Check failed: error == cudaSuccess (30 vs. 0) unknown error
*** Check failure stack trace: ***
</nowiki>

You may see this error when attempting to use Caffe on a node without GPUs or a GPU node without specifically requesting a GPU.

To resolve this problem, use a GPU node and request a GPU. Caffe cannot run without an available GPU.

== Cannot see GPUs in an LSF job ==

$ nvidia-smi
No devices were found

GPUs must be requested with the <code>-gpu -</code> option to bsub. See [[LSF#GPU_Computation]] for more information.

== Nested anaconda environments may cause strange behaviour ==

Some users have experienced strange behaviour when activating an anaconda environment within another environment. This may include permission errors, loading incorrect versions of software, or strange conflicts when attempting to install packages. If you encounter problems with a nested anaconda environment then first try deactivating all anaconda environments and activating just the desired environment.

LSF

2019-12-05T20:13:05Z

Cwhidden:

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory for at most 24 hours you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -W 24:0 -R "span[hosts=1] rusage[mem=256000]" <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
LSF has two different types of memory limits.
The scheduler memory limit <code>-R "rusage[mem=<memlimit>]"</code> requests <code><memlimit></code> amount of memory. Your job will not start until a compute node is available with that amount of memory. You are guaranteed to have this amount of memory available. If you exceed the requested amount then your job may be killed but it will only be killed if other jobs need that memory.

The job memory limit <code>-R "rusage[mem=<memlimit>]</code> will kill your job if it exceeds the given memory limit. Note that this option does not guarantee that you will have that amount of memory available.

The memory limits are specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code> and <code>-R "rusage[mem=256GB]"</code>.

If you are using more than a few GB of memory than you must specify the <code>-R "rusage[mem=<memlimit>]"</code> option or your job may be terminated. You may additionally want to use the <code>-M <memlimit></code> option to be sure you aren't using more memory than intended.

=== Time Limit ===
The runtime limit <code>-W hours:minutes</code> specifies the maximum length of time your job is allowed to run.
For example <code>-W 24:0</code> requests 24 hours of running time.
Your job will be terminated when the runtime limit is exceeded.

If you do not specify a runtime limit then the default runtime limit of 168 hours (7 days) will be used.
The maximum possible runtime limit is currently 30 days and may vary by queue in the future.

If there is a scheduled maintenance window announced then any job with a run time limit that could extend into the maintenance period will be listed as pending and will not run until the maintenance has concluded. Use a shorter run time limit that ends before the maintenance period to avoid this.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=shared:mps=no:j_exclusive=no"</code>

<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources. If <code>mode=exclusive_process</code> then mps should be set to yes.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs?

By default the <code>-gpu -</code> option will request one nonexclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs and use shared GPUs when possible. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

Note that output may not be written to the specified file immediately. You can use the <code>bpeek <jobid></code> command to view the output of a currently running job.

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB -R "rusage[mem=100MB]"</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_kc_cmd_ref.html here].

Known problems

2019-12-05T20:07:47Z

Cwhidden:

== Where is my job output file? ==
Output may not be written to the specified file immediately when using the <code>-o <filename></code> or <code>-oo <filename></code> options. There are two workarounds for this problem:

# You can use the bpeek <jobid> command to view the output of a currently running job.
# You can send your output to a file with the typical unix output specifications such as <code>> <filename></code> with your executed programs or by specifying output files in programs that support such options.

== Jupyter notebooks or other programs fail trying to access a /run directory ==

The default login shell is BASH. Make sure the following parameter is in your .bashrc file in your home directory, as it prevents a problem where some types of jobs fail when run through the LSF queue. This should be done automatically the first time you log onto DeepSense.

<code>echo 'unset XDG_RUNTIME_DIR' >> ~/.bashrc</code>

This line has been added to the default .bashrc file for new users but older user accounts may need this step to be done manually.

== Browser fails to connect to Jupyter Notebooks ==

On our MacBook Pros, Jupyter notebooks work in Chrome, but don't work in safari. Unfortunately, no error is given. Safari just fails to connect. Please let us know if you have issues with any other browsers, and we can add that info here.

== Cannot Install PyTorch dependencies ==

UnsatisfiableError: The following specifications were found to be in conflict:
- powerai-pytorch-prereqs=0.4.1_12295.5cb3523

You may see this error when attempting to install the pytorch dependencies in a local anaconda environment. This error indicates that some of your installed python packages are not compatible with the pytorch prequisites. In particular, we see this error when conda has been updated to version 4.6 (which may sometimes happen when installing the tensorflow dependencies first).

To resolve this problem, create a new environment with a 4.5.x conda version and then install the pytorch dependencies in that environment.

== Cannot use Caffe on login node or compute nodes without GPUs ==

<nowiki>Cuda number of devices: -579579216
Current device id: -579579216
Current device name:
[==========] Running 2207 tests from 293 test cases.
[----------] Global test environment set-up.
[----------] 9 tests from AccuracyLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN ] AccuracyLayerTest/0.TestSetup
E0206 15:59:26.604874 7990 common.cpp:121] Cannot create Cublas handle. Cublas won't be available.
E0206 15:59:26.611477 7990 common.cpp:128] Cannot create Curand generator. Curand won't be available.
F0206 15:59:26.611616 7990 syncedmem.cpp:500] Check failed: error == cudaSuccess (30 vs. 0) unknown error
*** Check failure stack trace: ***
</nowiki>

You may see this error when attempting to use Caffe on a node without GPUs or a GPU node without specifically requesting a GPU.

To resolve this problem, use a GPU node and request a GPU. Caffe cannot run without an available GPU.

== Cannot see GPUs in an LSF job ==

$ nvidia-smi
No devices were found

GPUs must be requested with the <code>-gpu -</code> option to bsub. See [[LSF#GPU_Computation]] for more information.

== Nested anaconda environments may cause strange behaviour ==

Some users have experienced strange behaviour when activating an anaconda environment within another environment. This may include permission errors, loading incorrect versions of software, or strange conflicts when attempting to install packages. If you encounter problems with a nested anaconda environment then first try deactivating all anaconda environments and activating just the desired environment.

LSF

2019-12-05T20:03:02Z

Cwhidden: /* Input and Output files */

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory for at most 24 hours you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -W 24:0 -R “span[hosts=1] rusage[mem=256000]” <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
LSF has two different types of memory limits.
The scheduler memory limit <code>-R "rusage[mem=<memlimit>]"</code> requests <code><memlimit></code> amount of memory. Your job will not start until a compute node is available with that amount of memory. You are guaranteed to have this amount of memory available. If you exceed the requested amount then your job may be killed but it will only be killed if other jobs need that memory.

The job memory limit <code>-R "rusage[mem=<memlimit>]</code> will kill your job if it exceeds the given memory limit. Note that this option does not guarantee that you will have that amount of memory available.

The memory limits are specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code> and <code>-R "rusage[mem=256GB]"</code>.

If you are using more than a few GB of memory than you must specify the <code>-R "rusage[mem=<memlimit>]"</code> option or your job may be terminated. You may additionally want to use the <code>-M <memlimit></code> option to be sure you aren't using more memory than intended.

=== Time Limit ===
The runtime limit <code>-W hours:minutes</code> specifies the maximum length of time your job is allowed to run.
For example <code>-W 24:0</code> requests 24 hours of running time.
Your job will be terminated when the runtime limit is exceeded.

If you do not specify a runtime limit then the default runtime limit of 168 hours (7 days) will be used.
The maximum possible runtime limit is currently 30 days and may vary by queue in the future.

If there is a scheduled maintenance window announced then any job with a run time limit that could extend into the maintenance period will be listed as pending and will not run until the maintenance has concluded. Use a shorter run time limit that ends before the maintenance period to avoid this.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=shared:mps=no:j_exclusive=no"</code>

<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources. If <code>mode=exclusive_process</code> then mps should be set to yes.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs?

By default the <code>-gpu -</code> option will request one nonexclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs and use shared GPUs when possible. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

Note that output may not be written to the specified file immediately. You can use the <code>bpeek <jobid></code> command to view the output of a currently running job.

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB -R "rusage[mem=100MB]"</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_kc_cmd_ref.html here].

Installing local software

2019-12-05T20:01:25Z

Cwhidden:

== Introduction ==

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact [[Contact_Information|Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation.

If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Contact_Information|Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le).

If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Contact_Information|Technical Support]]. We may install and support it systemwide if there is sufficient interest.

== Installing Anaconda Python in your home directory ==

=== Stop using systemwide anaconda ===

If you added the system anaconda environment to your <code>.bashrc</code> file then remove the line:
. /opt/anaconda2/etc/profile.d/conda.sh

=== Installing Anaconda with a python3 base ===

From your home directory run:
wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-ppc64le.sh
bash Anaconda3-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
. ~/anaconda3/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:
source ~/.bashrc

=== Adding a python2 environment ===
The previous instruction creates a python3 base environment. To add a python2 environment:
conda create -n py27 python=2.7

Activate this environment to use python3:
conda activate py27

note: if you receive an error message then you may need to deactivate the base conda environment first:
conda deactivate
conda activate py27

=== Adding a python3 environment ===
We recommend creating a separate python3 environment from the base environment. This makes it easier to install the specific packages required for IBM PowerAI.
conda create -n py36 python=3.6

Activate this environment to use python3:
conda activate py36

=== Install PowerAI dependencies ===

Warning: these scripts will install, update, and downgrade some packages to the recommended packages for the current version of PowerAI. You may want to create a separate python environment to use different versions of those packages with other software.

To use Tensorflow first install the Tensorflow dependencies:
/opt/DL/tensorflow/bin/install_dependencies

To use PyTorch first install the PyTorch dependencies:
/opt/DL/pytorch/bin/install_dependencies

The dependencies must be installed in whichever python environment you intend to use. We've encountered some problems installing the PyTorch dependencies directly in the base environment if the base conda environment has been updated to conda version 4.6.2. If you want to use PyTorch, be sure to use a conda environment with a lower version of conda.

=== Install other dependencies ===

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the <code>scikit-learn</code> package in your python3 environment.

First you need to activate the environment:
conda activate py36

Then you install the package
conda install scikit-learn

A list of recommended packages follows in the next section.

=== Recommended packages ===

==== Jupyter Notebooks for deep learning ====
conda install jupyter

=== Testing Deep Learning packages on the login nodes or non-GPU nodes ===

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:
source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

== Compiling Software for DeepSense ==

DeepSense uses IBM Power8 systems running RedHat Enterprise Linux. Code must be compiled for <code>ppc64le</code> which is PowerPC 64 bit Little Endian.

Some software may not have binaries available for <code>ppc64le</code> even if it does for other systems. If this happens then you (or [[Contact_Information|DeepSense support]]) will need to compile the software to run on DeepSense. Visit the web page for the software and see if the source code is available (e.g. through github). If so then follow the compilation instructions to run the software.

You may encounter errors when attempting to compile software for <code>ppc64le</code>. Often this occurs because of differences between <code>ppc64le</code> and other common architectures such as x86 and x86_64.

For example, one DeepSense user attempted to compile the rdkit software package from https://www.rdkit.org/ . This compilation failed when it attempted to use the gcc x86 optimization <code>-mpopcnt</code>. After replacing the optimization with the <code>ppc64le</code> equivalent <code>-mpopcntb</code> the software compiled successfully.

Getting started with Deep Learning

2019-12-05T19:57:43Z

Cwhidden: /* 1. Get started with DeepSense */

<div class="noautonum">

== 1. Get started with DeepSense ==

Follow all the steps from [[Getting started]]. This tutorial assumes you can log on to the DeepSense compute platform and have a version of Anaconda python on your path. We recommend installing Anaconda in your home directory before starting this tutorial (See [[Installing local software]]).

== 2. Download Caffe samples to your home directory ==

<code>/opt/DL/caffe/bin/caffe-install-samples</code>

== 3. Request an interactive session on a GPU compute node ==



<code>bsub -Is -gpu - bash</code>

== 4. Start a python2 Jupyter notebook ==

=== Source the Caffe deep learning toolkit ===

<code>source /opt/DL/caffe/bin/caffe-activate</code>

=== Start the notebook ===

<code>jupyter notebook --no-browser --ip=0.0.0.0</code>

=== Sample output ===
<pre>[I 13:32:23.937 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:32:23.937 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://ds-cmgpu-04:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba
</pre>

=== Copy the URL, host, and port ===

Copy the URL but don’t paste it in your browser yet.

Make a note of which compute host and port the notebook is running on (e.g. host ds-cmgpu-04 and port 8888 in this case)

== 5. Port Forwarding ==

In a separate terminal window from your local computer, forward your local port to the remote host.

=== ssh command port forwarding ===

<code> ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port></code>

for example, <code>ssh -l user1 login1.deepsense.ca -L 8888:ds-cmgpu-04:8888</code>

=== PuTTY port forwarding on Windows ===

If you are using a PuTTY terminal from a Windows computer to access DeepSense then you can still forward ports.

Before starting your session, scroll down to the option <code>Connection->SSH->Tunnels</code> in the Category pane.

Enter the <code>local_port</code> in the <code>Source port</code> field. For example, <code>8888</code>.

Enter <code><remote_host>:<remote_port></code> in the Destination field. For example, <code>ds-cmgpu-04:8888</code>.

Press the <code>Add</code> button to add the port forwarding rule to your PuTTY session.

Finally, open the session as usual.

== 6. Open the desired sample notebook ==

Enter the copied URL in your web browser but change the remote host name to “localhost” before pressing enter.

e.g <code>http://localhost:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba</code>

'''Note''': On our macs, this worked in Chrome, but not in Safari. Unfortunately, there was no error reported, it simply could not connect.

Be sure to enter the location of the “caffe-samples” directory in your home directory as your caffe-root in the Caffe example notebooks.

== 7. Enjoy Deep Learning on DeepSense! ==

== 8. More information ==

Go to Caffe's [http://caffe.berkeleyvision.org/ website] for tutorials and example programs that you can run to get started.
See the following links to a couple of the example programs:

[http://caffe.berkeleyvision.org/gathered/examples/mnist.html LeNet MNIST Tutorial] - Train a neural network to understand handwritten digits.

[http://caffe.berkeleyvision.org/gathered/examples/cifar10.html CIFAR-10 tutorial] - Train a convolutional neural network to classify small images.

== 9. Using another deep learning toolkit such as Tensorflow ==

* Ensure any Anaconda dependencies are installed
** for tensorflow, run <code>/opt/DL/tensorflow/bin/install_dependencies</code>
* Source the appropriate toolkit instead of caffe-activate
** e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>
* Download example notebooks for the deep learning toolkit to your home directory,
** e.g. <code> git clone https://github.com/aymericdamien/TensorFlow-Examples.git</code>

The TensorFlow [https://www.tensorflow.org/ home page] has various information, including Tutorials, How-To documents, and a Getting Started guide.

Additional tutorials and examples are available from the community, for example:

https://github.com/nlintz/TensorFlow-Tutorials

https://github.com/aymericdamien/TensorFlow-Examples

</div>

Getting started with Deep Learning

2019-12-03T18:08:24Z

Cwhidden: /* PuTTY port forwarding on Windows */

<div class="noautonum">

== 1. Get started with DeepSense ==

Follow all the steps from [[Getting started]]. This tutorial assumes you can log on to the DeepSense compute platform and have a version of Anaconda python on your path.

== 2. Download Caffe samples to your home directory ==

<code>/opt/DL/caffe/bin/caffe-install-samples</code>

== 3. Request an interactive session on a GPU compute node ==



<code>bsub -Is -gpu - bash</code>

== 4. Start a python2 Jupyter notebook ==

=== Source the Caffe deep learning toolkit ===

<code>source /opt/DL/caffe/bin/caffe-activate</code>

=== Start the notebook ===

<code>jupyter notebook --no-browser --ip=0.0.0.0</code>

=== Sample output ===
<pre>[I 13:32:23.937 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:32:23.937 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://ds-cmgpu-04:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba
</pre>

=== Copy the URL, host, and port ===

Copy the URL but don’t paste it in your browser yet.

Make a note of which compute host and port the notebook is running on (e.g. host ds-cmgpu-04 and port 8888 in this case)

== 5. Port Forwarding ==

In a separate terminal window from your local computer, forward your local port to the remote host.

=== ssh command port forwarding ===

<code> ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port></code>

for example, <code>ssh -l user1 login1.deepsense.ca -L 8888:ds-cmgpu-04:8888</code>

=== PuTTY port forwarding on Windows ===

If you are using a PuTTY terminal from a Windows computer to access DeepSense then you can still forward ports.

Before starting your session, scroll down to the option <code>Connection->SSH->Tunnels</code> in the Category pane.

Enter the <code>local_port</code> in the <code>Source port</code> field. For example, <code>8888</code>.

Enter <code><remote_host>:<remote_port></code> in the Destination field. For example, <code>ds-cmgpu-04:8888</code>.

Press the <code>Add</code> button to add the port forwarding rule to your PuTTY session.

Finally, open the session as usual.

== 6. Open the desired sample notebook ==

Enter the copied URL in your web browser but change the remote host name to “localhost” before pressing enter.

e.g <code>http://localhost:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba</code>

'''Note''': On our macs, this worked in Chrome, but not in Safari. Unfortunately, there was no error reported, it simply could not connect.

Be sure to enter the location of the “caffe-samples” directory in your home directory as your caffe-root in the Caffe example notebooks.

== 7. Enjoy Deep Learning on DeepSense! ==

== 8. More information ==

Go to Caffe's [http://caffe.berkeleyvision.org/ website] for tutorials and example programs that you can run to get started.
See the following links to a couple of the example programs:

[http://caffe.berkeleyvision.org/gathered/examples/mnist.html LeNet MNIST Tutorial] - Train a neural network to understand handwritten digits.

[http://caffe.berkeleyvision.org/gathered/examples/cifar10.html CIFAR-10 tutorial] - Train a convolutional neural network to classify small images.

== 9. Using another deep learning toolkit such as Tensorflow ==

* Ensure any Anaconda dependencies are installed
** for tensorflow, run <code>/opt/DL/tensorflow/bin/install_dependencies</code>
* Source the appropriate toolkit instead of caffe-activate
** e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>
* Download example notebooks for the deep learning toolkit to your home directory,
** e.g. <code> git clone https://github.com/aymericdamien/TensorFlow-Examples.git</code>

The TensorFlow [https://www.tensorflow.org/ home page] has various information, including Tutorials, How-To documents, and a Getting Started guide.

Additional tutorials and examples are available from the community, for example:

https://github.com/nlintz/TensorFlow-Tutorials

https://github.com/aymericdamien/TensorFlow-Examples

</div>

Getting started with Deep Learning

2019-12-03T16:46:20Z

Cwhidden:

<div class="noautonum">

== 1. Get started with DeepSense ==

Follow all the steps from [[Getting started]]. This tutorial assumes you can log on to the DeepSense compute platform and have a version of Anaconda python on your path.

== 2. Download Caffe samples to your home directory ==

<code>/opt/DL/caffe/bin/caffe-install-samples</code>

== 3. Request an interactive session on a GPU compute node ==



<code>bsub -Is -gpu - bash</code>

== 4. Start a python2 Jupyter notebook ==

=== Source the Caffe deep learning toolkit ===

<code>source /opt/DL/caffe/bin/caffe-activate</code>

=== Start the notebook ===

<code>jupyter notebook --no-browser --ip=0.0.0.0</code>

=== Sample output ===
<pre>[I 13:32:23.937 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:32:23.937 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://ds-cmgpu-04:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba
</pre>

=== Copy the URL, host, and port ===

Copy the URL but don’t paste it in your browser yet.

Make a note of which compute host and port the notebook is running on (e.g. host ds-cmgpu-04 and port 8888 in this case)

== 5. Port Forwarding ==

In a separate terminal window from your local computer, forward your local port to the remote host.

=== ssh command port forwarding ===

<code> ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port></code>

for example, <code>ssh -l user1 login1.deepsense.ca -L 8888:ds-cmgpu-04:8888</code>

=== PuTTY port forwarding on Windows ===

If you are using a PuTTY terminal from a Windows computer to access DeepSense then you can still forward ports.

Before starting your session, scroll down to the option <code>Connection->SSH->Tunnels</code> in the Category pane.

Enter the <code>local_port</code> in the <code>Source port</code> field. For example, <code>8888</code>.

Enter <code><remote_host>:<remote_port></code> in the Destination field. For example, <code>ds-cmgpu-04:8888</code>.

Finally, open the session as usual.

== 6. Open the desired sample notebook ==

Enter the copied URL in your web browser but change the remote host name to “localhost” before pressing enter.

e.g <code>http://localhost:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba</code>

'''Note''': On our macs, this worked in Chrome, but not in Safari. Unfortunately, there was no error reported, it simply could not connect.

Be sure to enter the location of the “caffe-samples” directory in your home directory as your caffe-root in the Caffe example notebooks.

== 7. Enjoy Deep Learning on DeepSense! ==

== 8. More information ==

Go to Caffe's [http://caffe.berkeleyvision.org/ website] for tutorials and example programs that you can run to get started.
See the following links to a couple of the example programs:

[http://caffe.berkeleyvision.org/gathered/examples/mnist.html LeNet MNIST Tutorial] - Train a neural network to understand handwritten digits.

[http://caffe.berkeleyvision.org/gathered/examples/cifar10.html CIFAR-10 tutorial] - Train a convolutional neural network to classify small images.

== 9. Using another deep learning toolkit such as Tensorflow ==

* Ensure any Anaconda dependencies are installed
** for tensorflow, run <code>/opt/DL/tensorflow/bin/install_dependencies</code>
* Source the appropriate toolkit instead of caffe-activate
** e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>
* Download example notebooks for the deep learning toolkit to your home directory,
** e.g. <code> git clone https://github.com/aymericdamien/TensorFlow-Examples.git</code>

The TensorFlow [https://www.tensorflow.org/ home page] has various information, including Tutorials, How-To documents, and a Getting Started guide.

Additional tutorials and examples are available from the community, for example:

https://github.com/nlintz/TensorFlow-Tutorials

https://github.com/aymericdamien/TensorFlow-Examples

</div>

Installing local software

2019-11-07T17:42:56Z

Cwhidden: change instructions to use anaconda3 as the base environment

== Introduction ==

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact [[Contact_Information|Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation.

If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Contact_Information|Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le).

If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Contact_Information|Technical Support]]. We may install and support it systemwide if there is sufficient interest.

== Installing Anaconda Python in your home directory ==

=== Stop using systemwide anaconda ===

If you added the system anaconda environment to your <code>.bashrc</code> file then remove the line:
. /opt/anaconda2/etc/profile.d/conda.sh

=== Installing Anaconda with a python3 base ===

From your home directory run:
wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-ppc64le.sh
bash Anaconda3-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
. ~/anaconda3/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:
source ~/.bashrc

=== Adding a python2 environment ===
The previous instruction creates a python3 base environment. To add a python2 environment:
conda create -n py27 python=2.7

Activate this environment to use python3:
conda activate py27

note: if you receive an error message then you may need to deactivate the base conda environment first:
conda deactivate
conda activate py27

=== Adding a python3 environment ===
We recommend creating a separate python3 environment from the base environment. This makes it easier to install the specific packages required for IBM PowerAI.
conda create -n py36 python=3.6

Activate this environment to use python3:
conda activate py36

=== Install PowerAI dependencies ===

Warning: these scripts will install, update, and downgrade some packages to the recommended packages for the current version of PowerAI. You may want to create a separate python environment to use different versions of those packages with other software.

To use Tensorflow first install the Tensorflow dependencies:
/opt/DL/tensorflow/bin/install_dependencies

To use PyTorch first install the PyTorch dependencies:
/opt/DL/pytorch/bin/install_dependencies

The dependencies must be installed in whichever python environment you intend to use. We've encountered some problems installing the PyTorch dependencies directly in the base environment if the base conda environment has been updated to conda version 4.6.2. If you want to use PyTorch, be sure to use a conda environment with a lower version of conda.

=== Install other dependencies ===

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the <code>scikit-learn</code> package in your python3 environment.

First you need to activate the environment:
conda activate py36

Then you install the package
conda install scikit-learn

=== Testing Deep Learning packages on the login nodes or non-GPU nodes ===

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:
source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

== Compiling Software for DeepSense ==

DeepSense uses IBM Power8 systems running RedHat Enterprise Linux. Code must be compiled for <code>ppc64le</code> which is PowerPC 64 bit Little Endian.

Some software may not have binaries available for <code>ppc64le</code> even if it does for other systems. If this happens then you (or [[Contact_Information|DeepSense support]]) will need to compile the software to run on DeepSense. Visit the web page for the software and see if the source code is available (e.g. through github). If so then follow the compilation instructions to run the software.

You may encounter errors when attempting to compile software for <code>ppc64le</code>. Often this occurs because of differences between <code>ppc64le</code> and other common architectures such as x86 and x86_64.

For example, one DeepSense user attempted to compile the rdkit software package from https://www.rdkit.org/ . This compilation failed when it attempted to use the gcc x86 optimization <code>-mpopcnt</code>. After replacing the optimization with the <code>ppc64le</code> equivalent <code>-mpopcntb</code> the software compiled successfully.

Installing local software

2019-11-06T15:30:08Z

Cwhidden: /* Installing Anaconda with a python2 base */

== Introduction ==

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact [[Contact_Information|Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation.

If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Contact_Information|Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le).

If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Contact_Information|Technical Support]]. We may install and support it systemwide if there is sufficient interest.

== Installing Anaconda Python in your home directory ==

=== Stop using systemwide anaconda ===

If you added the system anaconda environment to your <code>.bashrc</code> file then remove the line:
. /opt/anaconda2/etc/profile.d/conda.sh

=== Installing Anaconda with a python2 base ===

From your home directory run:
wget https://repo.continuum.io/archive/Anaconda2-5.2.0-Linux-ppc64le.sh
bash Anaconda2-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
. ~/anaconda2/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:
source ~/.bashrc

=== Adding a python3 environment ===
The previous instruction creates a python2 base environment. To add a python3 environment:
conda create -n py36 python=3.6

Activate this environment to use python3:
conda activate py36

note: if you receive an error message then you may need to deactivate the base conda environment first:
conda deactivate
conda activate py36

=== Adding a python2 environment ===
We recommend creating a separate python2 environment from the base environment. This makes it easier to install the specific packages required for IBM PowerAI.
conda create -n py27 python=2.7

Activate this environment to use python2:
conda activate py27

=== Install PowerAI dependencies ===

Warning: these scripts will install, update, and downgrade some packages to the recommended packages for the current version of PowerAI. You may want to create a separate python environment to use different versions of those packages with other software.

To use Tensorflow first install the Tensorflow dependencies:
/opt/DL/tensorflow/bin/install_dependencies

To use PyTorch first install the PyTorch dependencies:
/opt/DL/pytorch/bin/install_dependencies

The dependencies must be installed in whichever python environment you intend to use. We've encountered some problems installing the PyTorch dependencies directly in the base environment if the base conda environment has been updated to conda version 4.6.2. If you want to use PyTorch, be sure to use a conda environment with a lower version of conda.

=== Install other dependencies ===

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the <code>scikit-learn</code> package in your python3 environment.

First you need to activate the environment:
conda activate py36

Then you install the package
conda install scikit-learn

=== Testing Deep Learning packages on the login nodes or non-GPU nodes ===

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:
source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

== Compiling Software for DeepSense ==

DeepSense uses IBM Power8 systems running RedHat Enterprise Linux. Code must be compiled for <code>ppc64le</code> which is PowerPC 64 bit Little Endian.

Some software may not have binaries available for <code>ppc64le</code> even if it does for other systems. If this happens then you (or [[Contact_Information|DeepSense support]]) will need to compile the software to run on DeepSense. Visit the web page for the software and see if the source code is available (e.g. through github). If so then follow the compilation instructions to run the software.

You may encounter errors when attempting to compile software for <code>ppc64le</code>. Often this occurs because of differences between <code>ppc64le</code> and other common architectures such as x86 and x86_64.

For example, one DeepSense user attempted to compile the rdkit software package from https://www.rdkit.org/ . This compilation failed when it attempted to use the gcc x86 optimization <code>-mpopcnt</code>. After replacing the optimization with the <code>ppc64le</code> equivalent <code>-mpopcntb</code> the software compiled successfully.

Available software

2019-10-15T14:26:15Z

Cwhidden: /* Bioinformatics Software */

== Basic Software ==

* RedHat Enterprise Linux Server release 7.5 (RHEL)
* gcc 4.8.5
* glibc 2.17
* R 3.5.1

== Anaconda Python ==

Two Anaconda python environments are installed locally on each DeepSense compute node:
{| class="wikitable"
! Version
! Environment location
|-
|python 2.7.15
|/opt/anaconda2
|-
|python 3.6.8
|/opt/anaconda2/envs/py36
|}

These python environments have many packages installed, including prerequisite libraries for running the IBM PowerAI deep learning frameworks.

See [[Getting_started]] for instructions on using the shared anaconda python environments.

See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory.

== IBM PowerAI Deep Learning Packages ==

[https://developer.ibm.com/linuxonpower/deep-learning-powerai/ IBM PowerAI] includes multiple open source deep learning frameworks compiled for IBM Power8 systems.

IBM PowerAI Enterprise includes:

{| class="wikitable"
!Framework
!colspan="2"|Location
|-
|Caffe
|/opt/DL/caffe
|-
|cuDNN
|/opt/DL/cudnn
|-
|IBM Distributed Deep Learning (DDL)
|/opt/DL/ddl
|-
| HDF5
|/opt/DL/hdf5
|-
|NCCL
|/opt/DL/nccl
|/opt/DL/nccl2
|-
|openblas
|/opt/DL/openblas
|-
|protobuf
|/opt/DL/protobuf
|-
|pytorch
|/opt/DL/pytorch
|-
|snap-ml
|/opt/DL/snap-ml-local
|/opt/DL/snap-ml-mpi
|-
|Tensorflow 1.11 (including keras)
|/opt/DL/tensorflow
|/opt/DL/ddl-tensorflow
|-
|Tensorboard
|/opt/DL/tensorboard
|}

To use most of these frameworks you need to activate a python2 or python3 environment and then activate the relevant framework.

For example, to use tensorflow you can activate a python2 environment:
. /opt/anaconda2/etc/profile.d/conda.sh
conda activate

and then activate tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

You can then <code>import tensorflow as tf</code> in your python code.

See [[Getting started with Deep Learning]] for a tutorial on using Caffe and Tensorflow on Deep Sense.

== IBM Advance Toolchain ==

You may require newer versions of compilers such as GCC than are provided with RHEL.

The [https://developer.ibm.com/linuxonpower/advance-toolchain IBM Advance Toolchain for Linux on Power] is a set of open source compilers, runtime libraries, and development tools.

The IBM Advance Toolchain] includes recent versions of:
* GNU Compiler Collection (gcc, g++ and gfortran)
* GNU C library (glibc)
* GNU Binary Utilities (binutils)
* Decimal Floating Point Library (libdfp)
* IBM Power Architecture Facilities Library (PAFLib)
* GNU Debugger (gdb)
* Python
* Golang
* Performance analysis tools (oprofile, valgrind, itrace)
* Multi-core exploitation libraries (TBB, Userspace RCU, SPHDE)
* support libraries (libhugetlbfs, Boost, zlib, etc)

To use the the Advance Toolchain, first activate environment modules:
source /usr/local/Modules/init/bash

Then load the advance toolchain:
module load at12.0

To stop using the advance toolchain, unload the environment module:
module unload at12.0

Note that software dynamically compiled with the advance toolchain will only run with the advance toolchain loaded.

== Requested Software ==

Software packages that are requested for use by DeepSense projects will be available in several locations. Our preference is to use conda packages when available.

=== External conda channels ===
If a requested software is available for ppc64le systems from an externally maintained anaconda channel then we will simply list the channel. You can install such software into a local anaconda environment using:
conda install -c <channel> <software>

=== Internal conda packages ===
When possible, software compiled by DeepSense staff will compiled using conda build and placed in a subdirectory of <code>/software/conda-bld/</code> . You can install such software into a local anaconda environment using:
conda install -c file://software/conda-bld/ <software>

=== Shared software ===
Some software will simply be installed in its own subdirectory of <code>/software</code>. You can run this software directly from its subdirectory.

=== Bioinformatics Software ===

{| class="wikitable"
!Software
!Version
!Location
|-
|trimmomatic
|0.39
|/software/trimmomattic-0.39
|-
|cutadapt
|2.3
|/software/conda-bld/linux-ppc64le/
|-
|bowtie2
|
|biobuilds channel
|-
|LAST
|980
|/software/last-980
|-
|Burrows wheeler aligner
|0.7.15
|/software/bwa
|-
|pb-falcon
|2.2.0
|/software/conda-bld/linux-ppc64le/
|-
|MASURCA
|3.3.4
|/software/conda-bld/linux-ppc64le/
|-
|Samtools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|htslib
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|bcftools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|gatk
|4.1.2.0
|/software/conda-bld/noarch/
|-
|stacks
|2.4
|/software/conda-bld/linux-ppc64le/
|-
|angsd
|0.923
|/software/conda-bld/linux-ppc64le/
|-
|vcftools
|
|biobuilds channel
|-
|plink
|
|biobuilds channel
|-
|msprime
|0.7.0
|/software/conda-bld/linux-ppc64le/
|-
|slim
|3.3
|/software/slim-3.3
|-
|DeepGSR
|
|/software/DeepGSR
|}

== Requesting Additional Software ==

Contact DeepSense [[contact information|support]] to have additional software installed or for help installing or compiling software locally in your home directory.

Available software

2019-10-15T14:25:24Z

Cwhidden: /* Shared software */

== Basic Software ==

* RedHat Enterprise Linux Server release 7.5 (RHEL)
* gcc 4.8.5
* glibc 2.17
* R 3.5.1

== Anaconda Python ==

Two Anaconda python environments are installed locally on each DeepSense compute node:
{| class="wikitable"
! Version
! Environment location
|-
|python 2.7.15
|/opt/anaconda2
|-
|python 3.6.8
|/opt/anaconda2/envs/py36
|}

These python environments have many packages installed, including prerequisite libraries for running the IBM PowerAI deep learning frameworks.

See [[Getting_started]] for instructions on using the shared anaconda python environments.

See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory.

== IBM PowerAI Deep Learning Packages ==

[https://developer.ibm.com/linuxonpower/deep-learning-powerai/ IBM PowerAI] includes multiple open source deep learning frameworks compiled for IBM Power8 systems.

IBM PowerAI Enterprise includes:

{| class="wikitable"
!Framework
!colspan="2"|Location
|-
|Caffe
|/opt/DL/caffe
|-
|cuDNN
|/opt/DL/cudnn
|-
|IBM Distributed Deep Learning (DDL)
|/opt/DL/ddl
|-
| HDF5
|/opt/DL/hdf5
|-
|NCCL
|/opt/DL/nccl
|/opt/DL/nccl2
|-
|openblas
|/opt/DL/openblas
|-
|protobuf
|/opt/DL/protobuf
|-
|pytorch
|/opt/DL/pytorch
|-
|snap-ml
|/opt/DL/snap-ml-local
|/opt/DL/snap-ml-mpi
|-
|Tensorflow 1.11 (including keras)
|/opt/DL/tensorflow
|/opt/DL/ddl-tensorflow
|-
|Tensorboard
|/opt/DL/tensorboard
|}

To use most of these frameworks you need to activate a python2 or python3 environment and then activate the relevant framework.

For example, to use tensorflow you can activate a python2 environment:
. /opt/anaconda2/etc/profile.d/conda.sh
conda activate

and then activate tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

You can then <code>import tensorflow as tf</code> in your python code.

See [[Getting started with Deep Learning]] for a tutorial on using Caffe and Tensorflow on Deep Sense.

== IBM Advance Toolchain ==

You may require newer versions of compilers such as GCC than are provided with RHEL.

The [https://developer.ibm.com/linuxonpower/advance-toolchain IBM Advance Toolchain for Linux on Power] is a set of open source compilers, runtime libraries, and development tools.

The IBM Advance Toolchain] includes recent versions of:
* GNU Compiler Collection (gcc, g++ and gfortran)
* GNU C library (glibc)
* GNU Binary Utilities (binutils)
* Decimal Floating Point Library (libdfp)
* IBM Power Architecture Facilities Library (PAFLib)
* GNU Debugger (gdb)
* Python
* Golang
* Performance analysis tools (oprofile, valgrind, itrace)
* Multi-core exploitation libraries (TBB, Userspace RCU, SPHDE)
* support libraries (libhugetlbfs, Boost, zlib, etc)

To use the the Advance Toolchain, first activate environment modules:
source /usr/local/Modules/init/bash

Then load the advance toolchain:
module load at12.0

To stop using the advance toolchain, unload the environment module:
module unload at12.0

Note that software dynamically compiled with the advance toolchain will only run with the advance toolchain loaded.

== Bioinformatics Software ==

Software packages that are requested for use by DeepSense projects will be available in several locations. Our preference is to use conda packages when available.

=== External conda channels ===
If a requested software is available for ppc64le systems from an externally maintained anaconda channel then we will simply list the channel. You can install such software into a local anaconda environment using:
conda install -c <channel> <software>

=== Internal conda packages ===
When possible, software compiled by DeepSense staff will compiled using conda build and placed in a subdirectory of <code>/software/conda-bld/</code> . You can install such software into a local anaconda environment using:
conda install -c file://software/conda-bld/ <software>

=== Shared software ===
Some software will simply be installed in its own subdirectory of <code>/software</code>. You can run this software directly from its subdirectory.

{| class="wikitable"
!Software
!Version
!Location
|-
|trimmomatic
|0.39
|/software/trimmomattic-0.39
|-
|cutadapt
|2.3
|/software/conda-bld/linux-ppc64le/
|-
|bowtie2
|
|biobuilds channel
|-
|LAST
|980
|/software/last-980
|-
|Burrows wheeler aligner
|0.7.15
|/software/bwa
|-
|pb-falcon
|2.2.0
|/software/conda-bld/linux-ppc64le/
|-
|MASURCA
|3.3.4
|/software/conda-bld/linux-ppc64le/
|-
|Samtools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|htslib
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|bcftools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|gatk
|4.1.2.0
|/software/conda-bld/noarch/
|-
|stacks
|2.4
|/software/conda-bld/linux-ppc64le/
|-
|angsd
|0.923
|/software/conda-bld/linux-ppc64le/
|-
|vcftools
|
|biobuilds channel
|-
|plink
|
|biobuilds channel
|-
|msprime
|0.7.0
|/software/conda-bld/linux-ppc64le/
|-
|slim
|3.3
|/software/slim-3.3
|-
|DeepGSR
|
|/software/DeepGSR
|}

== Requesting Additional Software ==

Contact DeepSense [[contact information|support]] to have additional software installed or for help installing or compiling software locally in your home directory.

Available software

2019-10-15T14:22:11Z

Cwhidden: /* Shared software */

== Basic Software ==

* RedHat Enterprise Linux Server release 7.5 (RHEL)
* gcc 4.8.5
* glibc 2.17
* R 3.5.1

== Anaconda Python ==

Two Anaconda python environments are installed locally on each DeepSense compute node:
{| class="wikitable"
! Version
! Environment location
|-
|python 2.7.15
|/opt/anaconda2
|-
|python 3.6.8
|/opt/anaconda2/envs/py36
|}

These python environments have many packages installed, including prerequisite libraries for running the IBM PowerAI deep learning frameworks.

See [[Getting_started]] for instructions on using the shared anaconda python environments.

See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory.

== IBM PowerAI Deep Learning Packages ==

[https://developer.ibm.com/linuxonpower/deep-learning-powerai/ IBM PowerAI] includes multiple open source deep learning frameworks compiled for IBM Power8 systems.

IBM PowerAI Enterprise includes:

{| class="wikitable"
!Framework
!colspan="2"|Location
|-
|Caffe
|/opt/DL/caffe
|-
|cuDNN
|/opt/DL/cudnn
|-
|IBM Distributed Deep Learning (DDL)
|/opt/DL/ddl
|-
| HDF5
|/opt/DL/hdf5
|-
|NCCL
|/opt/DL/nccl
|/opt/DL/nccl2
|-
|openblas
|/opt/DL/openblas
|-
|protobuf
|/opt/DL/protobuf
|-
|pytorch
|/opt/DL/pytorch
|-
|snap-ml
|/opt/DL/snap-ml-local
|/opt/DL/snap-ml-mpi
|-
|Tensorflow 1.11 (including keras)
|/opt/DL/tensorflow
|/opt/DL/ddl-tensorflow
|-
|Tensorboard
|/opt/DL/tensorboard
|}

To use most of these frameworks you need to activate a python2 or python3 environment and then activate the relevant framework.

For example, to use tensorflow you can activate a python2 environment:
. /opt/anaconda2/etc/profile.d/conda.sh
conda activate

and then activate tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

You can then <code>import tensorflow as tf</code> in your python code.

See [[Getting started with Deep Learning]] for a tutorial on using Caffe and Tensorflow on Deep Sense.

== IBM Advance Toolchain ==

You may require newer versions of compilers such as GCC than are provided with RHEL.

The [https://developer.ibm.com/linuxonpower/advance-toolchain IBM Advance Toolchain for Linux on Power] is a set of open source compilers, runtime libraries, and development tools.

The IBM Advance Toolchain] includes recent versions of:
* GNU Compiler Collection (gcc, g++ and gfortran)
* GNU C library (glibc)
* GNU Binary Utilities (binutils)
* Decimal Floating Point Library (libdfp)
* IBM Power Architecture Facilities Library (PAFLib)
* GNU Debugger (gdb)
* Python
* Golang
* Performance analysis tools (oprofile, valgrind, itrace)
* Multi-core exploitation libraries (TBB, Userspace RCU, SPHDE)
* support libraries (libhugetlbfs, Boost, zlib, etc)

To use the the Advance Toolchain, first activate environment modules:
source /usr/local/Modules/init/bash

Then load the advance toolchain:
module load at12.0

To stop using the advance toolchain, unload the environment module:
module unload at12.0

Note that software dynamically compiled with the advance toolchain will only run with the advance toolchain loaded.

== Bioinformatics Software ==

Software packages that are requested for use by DeepSense projects will be available in several locations. Our preference is to use conda packages when available.

=== External conda channels ===
If a requested software is available for ppc64le systems from an externally maintained anaconda channel then we will simply list the channel. You can install such software into a local anaconda environment using:
conda install -c <channel> <software>

=== Internal conda packages ===
When possible, software compiled by DeepSense staff will compiled using conda build and placed in a subdirectory of <code>/software/conda-bld/</code> . You can install such software into a local anaconda environment using:
conda install -c file://software/conda-bld/ <software>

=== Shared software ===
Some software will simply be installed in its own subdirectory of <code>/software</code>. You can run this software directly from its subdirectory.

{| class="wikitable"
!Software
!Version
!Location
|-
|trimmomatic
|0.39
|/software/trimmomattic-0.39
|-
|cutadapt
|2.3
|/software/conda-bld/linux-ppc64le/
|-
|bowtie2
|
|biobuilds channel
|-
|LAST
|980
|/software/last-980
|-
|Burrows wheeler aligner
|0.7.15
|/software/bwa
|-
|pb-falcon
|2.2.0
|/software/conda-bld/linux-ppc64le/
|-
|MASURCA
|3.3.4
|/software/conda-bld/linux-ppc64le/
|-
|Samtools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|htslib
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|bcftools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|gatk
|4.1.2.0
|/software/conda-bld/noarch/
|-
|stacks
|2.4
|/software/conda-bld/linux-ppc64le/
|-
|angsd
|0.923
|/software/conda-bld/linux-ppc64le/
|-
|vcftools
|
|biobuilds channel
|-
|plink
|
|biobuilds channel
|-
|msprime
|0.7.0
|/software/conda-bld/linux-ppc64le/
|-
|slim
|3.3
|?
|-
|DeepGSR
|
|/software/DeepGSR
|}

== Requesting Additional Software ==

Contact DeepSense [[contact information|support]] to have additional software installed or for help installing or compiling software locally in your home directory.

Available software

2019-10-15T14:15:14Z

Cwhidden: Bioinformatics Software

== Basic Software ==

* RedHat Enterprise Linux Server release 7.5 (RHEL)
* gcc 4.8.5
* glibc 2.17
* R 3.5.1

== Anaconda Python ==

Two Anaconda python environments are installed locally on each DeepSense compute node:
{| class="wikitable"
! Version
! Environment location
|-
|python 2.7.15
|/opt/anaconda2
|-
|python 3.6.8
|/opt/anaconda2/envs/py36
|}

These python environments have many packages installed, including prerequisite libraries for running the IBM PowerAI deep learning frameworks.

See [[Getting_started]] for instructions on using the shared anaconda python environments.

See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory.

== IBM PowerAI Deep Learning Packages ==

[https://developer.ibm.com/linuxonpower/deep-learning-powerai/ IBM PowerAI] includes multiple open source deep learning frameworks compiled for IBM Power8 systems.

IBM PowerAI Enterprise includes:

{| class="wikitable"
!Framework
!colspan="2"|Location
|-
|Caffe
|/opt/DL/caffe
|-
|cuDNN
|/opt/DL/cudnn
|-
|IBM Distributed Deep Learning (DDL)
|/opt/DL/ddl
|-
| HDF5
|/opt/DL/hdf5
|-
|NCCL
|/opt/DL/nccl
|/opt/DL/nccl2
|-
|openblas
|/opt/DL/openblas
|-
|protobuf
|/opt/DL/protobuf
|-
|pytorch
|/opt/DL/pytorch
|-
|snap-ml
|/opt/DL/snap-ml-local
|/opt/DL/snap-ml-mpi
|-
|Tensorflow 1.11 (including keras)
|/opt/DL/tensorflow
|/opt/DL/ddl-tensorflow
|-
|Tensorboard
|/opt/DL/tensorboard
|}

To use most of these frameworks you need to activate a python2 or python3 environment and then activate the relevant framework.

For example, to use tensorflow you can activate a python2 environment:
. /opt/anaconda2/etc/profile.d/conda.sh
conda activate

and then activate tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

You can then <code>import tensorflow as tf</code> in your python code.

See [[Getting started with Deep Learning]] for a tutorial on using Caffe and Tensorflow on Deep Sense.

== IBM Advance Toolchain ==

You may require newer versions of compilers such as GCC than are provided with RHEL.

The [https://developer.ibm.com/linuxonpower/advance-toolchain IBM Advance Toolchain for Linux on Power] is a set of open source compilers, runtime libraries, and development tools.

The IBM Advance Toolchain] includes recent versions of:
* GNU Compiler Collection (gcc, g++ and gfortran)
* GNU C library (glibc)
* GNU Binary Utilities (binutils)
* Decimal Floating Point Library (libdfp)
* IBM Power Architecture Facilities Library (PAFLib)
* GNU Debugger (gdb)
* Python
* Golang
* Performance analysis tools (oprofile, valgrind, itrace)
* Multi-core exploitation libraries (TBB, Userspace RCU, SPHDE)
* support libraries (libhugetlbfs, Boost, zlib, etc)

To use the the Advance Toolchain, first activate environment modules:
source /usr/local/Modules/init/bash

Then load the advance toolchain:
module load at12.0

To stop using the advance toolchain, unload the environment module:
module unload at12.0

Note that software dynamically compiled with the advance toolchain will only run with the advance toolchain loaded.

== Bioinformatics Software ==

Software packages that are requested for use by DeepSense projects will be available in several locations. Our preference is to use conda packages when available.

=== External conda channels ===
If a requested software is available for ppc64le systems from an externally maintained anaconda channel then we will simply list the channel. You can install such software into a local anaconda environment using:
conda install -c <channel> <software>

=== Internal conda packages ===
When possible, software compiled by DeepSense staff will compiled using conda build and placed in a subdirectory of <code>/software/conda-bld/</code> . You can install such software into a local anaconda environment using:
conda install -c file://software/conda-bld/ <software>

=== Shared software ===
Some software will simply be installed in its own subdirectory of <code>/software</code>. You can run this software directly from its subdirectory.

{| class="wikitable"
!Software
!Version
!Location
|-
|trimmomatic
|0.39
|/software/trimmomattic-0.39
|-
|cutadapt
|2.3
|/software/conda-bld/linux-ppc64le/
|-
|bowtie2
|
|biobuilds channel
|-
|LAST
|980
|/software/last-980
|-
|Burrows wheeler aligner
|0.7.15
|/software/bwa
|-
|pb-falcon
|2.2.0
|/software/conda-bld/linux-ppc64le/
|-
|MASURCA
|3.3.4
|/software/conda-bld/linux-ppc64le/
|-
|Samtools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|htslib
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|bcftools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|gatk
|4.1.2.0
|/software/conda-bld/noarch/
|-
|stacks
|2.4
|/software/conda-bld/linux-ppc64le/
|-
|angsd
|0.923
|/software/conda-bld/linux-ppc64le/
|-
|vcftools
|
|biobuilds channel
|-
|plink
|
|biobuilds channel
|-
|msprime
|0.7.0
|/software/conda-bld/linux-ppc64le/
|-
|slim
|
|?
|-
|DeepGSR
|
|?
|}

== Requesting Additional Software ==

Contact DeepSense [[contact information|support]] to have additional software installed or for help installing or compiling software locally in your home directory.

Getting started

2019-09-23T17:59:38Z

Cwhidden: /* 3. Logging on */ clarify how to log on with ssh

Getting Started with DeepSense

<div class="noautonum">

== 1. Request access to DeepSense ==

If you belong to an approved DeepSense project but do not yet have an account then send an email to support@deepsense.ca with the subject "DeepSense Account Request" and provide your:
a) First and last name
b) Faculty of Computer Science username or requested FCS username
c) Dalhousie BannerID
d) Project ID
e) Project leader
f) Reason for requesting the account.

== 2. Change your password ==

If you require a new FCS username then your initial password is your BannerID. Please change it immediately upon receiving access to DeepSense.

You can change your password at https://www.cs.dal.ca/csid

Alternatively, you can contact cshelp@cs.dal.ca to reset your password.

== 3. Logging on ==

DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus.

For example, if your userid is <code>user1</code>, you can connect to deepsense by typing <code>ssh user1@login1.deepsense.ca</code> just like logging on to any other network computer.

From off campus you’ll need to use the Dalhousie VPN (https://wireless.dal.ca/vpnsoftware.php). If you are not a Dalhousie staff, student, or faculty but require offsite access and cannot use the Dalhousie VPN then contact your project leader or support@deepsense.ca to make different arrangements.

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes. Keep reading for instructions on how to submit compute jobs to dedicated compute nodes.

== 4. Transfer data ==

For more information, see [[Transferring Data]].

Deepsense has two protocol nodes, protocol1.deepsense.ca and protocol2.deepsense.ca . You can connect to these using the SAMBA transfer protocol, e.g. smb://protocol1.deepsense.ca with your username and password. Please contact your project leader or support@deepsense.ca if you need help transferring large amounts of data.

Data transferred through the protocol nodes will be located in the shared /data directory .

See [[Storage policies]] for more information about the available shared file systems, storage policies, and backup policies.

== 5. Configure your environment ==

DeepSense compute and management nodes are IBM Power8 computers (ppc64le) running Redhat Enterprise Linux. See [[Resources]] for more details on the available nodes.

=== 5.1 Loading a python environment ===

You have two options for using python on DeepSense. You can use the systemwide python install, managed by DeepSense administrators. This is recommended for users new to Linux. You will need to contact DeepSense support to have additional software packages installed in the systemwide python.

Alternatively, you can install an Anaconda python environment or other software in your home directory. This allows you to install or update packages or software without requesting and waiting for DeepSense staff.

==== Systemwide python (managed by DeepSense) ====

DeepSense nodes have anaconda2 python installed in /opt/anaconda2. To use this systemwide python add a parameter to your .bashrc file in your home directory:

<code>echo ". /opt/anaconda2/etc/profile.d/conda.sh" >> ~/.bashrc</code>

Then source your .bashrc file:
<code>source ~/.bashrc</code>

To load the python2 environment run <code>conda activate</code>

To use python3 you can activate the py36 environment:
<code>conda activate py36</code>

You can add either line to your .bashrc file to automatically load the desired environment when you log in.

==== Local python install (managed by individual user) ====

See [[Installing local software]] for more information.

== 6. Running compute jobs ==

DeepSense has two different methods of submitting compute jobs.

=== 6.1 Load Sharing Facility (LSF) ===

LSF is a set of command line tools for submitting compute jobs. You may be familiar with other similar software such as Sun Grid Engine or SLURM.

LSF jobs are submitted using the <code>bsub</code> command.

You can examine the progress of your currently running jobs with the <code>bjobs</code> command.

You can examine the available compute nodes and their available resources with the <code>bhosts</code> command.

For more information about using LSF see [[LSF]].

=== 6.2 Conductor with Spark (CWS) ===

CWS is an IBM web-based graphical interface for creating and running Apache Spark compute jobs.

To use CWS, connect to the IBM Spectrum Computing Cluster Management Console at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Log in with your username and password.

Note that currently you need to accept a self-signed web certificate. In the future this will be fixed.

For more information about using CWS see [[CWS]].

== 7. Deep Learning packages and other available software ==

DeepSense has a variety of Deep Learning packages installed as part of IBM PowerAI including Tensorflow, Caffe, and PyTorch. These packages are installed in /opt/DL/ on each compute node and typically need to be activated before using them, e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>.

Deep Learning packages are typically used on the GPU nodes but some deep learning packages can also be used on the login nodes and CPU-only nodes. This can be useful for testing your code or running CPU-bound workloads. To use the deep learning packages on the login or compute nodes you will also need to load the GPU libraries with <code>source /opt/DL/cudnn/bin/cudnn-activate</code>. Note that some deep learning packages may fail if run without a GPU, e.g. Caffe currently requires a GPU.

For a brief tutorial including running Caffe and Tensorflow in a Jupyter notebook see [[Getting started with Deep Learning]].

See [[Available software]] for the current list of installed software. If you require additional software you are welcome to install it locally in your home directory or contact DeepSense support.

== 8. Technical and research support ==

DeepSense has a dedicated support team of research scientists ready to help you with technical questions, installing software, or even research questions.

If you can't find the answer to your question on this wiki or need more extensive help then send an email to support@deepsense.ca .

See [[Technical support]] for more information about the support available.

</div>

Getting started with Deep Learning

2019-09-13T19:56:46Z

Cwhidden: /* 3. Request an interactive session on a GPU compute node */

<div class="noautonum">

== 1. Get started with DeepSense ==

Follow all the steps from [[Getting started]]. This tutorial assumes you can log on to the DeepSense compute platform and have a version of Anaconda python on your path.

== 2. Download Caffe samples to your home directory ==

<code>/opt/DL/caffe/bin/caffe-install-samples</code>

== 3. Request an interactive session on a GPU compute node ==



<code>bsub -Is -gpu - bash</code>

== 4. Start a python2 Jupyter notebook ==

=== Source the Caffe deep learning toolkit ===

<code>source /opt/DL/caffe/bin/caffe-activate</code>

=== Start the notebook ===

<code>jupyter notebook --no-browser --ip=0.0.0.0</code>

=== Sample output ===
<pre>[I 13:32:23.937 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 13:32:23.937 NotebookApp]

Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
http://ds-cmgpu-04:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba
</pre>

=== Copy the URL, host, and port ===

Copy the URL but don’t paste it in your browser yet.

Make a note of which compute host and port the notebook is running on (e.g. host ds-cmgpu-04 and port 8888 in this case)

== 5. Port Forwarding ==

In a separate terminal window from your local computer, forward your local port to the remote host:

<code> ssh -l <username> login1.deepsense.ca -L <port>:<remote_host>:<port></code>

for example, <code>ssh -l user1 login1.deepsense.ca -L 8888:ds-cmgpu-04:8888</code>

Enter the copied URL in your web browser but change the remote host name to “localhost” before pressing enter.

e.g <code>http://localhost:8888/?token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba&token=68042f40a10b500f3747ae0a232ee209fa4bf1aa384d29ba</code>

== 6. Open the desired sample notebook ==

Be sure to enter the location of the “caffe-samples” directory in your home directory as your caffe-root in the Caffe example notebooks.

== 7. Enjoy Deep Learning on DeepSense! ==

== 8. More information ==

Go to Caffe's [http://caffe.berkeleyvision.org/ website] for tutorials and example programs that you can run to get started.
See the following links to a couple of the example programs:

[http://caffe.berkeleyvision.org/gathered/examples/mnist.html LeNet MNIST Tutorial] - Train a neural network to understand handwritten digits.

[http://caffe.berkeleyvision.org/gathered/examples/cifar10.html CIFAR-10 tutorial] - Train a convolutional neural network to classify small images.

== 9. Using another deep learning toolkit such as Tensorflow ==

* Ensure any Anaconda dependencies are installed
** for tensorflow, run <code>/opt/DL/tensorflow/bin/install_dependencies</code>
* Source the appropriate toolkit instead of caffe-activate
** e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>
* Download example notebooks for the deep learning toolkit to your home directory,
** e.g. <code> git clone https://github.com/aymericdamien/TensorFlow-Examples.git</code>

The TensorFlow [https://www.tensorflow.org/ home page] has various information, including Tutorials, How-To documents, and a Getting Started guide.

Additional tutorials and examples are available from the community, for example:

https://github.com/nlintz/TensorFlow-Tutorials

https://github.com/aymericdamien/TensorFlow-Examples

</div>

Getting started

2019-08-27T17:04:19Z

Cwhidden: /* 6.2 Conductor with Spark (CWS) */

Getting Started with DeepSense

<div class="noautonum">

== 1. Request access to DeepSense ==

If you belong to an approved DeepSense project but do not yet have an account then send an email to support@deepsense.ca with the subject "DeepSense Account Request" and provide your:
a) First and last name
b) Faculty of Computer Science username or requested FCS username
c) Dalhousie BannerID
d) Project ID
e) Project leader
f) Reason for requesting the account.

== 2. Change your password ==

If you require a new FCS username then your initial password is your BannerID. Please change it immediately upon receiving access to DeepSense.

You can change your password at https://www.cs.dal.ca/csid

Alternatively, you can contact cshelp@cs.dal.ca to reset your password.

== 3. Logging on ==

DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus.

From off campus you’ll need to use the Dalhousie VPN (https://wireless.dal.ca/vpnsoftware.php). If you are not a Dalhousie staff, student, or faculty but require offsite access and cannot use the Dalhousie VPN then contact your project leader or support@deepsense.ca to make different arrangements.

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== 4. Transfer data ==

For more information, see [[Transferring Data]].

Deepsense has two protocol nodes, protocol1.deepsense.ca and protocol2.deepsense.ca . You can connect to these using the SAMBA transfer protocol, e.g. smb://protocol1.deepsense.ca with your username and password. Please contact your project leader or support@deepsense.ca if you need help transferring large amounts of data.

Data transferred through the protocol nodes will be located in the shared /data directory .

See [[Storage policies]] for more information about the available shared file systems, storage policies, and backup policies.

== 5. Configure your environment ==

DeepSense compute and management nodes are IBM Power8 computers (ppc64le) running Redhat Enterprise Linux. See [[Resources]] for more details on the available nodes.

=== 5.1 Loading a python environment ===

You have two options for using python on DeepSense. You can use the systemwide python install, managed by DeepSense administrators. This is recommended for users new to Linux. You will need to contact DeepSense support to have additional software packages installed in the systemwide python.

Alternatively, you can install an Anaconda python environment or other software in your home directory. This allows you to install or update packages or software without requesting and waiting for DeepSense staff.

==== Systemwide python (managed by DeepSense) ====

DeepSense nodes have anaconda2 python installed in /opt/anaconda2. To use this systemwide python add a parameter to your .bashrc file in your home directory:

<code>echo ". /opt/anaconda2/etc/profile.d/conda.sh" >> ~/.bashrc</code>

Then source your .bashrc file:
<code>source ~/.bashrc</code>

To load the python2 environment run <code>conda activate</code>

To use python3 you can activate the py36 environment:
<code>conda activate py36</code>

You can add either line to your .bashrc file to automatically load the desired environment when you log in.

==== Local python install (managed by individual user) ====

See [[Installing local software]] for more information.

== 6. Running compute jobs ==

DeepSense has two different methods of submitting compute jobs.

=== 6.1 Load Sharing Facility (LSF) ===

LSF is a set of command line tools for submitting compute jobs. You may be familiar with other similar software such as Sun Grid Engine or SLURM.

LSF jobs are submitted using the <code>bsub</code> command.

You can examine the progress of your currently running jobs with the <code>bjobs</code> command.

You can examine the available compute nodes and their available resources with the <code>bhosts</code> command.

For more information about using LSF see [[LSF]].

=== 6.2 Conductor with Spark (CWS) ===

CWS is an IBM web-based graphical interface for creating and running Apache Spark compute jobs.

To use CWS, connect to the IBM Spectrum Computing Cluster Management Console at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Log in with your username and password.

Note that currently you need to accept a self-signed web certificate. In the future this will be fixed.

For more information about using CWS see [[CWS]].

== 7. Deep Learning packages and other available software ==

DeepSense has a variety of Deep Learning packages installed as part of IBM PowerAI including Tensorflow, Caffe, and PyTorch. These packages are installed in /opt/DL/ on each compute node and typically need to be activated before using them, e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>.

Deep Learning packages are typically used on the GPU nodes but some deep learning packages can also be used on the login nodes and CPU-only nodes. This can be useful for testing your code or running CPU-bound workloads. To use the deep learning packages on the login or compute nodes you will also need to load the GPU libraries with <code>source /opt/DL/cudnn/bin/cudnn-activate</code>. Note that some deep learning packages may fail if run without a GPU, e.g. Caffe currently requires a GPU.

For a brief tutorial including running Caffe and Tensorflow in a Jupyter notebook see [[Getting started with Deep Learning]].

See [[Available software]] for the current list of installed software. If you require additional software you are welcome to install it locally in your home directory or contact DeepSense support.

== 8. Technical and research support ==

DeepSense has a dedicated support team of research scientists ready to help you with technical questions, installing software, or even research questions.

If you can't find the answer to your question on this wiki or need more extensive help then send an email to support@deepsense.ca .

See [[Technical support]] for more information about the support available.

</div>

LSF

2019-08-26T15:00:50Z

Cwhidden: /* Complicated Jobs */

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory for at most 24 hours you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -W 24:0 -R “span[hosts=1] rusage[mem=256000]” <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
LSF has two different types of memory limits.
The scheduler memory limit <code>-R "rusage[mem=<memlimit>]"</code> requests <code><memlimit></code> amount of memory. Your job will not start until a compute node is available with that amount of memory. You are guaranteed to have this amount of memory available. If you exceed the requested amount then your job may be killed but it will only be killed if other jobs need that memory.

The job memory limit <code>-R "rusage[mem=<memlimit>]</code> will kill your job if it exceeds the given memory limit. Note that this option does not guarantee that you will have that amount of memory available.

The memory limits are specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code> and <code>-R "rusage[mem=256GB]"</code>.

If you are using more than a few GB of memory than you must specify the <code>-R "rusage[mem=<memlimit>]"</code> option or your job may be terminated. You may additionally want to use the <code>-M <memlimit></code> option to be sure you aren't using more memory than intended.

=== Time Limit ===
The runtime limit <code>-W hours:minutes</code> specifies the maximum length of time your job is allowed to run.
For example <code>-W 24:0</code> requests 24 hours of running time.
Your job will be terminated when the runtime limit is exceeded.

If you do not specify a runtime limit then the default runtime limit of 168 hours (7 days) will be used.
The maximum possible runtime limit is currently 30 days and may vary by queue in the future.

If there is a scheduled maintenance window announced then any job with a run time limit that could extend into the maintenance period will be listed as pending and will not run until the maintenance has concluded. Use a shorter run time limit that ends before the maintenance period to avoid this.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=shared:mps=no:j_exclusive=no"</code>

<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources. If <code>mode=exclusive_process</code> then mps should be set to yes.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs?

By default the <code>-gpu -</code> option will request one nonexclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs and use shared GPUs when possible. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB -R "rusage[mem=100MB]"</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_kc_cmd_ref.html here].

LSF

2019-08-26T15:00:05Z

Cwhidden: /* Job Submission */ rusage memory limit

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory for at most 24 hours you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -W 24:0 -R “span[hosts=1] rusage[mem=256000]” <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
LSF has two different types of memory limits.
The scheduler memory limit <code>-R "rusage[mem=<memlimit>]"</code> requests <code><memlimit></code> amount of memory. Your job will not start until a compute node is available with that amount of memory. You are guaranteed to have this amount of memory available. If you exceed the requested amount then your job may be killed but it will only be killed if other jobs need that memory.

The job memory limit <code>-R "rusage[mem=<memlimit>]</code> will kill your job if it exceeds the given memory limit. Note that this option does not guarantee that you will have that amount of memory available.

The memory limits are specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code> and <code>-R "rusage[mem=256GB]"</code>.

If you are using more than a few GB of memory than you must specify the <code>-R "rusage[mem=<memlimit>]"</code> option or your job may be terminated. You may additionally want to use the <code>-M <memlimit></code> option to be sure you aren't using more memory than intended.

=== Time Limit ===
The runtime limit <code>-W hours:minutes</code> specifies the maximum length of time your job is allowed to run.
For example <code>-W 24:0</code> requests 24 hours of running time.
Your job will be terminated when the runtime limit is exceeded.

If you do not specify a runtime limit then the default runtime limit of 168 hours (7 days) will be used.
The maximum possible runtime limit is currently 30 days and may vary by queue in the future.

If there is a scheduled maintenance window announced then any job with a run time limit that could extend into the maintenance period will be listed as pending and will not run until the maintenance has concluded. Use a shorter run time limit that ends before the maintenance period to avoid this.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=shared:mps=no:j_exclusive=no"</code>

<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources. If <code>mode=exclusive_process</code> then mps should be set to yes.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs?

By default the <code>-gpu -</code> option will request one nonexclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs and use shared GPUs when possible. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_kc_cmd_ref.html here].

LSF

2019-07-03T19:23:09Z

Cwhidden: /* Time Limit */

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory for at most 24 hours you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -W 24:0 -R “span[hosts=1]” <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
The memory limit <code>-M</code> is specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code>.

=== Time Limit ===
The runtime limit <code>-W hours:minutes</code> specifies the maximum length of time your job is allowed to run.
For example <code>-W 24:0</code> requests 24 hours of running time.
Your job will be terminated when the runtime limit is exceeded.

If you do not specify a runtime limit then the default runtime limit of 168 hours (7 days) will be used.
The maximum possible runtime limit is currently 30 days and may vary by queue in the future.

If there is a scheduled maintenance window announced then any job with a run time limit that could extend into the maintenance period will be listed as pending and will not run until the maintenance has concluded. Use a shorter run time limit that ends before the maintenance period to avoid this.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=shared:mps=no:j_exclusive=no"</code>

<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources. If <code>mode=exclusive_process</code> then mps should be set to yes.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs?

By default the <code>-gpu -</code> option will request one nonexclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs and use shared GPUs when possible. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_kc_cmd_ref.html here].

LSF

2019-07-03T19:08:38Z

Cwhidden:

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory for at most 24 hours you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -W 24:0 -R “span[hosts=1]” <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
The memory limit <code>-M</code> is specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code>.

=== Time Limit ===
The runtime limit <code>-W hours:minutes</code> specifies the maximum length of time your job is allowed to run.
For example <code>-W 24:0</code> requests 24 hours of running time.
You job will be terminated when the runtime limit is exceeded.

If you do not specify a runtime limit then the default runtime limit of 168 hours (7 days) will be used.
The maximum possible runtime limit is currently 30 days and may vary by queue in the future.

If there is a scheduled maintenance window announced then any job with a run time limit that could extend into the maintenance period will be listed as pending and will not run until the maintenance has concluded. Use a shorter run time limit that ends before the maintenance period to avoid this.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=shared:mps=no:j_exclusive=no"</code>

<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources. If <code>mode=exclusive_process</code> then mps should be set to yes.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs?

By default the <code>-gpu -</code> option will request one nonexclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs and use shared GPUs when possible. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_kc_cmd_ref.html here].

Terms of Use

2019-05-30T15:48:21Z

Cwhidden: hide TOC

__NOTOC__

== General Principles ==
DeepSense resources are provided to enhance the ocean analytics economy in Nova Scotia.
Users must conform to ethical standards, follow all applicable laws and regulations, and follow all applicable Acceptable Use Policies

== University Acceptable Use Policies ==
The use of DeepSense resources is subject to the Dalhousie University and Faculty of Computer Science Acceptable Use Policies:

* [https://www.dal.ca/dept/university_secretariat/policies/information-management-and-technology/acceptable-use-policy-.html Dalhousie Acceptable Use Policy]
* [https://cdn.dal.ca/content/dam/dalhousie/pdf/faculty/computerscience/policies/fcs_policy_local.pdf Policy on Local Computer Resources]

== Resource Priority ==

DeepSense is intended for academic and commercial partnerships for oceans analytics and priority will be given to these projects. Others uses may be approved for training and entrepreneurial projects but are subject to available resources and it may be necessary at times to reduce the resources allocated to such use.

== Protection of Data ==

All your data and backups are stored in Canada.

Aspects of data management will follow industry best practices (as determined by DeepSense staff) unless specific arrangements are agreed for your project.

Your data will be stored and protected for the duration of your project and a short grace period thereafter. It is your responsibility to safeguard and delete your data after your project is terminated. Alternative arrangements can be made to accomodate longer storage or sharing data between projects.

=== Data Access ===
Members of your project team with deepsense accounts can access your team's data.

=== Data Storage ===
Data is stored for the duration of your project unless you request deletion sooner. Deletion requests will be confirmed with the project lead.

=== Confidentiality ===
All project data is confidential to your project team.

== Acknowledging DeepSense ==

Please [[Acknowledging_DeepSense|acknowledge DeepSense]] when publishing results that used DeepSense resources, including computing hardware, computing software, and staff expertise.

MediaWiki:Sidebar

2019-05-30T15:47:45Z

Cwhidden:

Terms of Use

2019-05-30T15:43:00Z

Cwhidden: Cwhidden moved page Acceptable Use Policy to Terms of Use

== General Principles ==
DeepSense resources are provided to enhance the ocean analytics economy in Nova Scotia.
Users must conform to ethical standards, follow all applicable laws and regulations, and follow all applicable Acceptable Use Policies

== University Acceptable Use Policies ==
The use of DeepSense resources is subject to the Dalhousie University and Faculty of Computer Science Acceptable Use Policies:

* [https://www.dal.ca/dept/university_secretariat/policies/information-management-and-technology/acceptable-use-policy-.html Dalhousie Acceptable Use Policy]
* [https://cdn.dal.ca/content/dam/dalhousie/pdf/faculty/computerscience/policies/fcs_policy_local.pdf Policy on Local Computer Resources]

== Resource Priority ==

DeepSense is intended for academic and commercial partnerships for oceans analytics and priority will be given to these projects. Others uses may be approved for training and entrepreneurial projects but are subject to available resources and it may be necessary at times to reduce the resources allocated to such use.

== Protection of Data ==

All your data and backups are stored in Canada.

Aspects of data management will follow industry best practices (as determined by DeepSense staff) unless specific arrangements are agreed for your project.

Your data will be stored and protected for the duration of your project and a short grace period thereafter. It is your responsibility to safeguard and delete your data after your project is terminated. Alternative arrangements can be made to accomodate longer storage or sharing data between projects.

=== Data Access ===
Members of your project team with deepsense accounts can access your team's data.

=== Data Storage ===
Data is stored for the duration of your project unless you request deletion sooner. Deletion requests will be confirmed with the project lead.

=== Confidentiality ===
All project data is confidential to your project team.

== Acknowledging DeepSense ==

Please [[Acknowledging_DeepSense|acknowledge DeepSense]] when publishing results that used DeepSense resources, including computing hardware, computing software, and staff expertise.

Acceptable Use Policy

2019-05-30T15:43:00Z

Cwhidden: Cwhidden moved page Acceptable Use Policy to Terms of Use

#REDIRECT [[Terms of Use]]

Terms of Use

2019-05-30T15:42:22Z

Cwhidden:

Terms of Use

2019-05-30T15:40:16Z

Cwhidden:

== General Principles ==
DeepSense resources are provided to enhance the ocean analytics economy in Nova Scotia.
Users must conform to ethical standards, follow all applicable laws and regulations, and follow all applicable Acceptable Use Policies

== Acceptable Use Policies ==
The use of DeepSense resources is subject to the Dalhousie University and Faculty of Computer Science Acceptable Use Policies:

* [https://www.dal.ca/dept/university_secretariat/policies/information-management-and-technology/acceptable-use-policy-.html Dalhousie Acceptable Use Policy]
* [https://cdn.dal.ca/content/dam/dalhousie/pdf/faculty/computerscience/policies/fcs_policy_local.pdf Policy on Local Computer Resources]

== Resource Priority ==

DeepSense is intended for academic and commercial partnerships for oceans analytics and priority will be given to these projects. Others uses may be approved for training and entrepreneurial projects but are subject to available resources and it may be necessary at times to reduce the resources allocated to such use.

== Protection of Data ==

All your data and backups are stored in Canada.

Aspects of data management will follow industry best practices (as determined by DeepSense staff) unless specific arrangements are agreed for your project.

Your data will be stored and protected for the duration of your project and a short grace period thereafter. It is your responsibility to safeguard and delete your data after your project is terminated. Alternative arrangements can be made to accomodate longer storage or sharing data between projects.

=== Data Access ===
Members of your project team with deepsense accounts can access your team's data.

=== Data Storage ===
Data is stored for the duration of your project unless you request deletion sooner. Deletion requests will be confirmed with the project lead.

=== Confidentiality ===
All project data is confidential to your project team.

== Acknowledging DeepSense ==

Please [[Acknowledging_DeepSense|acknowledge DeepSense]] when publishing results that used DeepSense resources, including computing hardware, computing software, and staff expertise.

Available software

2019-05-15T14:43:39Z

Cwhidden:

Installing local software

2019-04-15T18:08:22Z

Cwhidden: Compiling Software section

== Introduction ==

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact [[Contact_Information|Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation.

If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Contact_Information|Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le).

If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Contact_Information|Technical Support]]. We may install and support it systemwide if there is sufficient interest.

== Installing Anaconda Python in your home directory ==

=== Stop using systemwide anaconda ===

If you added the system anaconda environment to your <code>.bashrc</code> file then remove the line:
. /opt/anaconda2/etc/profile.d/conda.sh

=== Installing Anaconda with a python2 base ===

From your home directory run:
wget https://repo.continuum.io/archive/Anaconda2-5.2.0-Linux-ppc64le.sh
bash Anaconda2-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
~/anaconda2/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:
source ~/.bashrc

=== Adding a python3 environment ===
The previous instruction creates a python2 base environment. To add a python3 environment:
conda create -n py36 python=3.6

Activate this environment to use python3:
conda activate py36

note: if you receive an error message then you may need to deactivate the base conda environment first:
conda deactivate
conda activate py36

=== Adding a python2 environment ===
We recommend creating a separate python2 environment from the base environment. This makes it easier to install the specific packages required for IBM PowerAI.
conda create -n py27 python=2.7

Activate this environment to use python2:
conda activate py27

=== Install PowerAI dependencies ===

Warning: these scripts will install, update, and downgrade some packages to the recommended packages for the current version of PowerAI. You may want to create a separate python environment to use different versions of those packages with other software.

To use Tensorflow first install the Tensorflow dependencies:
/opt/DL/tensorflow/bin/install_dependencies

To use PyTorch first install the PyTorch dependencies:
/opt/DL/pytorch/bin/install_dependencies

The dependencies must be installed in whichever python environment you intend to use. We've encountered some problems installing the PyTorch dependencies directly in the base environment if the base conda environment has been updated to conda version 4.6.2. If you want to use PyTorch, be sure to use a conda environment with a lower version of conda.

=== Install other dependencies ===

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the <code>scikit-learn</code> package in your python3 environment.

First you need to activate the environment:
conda activate py36

Then you install the package
conda install scikit-learn

=== Testing Deep Learning packages on the login nodes or non-GPU nodes ===

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:
source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

== Compiling Software for DeepSense ==

DeepSense uses IBM Power8 systems running RedHat Enterprise Linux. Code must be compiled for <code>ppc64le</code> which is PowerPC 64 bit Little Endian.

Some software may not have binaries available for <code>ppc64le</code> even if it does for other systems. If this happens then you (or [[Contact_Information|DeepSense support]]) will need to compile the software to run on DeepSense. Visit the web page for the software and see if the source code is available (e.g. through github). If so then follow the compilation instructions to run the software.

You may encounter errors when attempting to compile software for <code>ppc64le</code>. Often this occurs because of differences between <code>ppc64le</code> and other common architectures such as x86 and x86_64.

For example, one DeepSense user attempted to compile the rdkit software package from https://www.rdkit.org/ . This compilation failed when it attempted to use the gcc x86 optimization <code>-mpopcnt</code>. After replacing the optimization with the <code>ppc64le</code> equivalent <code>-mpopcntb</code> the software compiled successfully.

LSF

2019-04-15T17:47:25Z

Cwhidden: /* Advanced Job Submission */ Added Interactive Jobs section

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -R “span[hosts=1]” <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
The memory limit <code>-M</code> is specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code>.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=shared:mps=no:j_exclusive=no"</code>

<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources. If <code>mode=exclusive_process</code> then mps should be set to yes.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs?

By default the <code>-gpu -</code> option will request one nonexclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs and use shared GPUs when possible. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_kc_cmd_ref.html here].

Known problems

2019-04-15T17:31:00Z

Cwhidden:

== Jupyter notebooks or other programs fail trying to access a /run directory ==

The default login shell is BASH. Make sure the following parameter is in your .bashrc file in your home directory, as it prevents a problem where some types of jobs fail when run through the LSF queue. This should be done automatically the first time you log onto DeepSense.

<code>echo 'unset XDG_RUNTIME_DIR' >> ~/.bashrc</code>

This line has been added to the default .bashrc file for new users but older user accounts may need this step to be done manually.

== Cannot Install PyTorch dependencies ==

UnsatisfiableError: The following specifications were found to be in conflict:
- powerai-pytorch-prereqs=0.4.1_12295.5cb3523

You may see this error when attempting to install the pytorch dependencies in a local anaconda environment. This error indicates that some of your installed python packages are not compatible with the pytorch prequisites. In particular, we see this error when conda has been updated to version 4.6 (which may sometimes happen when installing the tensorflow dependencies first).

To resolve this problem, create a new environment with a 4.5.x conda version and then install the pytorch dependencies in that environment.

== Cannot use Caffe on login node or compute nodes without GPUs ==

<nowiki>Cuda number of devices: -579579216
Current device id: -579579216
Current device name:
[==========] Running 2207 tests from 293 test cases.
[----------] Global test environment set-up.
[----------] 9 tests from AccuracyLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN ] AccuracyLayerTest/0.TestSetup
E0206 15:59:26.604874 7990 common.cpp:121] Cannot create Cublas handle. Cublas won't be available.
E0206 15:59:26.611477 7990 common.cpp:128] Cannot create Curand generator. Curand won't be available.
F0206 15:59:26.611616 7990 syncedmem.cpp:500] Check failed: error == cudaSuccess (30 vs. 0) unknown error
*** Check failure stack trace: ***
</nowiki>

You may see this error when attempting to use Caffe on a node without GPUs or a GPU node without specifically requesting a GPU.

To resolve this problem, use a GPU node and request a GPU. Caffe cannot run without an available GPU.

== Cannot see GPUs in an LSF job ==

$ nvidia-smi
No devices were found

GPUs must be requested with the <code>-gpu -</code> option to bsub. See [[LSF#GPU_Computation]] for more information.

== Nested anaconda environments may cause strange behaviour ==

Some users have experienced strange behaviour when activating an anaconda environment within another environment. This may include permission errors, loading incorrect versions of software, or strange conflicts when attempting to install packages. If you encounter problems with a nested anaconda environment then first try deactivating all anaconda environments and activating just the desired environment.

Installing local software

2019-04-10T16:04:51Z

Cwhidden: /* Introduction */

== Introduction ==

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact [[Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation.

If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le).

If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Technical Support]]. We may install and support it systemwide if there is sufficient interest.

== Installing Anaconda Python in your home directory ==

=== Stop using systemwide anaconda ===

If you added the system anaconda environment to your <code>.bashrc</code> file then remove the line:
. /opt/anaconda2/etc/profile.d/conda.sh

=== Installing Anaconda with a python2 base ===

From your home directory run:
wget https://repo.continuum.io/archive/Anaconda2-5.2.0-Linux-ppc64le.sh
bash Anaconda2-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
~/anaconda2/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:
source ~/.bashrc

=== Adding a python3 environment ===
The previous instruction creates a python2 base environment. To add a python3 environment:
conda create -n py36 python=3.6

Activate this environment to use python3:
conda activate py36

note: if you receive an error message then you may need to deactivate the base conda environment first:
conda deactivate
conda activate py36

=== Adding a python2 environment ===
We recommend creating a separate python2 environment from the base environment. This makes it easier to install the specific packages required for IBM PowerAI.
conda create -n py27 python=2.7

Activate this environment to use python2:
conda activate py27

=== Install PowerAI dependencies ===

Warning: these scripts will install, update, and downgrade some packages to the recommended packages for the current version of PowerAI. You may want to create a separate python environment to use different versions of those packages with other software.

To use Tensorflow first install the Tensorflow dependencies:
/opt/DL/tensorflow/bin/install_dependencies

To use PyTorch first install the PyTorch dependencies:
/opt/DL/pytorch/bin/install_dependencies

The dependencies must be installed in whichever python environment you intend to use. We've encountered some problems installing the PyTorch dependencies directly in the base environment if the base conda environment has been updated to conda version 4.6.2. If you want to use PyTorch, be sure to use a conda environment with a lower version of conda.

=== Install other dependencies ===

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the <code>scikit-learn</code> package in your python3 environment.

First you need to activate the environment:
conda activate py36

Then you install the package
conda install scikit-learn

=== Testing Deep Learning packages on the login nodes or non-GPU nodes ===

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:
source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

Getting started

2019-04-10T15:57:03Z

Cwhidden: /* 5. Configure your environment */ Separated systemwide and local python options so users don't follow the instructions to set up the systemwide python before learning that they can install it locally.

Getting Started with DeepSense

<div class="noautonum">

== 1. Request access to DeepSense ==

If you belong to an approved DeepSense project but do not yet have an account then send an email to support@deepsense.ca with the subject "DeepSense Account Request" and provide your:
a) First and last name
b) Faculty of Computer Science username or requested FCS username
c) Dalhousie BannerID
d) Project ID
e) Project leader
f) Reason for requesting the account.

== 2. Change your password ==

If you require a new FCS username then your initial password is your BannerID. Please change it immediately upon receiving access to DeepSense.

You can change your password at https://www.cs.dal.ca/csid

Alternatively, you can contact cshelp@cs.dal.ca to reset your password.

== 3. Logging on ==

DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus.

From off campus you’ll need to use the Dalhousie VPN (https://wireless.dal.ca/vpnsoftware.php). If you are not a Dalhousie staff, student, or faculty but require offsite access and cannot use the Dalhousie VPN then contact your project leader or support@deepsense.ca to make different arrangements.

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== 4. Transfer data ==

Deepsense has two protocol nodes, protocol1.deepsense.ca and protocol2.deepsense.ca . You can connect to these using the SAMBA transfer protocol, e.g. smb://protocol1.deepsense.ca with your username and password. Please contact your project leader or support@deepsense.ca if you need help transferring large amounts of data.

Data transferred through the protocol nodes will be located in the shared /data directory .

See [[Storage policies]] for more information about the available shared file systems, storage policies, and backup policies.

== 5. Configure your environment ==

DeepSense compute and management nodes are IBM Power8 computers (ppc64le) running Redhat Enterprise Linux. See [[Resources]] for more details on the available nodes.

=== 5.1 Loading a python environment ===

You have two options for using python on DeepSense. You can use the systemwide python install, managed by DeepSense administrators. This is recommended for users new to Linux. You will need to contact DeepSense support to have additional software packages installed in the systemwide python.

Alternatively, you can install an Anaconda python environment or other software in your home directory. This allows you to install or update packages or software without requesting and waiting for DeepSense staff.

==== Systemwide python (managed by DeepSense) ====

DeepSense nodes have anaconda2 python installed in /opt/anaconda2. To use this systemwide python add a parameter to your .bashrc file in your home directory:

<code>echo ". /opt/anaconda2/etc/profile.d/conda.sh" >> ~/.bashrc</code>

Then source your .bashrc file:
<code>source ~/.bashrc</code>

To load the python2 environment run <code>conda activate</code>

To use python3 you can activate the py36 environment:
<code>conda activate py36</code>

You can add either line to your .bashrc file to automatically load the desired environment when you log in.

==== Local python install (managed by individual user) ====

See [[Installing local software]] for more information.

== 6. Running compute jobs ==

DeepSense has two different methods of submitting compute jobs.

=== 6.1 Load Sharing Facility (LSF) ===

LSF is a set of command line tools for submitting compute jobs. You may be familiar with other similar software such as Sun Grid Engine or SLURM.

LSF jobs are submitted using the <code>bsub</code> command.

You can examine the progress of your currently running jobs with the <code>bjobs</code> command.

You can examine the available compute nodes and their available resources with the <code>bhosts</code> command.

For more information about using LSF see [[LSF]].

=== 6.2 Conductor with Spark (CWS) ===

CWS is an IBM web-based graphical interface for creating and running Apache Spark compute jobs.

To use CWS, connect to the IBM Spectrum Computing Cluster Management Console at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Log in with your username and password.

Note that currently you need to accept a self-signed web certificate. In the future this will be fixed.

For more information about using CWS see [[Conductor with Spark]].

== 7. Deep Learning packages and other available software ==

DeepSense has a variety of Deep Learning packages installed as part of IBM PowerAI including Tensorflow, Caffe, and PyTorch. These packages are installed in /opt/DL/ on each compute node and typically need to be activated before using them, e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>.

Deep Learning packages are typically used on the GPU nodes but some deep learning packages can also be used on the login nodes and CPU-only nodes. This can be useful for testing your code or running CPU-bound workloads. To use the deep learning packages on the login or compute nodes you will also need to load the GPU libraries with <code>source /opt/DL/cudnn/bin/cudnn-activate</code>. Note that some deep learning packages may fail if run without a GPU, e.g. Caffe currently requires a GPU.

For a brief tutorial including running Caffe and Tensorflow in a Jupyter notebook see [[Getting started with Deep Learning]].

See [[Available software]] for the current list of installed software. If you require additional software you are welcome to install it locally in your home directory or contact DeepSense support.

== 8. Technical and research support ==

DeepSense has a dedicated support team of research scientists ready to help you with technical questions, installing software, or even research questions.

If you can't find the answer to your question on this wiki or need more extensive help then send an email to support@deepsense.ca .

See [[Technical support]] for more information about the support available.

</div>

Contact information

2019-03-15T18:54:33Z

Cwhidden:

== For potential new industry or academic partners ==
If you are interested in the DeepSense ocean innovation platform then please visit https://deepsense.ca and contact info@deepsense.ca for more information on starting a DeepSense project.

== For students interested in working on current DeepSense projects ==
Please contact the scientific director of DeepSense, Evangelos Milios ([mailto:eem@cs.dal.ca eem@cs.dal.ca]).

== For technical support with the DeepSense platform ==

Contact [mailto:support@deepsense.ca support@deepsense.ca]. Your issue will be submitted to our ticket tracking queue and responded to promptly.

== DEEPSENSE-USERS Mailing List ==

We maintain a mailing list for deepsense users
https://listserv.dal.ca/index.cgi?A0=DEEPSENSE-USERS

The mailing list is primarily intended to provide information such as service interruptions or important updates.

During the onboarding process you should be added to the mailing list. If you are not on the mailing list and wish to be added you can visit the above link or send us an email using one of the above methods.

Available software

2019-03-15T18:49:02Z

Cwhidden: /* IBM PowerAI Deep Learning Packages */

== Basic Software ==

* RedHat Enterprise Linux Server release 7.5 (RHEL)
* gcc 4.8.5
* glibc 2.17

== Anaconda Python ==

Two Anaconda python environments are installed locally on each DeepSense compute node:
{| class="wikitable"
! Version
! Environment location
|-
|python 2.7.15
|/opt/anaconda2
|-
|python 3.6.8
|/opt/anaconda2/envs/py36
|}

These python environments have many packages installed, including prerequisite libraries for running the IBM PowerAI deep learning frameworks.

See [[Getting_started]] for instructions on using the shared anaconda python environments.

See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory.

== IBM PowerAI Deep Learning Packages ==

[https://developer.ibm.com/linuxonpower/deep-learning-powerai/ IBM PowerAI] includes multiple open source deep learning frameworks compiled for IBM Power8 systems.

IBM PowerAI Enterprise includes:

{| class="wikitable"
!Framework
!colspan="2"|Location
|-
|Caffe
|/opt/DL/caffe
|-
|cuDNN
|/opt/DL/cudnn
|-
|IBM Distributed Deep Learning (DDL)
|/opt/DL/ddl
|-
| HDF5
|/opt/DL/hdf5
|-
|NCCL
|/opt/DL/nccl
|/opt/DL/nccl2
|-
|openblas
|/opt/DL/openblas
|-
|protobuf
|/opt/DL/protobuf
|-
|pytorch
|/opt/DL/pytorch
|-
|snap-ml
|/opt/DL/snap-ml-local
|/opt/DL/snap-ml-mpi
|-
|Tensorflow 1.11 (including keras)
|/opt/DL/tensorflow
|/opt/DL/ddl-tensorflow
|-
|Tensorboard
|/opt/DL/tensorboard
|}

To use most of these frameworks you need to activate a python2 or python3 environment and then activate the relevant framework.

For example, to use tensorflow you can activate a python2 environment:
. /opt/anaconda2/etc/profile.d/conda.sh
conda activate

and then activate tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

You can then <code>import tensorflow as tf</code> in your python code.

See [[Getting started with Deep Learning]] for a tutorial on using Caffe and Tensorflow on Deep Sense.

== IBM Advance Toolchain ==

You may require newer versions of compilers such as GCC than are provided with RHEL.

The [https://developer.ibm.com/linuxonpower/advance-toolchain IBM Advance Toolchain for Linux on Power] is a set of open source compilers, runtime libraries, and development tools.

The IBM Advance Toolchain] includes recent versions of:
* GNU Compiler Collection (gcc, g++ and gfortran)
* GNU C library (glibc)
* GNU Binary Utilities (binutils)
* Decimal Floating Point Library (libdfp)
* IBM Power Architecture Facilities Library (PAFLib)
* GNU Debugger (gdb)
* Python
* Golang
* Performance analysis tools (oprofile, valgrind, itrace)
* Multi-core exploitation libraries (TBB, Userspace RCU, SPHDE)
* support libraries (libhugetlbfs, Boost, zlib, etc)

To use the the Advance Toolchain, first activate environment modules:
source /usr/local/Modules/init/bash

Then load the advance toolchain:
module load at12.0

To stop using the advance toolchain, unload the environment module:
module unload at12.0

Note that software dynamically compiled with the advance toolchain will only run with the advance toolchain loaded.

== Requesting Additional Software ==

Contact DeepSense [[contact information|support]] to have additional software installed or for help installing or compiling software locally in your home directory.

Known problems

2019-03-12T18:57:12Z

Cwhidden:

Getting started

2019-03-12T18:53:39Z

Cwhidden:

Getting Started with DeepSense

<div class="noautonum">

== 1. Request access to DeepSense ==

If you belong to an approved DeepSense project but do not yet have an account then send an email to support@deepsense.ca with the subject "DeepSense Account Request" and provide your:
a) First and last name
b) Faculty of Computer Science username or requested FCS username
c) Dalhousie BannerID
d) Project ID
e) Project leader
f) Reason for requesting the account.

== 2. Change your password ==

If you require a new FCS username then your initial password is your BannerID. Please change it immediately upon receiving access to DeepSense.

You can change your password at https://www.cs.dal.ca/csid

Alternatively, you can contact cshelp@cs.dal.ca to reset your password.

== 3. Logging on ==

DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus.

From off campus you’ll need to use the Dalhousie VPN (https://wireless.dal.ca/vpnsoftware.php). If you are not a Dalhousie staff, student, or faculty but require offsite access and cannot use the Dalhousie VPN then contact your project leader or support@deepsense.ca to make different arrangements.

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== 4. Transfer data ==

Deepsense has two protocol nodes, protocol1.deepsense.ca and protocol2.deepsense.ca . You can connect to these using the SAMBA transfer protocol, e.g. smb://protocol1.deepsense.ca with your username and password. Please contact your project leader or support@deepsense.ca if you need help transferring large amounts of data.

Data transferred through the protocol nodes will be located in the shared /data directory .

See [[Storage policies]] for more information about the available shared file systems, storage policies, and backup policies.

== 5. Configure your environment ==

DeepSense compute and management nodes are IBM Power8 computers (ppc64le) running Redhat Enterprise Linux. See [[Resources]] for more details on the available nodes.

=== 5.1 Loading a python environment ===

DeepSense nodes have anaconda2 python installed in /opt/anaconda2. To use this systemwide python add a parameter to your .bashrc file in your home directory:

<code>echo ". /opt/anaconda2/etc/profile.d/conda.sh" >> ~/.bashrc</code>

Then source your .bashrc file:
<code>source ~/.bashrc</code>

To load the python2 environment run <code>conda activate</code>

To use python3 you can activate the py36 environment:
<code>conda activate py36</code>

You can add either line to your .bashrc file to automatically load the desired environment when you log in.

Alternatively, you may wish to install Anaconda or other software locally in your home directory. This allows you to install or update packages or software without requesting and waiting for DeepSense staff. See [[Installing local software]] for more information.

== 6. Running compute jobs ==

DeepSense has two different methods of submitting compute jobs.

=== 6.1 Load Sharing Facility (LSF) ===

LSF is a set of command line tools for submitting compute jobs. You may be familiar with other similar software such as Sun Grid Engine or SLURM.

LSF jobs are submitted using the <code>bsub</code> command.

You can examine the progress of your currently running jobs with the <code>bjobs</code> command.

You can examine the available compute nodes and their available resources with the <code>bhosts</code> command.

For more information about using LSF see [[LSF]].

=== 6.2 Conductor with Spark (CWS) ===

CWS is an IBM web-based graphical interface for creating and running Apache Spark compute jobs.

To use CWS, connect to the IBM Spectrum Computing Cluster Management Console at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Log in with your username and password.

Note that currently you need to accept a self-signed web certificate. In the future this will be fixed.

For more information about using CWS see [[Conductor with Spark]].

== 7. Deep Learning packages and other available software ==

DeepSense has a variety of Deep Learning packages installed as part of IBM PowerAI including Tensorflow, Caffe, and PyTorch. These packages are installed in /opt/DL/ on each compute node and typically need to be activated before using them, e.g. <code>source /opt/DL/tensorflow/bin/tensorflow-activate</code>.

Deep Learning packages are typically used on the GPU nodes but some deep learning packages can also be used on the login nodes and CPU-only nodes. This can be useful for testing your code or running CPU-bound workloads. To use the deep learning packages on the login or compute nodes you will also need to load the GPU libraries with <code>source /opt/DL/cudnn/bin/cudnn-activate</code>. Note that some deep learning packages may fail if run without a GPU, e.g. Caffe currently requires a GPU.

For a brief tutorial including running Caffe and Tensorflow in a Jupyter notebook see [[Getting started with Deep Learning]].

See [[Available software]] for the current list of installed software. If you require additional software you are welcome to install it locally in your home directory or contact DeepSense support.

== 8. Technical and research support ==

DeepSense has a dedicated support team of research scientists ready to help you with technical questions, installing software, or even research questions.

If you can't find the answer to your question on this wiki or need more extensive help then send an email to support@deepsense.ca .

See [[Technical support]] for more information about the support available.

</div>