DeepSense Docs - User contributions [en-ca]

LSF

2022-06-10T18:13:38Z

Rdickson: /* LSF Command Reference */ replace dead link to LSF cmd ref

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory for at most 24 hours you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -W 24:0 -R "span[hosts=1] rusage[mem=256000]" <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

===Shell Scripts for Batch Jobs===
Users can just run a single command line to submit batch jobs. The job scheduler would take care of everything and users only need to check their output and/or errors. Users do not need to keep themselves logged in the systems when the jobs are running. An example job submission command is shown above.
However, if you will have to run your script in an environment that is not set as the default in your .bashrc file, you can write a simple shell script to set the environments. For example, you may want to use a specific Conda environment and/or Python version for your Python script, you would need to write a shell script to set the environments. Here is an example. I have a Python script whose name is "myPython.py" and it would need to use my anaconda3 and py36_tensorflow environments. I would create a shell script, say with name "myShellScript.sh", with the following contents:

#!/bin/bash
source ~/anaconda3/etc/profile.d/conda.sh
conda activate py36_tensorflow
python myPython.py
Then, save your edit and run the following command to make your shell script executable:
chmod +x myShellScript.sh
Then, submit your job:
bsub -gpu - /path/to/myShellScript.sh
Check if your job is submitted successfully.

===Check Job Progress of Batch Jobs===
When your batch jobs are not finished, you can check the progress of your jobs using command 'bpeek'. This command provides detailed information about your running jobs and it is very helpful for you to monitor and decide if you would need to terminate it for whatever reasons to save time.
You can always run 'bpeek jobid' to check the progress. You can also use 'man bpeek' to get more detailed usage of the command.
Be aware that this command doesn't work for interactive jobs.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
LSF has two different types of memory limits.
The scheduler memory limit <code>-R "rusage[mem=<memlimit>]"</code> requests <code><memlimit></code> amount of memory. Your job will not start until a compute node is available with that amount of memory. You are guaranteed to have this amount of memory available. If you exceed the requested amount then your job may be killed but it will only be killed if other jobs need that memory.

The job memory limit <code>-R "rusage[mem=<memlimit>]</code> will kill your job if it exceeds the given memory limit. Note that this option does not guarantee that you will have that amount of memory available.

The memory limits are specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code> and <code>-R "rusage[mem=256GB]"</code>.

If you are using more than a few GB of memory than you must specify the <code>-R "rusage[mem=<memlimit>]"</code> option or your job may be terminated. You may additionally want to use the <code>-M <memlimit></code> option to be sure you aren't using more memory than intended.

=== Time Limit ===
The runtime limit <code>-W hours:minutes</code> specifies the maximum length of time your job is allowed to run.
For example <code>-W 24:0</code> requests 24 hours of running time.
Your job will be terminated when the runtime limit is exceeded.

If you do not specify a runtime limit then the default runtime limit of 168 hours (7 days) will be used.
The maximum possible runtime limit is currently 30 days and may vary by queue in the future.

If there is a scheduled maintenance window announced then any job with a run time limit that could extend into the maintenance period will be listed as pending and will not run until the maintenance has concluded. Use a shorter run time limit that ends before the maintenance period to avoid this.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=exclusive_process:mps=yes:j_exclusive=yes"</code>
<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode. Yours jobs will be running on exclusive mode by default which means that no other jobs would share the gpus your jobs are using.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs? By default, it is 'yes' for your jobs.

By default the <code>-gpu -</code> option will request one exclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

Note that output may not be written to the specified file immediately. You can use the <code>bpeek <jobid></code> command to view the output of a currently running job.

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB -R "rusage[mem=100MB]"</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=reference-command here].

Installing Software

2022-03-08T15:39:39Z

Rdickson: /* 3.1 Using IBM-AI Deep Learning Anaconda Channel */ URL change on advice from CWhidden

<div class="noautonum">

== 1. Logging on ==

DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus.

For example, if your userid is <code>user1</code>, you can connect to deepsense by typing <code>ssh user1@login1.deepsense.ca</code> just like logging on to any other network computer.

'''Note''': The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes. Keep reading for instructions on how to submit compute jobs to dedicated compute nodes.

=== 1.1 VPN ===

To connect to the DeepSense platform from outside of the Dalhousie Campus, you'll need to use a VPN.
If you are are student, staff or faculty, you can use the Dalhousie VPN (https://wireless.dal.ca/vpnsoftware.php).

If you are not a Dalhousie staff, student, or faculty but require offsite access and cannot use the Dalhousie VPN then contact your project leader or ([mailto:support@deepsense.ca support@deepsense.ca]) to make different arrangements.

For more info, see [[VPN Setup]].

== 2. Configure your environment ==

DeepSense compute and management nodes are IBM Power8 computers (ppc64le) running Redhat Enterprise Linux. See [[Resources]] for more details on the available nodes.

=== 2.1 Loading a python environment ===

You have two options for using python on DeepSense. You can use the systemwide python install, managed by DeepSense administrators. This is recommended for users new to Linux. You will need to contact DeepSense support to have additional software packages installed in the systemwide python.

Alternatively, you can install an Anaconda python environment or other software in your home directory. This allows you to install or update packages or software without requesting and waiting for DeepSense staff.

==== Systemwide python (managed by DeepSense) ====

DeepSense has two Anaconda python environments. Anaconda 2 is installed on each compute node. While Anaconda 3 is installed in a shared directory that can be accessed from any machines in the cluster.

First one is anaconda2 installed in /opt/anaconda2 will provide you python 2.7.5. To use this systemwide python add a parameter to your .bashrc file in your home directory:

echo ". /opt/anaconda2/etc/profile.d/conda.sh" >> ~/.bashrc

Second is anaconda3 installed in /software/WMLA/anaconda3 will provide you python 3.7.4. To use this systemwide python add a parameter to your .bashrc file in your home directory:

echo ". /software/WMLA/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc

Then source your .bashrc file:
source ~/.bashrc

To load the python environment run <code>conda activate</code>

You can add either line to your .bashrc file to automatically load the desired environment when you log in.

==== Local python install (managed by individual user) ====

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact [[Contact_Information|Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation. If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Contact_Information|Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le). If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Contact_Information|Technical Support]]. We may install and support it systemwide if there is sufficient interest.

'''Stop using systemwide anaconda'''

If you added the system anaconda environment to your <code>.bashrc</code> file then remove the line:
. /opt/anaconda2/etc/profile.d/conda.sh

'''Installing Anaconda with a python3 base '''

From your home directory run:
wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-ppc64le.sh
bash Anaconda3-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
. ~/anaconda3/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:
source ~/.bashrc

'''Adding a python2 environment'''

The previous instruction creates a python3 base environment. To add a python2 environment:
conda create -n py27 python=2.7

Activate this environment to use python3:
conda activate py27

note: if you receive an error message then you may need to deactivate the base conda environment first:
conda deactivate
conda activate py27

'''Adding a python3 environment'''

We recommend creating a separate python3 environment from the base environment. This makes it easier to install the specific packages required for IBM WMLA/PowerAI.
conda create -n py36 python=3.6

Activate this environment to use python3:
conda activate py36

==3. Installation of Deep Learning packages==

===3.1 Using IBM-AI Deep Learning Anaconda Channel===

To use deep learning packages like Tensorflow or Pytorch on DeepSense you need to add the anaconda channel to your list of available software channels. You can add either from the following two channels depends upon the version you want. We suggest you to first check the version under these channels and then add.

conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
conda config --prepend channels https://ftp.osuosl.org/pub/open-ce/1.2.2/

Open-CE versions later than 1.2.2 are not supported at this time.

We suggest creating a new environment for each deep learning package you want to use. For example for Tensorflow:
conda create -n py36_tensorflow python=3.6
conda activate py36_tensorflow

Then install the anaconda package for the software you need. Again, with Tensorflow as an example:
conda install tensorflow

You can then use tensorflow or other deep learning packages as needed by simply activating that anaconda environment. Unlike the old method, you do not need to specifically activate tensorflow or other deep learning methods.

The above channel is still working, but the supported software version will not be updated any more. For example, you can only install TensorFlow 2.1.2 and PyTorch 1.3.1 using this channel. You can directly visit the IBM-AI anaconda channel URL to see a list of available software and their versions (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/). If you would like to install higher versions of software, you can refer to the next section.

===3.2 Install PyTorch 1.6.0 in a user's home directory on DeepSense ===

A DeepSense user can install PyTorch by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build PyTorch with those versions. 
Here are the steps that a normal DeepSense user install PyTorch 1.6.0 in his/her home directory.

1. Source the conda environment you would like to use. For example:

source ~/anaconda3/etc/profile.d/conda.sh

2. Activate the environment you would use to install PyTorch. If the environment hasn't been created, a user can create it and install PyTorch in one command line. For example, if you would create an environment with name "my-environment" (This is just an example. Please choose a meaningful name for yourself.) and install PyTorch, you would run the following command:

conda create -y -n my-environment python=3.6 pytorch -c file:////software/PyTorch-1.6.0-Build/condabuild

3. If the environment has been created, say the name of the environment is "my-environment", you would need to activate the environment first and then install PyTorch. For example:

conda activate my-environment
conda install pytorch -c file:////software/PyTorch-1.6.0-Build/condabuild

This should take about 2 minutes to install.

4. To test if your install is successful, issue python from the environment where PyTorch is installed. Then run "import torch" to see if there are any errors. For example:
[luy@ds-lg-01 ~]$ conda activate my-environment
(my-environment) [luy@ds-lg-01 ~]$ python
Python 3.6.12 |Anaconda, Inc.| (default, Sep 9 2020, 00:40:10) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> 

===3.3 Install Opencv 3.4.10 in a user's home directory on DeepSense ===
A DeepSense user can install Opencv by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build/opencv-feedstock. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build Opencv with those versions. 
Here are the steps that a normal DeepSense user installs Opencv 3.4.10 in his/her home directory.
1. Source the conda environment you would like to use. For example:
source ~/anaconda3/etc/profile.d/conda.sh

2. Activate the environment you would use to install Opencv. If the environment hasn't been created, a user should create one. Assume a user created an environment "my-environment" and activated it. To install Opencv, you would run the following command:
conda activate my-environment
conda install opencv -c file:////software/PyTorch-1.6.0-Build/opencv-feedstock/condabuild
This should take about 2 minutes to install.

3. To test if your install is successful, issue python from the environment where Opencv is installed. Then run "import cv2" to see if there are any errors. For example:
[luy@ds-lg-01 ~]$ conda activate my-environment
(my-environment) [luy@ds-lg-01 ~]$ python
Python 3.6.12 |Anaconda, Inc.| (default, Sep 9 2020, 00:40:10)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>>

==4. Installation of Visualization Software==

Users can easily install Dash in their home directory on DeepSense. The way to install it is very similar to installing a package using conda. Here we provide how to install and use Dash on DeepSense. The install process only takes about 10 minutes if you have already set up your conda environments in your home directory on DeepSense.

===4.1 Installing Dash on DeepSense ===
Activate the conda environment you would install Dash 
Example:
[luy@ds-lg-01 ~]$ conda activate py36-pytorch
Running the following command to install Dash:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda install -c conda-forge dash
After the installation is done, issue the following command to check if Dash is installed:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda list |grep dash
dash 1.18.1 pyhd8ed1ab_0 conda-forge
dash-core-components 1.14.1 pyhd8ed1ab_0 conda-forge
dash-html-components 1.1.1 pyh9f0ad1d_0 conda-forge
dash-renderer 1.8.3 pyhd8ed1ab_0 conda-forge
dash-table 4.11.1 pyhd8ed1ab_0 conda-forge

==5. Install other dependencies ==

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the <code>scikit-learn</code> package in your python3 environment.

First you need to activate the environment:
conda activate py36

Then you install the package
conda install scikit-learn

A list of recommended packages follows in the next section.

==6. Recommended packages ==

===6.1 Jupyter Notebooks for deep learning ===
conda install jupyter

===6.2 (Old Method) Testing Deep Learning packages on the login nodes or non-GPU nodes ===

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:
source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

==7. Compiling Software for DeepSense ==

DeepSense uses IBM Power8 systems running RedHat Enterprise Linux. Code must be compiled for <code>ppc64le</code> which is PowerPC 64 bit Little Endian.

Some software may not have binaries available for <code>ppc64le</code> even if it does for other systems. If this happens then you (or [[Contact_Information|DeepSense support]]) will need to compile the software to run on DeepSense. Visit the web page for the software and see if the source code is available (e.g. through github). If so then follow the compilation instructions to run the software.

You may encounter errors when attempting to compile software for <code>ppc64le</code>. Often this occurs because of differences between <code>ppc64le</code> and other common architectures such as x86 and x86_64.

For example, one DeepSense user attempted to compile the rdkit software package from https://www.rdkit.org/ . This compilation failed when it attempted to use the gcc x86 optimization <code>-mpopcnt</code>. After replacing the optimization with the <code>ppc64le</code> equivalent <code>-mpopcntb</code> the software compiled successfully.

== 8. Technical and research support ==

DeepSense has a dedicated support team of research scientists ready to help you with technical questions, installing software, or even research questions.

If you can't find the answer to your question on this wiki or need more extensive help then send an email to support@deepsense.ca .

See [[Technical support]] for more information about the support available.

</div>

Available software

2022-03-08T15:35:20Z

Rdickson: /* IBM-AI Deep Learning Anaconda Channel */ change URL on advice from CWhidden

== Basic Software ==

* RedHat Enterprise Linux Server release 7.7 (RHEL)
* gcc 4.8.5
* glibc 2.17
* R 3.6.0

== Anaconda Python ==

DeepSense has two Anaconda python environments. Anaconda 2 is installed on each compute node. While Anaconda 3 is installed in a shared directory that can be accessed from any machines in the cluster.
:
{| class="wikitable"
! Version
! Environment location
|-
|python 2.7.15
|/opt/anaconda2
|-
|python 3.7.4
|/software/WMLA/anaconda3/
|}

These python environments have many packages installed, including prerequisite libraries for running the IBM PowerAI deep learning frameworks.

See [[Getting_started]] for instructions on using the shared anaconda python environments.

See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory.

==IBM-AI Deep Learning Anaconda Channel==

To use deep learning packages like Tensorflow, pytorch, caffe on DeepSense you need to add the IBM-AI anaconda channel to your list of available software channels using below command.
conda config --prepend channels https://ftp.osuosl.org/pub/open-ce//1.2.2/
See also [https://docs.deepsense.ca/index.php?title=Installing_Software#3._Installation_of_Deep_Learning_packages| Installing Software].

== IBM WMLA Deep Learning Packages ==

[https://developer.ibm.com/linuxonpower/deep-learning-powerai/ WMLA] includes multiple open source deep learning frameworks compiled for IBM Power8 systems.
Anaconda under the WMLA install is a global install that can be accessed from any systems on DeepSense's platform. It is different from the anaconda install in users' home directories. Therefore, users cannot modify anything under the install of WMLA. However, users can use (i.e., read permission) the conda environments that have been created. If the environments created cannot satisfy your requirements, please feel free to ask the DeepSense support to create the environments for you. WMLA has following pre-developed environments which are ready to use by the users.

dlinsights
dlipy36-wmlce161
dlipy36-wmlce162
dlipy36-wmlce170
dlipy37-wmlce170

Users can check the version of the packages under these environments by activating the environment.
Below is the method for using these environments.

[luy@ds-lg-01 ~]$ ls /software/WMLA/anaconda3/envs
dlinsights dlipy36-wmlce161 dlipy36-wmlce162 dlipy36-wmlce170 dlipy37-wmlce170
[luy@ds-lg-01 ~]$ source /software/WMLA/anaconda3/etc/profile.d/conda.sh
[luy@ds-lg-01 ~]$ which conda
/software/WMLA/anaconda3/bin/conda
[luy@ds-lg-01 ~]$ conda activate dlipy36-wmlce170
(dlipy36-wmlce170) [luy@ds-lg-01 ~]$ which python
/software/WMLA/anaconda3/envs/dlipy36-wmlce170/bin/python
(dlipy36-wmlce170) [luy@ds-lg-01 ~]$ conda list

Users can find more frameworks at the following locations
{| class="wikitable"
!Framework
!colspan="2"|Location
|-
|Caffe
|/opt/DL/caffe
|
|-
|cuDNN
|/opt/DL/cudnn
|
|-
|IBM Distributed Deep Learning (DDL)
|/opt/DL/ddl
|
|-
| HDF5
|/opt/DL/hdf5
|
|-
|NCCL
|/opt/DL/nccl
|/opt/DL/nccl2
|-
|openblas
|/opt/DL/openblas
|
|-
|protobuf
|/opt/DL/protobuf
|
|-
|pytorch
|/opt/DL/pytorch
|
|-
|snap-ml
|/opt/DL/snap-ml-local
|/opt/DL/snap-ml-mpi
|-
|Tensorflow 1.11 (including keras)
|/opt/DL/tensorflow
|/opt/DL/ddl-tensorflow
|-
|Tensorboard
|/opt/DL/tensorboard
|
|}

To use most of these frameworks you need to activate a python2 or python3 environment and then activate the relevant framework.

For example, to use tensorflow you can activate a python2 environment:
. /opt/anaconda2/etc/profile.d/conda.sh
conda activate

and then activate tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

You can then <code>import tensorflow as tf</code> in your python code.

See [[Getting started with Deep Learning]] for a tutorial on using Caffe and Tensorflow on Deep Sense.

== IBM Advance Toolchain ==

You may require newer versions of compilers such as GCC than are provided with RHEL.

The [https://developer.ibm.com/linuxonpower/advance-toolchain IBM Advance Toolchain for Linux on Power] is a set of open source compilers, runtime libraries, and development tools.

The IBM Advance Toolchain] includes recent versions of:
* GNU Compiler Collection (gcc, g++ and gfortran)
* GNU C library (glibc)
* GNU Binary Utilities (binutils)
* Decimal Floating Point Library (libdfp)
* IBM Power Architecture Facilities Library (PAFLib)
* GNU Debugger (gdb)
* Python
* Golang
* Performance analysis tools (oprofile, valgrind, itrace)
* Multi-core exploitation libraries (TBB, Userspace RCU, SPHDE)
* support libraries (libhugetlbfs, Boost, zlib, etc)

To use the the Advance Toolchain, first activate environment modules:
source /usr/local/Modules/init/bash

Then load the advance toolchain:
module load at12.0

To stop using the advance toolchain, unload the environment module:
module unload at12.0

Note that software dynamically compiled with the advance toolchain will only run with the advance toolchain loaded.

== Requested Software ==

Software packages that are requested for use by DeepSense projects will be available in several locations. Our preference is to use conda packages when available.

=== External conda channels ===
If a requested software is available for ppc64le systems from an externally maintained anaconda channel then we will simply list the channel. You can install such software into a local anaconda environment using:
conda install -c <channel> <software>

=== Internal conda packages ===
When possible, software compiled by DeepSense staff will compiled using conda build and placed in a subdirectory of <code>/software/conda-bld/</code> . You can install such software into a local anaconda environment using:
conda install -c file://software/conda-bld/ <software>

=== Shared software ===
Some software will simply be installed in its own subdirectory of <code>/software</code>. You can run this software directly from its subdirectory.

=== Bioinformatics Software ===

{| class="wikitable"
!Software
!Version
!Location
|-
|trimmomatic
|0.39
|/software/trimmomattic-0.39
|-
|cutadapt
|2.3
|/software/conda-bld/linux-ppc64le/
|-
|bowtie2
|
|biobuilds channel
|-
|LAST
|980
|/software/last-980
|-
|Burrows wheeler aligner
|0.7.15
|/software/bwa
|-
|pb-falcon
|2.2.0
|/software/conda-bld/linux-ppc64le/
|-
|MASURCA
|3.3.4
|/software/conda-bld/linux-ppc64le/
|-
|Samtools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|htslib
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|bcftools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|gatk
|4.1.2.0
|/software/conda-bld/noarch/
|-
|stacks
|2.4
|/software/conda-bld/linux-ppc64le/
|-
|angsd
|0.923
|/software/conda-bld/linux-ppc64le/
|-
|vcftools
|
|biobuilds channel
|-
|plink
|
|biobuilds channel
|-
|msprime
|0.7.0
|/software/conda-bld/linux-ppc64le/
|-
|slim
|3.3
|/software/slim-3.3
|-
|DeepGSR
|
|/software/DeepGSR
|}

== Requesting Additional Software ==

Contact DeepSense [[contact information|support]] to have additional software installed or for help installing or compiling software locally in your home directory.

Resources

2022-03-08T15:11:43Z

Rdickson:

{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|+'''Compute Resources'''
|-
|'''Compute Hosts'''
|'''#'''
|'''Memory'''
|'''CPU'''
|'''GPU'''
|'''Interconnect'''
|'''Operation System'''
|-
|Large Memory
|20
|512GB
|style="text-align:left" | 20 Core IBM Power8 2.8GHz (ds-cmlm-01,...,ds-cmlm-20)
|N/A
|Mellanox SN 2700 –100GB Cluster Interconnect Switch
|Redhat Enterprise 7.7
|-
|Huge Memory
|2
|1TB
|style="text-align:left" |20 Core IBM Power8 3.4GHz (ds-cmhm-03,ds-cmhm-04)
|N/A
|Mellanox SN 2700 –100GB Cluster Interconnect Switch
|Redhat Enterprise 7.7
|-
|GPU
|10
|512GB
|style="text-align:left" |20 Core IBM Power8NVL 4.0GHz (ds-cmgpu-01,...,ds-cmgpu-10)
|2xP100
|Mellanox SN 2700 –100GB Cluster Interconnect Switch
|Redhat Enterprise 7.7
|}

Largest LSF job that can be hosted on a "512GB" node is 450GB,
''i.e.'' <code>bsub -R "rusage[mem=450GB]"</code>.
Largest job that can be hosted on a "1TB" node is 960GB,
''i.e.'' <code>bsub -R "rusage[mem=960GB]"</code>.

{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|+'''Storage Resources'''
|-
|'''Name'''
|'''Location'''
|'''Quota'''
|'''Function'''
|-
|Home
| /dshome
| 1TB
| Personal scripts/code/test data
|-
|Scratch
| /scratch
| 2TB
| Temporary space
|-
|Data
| /data
| 2TB
| Project group data
|}
See [[Storage policies]] for more information about the available shared file systems, storage policies, and backup policies.

Resources

2022-03-08T15:11:16Z

Rdickson:

{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|+'''Compute Resources'''
|-
|'''Compute Hosts'''
|'''#'''
|'''Memory'''
|'''CPU'''
|'''GPU'''
|'''Interconnect'''
|'''Operation System'''
|-
|Large Memory
|20
|512GB
|style="text-align:left" | 20 Core IBM Power8 2.8GHz (ds-cmlm-01,...,ds-cmlm-20)
|N/A
|Mellanox SN 2700 –100GB Cluster Interconnect Switch
|Redhat Enterprise 7.7
|-
|Huge Memory
|2
|1TB
|style="text-align:left" |20 Core IBM Power8 3.4GHz (ds-cmhm-03,ds-cmhm-04)
|N/A
|Mellanox SN 2700 –100GB Cluster Interconnect Switch
|Redhat Enterprise 7.7
|-
|GPU
|10
|512GB
|style="text-align:left" |20 Core IBM Power8NVL 4.0GHz (ds-cmgpu-01,...,ds-cmgpu-10)
|2xP100
|Mellanox SN 2700 –100GB Cluster Interconnect Switch
|Redhat Enterprise 7.7
|}

Largest LSF job that can be hosted on a "512GB" node is 450GB,
''i.e.'' <code>bsub -R "rusage[mem=450GB]"</code>.
Largest job that can be hosted on a "1TB" node is 960GB,
''i.e.'' <code>bsub -R "rusage[mem=960GB]"</code>.

{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|+'''Storage Resources'''
|-
|'''Name'''
|'''Location'''
|'''Quota'''
|'''Function'''
|-
|Home
| /dshome
| 1Tb
| Personal scripts/code/test data
|-
|Scratch
| /scratch
| 2Tb
| Temporary space
|-
|Data
| /data
| 2Tb
| Project group data
|}
See [[Storage policies]] for more information about the available shared file systems, storage policies, and backup policies.

DeepSense Documentation

2022-02-15T14:50:36Z

Rdickson: Reverted edits by Rdickson (talk) to last revision by Jnewport

'''Welcome to the DeepSense technical documentation wiki'''. This is the primary source for users with questions on the DeepSense equipment and services. You'll now find all of our content on the sidebar. Just below you can see the cluster status, and information about any planned outages we may have.

We routinely make changes and update the content. If you see anything missing, or have any suggestions for content, we would appreciate hearing from you. You can send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).

== Cluster Status ==

'''Cluster status'''
{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|'''Status'''
|style="width:20% | '''Planned Outage'''
|style="width:70% | '''Notes'''
|-
|style="Color:green" | Online
|
|
|}
Legend: 
Online: cluster is running normally 
Partially Online: cluster has some problems and is partially available 
Offline: cluster is offine and users are not able to log in

DeepSense Documentation

2022-02-14T13:43:12Z

Rdickson:

'''Welcome to the DeepSense technical documentation wiki'''. This is the primary source for users with questions on the DeepSense equipment and services. You'll now find all of our content on the sidebar. Just below you can see the cluster status, and information about any planned outages we may have.

We routinely make changes and update the content. If you see anything missing, or have any suggestions for content, we would appreciate hearing from you. You can send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).

== Cluster Status ==

'''Cluster status'''
{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|'''Status'''
|style="width:20% | '''Planned Outage'''
|style="width:70% | '''Notes'''
|-
|style="Color:red" | Offline
|Monday Feb 14, 2022
|Software updates, beginning ca 07h30 AST, ending later the same day
|}
Legend: 
Online: cluster is running normally 
Partially Online: cluster has some problems and is partially available 
Offline: cluster is offine and users are not able to log in

DeepSense Documentation

2022-02-11T17:00:08Z

Rdickson: /* Cluster Status */ announce Feb 14 maintenance

'''Welcome to the DeepSense technical documentation wiki'''. This is the primary source for users with questions on the DeepSense equipment and services. You'll now find all of our content on the sidebar. Just below you can see the cluster status, and information about any planned outages we may have.

We routinely make changes and update the content. If you see anything missing, or have any suggestions for content, we would appreciate hearing from you. You can send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).

== Cluster Status ==

'''Cluster status'''
{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|'''Status'''
|style="width:20% | '''Planned Outage'''
|style="width:70% | '''Notes'''
|-
|style="Color:green" | Online
|Monday Feb 14, 2022
|Software updates, beginning ca 07h30 AST, ending later the same day
|}
Legend: 
Online: cluster is running normally 
Partially Online: cluster has some problems and is partially available 
Offline: cluster is offine and users are not able to log in