Installing Software

1. Logging on

DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus.

For example, if your userid is user1, you can connect to deepsense by typing ssh user1@login1.deepsense.ca just like logging on to any other network computer.

Note: The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes. Keep reading for instructions on how to submit compute jobs to dedicated compute nodes.

1.1 VPN

To connect to the DeepSense platform from outside of the Dalhousie Campus, you'll need to use a VPN. If you are are student, staff or faculty, you can use the Dalhousie VPN (https://wireless.dal.ca/vpnsoftware.php).

If you are not a Dalhousie staff, student, or faculty but require offsite access and cannot use the Dalhousie VPN then contact your project leader or (support@deepsense.ca) to make different arrangements.

For more info, see VPN Setup.

2. Configure your environment

DeepSense compute and management nodes are IBM Power8 computers (ppc64le) running Redhat Enterprise Linux. See Resources for more details on the available nodes.

2.1 Loading a python environment

You have two options for using python on DeepSense. You can use the systemwide python install, managed by DeepSense administrators. This is recommended for users new to Linux. You will need to contact DeepSense support to have additional software packages installed in the systemwide python.

Alternatively, you can install an Anaconda python environment or other software in your home directory. This allows you to install or update packages or software without requesting and waiting for DeepSense staff.

Systemwide python (managed by DeepSense)

DeepSense has two Anaconda python environments. Anaconda 2 is installed on each compute node. While Anaconda 3 is installed in a shared directory that can be accessed from any machines in the cluster.

First one is anaconda2 installed in /opt/anaconda2 will provide you python 2.7.5. To use this systemwide python add a parameter to your .bashrc file in your home directory:

echo ". /opt/anaconda2/etc/profile.d/conda.sh" >> ~/.bashrc

Second is anaconda3 installed in /software/WMLA/anaconda3 will provide you python 3.7.4. To use this systemwide python add a parameter to your .bashrc file in your home directory:

echo ". /software/WMLA/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc

Then source your .bashrc file:

source ~/.bashrc

To load the python environment run conda activate

You can add either line to your .bashrc file to automatically load the desired environment when you log in.

Local python install (managed by individual user)

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact Technical Support. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation. If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact Technical Support. The package may not have been compiled for the DeepSense hardware architecture (ppc64le). If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at Technical Support. We may install and support it systemwide if there is sufficient interest.

Stop using systemwide anaconda

If you added the system anaconda environment to your .bashrc file then remove the line:

. /opt/anaconda2/etc/profile.d/conda.sh

Installing Anaconda with a python3 base

From your home directory run:

wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-ppc64le.sh
bash Anaconda3-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:

. ~/anaconda3/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:

source ~/.bashrc

Adding a python2 environment

The previous instruction creates a python3 base environment. To add a python2 environment:

conda create -n py27 python=2.7

Activate this environment to use python3:

conda activate py27

note: if you receive an error message then you may need to deactivate the base conda environment first:

conda deactivate
conda activate py27

Adding a python3 environment

We recommend creating a separate python3 environment from the base environment. This makes it easier to install the specific packages required for IBM WMLA/PowerAI.

conda create -n py36 python=3.6

Activate this environment to use python3:

conda activate py36

3. Installation of Deep Learning packages

3.1 Using IBM-AI Deep Learning Anaconda Channel

To use deep learning packages like Tensorflow or Pytorch on DeepSense you need to add the anaconda channel to your list of available software channels. You can add either from the following two channels depends upon the version you want. We suggest you to first check the version under these channels and then add.

conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
conda config --prepend channels https://ftp.osuosl.org/pub/open-ce/1.2.2/

Open-CE versions later than 1.2.2 are not supported at this time.

We suggest creating a new environment for each deep learning package you want to use. For example for Tensorflow:

conda create -n py36_tensorflow python=3.6
conda activate py36_tensorflow

Then install the anaconda package for the software you need. Again, with Tensorflow as an example:

conda install tensorflow

You can then use tensorflow or other deep learning packages as needed by simply activating that anaconda environment. Unlike the old method, you do not need to specifically activate tensorflow or other deep learning methods.

The above channel is still working, but the supported software version will not be updated any more. For example, you can only install TensorFlow 2.1.2 and PyTorch 1.3.1 using this channel. You can directly visit the IBM-AI anaconda channel URL to see a list of available software and their versions (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/). If you would like to install higher versions of software, you can refer to the next section.

3.2 Install PyTorch 1.6.0 in a user's home directory on DeepSense

A DeepSense user can install PyTorch by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build PyTorch with those versions.
Here are the steps that a normal DeepSense user install PyTorch 1.6.0 in his/her home directory.

1. Source the conda environment you would like to use. For example:

source ~/anaconda3/etc/profile.d/conda.sh

2. Activate the environment you would use to install PyTorch. If the environment hasn't been created, a user can create it and install PyTorch in one command line. For example, if you would create an environment with name "my-environment" (This is just an example. Please choose a meaningful name for yourself.) and install PyTorch, you would run the following command:

conda create -y -n my-environment python=3.6 pytorch -c file:////software/PyTorch-1.6.0-Build/condabuild

3. If the environment has been created, say the name of the environment is "my-environment", you would need to activate the environment first and then install PyTorch. For example:

conda activate my-environment

conda install pytorch -c  file:////software/PyTorch-1.6.0-Build/condabuild

This should take about 2 minutes to install.

4. To test if your install is successful, issue python from the environment where PyTorch is installed. Then run "import torch" to see if there are any errors. For example:

[luy@ds-lg-01 ~]$ conda activate my-environment

(my-environment) [luy@ds-lg-01 ~]$ python

Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:40:10) 

[GCC 7.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import torch

>>> 

3.3 Install Opencv 3.4.10 in a user's home directory on DeepSense

A DeepSense user can install Opencv by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build/opencv-feedstock. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build Opencv with those versions.
Here are the steps that a normal DeepSense user installs Opencv 3.4.10 in his/her home directory.
1. Source the conda environment you would like to use. For example:

source ~/anaconda3/etc/profile.d/conda.sh

2. Activate the environment you would use to install Opencv. If the environment hasn't been created, a user should create one. Assume a user created an environment "my-environment" and activated it. To install Opencv, you would run the following command:

conda activate my-environment
conda install opencv -c file:////software/PyTorch-1.6.0-Build/opencv-feedstock/condabuild

This should take about 2 minutes to install.

3. To test if your install is successful, issue python from the environment where Opencv is installed. Then run "import cv2" to see if there are any errors. For example:

[luy@ds-lg-01 ~]$ conda activate my-environment
(my-environment) [luy@ds-lg-01 ~]$ python
Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:40:10) 
[GCC 7.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> 

4. Installation of Visualization Software

Users can easily install Dash in their home directory on DeepSense. The way to install it is very similar to installing a package using conda. Here we provide how to install and use Dash on DeepSense. The install process only takes about 10 minutes if you have already set up your conda environments in your home directory on DeepSense.

4.1 Installing Dash on DeepSense

Activate the conda environment you would install Dash
Example:

[luy@ds-lg-01 ~]$ conda activate py36-pytorch

Running the following command to install Dash:

(py36-pytorch) [luy@ds-lg-01 ~]$ conda install -c conda-forge dash

After the installation is done, issue the following command to check if Dash is installed:

(py36-pytorch) [luy@ds-lg-01 ~]$ conda list |grep dash
dash                      1.18.1             pyhd8ed1ab_0    conda-forge
dash-core-components      1.14.1             pyhd8ed1ab_0    conda-forge
dash-html-components      1.1.1              pyh9f0ad1d_0    conda-forge
dash-renderer             1.8.3              pyhd8ed1ab_0    conda-forge
dash-table                4.11.1             pyhd8ed1ab_0    conda-forge

5. Install other dependencies

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the scikit-learn package in your python3 environment.

First you need to activate the environment:

conda activate py36

Then you install the package

conda install scikit-learn

A list of recommended packages follows in the next section.

6. Recommended packages

6.1 Jupyter Notebooks for deep learning

conda install jupyter

6.2 (Old Method) Testing Deep Learning packages on the login nodes or non-GPU nodes

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:

ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:

source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:

source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

7. Compiling Software for DeepSense

DeepSense uses IBM Power8 systems running RedHat Enterprise Linux. Code must be compiled for ppc64le which is PowerPC 64 bit Little Endian.

Some software may not have binaries available for ppc64le even if it does for other systems. If this happens then you (or DeepSense support) will need to compile the software to run on DeepSense. Visit the web page for the software and see if the source code is available (e.g. through github). If so then follow the compilation instructions to run the software.

You may encounter errors when attempting to compile software for ppc64le. Often this occurs because of differences between ppc64le and other common architectures such as x86 and x86_64.

For example, one DeepSense user attempted to compile the rdkit software package from https://www.rdkit.org/ . This compilation failed when it attempted to use the gcc x86 optimization -mpopcnt. After replacing the optimization with the ppc64le equivalent -mpopcntb the software compiled successfully.

8. Technical and research support

DeepSense has a dedicated support team of research scientists ready to help you with technical questions, installing software, or even research questions.

If you can't find the answer to your question on this wiki or need more extensive help then send an email to support@deepsense.ca .

See Technical support for more information about the support available.