Installing local software - Revision history

Bgeetika at 12:24, 30 October 2020

2020-10-30T12:24:09Z

Bgeetika at 12:02, 30 October 2020

2020-10-30T12:02:30Z

Bgeetika at 19:13, 26 October 2020

2020-10-26T19:13:46Z

Cwhidden: Update with information about IBM WMLA conda channel

2020-06-17T17:59:18Z

Update with information about IBM WMLA conda channel

Cwhidden at 20:01, 5 December 2019

2019-12-05T20:01:25Z

Cwhidden: change instructions to use anaconda3 as the base environment

2019-11-07T17:42:56Z

change instructions to use anaconda3 as the base environment

Cwhidden: /* Installing Anaconda with a python2 base */

2019-11-06T15:30:08Z

‎Installing Anaconda with a python2 base

Cwhidden: Compiling Software section

2019-04-15T18:08:22Z

Compiling Software section

Cwhidden: /* Introduction */

2019-04-10T16:04:51Z

‎Introduction

Cwhidden: /* Testing Deep Learning packages on the login nodes or non-GPU nodes */

2019-03-12T13:22:20Z

‎Testing Deep Learning packages on the login nodes or non-GPU nodes

@@ Line 104: / Line 104: @@
 	This should take about 2 minutes to install.</br>
-<b>4. To test if your install is successful, issue python from the environment where PyTorch is installed. Then run "import torch" to see if there are any errors. For example:</b>
+<b>4. To test if your install is successful, issue python from the environment where PyTorch is installed. Then run "import torch" to see if there are any errors. For example:</b></br>
 <code>[luy@ds-lg-01 ~]$ conda activate my-environment</br>
-	<code>[luy@ds-lg-01 ~]$ conda activate my-environment</br>
+(my-environment) [luy@ds-lg-01 ~]$ python</br>
-	(my-environment) [luy@ds-lg-01 ~]$ python</br>
+Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:40:10) </br>
-	Python 3.6.12 |Anaconda, Inc.| (default, Sep  9 2020, 00:40:10) </br>
+[GCC 7.3.0] on linux</br>
-	[GCC 7.3.0] on linux</br>
+Type "help", "copyright", "credits" or "license" for more information.</br>
-	Type "help", "copyright", "credits" or "license" for more information.</br>
+>>> import torch</br>
-	>>> import torch</br>
+>>> </code></br>
 === Install Opencv 3.4.10 in a user's home directory on DeepSense ===

← Older revision		Revision as of 12:02, 30 October 2020
Line 114:		Line 114:
	>>></code> </br>		>>></code> </br>

		+	=== Install Opencv 3.4.10 in a user's home directory on DeepSense ===
		+	A DeepSense user can install Opencv by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build/opencv-feedstock. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build Opencv with those versions. </br>
		+	Here are the steps that a normal DeepSense user installs Opencv 3.4.10 in his/her home directory.</br>
		+	<b>1. Source the conda environment you would like to use. For example:</b>
		+	source ~/anaconda3/etc/profile.d/conda.sh
		+
		+	<b>2. Activate the environment you would use to install Opencv. If the environment hasn't been created, a user should create one. Assume a user created an environment "my-environment" and activated it. To install Opencv, you would run the following command:</b>
		+	conda activate my-environment
		+	conda install opencv -c file:////software/PyTorch-1.6.0-Build/opencv-feedstock/condabuild
		+	This should take about 2 minutes to install.</br>
		+
		+	<b>3. To test if your install is successful, issue python from the environment where Opencv is installed. Then run "import cv2" to see if there are any errors. For example:</b>
		+	[luy@ds-lg-01 ~]$ conda activate my-environment
		+	(my-environment) [luy@ds-lg-01 ~]$ python
		+	Python 3.6.12 \|Anaconda, Inc.\| (default, Sep 9 2020, 00:40:10)
		+	[GCC 7.3.0] on linux</br>
		+	Type "help", "copyright", "credits" or "license" for more information.
		+	>>> import cv2
		+	>>>
		+
	=== Install other dependencies ===		=== Install other dependencies ===

← Older revision		Revision as of 19:13, 26 October 2020
Line 82:		Line 82:

	The dependencies must be installed in whichever python environment you intend to use. We've encountered some problems installing the PyTorch dependencies directly in the base environment if the base conda environment has been updated to conda version 4.6.2. If you want to use PyTorch, be sure to use a conda environment with a lower version of conda.		The dependencies must be installed in whichever python environment you intend to use. We've encountered some problems installing the PyTorch dependencies directly in the base environment if the base conda environment has been updated to conda version 4.6.2. If you want to use PyTorch, be sure to use a conda environment with a lower version of conda.
		+
		+
		+	=== Install PyTorch 1.6.0 in a user's home directory on DeepSense ===
		+
		+	A DeepSense user can install PyTorch by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build PyTorch with those versions. </br>
		+	Here are the steps that a normal DeepSense user install PyTorch 1.6.0 in his/her home directory.</br>
		+
		+	<b>1. Source the conda environment you would like to use. For example:</b>
		+
		+	<code>source anaconda3/etc/profile.d/conda.sh</code></br>
		+
		+	<b>2. Activate the environment you would use to install PyTorch. If the environment hasn't been created, a user can create it and install PyTorch in one command line. For example, if you would create an environment with name "my-environment" (This is just an example. Please choose a meaningful name for yourself.) and install PyTorch, you would run the following command:</b>
		+
		+	<code>conda create -y -n my-environment python=3.6 pytorch -c file:////software/PyTorch-1.6.0-Build/condabuild</code></br>
		+
		+	<b>3. If the environment has been created, say the name of the environment is "my-environment", you would need to activate the environment first and then install PyTorch. For example:</b>
		+
		+	<code>conda activate my-environment</br>
		+	conda install pytorch -c file:////software/PyTorch-1.6.0-Build/condabuild</code></br>
		+
		+	This should take about 2 minutes to install.</br>
		+
		+	<b>4. To test if your install is successful, issue python from the environment where PyTorch is installed. Then run "import torch" to see if there are any errors. For example:</b>
		+
		+	<code>[luy@ds-lg-01 ~]$ conda activate my-environment</br>
		+	(my-environment) [luy@ds-lg-01 ~]$ python</br>
		+	Python 3.6.12 \|Anaconda, Inc.\| (default, Sep 9 2020, 00:40:10) </br>
		+	[GCC 7.3.0] on linux</br>
		+	Type "help", "copyright", "credits" or "license" for more information.</br>
		+	>>> import torch</br>
		+	>>></code> </br>

	=== Install other dependencies ===		=== Install other dependencies ===

@@ Line 1: / Line 1: @@
 == Introduction ==
@@ Line 48: / Line 54: @@
   conda activate py36
-=== Install PowerAI dependencies ===
+=== (New Method) IBM-AI Deep Learning Anaconda Channel ===
 Warning: these scripts will install, update, and downgrade some packages to the recommended packages for the current version of PowerAI. You may want to create a separate python environment to use different versions of those packages with other software.
@@ Line 81: / Line 104: @@
   conda install jupyter
-=== Testing Deep Learning packages on the login nodes or non-GPU nodes ===
+=== (Old Method) Testing Deep Learning packages on the login nodes or non-GPU nodes ===
 You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

@@ Line 73: / Line 73: @@
 Then you install the package
   conda install scikit-learn
 === Testing Deep Learning packages on the login nodes or non-GPU nodes ===

@@ Line 18: / Line 18: @@
   . /opt/anaconda2/etc/profile.d/conda.sh
-=== Installing Anaconda with a python2 base ===
+=== Installing Anaconda with a python3 base ===
 From your home directory run:
-  wget https://repo.continuum.io/archive/Anaconda2-5.2.0-Linux-ppc64le.sh
+  wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-ppc64le.sh
-  bash Anaconda2-5.2.0-Linux-ppc64le.sh
+  bash Anaconda3-5.2.0-Linux-ppc64le.sh
 Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
-  . ~/anaconda2/etc/profile.d/conda.sh
+  . ~/anaconda3/etc/profile.d/conda.sh
 After the installer ends you need to either close and restart your terminal or run:
   source ~/.bashrc
-=== Adding a python3 environment ===
+=== Adding a python2 environment ===
-The previous instruction creates a python2 base environment. To add a python3 environment:
+The previous instruction creates a python3 base environment. To add a python2 environment:
-  conda create -n py36 python=3.6
+  conda create -n py27 python=2.7
 Activate this environment to use python3:
-  conda activate py36
+  conda activate py27
 note: if you receive an error message then you may need to deactivate the base conda environment first:
   conda deactivate
-  conda activate py36
+  conda activate py27
-=== Adding a python2 environment ===
+=== Adding a python3 environment ===
-We recommend creating a separate python2 environment from the base environment. This makes it easier to install the specific packages required for IBM PowerAI.
+We recommend creating a separate python3 environment from the base environment. This makes it easier to install the specific packages required for IBM PowerAI.
-  conda create -n py27 python=2.7
+  conda create -n py36 python=3.6
-Activate this environment to use python2:
+Activate this environment to use python3:
-  conda activate py27
+  conda activate py36
 === Install PowerAI dependencies ===

@@ Line 3: / Line 3: @@
 You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.
-For assistance installing or compiling software contact [[Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation.
+For assistance installing or compiling software contact [[Contact_Information|Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation.
-If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le).
+If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Contact_Information|Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le).
 If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>
-If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Technical Support]]. We may install and support it systemwide if there is sufficient interest.
+If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Contact_Information|Technical Support]]. We may install and support it systemwide if there is sufficient interest.
 == Installing Anaconda Python in your home directory ==
@@ Line 90: / Line 90: @@
 Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

@@ Line 74: / Line 74: @@
 You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.
-Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like PowerAI on the login nodes or CPU-only nodes then you will see errors like the following:
+Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
   ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory