Difference between revisions of "Available software"
(→Anaconda Python) |
(→IBM PowerAI Deep Learning Packages) |
||
| Line 26: | Line 26: | ||
See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory. | See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory. | ||
| − | == IBM | + | == IBM WMLA Deep Learning Packages == |
[https://developer.ibm.com/linuxonpower/deep-learning-powerai/ IBM PowerAI] includes multiple open source deep learning frameworks compiled for IBM Power8 systems. | [https://developer.ibm.com/linuxonpower/deep-learning-powerai/ IBM PowerAI] includes multiple open source deep learning frameworks compiled for IBM Power8 systems. | ||
| − | IBM | + | IBM WMLA Enterprise includes: |
{| class="wikitable" | {| class="wikitable" | ||
Revision as of 16:15, 19 October 2020
Contents
Basic Software
- RedHat Enterprise Linux Server release 7.7 (RHEL)
- gcc 4.8.5
- glibc 2.17
- R 3.6.0
Anaconda Python
Two Anaconda python environments are installed locally on each DeepSense compute node:
| Version | Environment location |
|---|---|
| python 2.7.15 | /opt/anaconda2 |
| python 3.7.4 | /software/WMLA/anaconda3/ |
These python environments have many packages installed, including prerequisite libraries for running the IBM PowerAI deep learning frameworks.
See Getting_started for instructions on using the shared anaconda python environments.
See Installing local software for instructions on installing and managing your own python environments in your home directory.
IBM WMLA Deep Learning Packages
IBM PowerAI includes multiple open source deep learning frameworks compiled for IBM Power8 systems.
IBM WMLA Enterprise includes:
| Framework | Location | |
|---|---|---|
| Caffe | /opt/DL/caffe | |
| cuDNN | /opt/DL/cudnn | |
| IBM Distributed Deep Learning (DDL) | /opt/DL/ddl | |
| HDF5 | /opt/DL/hdf5 | |
| NCCL | /opt/DL/nccl | /opt/DL/nccl2 |
| openblas | /opt/DL/openblas | |
| protobuf | /opt/DL/protobuf | |
| pytorch | /opt/DL/pytorch | |
| snap-ml | /opt/DL/snap-ml-local | /opt/DL/snap-ml-mpi |
| Tensorflow 1.11 (including keras) | /opt/DL/tensorflow | /opt/DL/ddl-tensorflow |
| Tensorboard | /opt/DL/tensorboard | |
To use most of these frameworks you need to activate a python2 or python3 environment and then activate the relevant framework.
For example, to use tensorflow you can activate a python2 environment:
. /opt/anaconda2/etc/profile.d/conda.sh conda activate
and then activate tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate
You can then import tensorflow as tf in your python code.
See Getting started with Deep Learning for a tutorial on using Caffe and Tensorflow on Deep Sense.
IBM Advance Toolchain
You may require newer versions of compilers such as GCC than are provided with RHEL.
The IBM Advance Toolchain for Linux on Power is a set of open source compilers, runtime libraries, and development tools.
The IBM Advance Toolchain] includes recent versions of:
- GNU Compiler Collection (gcc, g++ and gfortran)
- GNU C library (glibc)
- GNU Binary Utilities (binutils)
- Decimal Floating Point Library (libdfp)
- IBM Power Architecture Facilities Library (PAFLib)
- GNU Debugger (gdb)
- Python
- Golang
- Performance analysis tools (oprofile, valgrind, itrace)
- Multi-core exploitation libraries (TBB, Userspace RCU, SPHDE)
- support libraries (libhugetlbfs, Boost, zlib, etc)
To use the the Advance Toolchain, first activate environment modules:
source /usr/local/Modules/init/bash
Then load the advance toolchain:
module load at12.0
To stop using the advance toolchain, unload the environment module:
module unload at12.0
Note that software dynamically compiled with the advance toolchain will only run with the advance toolchain loaded.
Requested Software
Software packages that are requested for use by DeepSense projects will be available in several locations. Our preference is to use conda packages when available.
External conda channels
If a requested software is available for ppc64le systems from an externally maintained anaconda channel then we will simply list the channel. You can install such software into a local anaconda environment using:
conda install -c <channel> <software>
Internal conda packages
When possible, software compiled by DeepSense staff will compiled using conda build and placed in a subdirectory of /software/conda-bld/ . You can install such software into a local anaconda environment using:
conda install -c file://software/conda-bld/ <software>
Some software will simply be installed in its own subdirectory of /software. You can run this software directly from its subdirectory.
Bioinformatics Software
| Software | Version | Location |
|---|---|---|
| trimmomatic | 0.39 | /software/trimmomattic-0.39 |
| cutadapt | 2.3 | /software/conda-bld/linux-ppc64le/ |
| bowtie2 | biobuilds channel | |
| LAST | 980 | /software/last-980 |
| Burrows wheeler aligner | 0.7.15 | /software/bwa |
| pb-falcon | 2.2.0 | /software/conda-bld/linux-ppc64le/ |
| MASURCA | 3.3.4 | /software/conda-bld/linux-ppc64le/ |
| Samtools | 1.9 | /software/conda-bld/linux-ppc64le/ |
| htslib | 1.9 | /software/conda-bld/linux-ppc64le/ |
| bcftools | 1.9 | /software/conda-bld/linux-ppc64le/ |
| gatk | 4.1.2.0 | /software/conda-bld/noarch/ |
| stacks | 2.4 | /software/conda-bld/linux-ppc64le/ |
| angsd | 0.923 | /software/conda-bld/linux-ppc64le/ |
| vcftools | biobuilds channel | |
| plink | biobuilds channel | |
| msprime | 0.7.0 | /software/conda-bld/linux-ppc64le/ |
| slim | 3.3 | /software/slim-3.3 |
| DeepGSR | /software/DeepGSR |
Requesting Additional Software
Contact DeepSense support to have additional software installed or for help installing or compiling software locally in your home directory.
