DeepSense Docs - User contributions [en-ca]

Best Practices

2023-03-19T23:44:19Z

Lyang: Blanked the page

Workarounds

2023-03-19T23:44:10Z

Lyang: Blanked the page

Restrictions

2023-03-19T23:43:58Z

Lyang: Blanked the page

MediaWiki:Sidebar

2023-03-19T23:43:33Z

Lyang:

Overview of your Storage on DeepSense

2023-03-19T23:24:27Z

Lyang:

== Overview ==

DeepSense is a platform for AI/ML for oceans research data. You have store the project data for your DeepSense projects. DeepSense is not meant for long term data storage. Data will only be stored so long as your project is ongoing. Once a project is completed, it is expected that the users will remove their data in a timely fashion.

DeepSense is not intended to be used for data sharing. While each user in your project/group will have access to shared space, it won't be accessible by any other users.

Your data is stored on cloud storage. The cloud storage types are similar to the storage types in the on-premises data centers. You can store your data as objects in cloud buckets, cloud block devices like a hard drive on your computer, or even NFS shared file systems which provides very high performance.

Your data is very safe in the cloud storage because DeepSense applies security measures at every layer of the cloud architecture. DeepSense also helps you migrate your data from your local disk to the secured cloud storage. DeepSense provides you detailed instructions on how to access your and operate your data.

You may be confused which storage is the right choice for your workload. DeepSense will help you analyze the nature of your data and the requirements of your workload to provide you the best storage solutions.

Below are the explanations of the cloud storage solutions mentioned above.

== Amazon EBS - Elastic Block Store (hard drives in cloud) ==
Amazon Elastic Block Store (Amazon EBS) provides persistent block storage volumes in the AWS Cloud for use with Amazon EC2 instances. To protect you from component failure and to provide high availability and durability, each Amazon EBS volume is automatically replicated within its Availability Zone. Amazon EBS volumes provide the reliable and low-latency performance required to run your workloads. You can scale your usage up or down in minutes with Amazon EBS, all while paying a low price for only what you provision.

== Amazon S3 - Simple Storage Service (object storage in cloud) ==
Amazon S3 is an object storage service that provides industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can use it to store and protect any amount of data for a variety of use cases, including websites, mobile apps, backup and restore, archiving, enterprise applications, IoT devices, and big data analytics. Amazon S3 offers simple management features that allow you to organize your data and configure fine-grained access controls to meet your specific business, organizational, and compliance needs. Amazon S3 is built to have super high durability of 99.999999999% (11 9s) and stores data for millions of applications for businesses all over the world.
Amazon S3 supports versioning and deletion protections. You never worry the accidental deletion of your files. Also, you will never have to stop your workload after running for a long time due to storage quota issues that happen frequently on on-premises systems. Amazon S3 provides unlimited storage.

== Amazon S3 Storage Tiers ==
Amazon S3 provides several tiers of storage for different use cases. This can be very cost-effective if you choose a storage tier that matches your access pattern. For example, Amazon S3 Glacier (S3 Glacier) is a safe and long-lasting service for low-cost data archiving and backup. With S3 Glacier, you can store your data for months, years, or even decades at a low cost. You can consult DeepSense to learn the best storage solution for your use cases.

Overview of your Storage on DeepSense

2023-03-12T23:02:41Z

Lyang:

User Guide

2023-03-12T22:59:52Z

Lyang:

Welcome to the DeepSense cloud computing user guide. You don't have to have background of cloud computing, but it is helpful if you spend 5 minutes reading our wiki page "[[Introduction to Cloud Computing]]".

Unlike using on-premises systems, you don't have to have a Linux user account to ssh into a Linux boxes unless your workloads have to deploy to a virtual machine. What you will have are Identity and Access Management(IAM) accounts on AWS, GCP, or Azure. DeepSense will setup virtual machines, severless Machine Learning compute platform, data processing pipeline, and unlimited data storage for users. We will provide step by step guidance to help you develop your Machine Learning or data processing projects in cloud. You don't have to stick to any specific cloud. DeepSense will help you find the best solution to make your project development process straightforward in cloud.

The process of developing your projects are: 
1. Users and DeepSense will discuss the background of your projects. Users will provide their technical requirements, for example, data type and size, CPU, memory, GPU, and so on. 
2. DeepSense will transfer your data to a cloud storage and set up the permission that only the owner of the data can access. Your data is encrypted both in transit and at rest. 
3. DeepSense will set up the environment according to your requirement and create an IAM account for you in the cloud. 
4. DeepSense will notify you when the set up is finished and provide detailed instructions how to develop your projects. Using cloud's Platform as a Service (PAAS), you only need to provide your code to train your models or process your data. 
5. According to the specific environment set up for you, please select corresponding instructions below to start developing your projects. You can also find the instructions in the section "[[How to develop Machine Learning projects on cloud]]" in the navigation menu on the left side of this page. 

== Using AWS Sagemaker==
Please follow the instructions at "[[Using AWS Sagemaker]]".

== Using AWS EC2 Virtual Machine ==
Please follow the instructions at "[[Using AWS EC2]]".

We are configuring GCP Colab, High-Performance Computing clusters on GCP and AWS, and data processing pipeline on GCP and AWS. So more cloud computing platforms will be coming soon. We will keep you updated.

User Guide

2023-03-12T22:59:04Z

Lyang:

User Guide

2023-03-12T22:57:19Z

Lyang:

Welcome to the DeepSense cloud computing user guide. You don't have to have background of cloud computing, but it is helpful if you spend 5 minutes reading our wiki page "[[Introduction to Cloud Computing]]".

Unlike using on-premises systems, you don't have to have a Linux user account to ssh into a Linux boxes unless your workloads have to deploy to a virtual machine. What you will have are Identity and Access Management(IAM) accounts on AWS, GCP, or Azure. DeepSense will setup virtual machines, severless Machine Learning compute platform, data processing pipeline, and unlimited data storage for users. We will provide step by step guidance to help you develop your Machine Learning or data processing projects in cloud. You don't have to stick to any specific cloud. DeepSense will help you find the best solution to make your project development process straightforward in cloud.

The process of developing your projects are: 
1. Users and DeepSense will discuss the background of your projects. Users will provide their technical requirements, for example, data type and size, CPU, memory, GPU, and so on. 
2. DeepSense will transfer your data to a cloud storage and set up the permission that only the owner of the data can access. Your data is encrypted both in transit and at rest. 
3. DeepSense will set up the environment according to your requirement and create an IAM account for you in the cloud. 
4. DeepSense will notify you when the set up is finished and provide detailed instructions how to develop your projects. Using cloud's Platform as a Service (PAAS), you only need to provide your code to train your models or process your data. 
5. According to the specific environment set up for you, please select corresponding instructions below to start developing your projects. You can also find the instructions in the section "[[How to develop Machine Learning projects on cloud]]" in the navigation menu on the left side of this page. 

== Using AWS Sagemaker==
Please follow the instructions at "[[Using AWS Sagemaker]]".

== Using AWS EC2 Virtual Machine ==
Please following the instructions at "[[Using AWS EC2]]".

We are configuring GCP Colab, High-Performance Computing clusters on GCP and AWS, and data processing pipeline on GCP and AWS. We will keep you updated.

User Guide

2023-03-12T19:44:02Z

Lyang:

Welcome to the DeepSense cloud computing user guide. You don't have to have background of cloud computing, but it is helpful if you can have a 5 minute read of our wiki page "[[Introduction to Cloud Computing]]".

Unlike using on-premises systems, you don't have to have a Linux user account to ssh into a Linux boxes unless your workload would need a virtual machine. What you will have are IAM accounts on AWS, GCP, or Azure. DeepSense will setup virtual machines, severless Machine Learning compute platform, data processing pipeline, and unlimited data storage for users.

== Amazon Web Services (AWS) ==
* The cloud administrator will provide you with an IAM User account of AWS.
* Based on the project requirements, you can select available [[Resources]] from the table.
* Access the account by following the instructions given here.
* For S3 storage, an S3 bucket will be provided if needed to store project data.

== Google Cloud Platform (GCP) ==
* Google HPC cloud toolkit still under development for DeepSense.

Introduction to Cloud Computing

2023-03-06T00:17:57Z

Lyang:

Cloud computing is not the future, but already the present. Garner reports that by 2025, '''95%''' of new digital workloads will utilize cloud-native platforms, and '''more than 85%''' of organizations will embrace a cloud-first principle. Cloud revenue was also estimated to surpass non-cloud revenue bringing forth the need for skilled cloud computing professionals in the IT industry. 

'''What is Cloud Computing?''' 
In short words, cloud computing is the delivery of the computing resources as a service. The computing resources include storage (object and network shared storages), databases (SQL and NoSQL databases), compute power (virtual machines, serverless, and Kubernetes), and networking (virtual private cloud). A person can access all what the on-premise data centres could offer in cloud. 

'''Is cloud easy to use?''' 
It may cause a person a bit uncertain to use cloud for the first time. You, as a user, don't have to be a cloud expert to use the cloud services. This is very similar to that you can drive a car but don't know how to assemble a car. You simply tell us your technical requirements of your machine learning or data processing projects, and we will setup the cloud storage and compute environment for you. We will guide you through how to use it to train your machine learning models or process your data. The learning curve is way shorter than that of the on-premise data centres. 

'''Is my data secure in cloud?''' 
Security is always the biggest concern for everyone. Your data won't be safe anywhere as long as you don't have professionals to secure it. Almost all data breaches were caused by humans, but not technologies. At DeepSense, we apply the best practices of security measures to make sure your data is secure. Your data is encrypted both at rest and in transit. We also follow the principle of least privileges to make sure only you can access your data. 
When we use storage in cloud, we not only make your data secure, but also durable. For example, the super high durability of AWS S3 storage guarantees that the probability of the data loss is below 0.00000000001%. 

'''Do my jobs have to wait for compute resources in a queue?''' 
Due to the "unlimited" resources in cloud, you never wait for your resources in a queue. There are always available resources for you to use. As long as you make your data and machine learning models ready, the wait time is "zero". 

'''Here we list the characteristics of cloud computing if you would like to learn more about it.''' 
1. "Unlimited" storage 
2. High elasticity and scalability 
3. Adequate services 
4. Huge network access 
5. Pay-per-use pricing 
6. Hight resiliency and availability 
7. On-demand service 
8. Zero maintenance 
9. Fine-grained reporting service 
10. Security 
11. Automation 
12. Customizable and flexible payment structure 
13. Work from anywhere 
14. Cost-saving 
15. Highly customizable monitoring structure and logging system 

Please feel free to contact DeepSense for any questions!

Introduction to Cloud Computing

2023-03-06T00:09:00Z

Lyang:

Cloud computing is not the future, but already the present. Garner reports that by 2025, '''95%''' of new digital workloads will utilize cloud-native platforms, and '''more than 85%''' of organizations will embrace a cloud-first principle. Cloud revenue was also estimated to surpass non-cloud revenue bringing forth the need for skilled cloud computing professionals in the IT industry. 

'''What is Cloud Computing?''' 
In short words, cloud computing is the delivery of the computing resources as a service. The computing resources include storage (object and network shared storages), databases (SQL and NoSQL databases), compute power (virtual machines, serverless, and Kubernetes), and networking (virtual private cloud). A person can access all what the on-premise data centres could offer in cloud. 

'''Is cloud easy to use?''' 
It may cause a person a bit uncertain to use cloud for the first time. You, as a user, don't have to be a cloud expert to use the cloud services. This is very similar to that you can drive a car but don't know how to assemble a car. You simply tell us your technical requirements of your machine learning or data processing projects, and we will setup the cloud storage and compute environment for you. We will guide you through how to use it to train your machine learning models or process your data. The learning curve is way shorter than that of the on-premise data centres. 

'''Is my data secure in cloud?''' 
Security is always the biggest concern for everyone. Your data won't be safe anywhere as long as you don't have professionals to secure it. Almost all data breaches were caused by humans, but not technologies. At DeepSense, we apply the best practices of security measures to make sure your data is secure. Your data is encrypted both at rest and in transit. We also follow the principle of least privileges to make sure only you can access your data. 
When we use storage in cloud, we not only make your data secure, but also durable. For example, the super high durability of AWS S3 storage guarantees that the probability of the data loss is below 0.00000000001%. 

'''Do my jobs have to wait for compute resources in a queue?''' 
Due to the "unlimited" resources in cloud, you never wait for your resources in a queue. There are always available resources for you to use. As long as you make your data and machine learning models ready, the wait time is "zero". 

'''Here we list the characteristics of cloud computing if you are interested in understand more about it.''' 
1. "Unlimited" storage 
2. High elasticity and scalability 
3. Adequate services 
4. Huge network access 
5. Pay-per-use pricing 
6. Hight resiliency and availability 
7. On-demand service 
8. Zero maintenance 
9. Fine-grained reporting service 
10. Security 
11. Automation 
12. Customizable and flexible payment structure 
13. Work from anywhere 
14. Cost-saving 
15. Highly customizable monitoring structure and logging system 

Please feel free to contact DeepSense for any questions!

Introduction to Cloud Computing

2023-03-05T21:49:07Z

Lyang:

Cloud computing is not the future, but already the present. Garner reports that by 2025, '''95%''' of new digital workloads will utilize cloud-native platforms, and '''more than 85%''' of organizations will embrace a cloud-first principle. Cloud revenue was also estimated to surpass non-cloud revenue bringing forth the need for skilled cloud computing professionals in the IT industry. 

'''What is Cloud Computing?''' 
In short words, cloud computing is the delivery of the computing resources as a service. The computing resources include storage (object and network shared storages), databases (SQL and NoSQL databases), compute power (virtual machines, serverless, and Kubernetes), and networking (virtual private cloud). A person can access all what the on-premise data centres could offer in cloud. 

'''Is cloud easy to use?''' 
It may cause a person a bit uncertain to use cloud for the first time. You, as a user, don't have to be a cloud expert to use the cloud services. This is very similar to that you can drive a car but don't know how to fix a car. You simply tell us your technical requirements of your machine learning or data processing projects, and we will setup the cloud storage and compute environment for you. We will guide you through how to use it to train your machine learning models or process your data. The learning curve is way shorter than that of the on-premise data centres. 

'''Is my data secure in cloud?''' 
Security is always the biggest concern for everyone. Your data won't be safe anywhere as long as you don't have professionals to secure it. Almost all data breaches were caused by humans, but not technologies. At DeepSense, we apply the best practices of security measures to make sure your data is secure. Your data is encrypted both at rest and in transit. We also follow the principle of least privileges to make sure only you can access your data. 
When we use storage in cloud, we not only make your data secure, but also durable. For example, the super high durability of AWS S3 storage guarantees that the probability of the data loss is below 0.00000000001%. 

'''Do my jobs have to wait for compute resources in a queue?''' 
Due to the "unlimited" resources in cloud, you never wait for your resources in a queue. There are always available resources for you to use. As long as you make your data and machine learning models ready, the wait time is "zero". 

'''Here we list the characteristics of cloud computing if you are interested in understand more about it.''' 
1. "Unlimited" storage 
2. High elasticity and scalability 
3. Adequate services 
4. Huge network access 
5. Pay-per-use pricing 
6. Hight resiliency and availability 
7. On-demand service 
8. Zero maintenance 
9. Fine-grained reporting service 
10. Security 
11. Automation 
12. Customizable and flexible payment structure 
13. Work from anywhere 
14. Cost-saving 
15. Highly customizable monitoring structure and logging system 

Please feel free to contact DeepSense for any questions!

Introduction to Cloud Computing

2023-03-05T21:46:26Z

Lyang:

Cloud computing is not the future, but already the present. Garner reports that by 2025, '''95%''' of new digital workloads will utilize cloud-native platforms, and '''more than 85%''' of organizations will embrace a cloud-first principle. Cloud revenue was also estimated to surpass non-cloud revenue bringing forth the need for skilled cloud computing professionals in the IT industry. 

'''What is Cloud Computing?''' 
In short words, cloud computing is the delivery of the computing resources as a service. The computing resources include storage (object and network shared storages), databases (SQL and NoSQL databases), compute power (virtual machines, serverless, and Kubernetes), and networking (virtual private cloud). A person can access all what the on-premise data centres could offer in cloud. 

'''Is cloud easy to use?''' 
It may cause a person a bit uncertain to use cloud for the first time. You, as a user, don't have to be a cloud expert to use the cloud services. This is very similar to that you can drive a car but don't know how to fix a car. You simply tell us your technical requirements of your machine learning or data processing projects, and we will setup the cloud storage and compute environment for you. We will guide you through how to use it to train your machine learning models or process your data. The learning curve is way shorter than that of the on-premise data centres. 

'''Is my data secure in cloud?''' 
Security is always the biggest concern for everyone. Your data won't be safe anywhere as long as you don't have professionals to secure it. Almost all data breaches were caused by humans, but not technologies. At DeepSense, we apply the best practices of security measures to make sure your data is secure. 
When we use storage in cloud, we not only make your data secure, but also durable. For example, the super high durability of AWS S3 storage guarantees that the probability of the data loss is below 0.00000000001%. 

'''Do my jobs have to wait for compute resources in a queue?''' 
Due to the "unlimited" resources in cloud, you never wait for your resources in a queue. There are always available resources for you to use. As long as you make your data and machine learning models ready, the wait time is "zero". 

'''Here we list the characteristics of cloud computing if you are interested in understand more about it.''' 
1. "Unlimited" storage 
2. High elasticity and scalability 
3. Adequate services 
4. Huge network access 
5. Pay-per-use pricing 
6. Hight resiliency and availability 
7. On-demand service 
8. Zero maintenance 
9. Fine-grained reporting service 
10. Security 
11. Automation 
12. Customizable and flexible payment structure 
13. Work from anywhere 
14. Cost-saving 
15. Highly customizable monitoring structure and logging system 

Please feel free to contact DeepSense for any questions!

Introduction to Cloud Computing

2023-03-05T21:41:46Z

Lyang:

Cloud computing is not the future, but already the present. Garner reports that by 2025, '''95%''' of new digital workloads will utilize cloud-native platforms, and '''more than 85%''' of organizations will embrace a cloud-first principle. Cloud revenue was also estimated to surpass non-cloud revenue bringing forth the need for skilled cloud computing professionals in the IT industry. 

'''What is Cloud Computing?''' 
In short words, cloud computing is the delivery of the computing resources as a service. The computing resources include storage (object and network shared storages), databases (SQL and NoSQL databases), compute power (virtual machines, serverless, and Kubernetes), and networking (virtual private cloud). A person can access all what the on-premise data centres could offer in cloud. 

'''Is it easy to use?''' 
It may cause a person a bit uncertain to use cloud for the first time. Security is always the biggest concern. You, as a user, don't have to be a cloud expert to use the cloud services. This is very similar to that you can drive a car but don't know how to fix a car. You simply tell us your technical requirements of your machine learning or data processing projects, and we will setup the cloud storage and compute environment for you. We will guide you through how to use it to train your machine learning models or process your data. The learning curve is way shorter than that of the on-premise data centre. 

'''Is my data secure in cloud?''' 
Your data won't be safe anywhere as long as you don't have professionals to secure it. Almost all data breaches were caused by humans, but not technologies. At DeepSense, we apply the best practices of security measures to make sure your data is secure. 
When we use storage on cloud, we not only make your data secure, but also durable. For example, the super high durability of AWS S3 storage guarantees that the probability of the data loss is below 0.00000000001%. 

'''Do my jobs have to wait for compute resources in a queue?''' 
Due to the "unlimited" resources in cloud, you never wait for your resources in a queue. There are always available resources for you to use. As long as you make your data and machine learning models ready, the wait time is "zero". 

'''Here we list the characteristics of cloud computing if you are interested in understand more about it.''' 
1. "Unlimited" storage 
2. High elasticity and scalability 
3. Adequate services 
4. Huge network access 
5. Pay-per-use pricing 
6. Hight resiliency and availability 
7. On-demand service 
8. Zero maintenance 
9. Fine-grained reporting service 
10. Security 
11. Automation 
12. Customizable and flexible payment structure 
13. Work from anywhere 
14. Cost-saving 
15. Highly customizable monitoring structure and logging system 

Please feel free to contact DeepSense for any questions!

Introduction to Cloud Computing

2023-03-05T19:25:23Z

Lyang:

Cloud computing is not the future, but already the present. Garner reports that by 2025, '''95%''' of new digital workloads will utilize cloud-native platforms, and '''more than 85%''' of organizations will embrace a cloud-first principle. Cloud revenue was also estimated to surpass non-cloud revenue bringing forth the need for skilled cloud computing professionals in the IT industry. 

'''What is Cloud Computing?''' 
In short words, cloud computing is the delivery of the computing resources as a service. The computing resources include storage (object and network shared storages), databases (SQL and NoSQL databases), compute power (virtual machines, serverless, and Kubernetes), and networking (virtual private cloud). A person can obtain all what the on-premise data centres offer in cloud. 

'''Is it easy to use?''' 
It may cause a person a bit uncertain to use cloud for the first time. However, you don't have to be a cloud expert to use the cloud services. You simply tell us your technical requirements of your machine learning or data processing projects, and we will setup the cloud storage and compute environment for you. We will guide you through how to use it to train your machine learning models or process your data. The learning curve is way shorter than that of the on-premise data centre. 

'''Is my data secure in cloud?''' 
Your data won't be safe anywhere as long as you don't have professionals to secure it. Almost all data breaches were caused by humans, but not technologies. At DeepSense, we apply the best practices of security measures to make sure your data is secure. 
When we use storage on cloud, we not only make your data secure, but also durable. For example, the super high durability of AWS S3 storage guarantees that the probability of your data loss is 0.00000000001%. 

'''Do my jobs have to wait for compute resources in a queue?''' 
Due to the "unlimited" resources in cloud, you never wait for your resources in a queue. There are always available resources for you to use. As long as you make your data and machine learning models ready, the wait time is "zero". 

'''Here we list the characteristics of cloud computing if you are interested in understand more about it.''' 
1. "Unlimited" storage 
2. High elasticity and scalability 
3. Adequate services 
4. Huge network access 
5. Pay-per-use pricing 
6. Hight resiliency and availability 
7. On-demand service 
8. Zero maintenance 
9. Fine-grained reporting service 
10. Security 
11. Automation 
12. Customizable and flexible payment structure 
13. Work from anywhere 
14. Cost-saving 
15. Highly customizable monitoring structure and logging system 

Please feel free to contact DeepSense for any questions!

Introduction to Cloud Computing

2023-03-05T19:18:34Z

Lyang:

Cloud computing is not the future, but already the present. Garner reports that by 2025, '''95%''' of new digital workloads will utilize cloud-native platforms, and '''more than 85%''' of organizations will embrace a cloud-first principle. Cloud revenue was also estimated to surpass non-cloud revenue bringing forth the need for skilled cloud computing professionals in the IT industry. 

'''What is Cloud Computing?''' 
In short words, cloud computing is the delivery of the computing resources as a service. The computing resources include storage (object and network shared storages), databases (SQL and NoSQL databases), compute power (virtual machines, serverless, and Kubernetes), and networking (virtual private cloud). A person can obtain all what the on-premise data centres offer in cloud. 

'''Is it easy to use?''' 
It may cause a person a bit uncertain to use cloud for the first time. However, you don't have to be a cloud expert to use the cloud services. You simply tell us your technical requirements of your machine learning or data processing projects, and we will setup the cloud storage and compute environment for you. We will guide you through how to use it to train your machine learning models or process your data. The learning curve is way shorter than that of the on-premise data centre. 

'''Is my data secure in cloud?''' 
Your data won't be safe anywhere as long as you don't have professionals to secure it. Almost all data breach is caused by humans, but not technologies. At DeepSense, we apply the best practices of security measures to make sure your data is secure. 
When we use storage on cloud, we not only make your data secure, but also durable. For example, the super high durability of AWS S3 storage guarantees that the probability of your data loss is 0.00000000001%. 

'''Do my jobs have to wait for compute resources in a queue?''' 
Due to the "unlimited" resources in cloud, you never wait for your resources in a queue. There are always available resources for you to use. As long as you make your data and machine learning models ready, the wait time is "zero". 

'''Here we list the characteristics of cloud computing if you are interested in understand more about it.''' 
1. "Unlimited" storage 
2. High elasticity and scalability 
3. Adequate services 
4. Huge network access 
5. Pay-per-use pricing 
6. Hight resiliency and availability 
7. On-demand service 
8. Zero maintenance 
9. Fine-grained reporting service 
10. Security 
11. Automation 
12. Customizable and flexible payment structure 
13. Work from anywhere 
14. Cost-saving 
15. Highly customizable monitoring structure and logging system

Introduction to Cloud Computing

2023-03-05T18:27:03Z

Lyang:

Cloud computing is not the future, but already the present. Garner reports that by 2025, 95 percent of new digital workloads will utilize cloud-native platforms. Cloud revenue was also estimated to surpass non-cloud revenue bringing forth the need for skilled cloud computing professionals in the IT industry. 

'''What is Cloud Computing?''' 
In short words, cloud computing is the delivery of the computing resources as a service. The computing resources include storage (object and network shared storages), databases (SQL and NoSQL databases), compute power (virtual machines, serverless, and Kubernetes), and networking (virtual private cloud). A person can obtain all the functionalities for his/her workloads from the on-premise data centres. 

'''Is it easy to use?''' 
As a user of cloud computing to develop your ocean AI projects, you don't need to be a cloud expert. You simply tell us your technical requirements and we will setup the storage and compute environment for you. We then guide you through how to use it to train your machine learning models or process your data. The learning curve is way shorter than that of the on-premise data center. 

'''Is my data secure in cloud?''' 
Your data won't be safe anywhere as long as you don't have professionals to secure it. Almost all data breach is caused by humans, but not technologies. At DeepSense, we apply the best practices of security measures of AWS, GCP, and Azure to make your data very secure. 
When we use storage on cloud, we not only make your data secure, but also super durable. For example, the super high durability of AWS S3 storage guarantees that the probability of your data loss is 0.00000000001%. 

'''Do my jobs have to wait for compute resources in a queue?''' 
Due the "unlimited" resources in cloud, you never wait for your resources in a queue. There are always available resources for you to use. As long as you make your data and machine learning models ready, the wait time is "zero". 

'''There are some characteristics of cloud computing that you may not use, but it is good to understand them. They are:''' 
1. "Unlimited" storage 
2. High elasticity and scalability 
3. Adequate services 
4. Huge network access 
5. Pay-per-use pricing 
6. Hight resiliency and availability 
7. On-demand service 
8. Zero maintenance 
9. Fine-grained reporting service 
10. Security 
11. Automation 
12. Customizable and flexible payment structure 
13. Work from anywhere 
14. Cost-saving 
15. Highly customizable monitoring structure and logging system

Introduction to Cloud Computing

2023-03-05T18:25:34Z

Lyang:

Cloud computing is not the future, but already the present. Garner reports that by 2025, 95 percent of new digital workloads will utilize cloud-native platforms. Cloud revenue was also estimated to surpass non-cloud revenue bringing forth the need for skilled cloud computing professionals in the IT industry. 

What is Cloud Computing? 
In short words, cloud computing is the delivery of the computing resources as a service. The computing resources include storage (object and network shared storages), databases (SQL and NoSQL databases), compute power (virtual machines, serverless, and Kubernetes), and networking (virtual private cloud). A person can obtain all the functionalities for his/her workloads from the on-premise data centres. 

Is it easy to use? 
As a user of cloud computing to develop your ocean AI projects, you don't need to be a cloud expert. You simply tell us your technical requirements and we will setup the storage and compute environment for you. We then guide you through how to use it to train your machine learning models or process your data. The learning curve is way shorter than that of the on-premise data center. 

Is the cloud secure enough for my data? 
Your data won't be safe anywhere as long as you don't have professionals to secure it. Almost all data breach is caused by humans, but not technologies. At DeepSense, we apply the best practices of security measures of AWS, GCP, and Azure to make your data very secure. 
When we use storage on cloud, we not only make your data secure, but also super durable. For example, the super high durability of AWS S3 storage guarantees that the probability of your data loss is 0.00000000001%. 

Do my jobs have to wait for compute resources in a queue? 
Due the "unlimited" resources in cloud, you never wait for your resources in a queue. There are always available resources for you to use. As long as you make your data and machine learning models ready, the wait time is "zero". 

There are some characteristics of cloud computing that you may not use, but it is good to understand them. They are: 
1. "Unlimited" storage 
2. High elasticity and scalability 
3. Adequate services 
4. Huge network access 
5. Pay-per-use pricing 
6. Hight resiliency and availability 
7. On-demand service 
8. Zero maintenance 
9. Fine-grained reporting service 
10. Security 
11. Automation 
12. Customizable and flexible payment structure 
13. Work from anywhere 
14. Cost-saving 
15. Highly customizable monitoring structure and logging system

Introduction to Cloud Computing

2023-03-05T17:57:08Z

Lyang: Created page with "Cloud computing is not the future, but already the present. Garner reports that by 2025, 95 percent of new digital workloads will utilize cloud-native platforms. Cloud revenue..."

DeepSense Documentation

2023-02-28T18:36:35Z

Lyang:

'''Welcome to the DeepSense technical cloud documentation wiki'''. This is the primary source for users with questions on the DeepSense cloud equipment and services. DeepSense uses cloud services from various cloud vendors for the development of AI projects. You'll now find all of our content on the sidebar. Just below you can see the available cloud services, and information about any planned outages we may have.
Due to the nature of "unlimited resources" of cloud computing, DeepSense doesn't limit any cloud services that the projects need. The cloud services listed in the following tables are our currently running resources. This doesn't necessarily indicate we cannot use other cloud resources. DeepSense users are encouraged to contact us to apply for required cloud computing services.
We continuously explore the best cloud solutions to the AI projects. We don't lock our solutions in any specific cloud vendors. The tables below show the cloud services we have tested and are developing projects on. More cloud services will be coming soon.

We routinely make changes and update the content. If you see anything missing, or have any suggestions for content, we would appreciate hearing from you. You can send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).
You can click on "Resources" on the navigation panel to find the technical details of the virtual machines and serverless computing services.

== DeepSense Cloud Computing Services ==

'''Amazon Web Services (AWS)'''
{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|'''Availability'''
|style="width:20% | '''Service'''
|style="width:70% | '''Usage'''
|-
|style="Color:green" | Available
| S3 (Simple Storage Service)
| Amazon S3 can be used to store and retrieve any amount of data. Mainly use it for backing up your data.
|-
|style="Color:green" | Available
| EC2 (Elastic Compute Cloud)
| Amazon EC2 can be used to create virtual machines for training models in a manually configured environment.
|-
|style="Color:green" | Available
| SageMaker Notebook
| Amazon SageMaker notebook instance is a machine learning (ML) compute instance that runs the Jupyter Notebook App.
|}

MediaWiki:Sidebar

2023-02-28T18:35:33Z

Lyang:

DeepSense Documentation

2023-02-28T18:26:33Z

Lyang:

'''
== Attention! We are reconstructing our wiki pages to update all the amazing cloud computing resources we are offering to our users. We are sorry that the pages are messy during the reconstruction. We will keep you updated. Stay tuned... ==
'''

'''Welcome to the DeepSense technical cloud documentation wiki'''. This is the primary source for users with questions on the DeepSense cloud equipment and services. DeepSense uses cloud services from various cloud vendors for the development of AI projects. You'll now find all of our content on the sidebar. Just below you can see the available cloud services, and information about any planned outages we may have.
Due to the nature of "unlimited resources" of cloud computing, DeepSense doesn't limit any cloud services that the projects need. The cloud services listed in the following tables are our currently running resources. This doesn't necessarily indicate we cannot use other cloud resources. DeepSense users are encouraged to contact us to apply for required cloud computing services.
We continuously explore the best cloud solutions to the AI projects. We don't lock our solutions in any specific cloud vendors. The tables below show the cloud services we have tested and are developing projects on. More cloud services will be coming soon.

We routinely make changes and update the content. If you see anything missing, or have any suggestions for content, we would appreciate hearing from you. You can send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).
You can click on "Resources" on the navigation panel to find the technical details of the virtual machines and serverless computing services.

== DeepSense Cloud Computing Services ==

'''Amazon Web Services (AWS)'''
{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|'''Availability'''
|style="width:20% | '''Service'''
|style="width:70% | '''Usage'''
|-
|style="Color:green" | Available
| S3 (Simple Storage Service)
| Amazon S3 can be used to store and retrieve any amount of data. Mainly use it for backing up your data.
|-
|style="Color:green" | Available
| EC2 (Elastic Compute Cloud)
| Amazon EC2 can be used to create virtual machines for training models in a manually configured environment.
|-
|style="Color:green" | Available
| SageMaker Notebook
| Amazon SageMaker notebook instance is a machine learning (ML) compute instance that runs the Jupyter Notebook App.
|}

DeepSense Documentation

2023-02-28T18:16:43Z

Lyang:

'''
== Attention! We are reconstructing our wiki pages to update all the amazing cloud computing resources we are offering to our users. We are sorry that the pages are messy during the reconstruction. We will keep you updated. Stay tuned... ==
'''

'''Welcome to the DeepSense technical cloud documentation wiki'''. This is the primary source for users with questions on the DeepSense cloud equipment and services. DeepSense uses cloud services from various cloud vendors for the development of AI projects. You'll now find all of our content on the sidebar. Just below you can see the available cloud services, and information about any planned outages we may have.
Due to the nature of "unlimited resources" of cloud computing, DeepSense doesn't limit any cloud services that the projects need. The cloud services listed in the following tables are our currently running resources. This doesn't necessarily indicate we cannot use other cloud resources. DeepSense users are encouraged to contact us to apply for required cloud computing services.
We continuously explore the best cloud solutions to the AI projects. We don't lock our solutions in any specific cloud vendors. The tables below show the cloud services we have tested and are developing projects on. More cloud services will be coming soon.

We routinely make changes and update the content. If you see anything missing, or have any suggestions for content, we would appreciate hearing from you. You can send us an email at ([mailto:support@deepsense.ca support@deepsense.ca]).

== Cloud Services ==

'''Amazon Web Services (AWS)'''
{|class="wikitable" style="text-align: center; color: black; font-style:bold"
|'''Availability'''
|style="width:20% | '''Service'''
|style="width:70% | '''Usage'''
|-
|style="Color:green" | Available
| S3 (Simple Storage Service)
| Amazon S3 can be used to store and retrieve any amount of data. Mainly use it for backing up your data.
|-
|style="Color:green" | Available
| EC2 (Elastic Compute Cloud)
| Amazon EC2 can be used to create virtual machines for training models in a manually configured environment.
|-
|style="Color:green" | Available
| SageMaker Notebook
| Amazon SageMaker notebook instance is a machine learning (ML) compute instance that runs the Jupyter Notebook App.
|}

MediaWiki:Sidebar

2023-02-23T12:13:21Z

Lyang:

* navigation
** mainpage|mainpage-description
* DeepSense Cloud Computing Platform
** mainpage | Available Cloud Services
** Resources | Resources
* Getting Support
** Contact information | Contact - Support email
* Getting Started
** Introduction to Cloud Computing | Introduction to Cloud Computing
** User Guide | User Guide
** Video Tutorials | Video Tutorials
** Accessing Systems | Accessing Systems
*** VPN Setup | VPN Setup
*** SSH client setup | SSH client setup
*** Basic Linux | Basic Linux
*** Glossary | Glossary for Clusters
*** Info for first time cluster users | Intro to Clusters
** LSF | Basic LSF Jobs
** CWS | Conductor with Spark
** Visualization | Visualization
* How to develop Machine Learning projects on cloud
** Deep Learning Frameworks | ML/DL Frameworks
** Software | Software
*** Available software | Available Software
*** Installing Software | Installing Software
*** Getting started with Deep Learning | Using Software
** Running ML Jobs | Running ML jobs
***Submitting Jobs | Submitting Jobs
***Checking Job Status | Checking Job Status
***Writing Script| Writing Script
** Getting started with Jupyter Notebook | Using Jupyter Notebook
** Deep Learning Tutorials | ML/DL Tutorials
* Storage on cloud
** Overview of your Storage on DeepSense | Storage Overview
** How to Transfer Data | How to Transfer Data
** Backup Policies | Backup Policies
** Quota Information and Management | Storage Quotas
* FAQ
** Restrictions | Restrictions
** Workarounds | Workarounds
** Best Practices | Best Practices
*** Your Accounts | Your Accounts
*** Data Storage | Data Storage
*** LSF Jobs | LSF Jobs
* Writing Tips
** Mitacs Accelerate Proposals | Mitacs Accelerate Proposals
** Paper Prep | Paper Prep
* DeepSense
** https://deepsense.ca | DeepSense home page
** Acknowledging DeepSense | Acknowledging DeepSense
** Terms of Use | Terms of use
* Self Directed Projects
** Training Projects | Training Projects
* Additional Resources
** Related Links | Related Links
** External Links | External Links
** External Data Sources | External Data Sources

MediaWiki:Sidebar

2023-02-23T12:11:09Z

Lyang:

* navigation
** mainpage|mainpage-description
* DeepSense Cloud Computing Platform
** mainpage | Available Cloud Services
** Resources | Resources
* Getting Support
** Contact information | Contact - Support email
* Getting Started
** Introduction to Cloud Computing
** User Guide | User Guide
** Video Tutorials | Video Tutorials
** Accessing Systems | Accessing Systems
*** VPN Setup | VPN Setup
*** SSH client setup | SSH client setup
*** Basic Linux | Basic Linux
*** Glossary | Glossary for Clusters
*** Info for first time cluster users | Intro to Clusters
** LSF | Basic LSF Jobs
** CWS | Conductor with Spark
** Visualization | Visualization
* How to develop Machine Learning projects on cloud
** Deep Learning Frameworks | ML/DL Frameworks
** Software | Software
*** Available software | Available Software
*** Installing Software | Installing Software
*** Getting started with Deep Learning | Using Software
** Running ML Jobs | Running ML jobs
***Submitting Jobs | Submitting Jobs
***Checking Job Status | Checking Job Status
***Writing Script| Writing Script
** Getting started with Jupyter Notebook | Using Jupyter Notebook
** Deep Learning Tutorials | ML/DL Tutorials
* Storage on cloud
** Overview of your Storage on DeepSense | Storage Overview
** How to Transfer Data | How to Transfer Data
** Backup Policies | Backup Policies
** Quota Information and Management | Storage Quotas
* FAQ
** Restrictions | Restrictions
** Workarounds | Workarounds
** Best Practices | Best Practices
*** Your Accounts | Your Accounts
*** Data Storage | Data Storage
*** LSF Jobs | LSF Jobs
* Writing Tips
** Mitacs Accelerate Proposals | Mitacs Accelerate Proposals
** Paper Prep | Paper Prep
* DeepSense
** https://deepsense.ca | DeepSense home page
** Acknowledging DeepSense | Acknowledging DeepSense
** Terms of Use | Terms of use
* Self Directed Projects
** Training Projects | Training Projects
* Additional Resources
** Related Links | Related Links
** External Links | External Links
** External Data Sources | External Data Sources

DeepSense Documentation

2023-02-21T13:04:50Z

Lyang:

LSF

2021-06-02T16:09:55Z

Lyang:

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory for at most 24 hours you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -W 24:0 -R "span[hosts=1] rusage[mem=256000]" <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

===Shell Scripts for Batch Jobs===
Users can just run a single command line to submit batch jobs. The job scheduler would take care of everything and users only need to check their output and/or errors. Users do not need to keep themselves logged in the systems when the jobs are running. An example job submission command is shown above.
However, if you will have to run your script in an environment that is not set as the default in your .bashrc file, you can write a simple shell script to set the environments. For example, you may want to use a specific Conda environment and/or Python version for your Python script, you would need to write a shell script to set the environments. Here is an example. I have a Python script whose name is "myPython.py" and it would need to use my anaconda3 and py36_tensorflow environments. I would create a shell script, say with name "myShellScript.sh", with the following contents:

#!/bin/bash
source ~/anaconda3/etc/profile.d/conda.sh
conda activate py36_tensorflow
python myPython.py
Then, save your edit and run the following command to make your shell script executable:
chmod +x myShellScript.sh
Then, submit your job:
bsub -gpu - /path/to/myShellScript.sh
Check if your job is submitted successfully.

===Check Job Progress of Batch Jobs===
When your batch jobs are not finished, you can check the progress of your jobs using command 'bpeek'. This command provides detailed information about your running jobs and it is very helpful for you to monitor and decide if you would need to terminate it for whatever reasons to save time.
You can always run 'bpeek jobid' to check the progress. You can also use 'man bpeek' to get more detailed usage of the command.
Be aware that this command doesn't work for interactive jobs.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
LSF has two different types of memory limits.
The scheduler memory limit <code>-R "rusage[mem=<memlimit>]"</code> requests <code><memlimit></code> amount of memory. Your job will not start until a compute node is available with that amount of memory. You are guaranteed to have this amount of memory available. If you exceed the requested amount then your job may be killed but it will only be killed if other jobs need that memory.

The job memory limit <code>-R "rusage[mem=<memlimit>]</code> will kill your job if it exceeds the given memory limit. Note that this option does not guarantee that you will have that amount of memory available.

The memory limits are specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code> and <code>-R "rusage[mem=256GB]"</code>.

If you are using more than a few GB of memory than you must specify the <code>-R "rusage[mem=<memlimit>]"</code> option or your job may be terminated. You may additionally want to use the <code>-M <memlimit></code> option to be sure you aren't using more memory than intended.

=== Time Limit ===
The runtime limit <code>-W hours:minutes</code> specifies the maximum length of time your job is allowed to run.
For example <code>-W 24:0</code> requests 24 hours of running time.
Your job will be terminated when the runtime limit is exceeded.

If you do not specify a runtime limit then the default runtime limit of 168 hours (7 days) will be used.
The maximum possible runtime limit is currently 30 days and may vary by queue in the future.

If there is a scheduled maintenance window announced then any job with a run time limit that could extend into the maintenance period will be listed as pending and will not run until the maintenance has concluded. Use a shorter run time limit that ends before the maintenance period to avoid this.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=exclusive_process:mps=yes:j_exclusive=yes"</code>
<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode. Yours jobs will be running on exclusive mode by default which means that no other jobs would share the gpus your jobs are using.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs? By default, it is 'yes' for your jobs.

By default the <code>-gpu -</code> option will request one exclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

Note that output may not be written to the specified file immediately. You can use the <code>bpeek <jobid></code> command to view the output of a currently running job.

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB -R "rusage[mem=100MB]"</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_kc_cmd_ref.html here].

LSF

2021-05-31T11:20:17Z

Lyang:

[https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/ IBM Spectrum LSF] is the command line job submission system for submitting batch and interactive jobs on DeepSense computing hardware.

== Test code and short computation ==
DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus. From off campus you’ll need to use the [https://wireless.dal.ca/vpnsoftware.php Dalhousie VPN].

The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes.

== Job Submission ==
When you have a small example working with your code and are ready to run a real workload, use the LSF queue to submit your jobs to the cluster (https://www.ibm.com/support/knowledgecenter/SSWRJV_10.1.0/lsf_users_guide/batch_jobs_about.html). If you’ve used other queuing systems like slurm or Sun Grid Engine before then LSF will seem very familiar.

To submit a job you use the <code>bsub</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.man_top.1.html).

For example, to submit a shared memory job using 20 processors and 256GB of memory for at most 24 hours you would run:

<code>bsub -oo <output_file> -n 20 -M 256000 -W 24:0 -R "span[hosts=1] rusage[mem=256000]" <executable> [options]</code>

For openMP jobs, please make sure that you use <code>OMP_NUM_THREADS</code> to limit the number of threads your program uses and that you set this variable in your code that will run on the server. LSF sets a variable <code>$LSB_DJOB_NUMPROC</code> that you can use if you don’t want to hardcode <code>OMP_NUM_THREADS</code> or set it with your own variable.

===Shell Scripts for Batch Jobs===
Users can just run a single command line to submit batch jobs. The job scheduler would take care of everything and users only need to check their output and/or errors. Users do not need to keep themselves logged in the systems when the jobs are running. An example job submission command is shown above.
However, if you will have to run your script in an environment that is not set as the default in your .bashrc file, you can write a simple shell script to set the environments. For example, you may want to use a specific Conda environment and/or Python version for your Python script, you would need to write a shell script to set the environments. Here is an example. I have a Python script whose name is "myPython.py" and it would need to use my anaconda3 and py36_tensorflow environments. I would create a shell script, say with name "myShellScript.sh", with the following contents:

#!/bin/bash
source ~/anaconda3/etc/profile.d/conda.sh
conda activate py36_tensorflow
python myPython.py
Then, save your edit and run the following command to make your shell script executable:
chmod +x myShellScript.sh
Then, submit your job:
bsub -gpu - /path/to/myShellScript.sh
Check if your job is submitted successfully.

=== CPU Limit ===
The number of requested processors is specified with the option <code>-n</code>.

The resource request <code>-R "span[hosts=1]"</code> requires that all processors are on the same compute host, i.e. a shared memory job.

LSF can also be used to run compute jobs across multiple hosts such as MPI jobs. Examples will be included here at a later date.

=== Memory Limit ===
LSF has two different types of memory limits.
The scheduler memory limit <code>-R "rusage[mem=<memlimit>]"</code> requests <code><memlimit></code> amount of memory. Your job will not start until a compute node is available with that amount of memory. You are guaranteed to have this amount of memory available. If you exceed the requested amount then your job may be killed but it will only be killed if other jobs need that memory.

The job memory limit <code>-R "rusage[mem=<memlimit>]</code> will kill your job if it exceeds the given memory limit. Note that this option does not guarantee that you will have that amount of memory available.

The memory limits are specified in MB by default. You can also specify units, e.g. <code>-M 256GB</code> and <code>-R "rusage[mem=256GB]"</code>.

If you are using more than a few GB of memory than you must specify the <code>-R "rusage[mem=<memlimit>]"</code> option or your job may be terminated. You may additionally want to use the <code>-M <memlimit></code> option to be sure you aren't using more memory than intended.

=== Time Limit ===
The runtime limit <code>-W hours:minutes</code> specifies the maximum length of time your job is allowed to run.
For example <code>-W 24:0</code> requests 24 hours of running time.
Your job will be terminated when the runtime limit is exceeded.

If you do not specify a runtime limit then the default runtime limit of 168 hours (7 days) will be used.
The maximum possible runtime limit is currently 30 days and may vary by queue in the future.

If there is a scheduled maintenance window announced then any job with a run time limit that could extend into the maintenance period will be listed as pending and will not run until the maintenance has concluded. Use a shorter run time limit that ends before the maintenance period to avoid this.

=== GPU Computation ===

To request access to a GPU use the <code>-gpu -</code> options.

Note the trailing dash, which specifies the default GPU arguments. The following options can be used in place of that dash.

The default GPU arguments are <code>"num=1:mode=exclusive_process:mps=yes:j_exclusive=yes"</code>
<code>num=num_gpus</code> is the number of requested GPUs on each host.

<code>mode=shared | exclusive_process</code> specifies the GPU mode. Yours jobs will be running on exclusive mode by default which means that no other jobs would share the gpus your jobs are using.

<code>mps=yes | no</code> use the Nvidia Multi-Process Server (MPS). MPS enables better sharing of GPU resources.

<code>j_exclusive=yes | no</code> Is the GPU exclusive to this job and prevented from being used by other jobs? By default, it is 'yes' for your jobs.

By default the <code>-gpu -</code> option will request one exclusive GPU. Please limit your usage of GPU resources to a reasonable number of concurrently used GPUs. We may enact limits on GPU use in the feature if necessary.

See the [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bsub.gpu.1.html bsub.gpu] documentation for more information on submitting GPU jobs.

=== Input and Output files ===
If you do not specify an output file with <code>-o</code> (append) or <code>-oo</code> (overwrite) then the output will be lost. Note that LSF will prepend submission information to this file. You can use typical linux options like <code>> output_file2</code> in which case the file specified with <code>-oo</code> will just contain any errors and submission information.

You can specify an input file with the <code>-i</code> option or the typical linux option <code>< <input_file></code>

Note that output may not be written to the specified file immediately. You can use the <code>bpeek <jobid></code> command to view the output of a currently running job.

== Advanced Job Submission ==

=== Array Jobs ===
To run the same program multiple time with different input and output files you can use [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/job_arrays_lsf.html LSF Array Jobs].

An example command in the LSF documentation is given as:

<code> bsub -J "myArray[1-1000]" -i "input.%I" -o "output.%I" myJob</code>

This command uses only one line to submit 1000 jobs running the script myJob with the input file <code>input.1, input.2, ... input.1000</code> with the output of each job placed in the files <code>output.1, output.2, ... output.1000</code>

=== Complicated Jobs ===
To run the same program with multiple files, possibly with different options, you can create a job submission script that iterates over the files and submits the jobs.

For example, suppose you have <code>programA</code> and want to process <code>input.1, input.2, ... input.N</code> with output in <code>output.1, output.2, ... output.N</code>, as in the array example.

Create a bash script <code>do_submit_programA.bash</code> that looks something like:

n=<N>
arguments=<nodes, memory, time constraints, etc>
for ((i=1; i<=$n; i++)); do
bsub -oo log.$i $arguments programA < input.$i > output.$i
done

Note that everything in triangle braces here is not real code. For example <code>N</code> might be read from a command line argument or hardcoded as say 10. The arguments will be something like <code>-n 1 -M 100MB -R "rusage[mem=100MB]"</code> and any other desired options. You can run multiple types of jobs with complex arguments.

You may wish to create separate directories for the log files, input files, and output files if there are more than a handful of jobs.

If each job requires nontrivial processing (e.g. changing into different directories for each job) then you may want to create a second script that generates the jobfiles and then use a similar kind of submit script.

=== Interactive Jobs ===

Some jobs may require user input such as testing code on a gpu system or an interactive analytics program.

<code>bsub -I</code> requests an interactive job that will print its output to your terminal.

<code>bsub -Ip</code> requests an interactive job with a pseudo terminal. For example, this can be used to schedule a console program that takes user input and output.

<code>bsub -Is</code> requests an interactive job with a shell. This can be used to test code on one of the gpu nodes or for more resource intensive development than is allowed on the login nodes.

Note that interactive jobs are still subject to time and memory constraints as typical batch jobs. Please be careful not to interfere with other jobs running on a node and that your interactive job does not attempt to use more resources than you have requested. Please do not leave interactive jobs running for long periods and do not leave interactive jobs idle when you are not using them.

We do not currently treat interactive jobs different than any other jobs. As DeepSense becomes more heavily utilized we may need to limit the number of interactive jobs run by a user, project, or on a given node. We may need to limit the time or other resources used by interactive jobs.

== Job Information ==

=== Running Jobs ===

To examine currently running jobs you use the <code>bjobs</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bjobs.man_top.1.html)

<code>bjobs -l</code> or <code>bjobs -l <jobid></code> shows additional job information including job status and resource usage.

=== Past Jobs ===

To examine current and past jobs use the <code>bhist</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhist.1.html).

The following options will show jobs with the specified status:
-a all
-d finished
-e exited
-p pending
-r running
-s suspended

You can use options like <code>-S start_time,end_time</code> and <code>-C start_time,end_time</code> to find jobs that were submitted or completed between the specified time intervals. These options require using the <code>-a</code> option.

As with bjobs, you can use the <code>-l</code> option for additional information and can also specify a specific known jobid as the last command argument.

=== Available Hosts ===

To see the available hosts and how busy they are you use the <code>bhosts</code> command (https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_command_ref/bhosts.1.html)

== LSF Command Reference ==

The complete list of LSF commands with description is available [https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_kc_cmd_ref.html here].

Installing Software

2020-12-22T13:54:10Z

Lyang: /* 3. Installation of Deep Learning packages */

<div class="noautonum">

== 1. Logging on ==

DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus.

For example, if your userid is <code>user1</code>, you can connect to deepsense by typing <code>ssh user1@login1.deepsense.ca</code> just like logging on to any other network computer.

'''Note''': The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes. Keep reading for instructions on how to submit compute jobs to dedicated compute nodes.

=== 1.1 VPN ===

To connect to the DeepSense platform from outside of the Dalhousie Campus, you'll need to use a VPN.
If you are are student, staff or faculty, you can use the Dalhousie VPN (https://wireless.dal.ca/vpnsoftware.php).

If you are not a Dalhousie staff, student, or faculty but require offsite access and cannot use the Dalhousie VPN then contact your project leader or ([mailto:support@deepsense.ca support@deepsense.ca]) to make different arrangements.

For more info, see [[VPN Setup]].

== 2. Configure your environment ==

DeepSense compute and management nodes are IBM Power8 computers (ppc64le) running Redhat Enterprise Linux. See [[Resources]] for more details on the available nodes.

=== 2.1 Loading a python environment ===

You have two options for using python on DeepSense. You can use the systemwide python install, managed by DeepSense administrators. This is recommended for users new to Linux. You will need to contact DeepSense support to have additional software packages installed in the systemwide python.

Alternatively, you can install an Anaconda python environment or other software in your home directory. This allows you to install or update packages or software without requesting and waiting for DeepSense staff.

==== Systemwide python (managed by DeepSense) ====

DeepSense has two Anaconda python environments. Anaconda 2 is installed on each compute node. While Anaconda 3 is installed in a shared directory that can be accessed from any machines in the cluster.

First one is anaconda2 installed in /opt/anaconda2 will provide you python 2.7.5. To use this systemwide python add a parameter to your .bashrc file in your home directory:

echo ". /opt/anaconda2/etc/profile.d/conda.sh" >> ~/.bashrc

Second is anaconda3 installed in /software/WMLA/anaconda3 will provide you python 3.7.4. To use this systemwide python add a parameter to your .bashrc file in your home directory:

echo ". /software/WMLA/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc

Then source your .bashrc file:
source ~/.bashrc

To load the python environment run <code>conda activate</code>

You can add either line to your .bashrc file to automatically load the desired environment when you log in.

==== Local python install (managed by individual user) ====

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact [[Contact_Information|Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation. If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Contact_Information|Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le). If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Contact_Information|Technical Support]]. We may install and support it systemwide if there is sufficient interest.

'''Stop using systemwide anaconda'''

If you added the system anaconda environment to your <code>.bashrc</code> file then remove the line:
. /opt/anaconda2/etc/profile.d/conda.sh

'''Installing Anaconda with a python3 base '''

From your home directory run:
wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-ppc64le.sh
bash Anaconda3-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
. ~/anaconda3/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:
source ~/.bashrc

'''Adding a python2 environment'''

The previous instruction creates a python3 base environment. To add a python2 environment:
conda create -n py27 python=2.7

Activate this environment to use python3:
conda activate py27

note: if you receive an error message then you may need to deactivate the base conda environment first:
conda deactivate
conda activate py27

'''Adding a python3 environment'''

We recommend creating a separate python3 environment from the base environment. This makes it easier to install the specific packages required for IBM WMLA/PowerAI.
conda create -n py36 python=3.6

Activate this environment to use python3:
conda activate py36

==3. Installation of Deep Learning packages==

===3.1 Using IBM-AI Deep Learning Anaconda Channel===

To use deep learning packages like Tensorflow on DeepSense you need to add the IBM-AI anaconda channel to your list of available software channels.
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

We suggest creating a new environment for each deep learning package you want to use. For example for Tensorflow:
conda create -n py36_tensorflow python=3.6
conda activate py36_tensorflow

Then install the anaconda package for the software you need. Again, with Tensorflow as an example:
conda install tensorflow

You can then use tensorflow or other deep learning packages as needed by simply activating that anaconda environment. Unlike the old method, you do not need to specifically activate tensorflow or other deep learning methods.

The above channel is still working, but the supported software version will not be updated any more. For example, you can only install TensorFlow 2.1.2 and PyTorch 1.3.1 using this channel. You can directly visit the IBM-AI anaconda channel URL to see a list of available software and their versions (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/). If you would like to install higher versions of software, you can refer to the next section.

===3.2 Install PyTorch 1.6.0 in a user's home directory on DeepSense ===

A DeepSense user can install PyTorch by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build PyTorch with those versions. 
Here are the steps that a normal DeepSense user install PyTorch 1.6.0 in his/her home directory.

1. Source the conda environment you would like to use. For example:

source ~/anaconda3/etc/profile.d/conda.sh

2. Activate the environment you would use to install PyTorch. If the environment hasn't been created, a user can create it and install PyTorch in one command line. For example, if you would create an environment with name "my-environment" (This is just an example. Please choose a meaningful name for yourself.) and install PyTorch, you would run the following command:

conda create -y -n my-environment python=3.6 pytorch -c file:////software/PyTorch-1.6.0-Build/condabuild

3. If the environment has been created, say the name of the environment is "my-environment", you would need to activate the environment first and then install PyTorch. For example:

conda activate my-environment
conda install pytorch -c file:////software/PyTorch-1.6.0-Build/condabuild

This should take about 2 minutes to install.

4. To test if your install is successful, issue python from the environment where PyTorch is installed. Then run "import torch" to see if there are any errors. For example:
[luy@ds-lg-01 ~]$ conda activate my-environment
(my-environment) [luy@ds-lg-01 ~]$ python
Python 3.6.12 |Anaconda, Inc.| (default, Sep 9 2020, 00:40:10) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> 

===3.3 Install Opencv 3.4.10 in a user's home directory on DeepSense ===
A DeepSense user can install Opencv by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build/opencv-feedstock. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build Opencv with those versions. 
Here are the steps that a normal DeepSense user installs Opencv 3.4.10 in his/her home directory.
1. Source the conda environment you would like to use. For example:
source ~/anaconda3/etc/profile.d/conda.sh

2. Activate the environment you would use to install Opencv. If the environment hasn't been created, a user should create one. Assume a user created an environment "my-environment" and activated it. To install Opencv, you would run the following command:
conda activate my-environment
conda install opencv -c file:////software/PyTorch-1.6.0-Build/opencv-feedstock/condabuild
This should take about 2 minutes to install.

3. To test if your install is successful, issue python from the environment where Opencv is installed. Then run "import cv2" to see if there are any errors. For example:
[luy@ds-lg-01 ~]$ conda activate my-environment
(my-environment) [luy@ds-lg-01 ~]$ python
Python 3.6.12 |Anaconda, Inc.| (default, Sep 9 2020, 00:40:10)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>>

==4. Install other dependencies ==

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the <code>scikit-learn</code> package in your python3 environment.

First you need to activate the environment:
conda activate py36

Then you install the package
conda install scikit-learn

A list of recommended packages follows in the next section.

==5. Recommended packages ==

===5.1 Jupyter Notebooks for deep learning ===
conda install jupyter

===5.2 (Old Method) Testing Deep Learning packages on the login nodes or non-GPU nodes ===

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:
source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

==6. Compiling Software for DeepSense ==

DeepSense uses IBM Power8 systems running RedHat Enterprise Linux. Code must be compiled for <code>ppc64le</code> which is PowerPC 64 bit Little Endian.

Some software may not have binaries available for <code>ppc64le</code> even if it does for other systems. If this happens then you (or [[Contact_Information|DeepSense support]]) will need to compile the software to run on DeepSense. Visit the web page for the software and see if the source code is available (e.g. through github). If so then follow the compilation instructions to run the software.

You may encounter errors when attempting to compile software for <code>ppc64le</code>. Often this occurs because of differences between <code>ppc64le</code> and other common architectures such as x86 and x86_64.

For example, one DeepSense user attempted to compile the rdkit software package from https://www.rdkit.org/ . This compilation failed when it attempted to use the gcc x86 optimization <code>-mpopcnt</code>. After replacing the optimization with the <code>ppc64le</code> equivalent <code>-mpopcntb</code> the software compiled successfully.

== 7. Technical and research support ==

DeepSense has a dedicated support team of research scientists ready to help you with technical questions, installing software, or even research questions.

If you can't find the answer to your question on this wiki or need more extensive help then send an email to support@deepsense.ca .

See [[Technical support]] for more information about the support available.

</div>

Installing Software

2020-12-22T13:51:19Z

Lyang: /* 3.1 Using IBM-AI Deep Learning Anaconda Channel */

<div class="noautonum">

== 1. Logging on ==

DeepSense has two login nodes, login1.deepsense.ca and login2.deepsense.ca . You can access these through SSH with your username and password from any computer on campus.

For example, if your userid is <code>user1</code>, you can connect to deepsense by typing <code>ssh user1@login1.deepsense.ca</code> just like logging on to any other network computer.

'''Note''': The login nodes are intended for testing and compiling code. Please don’t run long or intensive computation on these nodes. Keep reading for instructions on how to submit compute jobs to dedicated compute nodes.

=== 1.1 VPN ===

To connect to the DeepSense platform from outside of the Dalhousie Campus, you'll need to use a VPN.
If you are are student, staff or faculty, you can use the Dalhousie VPN (https://wireless.dal.ca/vpnsoftware.php).

If you are not a Dalhousie staff, student, or faculty but require offsite access and cannot use the Dalhousie VPN then contact your project leader or ([mailto:support@deepsense.ca support@deepsense.ca]) to make different arrangements.

For more info, see [[VPN Setup]].

== 2. Configure your environment ==

DeepSense compute and management nodes are IBM Power8 computers (ppc64le) running Redhat Enterprise Linux. See [[Resources]] for more details on the available nodes.

=== 2.1 Loading a python environment ===

You have two options for using python on DeepSense. You can use the systemwide python install, managed by DeepSense administrators. This is recommended for users new to Linux. You will need to contact DeepSense support to have additional software packages installed in the systemwide python.

Alternatively, you can install an Anaconda python environment or other software in your home directory. This allows you to install or update packages or software without requesting and waiting for DeepSense staff.

==== Systemwide python (managed by DeepSense) ====

DeepSense has two Anaconda python environments. Anaconda 2 is installed on each compute node. While Anaconda 3 is installed in a shared directory that can be accessed from any machines in the cluster.

First one is anaconda2 installed in /opt/anaconda2 will provide you python 2.7.5. To use this systemwide python add a parameter to your .bashrc file in your home directory:

echo ". /opt/anaconda2/etc/profile.d/conda.sh" >> ~/.bashrc

Second is anaconda3 installed in /software/WMLA/anaconda3 will provide you python 3.7.4. To use this systemwide python add a parameter to your .bashrc file in your home directory:

echo ". /software/WMLA/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc

Then source your .bashrc file:
source ~/.bashrc

To load the python environment run <code>conda activate</code>

You can add either line to your .bashrc file to automatically load the desired environment when you log in.

==== Local python install (managed by individual user) ====

You are welcome to install software locally in your home directory. This allows you to use specific versions of software instead of the cluster wide versions. For example you may need an older version of a specific package or a newly released version that isn't yet installed on DeepSense.

For assistance installing or compiling software contact [[Contact_Information|Technical Support]]. We will support locally installed software to the best of our ability, although we can not guarantee that all software will run on the DeepSense platform. In the event that desired software will not run, we can help you determine alternatives such as different software or using a different system for some of your computation. If you attempt to install compiled software (e.g. an anaconda package) but the package cannot be found then also contact [[Contact_Information|Technical Support]]. The package may not have been compiled for the DeepSense hardware architecture (ppc64le). If your project has specific software you want to share between members then we can create a shared directory for your group in /software/<project>

If you have locally compiled software that you think may be useful for other DeepSense users then let us know at [[Contact_Information|Technical Support]]. We may install and support it systemwide if there is sufficient interest.

'''Stop using systemwide anaconda'''

If you added the system anaconda environment to your <code>.bashrc</code> file then remove the line:
. /opt/anaconda2/etc/profile.d/conda.sh

'''Installing Anaconda with a python3 base '''

From your home directory run:
wget https://repo.continuum.io/archive/Anaconda3-5.2.0-Linux-ppc64le.sh
bash Anaconda3-5.2.0-Linux-ppc64le.sh

Note: please enter "yes" when asked if you want to add anaconda to your .bashrc file. If you do not then you will need to add the following command to your .bashrc file or run it each time before using anaconda:
. ~/anaconda3/etc/profile.d/conda.sh

After the installer ends you need to either close and restart your terminal or run:
source ~/.bashrc

'''Adding a python2 environment'''

The previous instruction creates a python3 base environment. To add a python2 environment:
conda create -n py27 python=2.7

Activate this environment to use python3:
conda activate py27

note: if you receive an error message then you may need to deactivate the base conda environment first:
conda deactivate
conda activate py27

'''Adding a python3 environment'''

We recommend creating a separate python3 environment from the base environment. This makes it easier to install the specific packages required for IBM WMLA/PowerAI.
conda create -n py36 python=3.6

Activate this environment to use python3:
conda activate py36

==3. Installation of Deep Learning packages==

===3.1 Using IBM-AI Deep Learning Anaconda Channel===

To use deep learning packages like Tensorflow on DeepSense you need to add the IBM-AI anaconda channel to your list of available software channels.
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/

We suggest creating a new environment for each deep learning package you want to use. For example for Tensorflow:
conda create -n py36_tensorflow python=3.6
conda activate py36_tensorflow

Then install the anaconda package for the software you need. Again, with Tensorflow as an example:
conda install tensorflow

You can then use tensorflow or other deep learning packages as needed by simply activating that anaconda environment. Unlike the old method, you do not need to specifically activate tensorflow or other deep learning methods.

The above channel is still working, but the supported software version will not be updated any more. For example, you can only install TensorFlow 2.1.2 and PyTorch 1.3.1 using this channel.You can directly visit the IBM-AI anaconda channel URL to see a list of available software (https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/). If you would like to install higher versions of software, you can refer to the next section.

===3.2 Install PyTorch 1.6.0 in a user's home directory on DeepSense ===

A DeepSense user can install PyTorch by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build PyTorch with those versions. 
Here are the steps that a normal DeepSense user install PyTorch 1.6.0 in his/her home directory.

1. Source the conda environment you would like to use. For example:

source ~/anaconda3/etc/profile.d/conda.sh

2. Activate the environment you would use to install PyTorch. If the environment hasn't been created, a user can create it and install PyTorch in one command line. For example, if you would create an environment with name "my-environment" (This is just an example. Please choose a meaningful name for yourself.) and install PyTorch, you would run the following command:

conda create -y -n my-environment python=3.6 pytorch -c file:////software/PyTorch-1.6.0-Build/condabuild

3. If the environment has been created, say the name of the environment is "my-environment", you would need to activate the environment first and then install PyTorch. For example:

conda activate my-environment
conda install pytorch -c file:////software/PyTorch-1.6.0-Build/condabuild

This should take about 2 minutes to install.

4. To test if your install is successful, issue python from the environment where PyTorch is installed. Then run "import torch" to see if there are any errors. For example:
[luy@ds-lg-01 ~]$ conda activate my-environment
(my-environment) [luy@ds-lg-01 ~]$ python
Python 3.6.12 |Anaconda, Inc.| (default, Sep 9 2020, 00:40:10) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> 

===3.3 Install Opencv 3.4.10 in a user's home directory on DeepSense ===
A DeepSense user can install Opencv by him/herself in his/her home directory using the already built packages in /sofware/PyTorch-1.6.0-Build/opencv-feedstock. The current build only works with Python 3.6. So, a user needs to create a conda environment with Python 3.6. If a user would like to use higher versions of Python, they would need to ask DeepSense team to build Opencv with those versions. 
Here are the steps that a normal DeepSense user installs Opencv 3.4.10 in his/her home directory.
1. Source the conda environment you would like to use. For example:
source ~/anaconda3/etc/profile.d/conda.sh

2. Activate the environment you would use to install Opencv. If the environment hasn't been created, a user should create one. Assume a user created an environment "my-environment" and activated it. To install Opencv, you would run the following command:
conda activate my-environment
conda install opencv -c file:////software/PyTorch-1.6.0-Build/opencv-feedstock/condabuild
This should take about 2 minutes to install.

3. To test if your install is successful, issue python from the environment where Opencv is installed. Then run "import cv2" to see if there are any errors. For example:
[luy@ds-lg-01 ~]$ conda activate my-environment
(my-environment) [luy@ds-lg-01 ~]$ python
Python 3.6.12 |Anaconda, Inc.| (default, Sep 9 2020, 00:40:10)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>>

==4. Install other dependencies ==

If you need additional python libraries then you can install them in your python environment.

The base package comes with several python libraries but you may want a newer version or additional libraries. Also, when you create a new environment it does not automatically get all of the same libraries as the base environment.

For example, suppose you want to install the <code>scikit-learn</code> package in your python3 environment.

First you need to activate the environment:
conda activate py36

Then you install the package
conda install scikit-learn

A list of recommended packages follows in the next section.

==5. Recommended packages ==

===5.1 Jupyter Notebooks for deep learning ===
conda install jupyter

===5.2 (Old Method) Testing Deep Learning packages on the login nodes or non-GPU nodes ===

You may wish to run PowerAI software on the login nodes for testing on the CPU-only nodes for some workflows.

Only the GPU nodes have graphics cards and graphics drivers installed. If you attempt to run the deep learning software like Tensorflow on the login nodes or CPU-only nodes then you will see errors like the following:
ImportError: libcublas.so.9.2: cannot open shared object file: No such file or directory

You need to load the GPU drivers with the following command:
source /opt/DL/cudnn/bin/cudnn-activate

Then you can activate the deep learning package, e.g. for Tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

Note that some deep learning software may be much slower or refuse to run without GPU access. Tensorflow works but Caffe does not.

Keep in mind you need to activate the GPU drivers and deep learning package in each browser shell before you are able to use the package in your code or LSF jobs.

==6. Compiling Software for DeepSense ==

DeepSense uses IBM Power8 systems running RedHat Enterprise Linux. Code must be compiled for <code>ppc64le</code> which is PowerPC 64 bit Little Endian.

Some software may not have binaries available for <code>ppc64le</code> even if it does for other systems. If this happens then you (or [[Contact_Information|DeepSense support]]) will need to compile the software to run on DeepSense. Visit the web page for the software and see if the source code is available (e.g. through github). If so then follow the compilation instructions to run the software.

You may encounter errors when attempting to compile software for <code>ppc64le</code>. Often this occurs because of differences between <code>ppc64le</code> and other common architectures such as x86 and x86_64.

For example, one DeepSense user attempted to compile the rdkit software package from https://www.rdkit.org/ . This compilation failed when it attempted to use the gcc x86 optimization <code>-mpopcnt</code>. After replacing the optimization with the <code>ppc64le</code> equivalent <code>-mpopcntb</code> the software compiled successfully.

== 7. Technical and research support ==

DeepSense has a dedicated support team of research scientists ready to help you with technical questions, installing software, or even research questions.

If you can't find the answer to your question on this wiki or need more extensive help then send an email to support@deepsense.ca .

See [[Technical support]] for more information about the support available.

</div>

Available software

2020-12-22T13:34:08Z

Lyang: /* IBM WMLA Deep Learning Packages */

== Basic Software ==

* RedHat Enterprise Linux Server release 7.7 (RHEL)
* gcc 4.8.5
* glibc 2.17
* R 3.6.0

== Anaconda Python ==

DeepSense has two Anaconda python environments. Anaconda 2 is installed on each compute node. While Anaconda 3 is installed in a shared directory that can be accessed from any machines in the cluster.
:
{| class="wikitable"
! Version
! Environment location
|-
|python 2.7.15
|/opt/anaconda2
|-
|python 3.7.4
|/software/WMLA/anaconda3/
|}

These python environments have many packages installed, including prerequisite libraries for running the IBM PowerAI deep learning frameworks.

See [[Getting_started]] for instructions on using the shared anaconda python environments.

See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory.

==IBM-AI Deep Learning Anaconda Channel==

To use deep learning packages like Tensorflow, pytorch, caffe on DeepSense you need to add the IBM-AI anaconda channel to your list of available software channels using below command.
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
The above channel is still working, but the supported software version will not be updated any more. For example, you can only install TensorFlow 2.1.2 and PyTorch 1.3.1 using this channel. If you would like to install higher versions of software, please refer to [https://docs.deepsense.ca/index.php?title=Installing_Software#3._Installation_of_Deep_Learning_packages| Installing Software].

== IBM WMLA Deep Learning Packages ==

[https://developer.ibm.com/linuxonpower/deep-learning-powerai/ IBM PowerAI] includes multiple open source deep learning frameworks compiled for IBM Power8 systems.

IBM WMLA Enterprise includes:

{| class="wikitable"
!Framework
!colspan="2"|Location
|-
|Caffe
|/opt/DL/caffe
|
|-
|cuDNN
|/opt/DL/cudnn
|
|-
|IBM Distributed Deep Learning (DDL)
|/opt/DL/ddl
|
|-
| HDF5
|/opt/DL/hdf5
|
|-
|NCCL
|/opt/DL/nccl
|/opt/DL/nccl2
|-
|openblas
|/opt/DL/openblas
|
|-
|protobuf
|/opt/DL/protobuf
|
|-
|pytorch
|/opt/DL/pytorch
|
|-
|snap-ml
|/opt/DL/snap-ml-local
|/opt/DL/snap-ml-mpi
|-
|Tensorflow 1.11 (including keras)
|/opt/DL/tensorflow
|/opt/DL/ddl-tensorflow
|-
|Tensorboard
|/opt/DL/tensorboard
|
|}

To use most of these frameworks you need to activate a python2 or python3 environment and then activate the relevant framework.

For example, to use tensorflow you can activate a python2 environment:
. /opt/anaconda2/etc/profile.d/conda.sh
conda activate

and then activate tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

You can then <code>import tensorflow as tf</code> in your python code.

See [[Getting started with Deep Learning]] for a tutorial on using Caffe and Tensorflow on Deep Sense.

== IBM Advance Toolchain ==

You may require newer versions of compilers such as GCC than are provided with RHEL.

The [https://developer.ibm.com/linuxonpower/advance-toolchain IBM Advance Toolchain for Linux on Power] is a set of open source compilers, runtime libraries, and development tools.

The IBM Advance Toolchain] includes recent versions of:
* GNU Compiler Collection (gcc, g++ and gfortran)
* GNU C library (glibc)
* GNU Binary Utilities (binutils)
* Decimal Floating Point Library (libdfp)
* IBM Power Architecture Facilities Library (PAFLib)
* GNU Debugger (gdb)
* Python
* Golang
* Performance analysis tools (oprofile, valgrind, itrace)
* Multi-core exploitation libraries (TBB, Userspace RCU, SPHDE)
* support libraries (libhugetlbfs, Boost, zlib, etc)

To use the the Advance Toolchain, first activate environment modules:
source /usr/local/Modules/init/bash

Then load the advance toolchain:
module load at12.0

To stop using the advance toolchain, unload the environment module:
module unload at12.0

Note that software dynamically compiled with the advance toolchain will only run with the advance toolchain loaded.

== Requested Software ==

Software packages that are requested for use by DeepSense projects will be available in several locations. Our preference is to use conda packages when available.

=== External conda channels ===
If a requested software is available for ppc64le systems from an externally maintained anaconda channel then we will simply list the channel. You can install such software into a local anaconda environment using:
conda install -c <channel> <software>

=== Internal conda packages ===
When possible, software compiled by DeepSense staff will compiled using conda build and placed in a subdirectory of <code>/software/conda-bld/</code> . You can install such software into a local anaconda environment using:
conda install -c file://software/conda-bld/ <software>

=== Shared software ===
Some software will simply be installed in its own subdirectory of <code>/software</code>. You can run this software directly from its subdirectory.

=== Bioinformatics Software ===

{| class="wikitable"
!Software
!Version
!Location
|-
|trimmomatic
|0.39
|/software/trimmomattic-0.39
|-
|cutadapt
|2.3
|/software/conda-bld/linux-ppc64le/
|-
|bowtie2
|
|biobuilds channel
|-
|LAST
|980
|/software/last-980
|-
|Burrows wheeler aligner
|0.7.15
|/software/bwa
|-
|pb-falcon
|2.2.0
|/software/conda-bld/linux-ppc64le/
|-
|MASURCA
|3.3.4
|/software/conda-bld/linux-ppc64le/
|-
|Samtools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|htslib
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|bcftools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|gatk
|4.1.2.0
|/software/conda-bld/noarch/
|-
|stacks
|2.4
|/software/conda-bld/linux-ppc64le/
|-
|angsd
|0.923
|/software/conda-bld/linux-ppc64le/
|-
|vcftools
|
|biobuilds channel
|-
|plink
|
|biobuilds channel
|-
|msprime
|0.7.0
|/software/conda-bld/linux-ppc64le/
|-
|slim
|3.3
|/software/slim-3.3
|-
|DeepGSR
|
|/software/DeepGSR
|}

== Requesting Additional Software ==

Contact DeepSense [[contact information|support]] to have additional software installed or for help installing or compiling software locally in your home directory.

Available software

2020-12-22T13:27:51Z

Lyang: /* IBM-AI Deep Learning Anaconda Channel */

== Basic Software ==

* RedHat Enterprise Linux Server release 7.7 (RHEL)
* gcc 4.8.5
* glibc 2.17
* R 3.6.0

== Anaconda Python ==

DeepSense has two Anaconda python environments. Anaconda 2 is installed on each compute node. While Anaconda 3 is installed in a shared directory that can be accessed from any machines in the cluster.
:
{| class="wikitable"
! Version
! Environment location
|-
|python 2.7.15
|/opt/anaconda2
|-
|python 3.7.4
|/software/WMLA/anaconda3/
|}

These python environments have many packages installed, including prerequisite libraries for running the IBM PowerAI deep learning frameworks.

See [[Getting_started]] for instructions on using the shared anaconda python environments.

See [[Installing local software]] for instructions on installing and managing your own python environments in your home directory.

==IBM-AI Deep Learning Anaconda Channel==

To use deep learning packages like Tensorflow, pytorch, caffe on DeepSense you need to add the IBM-AI anaconda channel to your list of available software channels using below command.
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
The above channel is still working, but the supported software version will not be updated any more. For example, you can only install TensorFlow 2.1.2 and PyTorch 1.3.1 using this channel. If you would like to install higher versions of software, please refer to [https://docs.deepsense.ca/index.php?title=Installing_Software#3._Installation_of_Deep_Learning_packages| Installing Software].

== IBM WMLA Deep Learning Packages ==

[https://developer.ibm.com/linuxonpower/deep-learning-powerai/ IBM PowerAI] includes multiple open source deep learning frameworks compiled for IBM Power8 systems.

IBM WMLA Enterprise includes:

{| class="wikitable"
!Framework
!colspan="2"|Location
|-
|Caffe
|/opt/DL/caffe
|-
|cuDNN
|/opt/DL/cudnn
|-
|IBM Distributed Deep Learning (DDL)
|/opt/DL/ddl
|-
| HDF5
|/opt/DL/hdf5
|-
|NCCL
|/opt/DL/nccl
|/opt/DL/nccl2
|-
|openblas
|/opt/DL/openblas
|-
|protobuf
|/opt/DL/protobuf
|-
|pytorch
|/opt/DL/pytorch
|-
|snap-ml
|/opt/DL/snap-ml-local
|/opt/DL/snap-ml-mpi
|-
|Tensorflow 1.11 (including keras)
|/opt/DL/tensorflow
|/opt/DL/ddl-tensorflow
|-
|Tensorboard
|/opt/DL/tensorboard
|}

To use most of these frameworks you need to activate a python2 or python3 environment and then activate the relevant framework.

For example, to use tensorflow you can activate a python2 environment:
. /opt/anaconda2/etc/profile.d/conda.sh
conda activate

and then activate tensorflow:
source /opt/DL/tensorflow/bin/tensorflow-activate

You can then <code>import tensorflow as tf</code> in your python code.

See [[Getting started with Deep Learning]] for a tutorial on using Caffe and Tensorflow on Deep Sense.

== IBM Advance Toolchain ==

You may require newer versions of compilers such as GCC than are provided with RHEL.

The [https://developer.ibm.com/linuxonpower/advance-toolchain IBM Advance Toolchain for Linux on Power] is a set of open source compilers, runtime libraries, and development tools.

The IBM Advance Toolchain] includes recent versions of:
* GNU Compiler Collection (gcc, g++ and gfortran)
* GNU C library (glibc)
* GNU Binary Utilities (binutils)
* Decimal Floating Point Library (libdfp)
* IBM Power Architecture Facilities Library (PAFLib)
* GNU Debugger (gdb)
* Python
* Golang
* Performance analysis tools (oprofile, valgrind, itrace)
* Multi-core exploitation libraries (TBB, Userspace RCU, SPHDE)
* support libraries (libhugetlbfs, Boost, zlib, etc)

To use the the Advance Toolchain, first activate environment modules:
source /usr/local/Modules/init/bash

Then load the advance toolchain:
module load at12.0

To stop using the advance toolchain, unload the environment module:
module unload at12.0

Note that software dynamically compiled with the advance toolchain will only run with the advance toolchain loaded.

== Requested Software ==

Software packages that are requested for use by DeepSense projects will be available in several locations. Our preference is to use conda packages when available.

=== External conda channels ===
If a requested software is available for ppc64le systems from an externally maintained anaconda channel then we will simply list the channel. You can install such software into a local anaconda environment using:
conda install -c <channel> <software>

=== Internal conda packages ===
When possible, software compiled by DeepSense staff will compiled using conda build and placed in a subdirectory of <code>/software/conda-bld/</code> . You can install such software into a local anaconda environment using:
conda install -c file://software/conda-bld/ <software>

=== Shared software ===
Some software will simply be installed in its own subdirectory of <code>/software</code>. You can run this software directly from its subdirectory.

=== Bioinformatics Software ===

{| class="wikitable"
!Software
!Version
!Location
|-
|trimmomatic
|0.39
|/software/trimmomattic-0.39
|-
|cutadapt
|2.3
|/software/conda-bld/linux-ppc64le/
|-
|bowtie2
|
|biobuilds channel
|-
|LAST
|980
|/software/last-980
|-
|Burrows wheeler aligner
|0.7.15
|/software/bwa
|-
|pb-falcon
|2.2.0
|/software/conda-bld/linux-ppc64le/
|-
|MASURCA
|3.3.4
|/software/conda-bld/linux-ppc64le/
|-
|Samtools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|htslib
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|bcftools
|1.9
|/software/conda-bld/linux-ppc64le/
|-
|gatk
|4.1.2.0
|/software/conda-bld/noarch/
|-
|stacks
|2.4
|/software/conda-bld/linux-ppc64le/
|-
|angsd
|0.923
|/software/conda-bld/linux-ppc64le/
|-
|vcftools
|
|biobuilds channel
|-
|plink
|
|biobuilds channel
|-
|msprime
|0.7.0
|/software/conda-bld/linux-ppc64le/
|-
|slim
|3.3
|/software/slim-3.3
|-
|DeepGSR
|
|/software/DeepGSR
|}

== Requesting Additional Software ==

Contact DeepSense [[contact information|support]] to have additional software installed or for help installing or compiling software locally in your home directory.

Visualization

2020-12-18T20:57:26Z

Lyang: /* Using Dash on DeepSense */

In this section, we would not introduce those popular plotting libraries like Matplotlib, Pandas, Seaborn, ggplot, Plotly, and so on. We would try to introduce and provide information about some visualization frameworks and libraries that have more features. Using the features in these frameworks and libraries, you may make your data more understandable and interactive.

== Dash ==
=== Introduction ===
We would make use of the introduction of Dash directly from [https://dash.plotly.com/introduction| Dash's official webpage]. Users can obtain more details about Dash accessing the official webpage. 
"Dash is a productive Python framework for building web analytic applications. 
Written on top of Flask, Plotly.js, and React.js, Dash is ideal for building data visualization apps with highly custom user interfaces in pure Python. It's particularly suited for anyone who works with data in Python. 
Through a couple of simple patterns, Dash abstracts away all of the technologies and protocols that are required to build an interactive web-based application. Dash is simple enough that you can bind a user interface around your Python code in an afternoon. 
Dash apps are rendered in the web browser. You can deploy your apps to servers and then share them through URLs. Since Dash apps are viewed in the web browser, Dash is inherently cross-platform and mobile ready. 
There is a lot behind the framework. To learn more about how it is built and what motivated Dash, watch our talk from Plotcon below or read our announcement letter. 
Dash is an open source library, released under the permissive MIT license. Plotly develops Dash and offers a platform for managing Dash apps in an enterprise environment." 

Users can easily install Dash in their home directory on DeepSense. The way to install it is very similar to installing a package using conda. Here we provide how to install and use Dash on DeepSense. The install process only takes about 10 minutes if you have already set up your conda environments in your home directory on DeepSense.

=== Installing Dash on DeepSense ===
Activate the conda environment you would install Dash 
Example:
[luy@ds-lg-01 ~]$ conda activate py36-pytorch
Running the following command to install Dash:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda install -c conda-forge dash
After the installation is done, issue the following command to check if Dash is installed:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda list |grep dash
dash 1.18.1 pyhd8ed1ab_0 conda-forge
dash-core-components 1.14.1 pyhd8ed1ab_0 conda-forge
dash-html-components 1.1.1 pyh9f0ad1d_0 conda-forge
dash-renderer 1.8.3 pyhd8ed1ab_0 conda-forge
dash-table 4.11.1 pyhd8ed1ab_0 conda-forge

=== Learning Dash ===
Users can learn Dash on its [https://dash.plotly.com/basic-callbacks| official webpage]. There are also a lot of Dash tutorials on youtube. You would be able to see how powerful Dash can do for visualizing your data.

=== Using Dash on DeepSense ===
1. Creating a sample Python script using Dash 
Now you can develop your projects using Dash. Below is a simple example script showing how Dash works on DeepSense platform. Users can copy and paste the contents of the script and create a Python script in your home directory on DeepSense. Let's name it 'dashTest.py'.
import torch
import torchvision
from platform import python_version
import dash
import dash_html_components as html
import subprocess
app = dash.Dash(__name__)
gpu_detail=str(subprocess.run(['nvidia-smi'], stdout=subprocess.PIPE).stdout.decode('utf-8'))
gpu_name=' '.join(gpu_detail.split('\n')[7].split()[2:4])
content=[
html.H6(children='Hello world!'),
html.H6(children="Python Version: " + str(python_version())),
html.H6(children="PyTorch Version: " + str(torch.__version__)),
html.H6(children="Torchvision Version: " + str(torchvision.__version__)),
html.H6(children="GPU name: " + gpu_name)]
app.layout = html.Div(children=content)
if __name__ == '__main__':
app.run_server(host='0.0.0.0')

2. Getting an interactive session via LSF 
If you will be using cpus for your jobs, please run the following command to submit a new LSF interactive job:
(py36-pytorch) [luy@ds-lg-01 ~]$ bsub -Is bash
Above is just an example command. We would use the command for gpu jobs to show more details of the command.
If you will be using gpus for your jobs, please run the following command to submit a new LSF interactive job:
(py36-pytorch) [luy@ds-lg-01 ~]$ bsub -Is -gpu - bash
Job <6008> is submitted to queue <gpu>.
<<Waiting for dispatch ...>>
<<Starting on ds-cmgpu-01>>
[luy@ds-cmgpu-01 ~]$
Now you have an interactive session opened on the compute host ds-cmgpu-01. You can run the sample script on the command line:
[luy@ds-cmgpu-01 ~]$ python dashTest.py
Dash is running on http://0.0.0.0:8050/
* Serving Flask app "dashTest" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:8050/ (Press CTRL+C to quit)

3. See the output on a webpage 
Keep the session open. This means you should not press CTRL+C. You would need to do port forwarding to open the webpage on the web browser on your local machine. This is very similar to how to use notebook on DeepSense. Below is an example: 
Lus-MacBook-Pro:~ lyang$ ssh -l luy login1.deepsense.ca -L 8050:ds-cmgpu-01:8050
luy@login1.deepsense.ca's password:
Last login: Fri Dec 18 08:41:15 2020 from vpn.deepsense.cs.dal.ca
**********************************************************************
IMPORTANT: Dear DeepSense users, jobs running on the head nodes (login
node 1 and login node 2) must be less than 10 minutes and have small
memory and CPU requirements. Jobs violating this will be terminated
without warning.
**********************************************************************
[luy@ds-lg-01 ~]$
Be careful that we obtain the compute host name 'ds-cmgpu-01' and port number '8050' from step 2. The host name and port number may change when you submit your jobs. After the port forwarding is finished, you can copy the url 'http://0.0.0.0:8050/' found at the last line of step 2 and paste it to your local web browser. The above example script would output the following contents on a HTML web page:<br.
Hello world!

Python Version: 3.6.12

PyTorch Version: 1.6.0a0+445c276

Torchvision Version: 0.7.0a0

GPU name: Tesla P100-SXM2...

Please feel free to contact DeepSense at support@deepsense.ca for any questions.

Visualization

2020-12-18T15:17:44Z

Lyang: /* Installing Dash */

In this section, we would not introduce those popular plotting libraries like Matplotlib, Pandas, Seaborn, ggplot, Plotly, and so on. We would try to introduce and provide information about some visualization frameworks and libraries that have more features. Using the features in these frameworks and libraries, you may make your data more understandable and interactive.

== Dash ==
=== Introduction ===
We would make use of the introduction of Dash directly from [https://dash.plotly.com/introduction| Dash's official webpage]. Users can obtain more details about Dash accessing the official webpage. 
"Dash is a productive Python framework for building web analytic applications. 
Written on top of Flask, Plotly.js, and React.js, Dash is ideal for building data visualization apps with highly custom user interfaces in pure Python. It's particularly suited for anyone who works with data in Python. 
Through a couple of simple patterns, Dash abstracts away all of the technologies and protocols that are required to build an interactive web-based application. Dash is simple enough that you can bind a user interface around your Python code in an afternoon. 
Dash apps are rendered in the web browser. You can deploy your apps to servers and then share them through URLs. Since Dash apps are viewed in the web browser, Dash is inherently cross-platform and mobile ready. 
There is a lot behind the framework. To learn more about how it is built and what motivated Dash, watch our talk from Plotcon below or read our announcement letter. 
Dash is an open source library, released under the permissive MIT license. Plotly develops Dash and offers a platform for managing Dash apps in an enterprise environment." 

Users can easily install Dash in their home directory on DeepSense. The way to install it is very similar to installing a package using conda. Here we provide how to install and use Dash on DeepSense. The install process only takes about 10 minutes if you have already set up your conda environments in your home directory on DeepSense.

=== Installing Dash on DeepSense ===
Activate the conda environment you would install Dash 
Example:
[luy@ds-lg-01 ~]$ conda activate py36-pytorch
Running the following command to install Dash:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda install -c conda-forge dash
After the installation is done, issue the following command to check if Dash is installed:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda list |grep dash
dash 1.18.1 pyhd8ed1ab_0 conda-forge
dash-core-components 1.14.1 pyhd8ed1ab_0 conda-forge
dash-html-components 1.1.1 pyh9f0ad1d_0 conda-forge
dash-renderer 1.8.3 pyhd8ed1ab_0 conda-forge
dash-table 4.11.1 pyhd8ed1ab_0 conda-forge

=== Learning Dash ===
Users can learn Dash on its [https://dash.plotly.com/basic-callbacks| official webpage]. There are also a lot of Dash tutorials on youtube. You would be able to see how powerful Dash can do for visualizing your data.

=== Using Dash on DeepSense ===
1. Creating a sample Python script using Dash 
Now you can develop your projects using Dash. Below is a simple example script showing how Dash works on DeepSense platform. Users can copy and paste the contents of the script and create a Python script in your home directory on DeepSense. Let's name it 'dashTest.py'.
import torch
import torchvision
from platform import python_version
import dash
import dash_html_components as html
import subprocess
app = dash.Dash(__name__)
gpu_detail=str(subprocess.run(['nvidia-smi'], stdout=subprocess.PIPE).stdout.decode('utf-8'))
gpu_name=' '.join(gpu_detail.split('\n')[7].split()[2:4])
content=[
html.H6(children='Hello world!'),
html.H6(children="Python Version: " + str(python_version())),
html.H6(children="PyTorch Version: " + str(torch.__version__)),
html.H6(children="Torchvision Version: " + str(torchvision.__version__)),
html.H6(children="GPU name: " + gpu_name)]
app.layout = html.Div(children=content)
if __name__ == '__main__':
app.run_server(host='0.0.0.0')

2. Getting an interactive session via LSF 
Running the following command to submit a new LSF interactive job:
(py36-pytorch) [luy@ds-lg-01 ~]$ bsub -Is -gpu - bash
Job <6008> is submitted to queue <gpu>.
<<Waiting for dispatch ...>>
<<Starting on ds-cmgpu-01>>
[luy@ds-cmgpu-01 ~]$
Now you have an interactive session opened on the compute host ds-cmgpu-01. You can run the sample script on the command line:
[luy@ds-cmgpu-01 ~]$ python dashTest.py
Dash is running on http://0.0.0.0:8050/
* Serving Flask app "dashTest" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:8050/ (Press CTRL+C to quit)

3. See the output on a webpage 
Keep the session open. This means you should not press CTRL+C. You would need to do port forwarding to open the webpage on the web browser on your local machine. This is very similar to how to use notebook on DeepSense. Below is an example: 
Lus-MacBook-Pro:~ lyang$ ssh -l luy login1.deepsense.ca -L 8050:ds-cmgpu-01:8050
luy@login1.deepsense.ca's password:
Last login: Fri Dec 18 08:41:15 2020 from vpn.deepsense.cs.dal.ca
**********************************************************************
IMPORTANT: Dear DeepSense users, jobs running on the head nodes (login
node 1 and login node 2) must be less than 10 minutes and have small
memory and CPU requirements. Jobs violating this will be terminated
without warning.
**********************************************************************
[luy@ds-lg-01 ~]$
Be careful that we obtain the compute host name 'ds-cmgpu-01' and port number '8050' from step 2. The host name and port number may change when you submit your jobs. After the port forwarding is finished, you can copy the url 'http://0.0.0.0:8050/' found at the last line of step 2 and paste it to your local web browser. The above example script would output the following contents on a HTML web page:<br.
Hello world!

Python Version: 3.6.12

PyTorch Version: 1.6.0a0+445c276

Torchvision Version: 0.7.0a0

GPU name: Tesla P100-SXM2...

Please feel free to contact DeepSense at support@deepsense.ca for any questions.

Visualization

2020-12-18T15:15:08Z

Lyang: /* Using Dash on DeepSense */

In this section, we would not introduce those popular plotting libraries like Matplotlib, Pandas, Seaborn, ggplot, Plotly, and so on. We would try to introduce and provide information about some visualization frameworks and libraries that have more features. Using the features in these frameworks and libraries, you may make your data more understandable and interactive.

== Dash ==
=== Introduction ===
We would make use of the introduction of Dash directly from [https://dash.plotly.com/introduction| Dash's official webpage]. Users can obtain more details about Dash accessing the official webpage. 
"Dash is a productive Python framework for building web analytic applications. 
Written on top of Flask, Plotly.js, and React.js, Dash is ideal for building data visualization apps with highly custom user interfaces in pure Python. It's particularly suited for anyone who works with data in Python. 
Through a couple of simple patterns, Dash abstracts away all of the technologies and protocols that are required to build an interactive web-based application. Dash is simple enough that you can bind a user interface around your Python code in an afternoon. 
Dash apps are rendered in the web browser. You can deploy your apps to servers and then share them through URLs. Since Dash apps are viewed in the web browser, Dash is inherently cross-platform and mobile ready. 
There is a lot behind the framework. To learn more about how it is built and what motivated Dash, watch our talk from Plotcon below or read our announcement letter. 
Dash is an open source library, released under the permissive MIT license. Plotly develops Dash and offers a platform for managing Dash apps in an enterprise environment." 

Users can easily install Dash in their home directory on DeepSense. The way to install it is very similar to installing a package using conda. Here we provide how to install and use Dash on DeepSense. The install process only takes about 10 minutes if you have already set up your conda environments in your home directory on DeepSense.

=== Installing Dash ===
Activate the conda environment you would install Dash 
Example:
[luy@ds-lg-01 ~]$ conda activate py36-pytorch
Running the following command to install Dash:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda install -c conda-forge dash
After the installation is done, issue the following command to check if Dash is installed:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda list |grep dash
dash 1.18.1 pyhd8ed1ab_0 conda-forge
dash-core-components 1.14.1 pyhd8ed1ab_0 conda-forge
dash-html-components 1.1.1 pyh9f0ad1d_0 conda-forge
dash-renderer 1.8.3 pyhd8ed1ab_0 conda-forge
dash-table 4.11.1 pyhd8ed1ab_0 conda-forge

=== Learning Dash ===
Users can learn Dash on its [https://dash.plotly.com/basic-callbacks| official webpage]. There are also a lot of Dash tutorials on youtube. You would be able to see how powerful Dash can do for visualizing your data.

=== Using Dash on DeepSense ===
1. Creating a sample Python script using Dash 
Now you can develop your projects using Dash. Below is a simple example script showing how Dash works on DeepSense platform. Users can copy and paste the contents of the script and create a Python script in your home directory on DeepSense. Let's name it 'dashTest.py'.
import torch
import torchvision
from platform import python_version
import dash
import dash_html_components as html
import subprocess
app = dash.Dash(__name__)
gpu_detail=str(subprocess.run(['nvidia-smi'], stdout=subprocess.PIPE).stdout.decode('utf-8'))
gpu_name=' '.join(gpu_detail.split('\n')[7].split()[2:4])
content=[
html.H6(children='Hello world!'),
html.H6(children="Python Version: " + str(python_version())),
html.H6(children="PyTorch Version: " + str(torch.__version__)),
html.H6(children="Torchvision Version: " + str(torchvision.__version__)),
html.H6(children="GPU name: " + gpu_name)]
app.layout = html.Div(children=content)
if __name__ == '__main__':
app.run_server(host='0.0.0.0')

2. Getting an interactive session via LSF 
Running the following command to submit a new LSF interactive job:
(py36-pytorch) [luy@ds-lg-01 ~]$ bsub -Is -gpu - bash
Job <6008> is submitted to queue <gpu>.
<<Waiting for dispatch ...>>
<<Starting on ds-cmgpu-01>>
[luy@ds-cmgpu-01 ~]$
Now you have an interactive session opened on the compute host ds-cmgpu-01. You can run the sample script on the command line:
[luy@ds-cmgpu-01 ~]$ python dashTest.py
Dash is running on http://0.0.0.0:8050/
* Serving Flask app "dashTest" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:8050/ (Press CTRL+C to quit)

3. See the output on a webpage 
Keep the session open. This means you should not press CTRL+C. You would need to do port forwarding to open the webpage on the web browser on your local machine. This is very similar to how to use notebook on DeepSense. Below is an example: 
Lus-MacBook-Pro:~ lyang$ ssh -l luy login1.deepsense.ca -L 8050:ds-cmgpu-01:8050
luy@login1.deepsense.ca's password:
Last login: Fri Dec 18 08:41:15 2020 from vpn.deepsense.cs.dal.ca
**********************************************************************
IMPORTANT: Dear DeepSense users, jobs running on the head nodes (login
node 1 and login node 2) must be less than 10 minutes and have small
memory and CPU requirements. Jobs violating this will be terminated
without warning.
**********************************************************************
[luy@ds-lg-01 ~]$
Be careful that we obtain the compute host name 'ds-cmgpu-01' and port number '8050' from step 2. The host name and port number may change when you submit your jobs. After the port forwarding is finished, you can copy the url 'http://0.0.0.0:8050/' found at the last line of step 2 and paste it to your local web browser. The above example script would output the following contents on a HTML web page:<br.
Hello world!

Python Version: 3.6.12

PyTorch Version: 1.6.0a0+445c276

Torchvision Version: 0.7.0a0

GPU name: Tesla P100-SXM2...

Please feel free to contact DeepSense at support@deepsense.ca for any questions.

Visualization

2020-12-18T15:09:27Z

Lyang: /* Learning Dash */

In this section, we would not introduce those popular plotting libraries like Matplotlib, Pandas, Seaborn, ggplot, Plotly, and so on. We would try to introduce and provide information about some visualization frameworks and libraries that have more features. Using the features in these frameworks and libraries, you may make your data more understandable and interactive.

== Dash ==
=== Introduction ===
We would make use of the introduction of Dash directly from [https://dash.plotly.com/introduction| Dash's official webpage]. Users can obtain more details about Dash accessing the official webpage. 
"Dash is a productive Python framework for building web analytic applications. 
Written on top of Flask, Plotly.js, and React.js, Dash is ideal for building data visualization apps with highly custom user interfaces in pure Python. It's particularly suited for anyone who works with data in Python. 
Through a couple of simple patterns, Dash abstracts away all of the technologies and protocols that are required to build an interactive web-based application. Dash is simple enough that you can bind a user interface around your Python code in an afternoon. 
Dash apps are rendered in the web browser. You can deploy your apps to servers and then share them through URLs. Since Dash apps are viewed in the web browser, Dash is inherently cross-platform and mobile ready. 
There is a lot behind the framework. To learn more about how it is built and what motivated Dash, watch our talk from Plotcon below or read our announcement letter. 
Dash is an open source library, released under the permissive MIT license. Plotly develops Dash and offers a platform for managing Dash apps in an enterprise environment." 

Users can easily install Dash in their home directory on DeepSense. The way to install it is very similar to installing a package using conda. Here we provide how to install and use Dash on DeepSense. The install process only takes about 10 minutes if you have already set up your conda environments in your home directory on DeepSense.

=== Installing Dash ===
Activate the conda environment you would install Dash 
Example:
[luy@ds-lg-01 ~]$ conda activate py36-pytorch
Running the following command to install Dash:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda install -c conda-forge dash
After the installation is done, issue the following command to check if Dash is installed:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda list |grep dash
dash 1.18.1 pyhd8ed1ab_0 conda-forge
dash-core-components 1.14.1 pyhd8ed1ab_0 conda-forge
dash-html-components 1.1.1 pyh9f0ad1d_0 conda-forge
dash-renderer 1.8.3 pyhd8ed1ab_0 conda-forge
dash-table 4.11.1 pyhd8ed1ab_0 conda-forge

=== Learning Dash ===
Users can learn Dash on its [https://dash.plotly.com/basic-callbacks| official webpage]. There are also a lot of Dash tutorials on youtube. You would be able to see how powerful Dash can do for visualizing your data.

=== Using Dash on DeepSense ===
1. Creating a sample Python script using Dash 
Now you can develop your projects using Dash. Below is simple example script showing how Dash works on DeepSense platform. Users can copy and paste the contents of the script and create a Python script in your home directory on DeepSense. Let's name it 'dashTest.py'.
import torch
import torchvision
from platform import python_version
import dash
import dash_html_components as html
import subprocess
app = dash.Dash(__name__)
gpu_detail=str(subprocess.run(['nvidia-smi'], stdout=subprocess.PIPE).stdout.decode('utf-8'))
gpu_name=' '.join(gpu_detail.split('\n')[7].split()[2:4])
content=[
html.H6(children='Hello world!'),
html.H6(children="Python Version: " + str(python_version())),
html.H6(children="PyTorch Version: " + str(torch.__version__)),
html.H6(children="Torchvision Version: " + str(torchvision.__version__)),
html.H6(children="GPU name: " + gpu_name)]
app.layout = html.Div(children=content)
if __name__ == '__main__':
app.run_server(host='0.0.0.0')

2. Getting an interactive session via LSF 
Running the following command to submit a new LSF interactive job:
(py36-pytorch) [luy@ds-lg-01 ~]$ bsub -Is -gpu - bash
Job <6008> is submitted to queue <gpu>.
<<Waiting for dispatch ...>>
<<Starting on ds-cmgpu-01>>
[luy@ds-cmgpu-01 ~]$
Now you have an interactive session opened on the compute host ds-cmgpu-01. You can run the sample script on the command line:
[luy@ds-cmgpu-01 ~]$ python dashTest.py
Dash is running on http://0.0.0.0:8050/
* Serving Flask app "dashTest" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:8050/ (Press CTRL+C to quit)

3. See the result on a webpage<br.
Keep the session open. This means you should not press CTRL+C. You would need to do port forwarding to open the webpage on the web browser on your local machine. This is very similar to how to use notebook on DeepSense. Below is an example: 
Lus-MacBook-Pro:~ lyang$ ssh -l luy login1.deepsense.ca -L 8050:ds-cmgpu-01:8050
luy@login1.deepsense.ca's password:
Last login: Fri Dec 18 08:41:15 2020 from vpn.deepsense.cs.dal.ca
**********************************************************************
IMPORTANT: Dear DeepSense users, jobs running on the head nodes (login
node 1 and login node 2) must be less than 10 minutes and have small
memory and CPU requirements. Jobs violating this will be terminated
without warning.
**********************************************************************
[luy@ds-lg-01 ~]$
Be careful that we obtain the compute host name 'ds-cmgpu-01' and port number '8050' from step 2. The host name and port number may change when your submit your jobs. After the port forwarding is finished, you can copy the url 'http://0.0.0.0:8050/' found at the last line of step 2 and paste it to your local web browser. The above example script would output the following contents on a HTML web page:<br.
Hello world!

Python Version: 3.6.12

PyTorch Version: 1.6.0a0+445c276

Torchvision Version: 0.7.0a0

GPU name: Tesla P100-SXM2...

Please feel free to contact DeepSense at support@deepsense.ca for any questions.

Visualization

2020-12-18T15:08:46Z

Lyang: /* Installing Dash */

In this section, we would not introduce those popular plotting libraries like Matplotlib, Pandas, Seaborn, ggplot, Plotly, and so on. We would try to introduce and provide information about some visualization frameworks and libraries that have more features. Using the features in these frameworks and libraries, you may make your data more understandable and interactive.

== Dash ==
=== Introduction ===
We would make use of the introduction of Dash directly from [https://dash.plotly.com/introduction| Dash's official webpage]. Users can obtain more details about Dash accessing the official webpage. 
"Dash is a productive Python framework for building web analytic applications. 
Written on top of Flask, Plotly.js, and React.js, Dash is ideal for building data visualization apps with highly custom user interfaces in pure Python. It's particularly suited for anyone who works with data in Python. 
Through a couple of simple patterns, Dash abstracts away all of the technologies and protocols that are required to build an interactive web-based application. Dash is simple enough that you can bind a user interface around your Python code in an afternoon. 
Dash apps are rendered in the web browser. You can deploy your apps to servers and then share them through URLs. Since Dash apps are viewed in the web browser, Dash is inherently cross-platform and mobile ready. 
There is a lot behind the framework. To learn more about how it is built and what motivated Dash, watch our talk from Plotcon below or read our announcement letter. 
Dash is an open source library, released under the permissive MIT license. Plotly develops Dash and offers a platform for managing Dash apps in an enterprise environment." 

Users can easily install Dash in their home directory on DeepSense. The way to install it is very similar to installing a package using conda. Here we provide how to install and use Dash on DeepSense. The install process only takes about 10 minutes if you have already set up your conda environments in your home directory on DeepSense.

=== Installing Dash ===
Activate the conda environment you would install Dash 
Example:
[luy@ds-lg-01 ~]$ conda activate py36-pytorch
Running the following command to install Dash:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda install -c conda-forge dash
After the installation is done, issue the following command to check if Dash is installed:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda list |grep dash
dash 1.18.1 pyhd8ed1ab_0 conda-forge
dash-core-components 1.14.1 pyhd8ed1ab_0 conda-forge
dash-html-components 1.1.1 pyh9f0ad1d_0 conda-forge
dash-renderer 1.8.3 pyhd8ed1ab_0 conda-forge
dash-table 4.11.1 pyhd8ed1ab_0 conda-forge

=== Learning Dash ===
Users can learn the features of Dash on its [https://dash.plotly.com/basic-callbacks| official webpage]. There are also a lot of Dash tutorials on youtube. You would be able to see how powerful Dash can do for visualizing your data.

=== Using Dash on DeepSense ===
1. Creating a sample Python script using Dash 
Now you can develop your projects using Dash. Below is simple example script showing how Dash works on DeepSense platform. Users can copy and paste the contents of the script and create a Python script in your home directory on DeepSense. Let's name it 'dashTest.py'.
import torch
import torchvision
from platform import python_version
import dash
import dash_html_components as html
import subprocess
app = dash.Dash(__name__)
gpu_detail=str(subprocess.run(['nvidia-smi'], stdout=subprocess.PIPE).stdout.decode('utf-8'))
gpu_name=' '.join(gpu_detail.split('\n')[7].split()[2:4])
content=[
html.H6(children='Hello world!'),
html.H6(children="Python Version: " + str(python_version())),
html.H6(children="PyTorch Version: " + str(torch.__version__)),
html.H6(children="Torchvision Version: " + str(torchvision.__version__)),
html.H6(children="GPU name: " + gpu_name)]
app.layout = html.Div(children=content)
if __name__ == '__main__':
app.run_server(host='0.0.0.0')

2. Getting an interactive session via LSF 
Running the following command to submit a new LSF interactive job:
(py36-pytorch) [luy@ds-lg-01 ~]$ bsub -Is -gpu - bash
Job <6008> is submitted to queue <gpu>.
<<Waiting for dispatch ...>>
<<Starting on ds-cmgpu-01>>
[luy@ds-cmgpu-01 ~]$
Now you have an interactive session opened on the compute host ds-cmgpu-01. You can run the sample script on the command line:
[luy@ds-cmgpu-01 ~]$ python dashTest.py
Dash is running on http://0.0.0.0:8050/
* Serving Flask app "dashTest" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:8050/ (Press CTRL+C to quit)

3. See the result on a webpage<br.
Keep the session open. This means you should not press CTRL+C. You would need to do port forwarding to open the webpage on the web browser on your local machine. This is very similar to how to use notebook on DeepSense. Below is an example: 
Lus-MacBook-Pro:~ lyang$ ssh -l luy login1.deepsense.ca -L 8050:ds-cmgpu-01:8050
luy@login1.deepsense.ca's password:
Last login: Fri Dec 18 08:41:15 2020 from vpn.deepsense.cs.dal.ca
**********************************************************************
IMPORTANT: Dear DeepSense users, jobs running on the head nodes (login
node 1 and login node 2) must be less than 10 minutes and have small
memory and CPU requirements. Jobs violating this will be terminated
without warning.
**********************************************************************
[luy@ds-lg-01 ~]$
Be careful that we obtain the compute host name 'ds-cmgpu-01' and port number '8050' from step 2. The host name and port number may change when your submit your jobs. After the port forwarding is finished, you can copy the url 'http://0.0.0.0:8050/' found at the last line of step 2 and paste it to your local web browser. The above example script would output the following contents on a HTML web page:<br.
Hello world!

Python Version: 3.6.12

PyTorch Version: 1.6.0a0+445c276

Torchvision Version: 0.7.0a0

GPU name: Tesla P100-SXM2...

Please feel free to contact DeepSense at support@deepsense.ca for any questions.

Visualization

2020-12-18T14:29:20Z

Lyang: Created page with "In this section, we would not introduce those popular plotting libraries like Matplotlib, Pandas, Seaborn, ggplot, Plotly, and so on. We would try to introduce and provide inf..."

In this section, we would not introduce those popular plotting libraries like Matplotlib, Pandas, Seaborn, ggplot, Plotly, and so on. We would try to introduce and provide information about some visualization frameworks and libraries that have more features. Using the features in these frameworks and libraries, you may make your data more understandable and interactive.

== Dash ==
=== Introduction ===
We would make use of the introduction of Dash directly from [https://dash.plotly.com/introduction| Dash's official webpage]. Users can obtain more details about Dash accessing the official webpage. 
"Dash is a productive Python framework for building web analytic applications. 
Written on top of Flask, Plotly.js, and React.js, Dash is ideal for building data visualization apps with highly custom user interfaces in pure Python. It's particularly suited for anyone who works with data in Python. 
Through a couple of simple patterns, Dash abstracts away all of the technologies and protocols that are required to build an interactive web-based application. Dash is simple enough that you can bind a user interface around your Python code in an afternoon. 
Dash apps are rendered in the web browser. You can deploy your apps to servers and then share them through URLs. Since Dash apps are viewed in the web browser, Dash is inherently cross-platform and mobile ready. 
There is a lot behind the framework. To learn more about how it is built and what motivated Dash, watch our talk from Plotcon below or read our announcement letter. 
Dash is an open source library, released under the permissive MIT license. Plotly develops Dash and offers a platform for managing Dash apps in an enterprise environment." 

Users can easily install Dash in their home directory on DeepSense. The way to install it is very similar to installing a package using conda. Here we provide how to install and use Dash on DeepSense. The install process only takes about 10 minutes if you have already set up your conda environments in your home directory on DeepSense.

=== Installing Dash ===
Activate the conda environment you would install Dash
Example:
[luy@ds-lg-01 ~]$ conda activate py36-pytorch
Running the following command to install Dash:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda install -c conda-forge dash
After the installation is done, issue the following command to check if Dash is installed:
(py36-pytorch) [luy@ds-lg-01 ~]$ conda list |grep dash
dash 1.18.1 pyhd8ed1ab_0 conda-forge
dash-core-components 1.14.1 pyhd8ed1ab_0 conda-forge
dash-html-components 1.1.1 pyh9f0ad1d_0 conda-forge
dash-renderer 1.8.3 pyhd8ed1ab_0 conda-forge
dash-table 4.11.1 pyhd8ed1ab_0 conda-forge

=== Learning Dash ===
Users can learn the features of Dash on its [https://dash.plotly.com/basic-callbacks| official webpage]. There are also a lot of Dash tutorials on youtube. You would be able to see how powerful Dash can do for visualizing your data.

=== Using Dash on DeepSense ===
1. Creating a sample Python script using Dash 
Now you can develop your projects using Dash. Below is simple example script showing how Dash works on DeepSense platform. Users can copy and paste the contents of the script and create a Python script in your home directory on DeepSense. Let's name it 'dashTest.py'.
import torch
import torchvision
from platform import python_version
import dash
import dash_html_components as html
import subprocess
app = dash.Dash(__name__)
gpu_detail=str(subprocess.run(['nvidia-smi'], stdout=subprocess.PIPE).stdout.decode('utf-8'))
gpu_name=' '.join(gpu_detail.split('\n')[7].split()[2:4])
content=[
html.H6(children='Hello world!'),
html.H6(children="Python Version: " + str(python_version())),
html.H6(children="PyTorch Version: " + str(torch.__version__)),
html.H6(children="Torchvision Version: " + str(torchvision.__version__)),
html.H6(children="GPU name: " + gpu_name)]
app.layout = html.Div(children=content)
if __name__ == '__main__':
app.run_server(host='0.0.0.0')

2. Getting an interactive session via LSF 
Running the following command to submit a new LSF interactive job:
(py36-pytorch) [luy@ds-lg-01 ~]$ bsub -Is -gpu - bash
Job <6008> is submitted to queue <gpu>.
<<Waiting for dispatch ...>>
<<Starting on ds-cmgpu-01>>
[luy@ds-cmgpu-01 ~]$
Now you have an interactive session opened on the compute host ds-cmgpu-01. You can run the sample script on the command line:
[luy@ds-cmgpu-01 ~]$ python dashTest.py
Dash is running on http://0.0.0.0:8050/
* Serving Flask app "dashTest" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://0.0.0.0:8050/ (Press CTRL+C to quit)

3. See the result on a webpage<br.
Keep the session open. This means you should not press CTRL+C. You would need to do port forwarding to open the webpage on the web browser on your local machine. This is very similar to how to use notebook on DeepSense. Below is an example: 
Lus-MacBook-Pro:~ lyang$ ssh -l luy login1.deepsense.ca -L 8050:ds-cmgpu-01:8050
luy@login1.deepsense.ca's password:
Last login: Fri Dec 18 08:41:15 2020 from vpn.deepsense.cs.dal.ca
**********************************************************************
IMPORTANT: Dear DeepSense users, jobs running on the head nodes (login
node 1 and login node 2) must be less than 10 minutes and have small
memory and CPU requirements. Jobs violating this will be terminated
without warning.
**********************************************************************
[luy@ds-lg-01 ~]$
Be careful that we obtain the compute host name 'ds-cmgpu-01' and port number '8050' from step 2. The host name and port number may change when your submit your jobs. After the port forwarding is finished, you can copy the url 'http://0.0.0.0:8050/' found at the last line of step 2 and paste it to your local web browser. The above example script would output the following contents on a HTML web page:<br.
Hello world!

Python Version: 3.6.12

PyTorch Version: 1.6.0a0+445c276

Torchvision Version: 0.7.0a0

GPU name: Tesla P100-SXM2...

Please feel free to contact DeepSense at support@deepsense.ca for any questions.

MediaWiki:Sidebar

2020-12-18T13:14:09Z

Lyang:

CWS

2020-12-16T15:12:14Z

Lyang: /* Setup your own conda environment */

[https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/conductorwithspark_kc_welcome.html IBM Spectrum Conductor with Spark (CWS)] enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.

== Accessing CWS ==
In order to access the Spectrum Conductor and get started with Spark application you can either use the web interface management console or the command-line interface.

=== Management Console ===
The [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/getting_started/management_console_overview.html management console], which is the web interface to IBM Spectrum Conductor with Spark, provides a single point of access to key system components for cluster monitoring and control, configuration, and troubleshooting. The web interface to the DeepSense IBM CWS Management Console is at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Go to the url and log in using your DeepSense account information.

=== Command Line Option ===
Spectrum Conductor with Spark also includes a Command-Line Interface (CLI) for administration. You can launch the CLI by starting a command console and source the environment for your shell.

Steps to launch the command console:

* From the login node, ssh to ‘ds-cmhm-02.deepsense.cs.dal.ca’
** <code> ssh ds-cmhm-02.deepsense.cs.dal.ca </code>
* Source the environment for your shell:
** <code> source /software/WMLA/wmla/profile.platform </code>
* Login using your account:
** <code> egosh user logon -u ‘user_name’ -x ‘password’ </code>
* You can see the list of available resources:
** <code> egosh resource list </code>
* View current activity:
** <code> egosh activity view </code>

The complete list of the CLI commands with details is available at the [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/reference/commands.html IBM Knowledge Center].

== Spark Workload ==
To create a Spark Instance Group (SIG), go to the management console and click Workload -> Spark -> Spark Instance Group. For the basic configuration you will need to specify the name, deployment directory and execution user. For how to create and manage a SIG, please refer to the following [https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/developing_instances/sig.html instructions].

=== Spark Versions ===
After specifying the basic configuration, you can choose one of the Spark versions to deploy. Currently, the following Spark versions are available:
* Spark 2.4.3
* Spark 2.3.3

== Setup your own conda environment ==
A conductor user can add his/her created conda environments to the conductor such that his/her ML scripts or notebooks can use of them. Below is the detailed instruction about how a user add his/her customized conda environments to the conductor. 
On the GUI of the conductor, go to "Resources"-> "Frameworks" -> "Anaconda Management":

[[File:Conda-mgmt.png|none|500px|Anaconda Management]]

On the "Anaconda Management" webpage, click on the "Add" button to add your conda environments. You will see a form asking you to type in the information required. Below is the summary of a created Anaconda instance. You can refer to this example to input the information of your conda environment:

[[File:sample-conda-instance.png|none|800px|Anaconda instance example]]

You will have to finish the conda instance creation before you create your notebooks in the next step. During the creation of notebooks, you would be asked to input your conda environments.

== Notebooks ==
Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. Below is an instruction how to setup and use notebook on the Conductor. 
When you create your SIG, in the section "Enable notebooks", you would need to check the box of "Jupyter 5.4.0 and select your "Anaconda distribution instance" and "Conda environment". The following screenshot is an example:

[[File:SIG-Notebook.png|none|500px|Notebook creation when creating a SIG]]

Finish the creation the SIG and then deploy and start it before you are able to use the notebook. The example SIG here is named "Lu-Jun-Test". Click on the SIG name and open it. We can see the Notebooks tab in the following screenshot:

[[File:Notebook-Tab.png|none|500px|Notebook tab opening a SIG]]

Click the "Notebooks" tab, then click the green button "Create Notebooks for Users", select yourself as the the user, and click the "Create" button:

[[File:Create-Notebooks.png|none|500px|Create notebooks]]

After you create the Notebooks, you can click on the button "My Notebooks" and select your notebook:

[[File:My-notebook.png|none|700px|My created notebook]]

After you click the Jupyter notebook you just created, you would be directed to a new web browser. Close the browser. You will need to go to the directory where your SIG is deployed to find the url with the token created and then copy and paste the url to a web browser to open the notebook. Here's an example:
The test SIG is deployed in /dshome/faculty/luy/Lu-Jun-Test/. In the deploy directory, you will be able to find a directory similar to this:
/dshome/faculty/luy/Lu-Jun-Test/Jupyter-5.4.0/Lu-Jun-Test/ad2a4cea-0279-4e30-8c99-b2d2e35a79ca/Jupyter-5-4-0-1/logs
In this directory, you are able to find a file named "ipython.log". Open this file, you would able to find the url similar to the following:
https://127.0.0.1:8890/?token=8fa719b4e98e34e555f864f73113eca1acbe45ad4ec03618
This would be the url you copy and paste to your web browser. However, before you copy and paste the url to your web browser, you need to forward the port to your local machine. This is the same as we open notebook when we submit jobs using LSF:
ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port>
for example,
ssh -l user1 login1.deepsense.ca -L 8890:ds-cmgpu-04:8890
Note that you may need to use a different <local_port> than 8890 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8890 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8890 or another unused port.

After the port forwarding is done, you can copy and paste the url to your web browser. The notebook would be opened for you.

File:Sample-conda-instance.png

2020-12-16T15:10:19Z

Lyang:

File:Conda-mgmt.png

2020-12-16T15:04:30Z

Lyang:

CWS

2020-12-16T14:57:02Z

Lyang:

[https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/conductorwithspark_kc_welcome.html IBM Spectrum Conductor with Spark (CWS)] enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.

== Accessing CWS ==
In order to access the Spectrum Conductor and get started with Spark application you can either use the web interface management console or the command-line interface.

=== Management Console ===
The [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/getting_started/management_console_overview.html management console], which is the web interface to IBM Spectrum Conductor with Spark, provides a single point of access to key system components for cluster monitoring and control, configuration, and troubleshooting. The web interface to the DeepSense IBM CWS Management Console is at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Go to the url and log in using your DeepSense account information.

=== Command Line Option ===
Spectrum Conductor with Spark also includes a Command-Line Interface (CLI) for administration. You can launch the CLI by starting a command console and source the environment for your shell.

Steps to launch the command console:

* From the login node, ssh to ‘ds-cmhm-02.deepsense.cs.dal.ca’
** <code> ssh ds-cmhm-02.deepsense.cs.dal.ca </code>
* Source the environment for your shell:
** <code> source /software/WMLA/wmla/profile.platform </code>
* Login using your account:
** <code> egosh user logon -u ‘user_name’ -x ‘password’ </code>
* You can see the list of available resources:
** <code> egosh resource list </code>
* View current activity:
** <code> egosh activity view </code>

The complete list of the CLI commands with details is available at the [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/reference/commands.html IBM Knowledge Center].

== Spark Workload ==
To create a Spark Instance Group (SIG), go to the management console and click Workload -> Spark -> Spark Instance Group. For the basic configuration you will need to specify the name, deployment directory and execution user. For how to create and manage a SIG, please refer to the following [https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/developing_instances/sig.html instructions].

=== Spark Versions ===
After specifying the basic configuration, you can choose one of the Spark versions to deploy. Currently, the following Spark versions are available:
* Spark 2.4.3
* Spark 2.3.3

== Setup your own conda environment ==

== Notebooks ==
Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. Below is an instruction how to setup and use notebook on the Conductor. 
When you create your SIG, in the section "Enable notebooks", you would need to check the box of "Jupyter 5.4.0 and select your "Anaconda distribution instance" and "Conda environment". The following screenshot is an example:

[[File:SIG-Notebook.png|none|500px|Notebook creation when creating a SIG]]

Finish the creation the SIG and then deploy and start it before you are able to use the notebook. The example SIG here is named "Lu-Jun-Test". Click on the SIG name and open it. We can see the Notebooks tab in the following screenshot:

[[File:Notebook-Tab.png|none|500px|Notebook tab opening a SIG]]

Click the "Notebooks" tab, then click the green button "Create Notebooks for Users", select yourself as the the user, and click the "Create" button:

[[File:Create-Notebooks.png|none|500px|Create notebooks]]

After you create the Notebooks, you can click on the button "My Notebooks" and select your notebook:

[[File:My-notebook.png|none|700px|My created notebook]]

After you click the Jupyter notebook you just created, you would be directed to a new web browser. Close the browser. You will need to go to the directory where your SIG is deployed to find the url with the token created and then copy and paste the url to a web browser to open the notebook. Here's an example:
The test SIG is deployed in /dshome/faculty/luy/Lu-Jun-Test/. In the deploy directory, you will be able to find a directory similar to this:
/dshome/faculty/luy/Lu-Jun-Test/Jupyter-5.4.0/Lu-Jun-Test/ad2a4cea-0279-4e30-8c99-b2d2e35a79ca/Jupyter-5-4-0-1/logs
In this directory, you are able to find a file named "ipython.log". Open this file, you would able to find the url similar to the following:
https://127.0.0.1:8890/?token=8fa719b4e98e34e555f864f73113eca1acbe45ad4ec03618
This would be the url you copy and paste to your web browser. However, before you copy and paste the url to your web browser, you need to forward the port to your local machine. This is the same as we open notebook when we submit jobs using LSF:
ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port>
for example,
ssh -l user1 login1.deepsense.ca -L 8890:ds-cmgpu-04:8890
Note that you may need to use a different <local_port> than 8890 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8890 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8890 or another unused port.

After the port forwarding is done, you can copy and paste the url to your web browser. The notebook would be opened for you.

CWS

2020-12-16T14:54:59Z

Lyang: /* Notebooks */

[https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/conductorwithspark_kc_welcome.html IBM Spectrum Conductor with Spark (CWS)] enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.

== Accessing CWS ==
In order to access the Spectrum Conductor and get started with Spark application you can either use the web interface management console or the command-line interface.

=== Management Console ===
The [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/getting_started/management_console_overview.html management console], which is the web interface to IBM Spectrum Conductor with Spark, provides a single point of access to key system components for cluster monitoring and control, configuration, and troubleshooting. The web interface to the DeepSense IBM CWS Management Console is at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Go to the url and log in using your DeepSense account information.

=== Command Line Option ===
Spectrum Conductor with Spark also includes a Command-Line Interface (CLI) for administration. You can launch the CLI by starting a command console and source the environment for your shell.

Steps to launch the command console:

* From the login node, ssh to ‘ds-cmhm-02.deepsense.cs.dal.ca’
** <code> ssh ds-cmhm-02.deepsense.cs.dal.ca </code>
* Source the environment for your shell:
** <code> source /software/WMLA/wmla/profile.platform </code>
* Login using your account:
** <code> egosh user logon -u ‘user_name’ -x ‘password’ </code>
* You can see the list of available resources:
** <code> egosh resource list </code>
* View current activity:
** <code> egosh activity view </code>

The complete list of the CLI commands with details is available at the [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/reference/commands.html IBM Knowledge Center].

== Spark Workload ==
To create a Spark Instance Group (SIG), go to the management console and click Workload -> Spark -> Spark Instance Group. For the basic configuration you will need to specify the name, deployment directory and execution user. For how to create and manage a SIG, please refer to the following [https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/developing_instances/sig.html instructions].

=== Spark Versions ===
After specifying the basic configuration, you can choose one of the Spark versions to deploy. Currently, the following Spark versions are available:
* Spark 2.4.3
* Spark 2.3.3

== Notebooks ==
Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. Below is an instruction how to setup and use notebook on the Conductor. 
When you create your SIG, in the section "Enable notebooks", you would need to check the box of "Jupyter 5.4.0 and select your "Anaconda distribution instance" and "Conda environment". The following screenshot is an example:

[[File:SIG-Notebook.png|none|500px|Notebook creation when creating a SIG]]

Finish the creation the SIG and then deploy and start it before you are able to use the notebook. The example SIG here is named "Lu-Jun-Test". Click on the SIG name and open it. We can see the Notebooks tab in the following screenshot:

[[File:Notebook-Tab.png|none|500px|Notebook tab opening a SIG]]

Click the "Notebooks" tab, then click the green button "Create Notebooks for Users", select yourself as the the user, and click the "Create" button:

[[File:Create-Notebooks.png|none|500px|Create notebooks]]

After you create the Notebooks, you can click on the button "My Notebooks" and select your notebook:

[[File:My-notebook.png|none|700px|My created notebook]]

After you click the Jupyter notebook you just created, you would be directed to a new web browser. Close the browser. You will need to go to the directory where your SIG is deployed to find the url with the token created and then copy and paste the url to a web browser to open the notebook. Here's an example:
The test SIG is deployed in /dshome/faculty/luy/Lu-Jun-Test/. In the deploy directory, you will be able to find a directory similar to this:
/dshome/faculty/luy/Lu-Jun-Test/Jupyter-5.4.0/Lu-Jun-Test/ad2a4cea-0279-4e30-8c99-b2d2e35a79ca/Jupyter-5-4-0-1/logs
In this directory, you are able to find a file named "ipython.log". Open this file, you would able to find the url similar to the following:
https://127.0.0.1:8890/?token=8fa719b4e98e34e555f864f73113eca1acbe45ad4ec03618
This would be the url you copy and paste to your web browser. However, before you copy and paste the url to your web browser, you need to forward the port to your local machine. This is the same as we open notebook when we submit jobs using LSF:
ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port>
for example,
ssh -l user1 login1.deepsense.ca -L 8890:ds-cmgpu-04:8890
Note that you may need to use a different <local_port> than 8890 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8890 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8890 or another unused port.

After the port forwarding is done, you can copy and paste the url to your web browser. The notebook would be opened for you.

CWS

2020-12-16T14:52:36Z

Lyang: /* Notebooks */

[https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/conductorwithspark_kc_welcome.html IBM Spectrum Conductor with Spark (CWS)] enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.

== Accessing CWS ==
In order to access the Spectrum Conductor and get started with Spark application you can either use the web interface management console or the command-line interface.

=== Management Console ===
The [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/getting_started/management_console_overview.html management console], which is the web interface to IBM Spectrum Conductor with Spark, provides a single point of access to key system components for cluster monitoring and control, configuration, and troubleshooting. The web interface to the DeepSense IBM CWS Management Console is at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Go to the url and log in using your DeepSense account information.

=== Command Line Option ===
Spectrum Conductor with Spark also includes a Command-Line Interface (CLI) for administration. You can launch the CLI by starting a command console and source the environment for your shell.

Steps to launch the command console:

* From the login node, ssh to ‘ds-cmhm-02.deepsense.cs.dal.ca’
** <code> ssh ds-cmhm-02.deepsense.cs.dal.ca </code>
* Source the environment for your shell:
** <code> source /software/WMLA/wmla/profile.platform </code>
* Login using your account:
** <code> egosh user logon -u ‘user_name’ -x ‘password’ </code>
* You can see the list of available resources:
** <code> egosh resource list </code>
* View current activity:
** <code> egosh activity view </code>

The complete list of the CLI commands with details is available at the [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/reference/commands.html IBM Knowledge Center].

== Spark Workload ==
To create a Spark Instance Group (SIG), go to the management console and click Workload -> Spark -> Spark Instance Group. For the basic configuration you will need to specify the name, deployment directory and execution user. For how to create and manage a SIG, please refer to the following [https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/developing_instances/sig.html instructions].

=== Spark Versions ===
After specifying the basic configuration, you can choose one of the Spark versions to deploy. Currently, the following Spark versions are available:
* Spark 2.4.3
* Spark 2.3.3

== Notebooks ==
Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. Below is an instruction how to setup and use notebook on the Conductor. 
When you create your SIG, in the section "Enable notebooks", you would need to check the box of "Jupyter 5.4.0 and select your "Anaconda distribution instance" and "Conda environment". The following screenshot is an example:

[[File:SIG-Notebook.png|none|500px|Notebook creation when creating a SIG]]

Finish the creation the SIG and then deploy and start it before you are able to use the notebook. The example SIG here is named "Lu-Jun-Test". Click on the SIG name and open it. We can see the Notebooks tab in the following screenshot:

[[File:Notebook-Tab.png|none|500px|Notebook tab opening a SIG]]

Click the "Notebooks" tab, then click the green button "Create Notebooks for Users", select yourself as the the user, and click the "Create" button:

[[File:Create-Notebooks.png|none|500px|Create notebooks]]

After you create the Notebooks, you can click on the button "My Notebooks" and select your notebook:

[[File:My-notebook.png|none|700px|My created notebook]]

After you click the Jupyter notebook you just created, you would be directed to a new web browser. Close the browser. You will need to go to the directory where your SIG is deployed to find the url with the token created and then copy and paste the url to a web browser to open the notebook. Here's an example:
The test SIG is deployed in /dshome/faculty/luy/Lu-Jun-Test/. In the deploy directory, you will be able to find a directory similar to this: /dshome/faculty/luy/Lu-Jun-Test/Jupyter-5.4.0/Lu-Jun-Test/ad2a4cea-0279-4e30-8c99-b2d2e35a79ca/Jupyter-5-4-0-1/logs. In this directory, you are able to find a file named "ipython.log". Open this file, you would able to find the url similar to the following:
https://127.0.0.1:8890/?token=8fa719b4e98e34e555f864f73113eca1acbe45ad4ec03618
This would be the url you copy and paste to your web browser. However, before you copy and paste the url to your web browser, you need to forward the port to your local machine. This is the same as we open notebook when we submit jobs using LSF:
ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port>
for example,
ssh -l user1 login1.deepsense.ca -L 8890:ds-cmgpu-04:8890
Note that you may need to use a different <local_port> than 8890 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8888 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8890 or another unused port.

After the port forwarding is done, you can copy and paste the url to your web browser. The notebook would be opened for you.

CWS

2020-12-16T14:39:19Z

Lyang: /* Notebooks */

CWS

2020-12-16T14:37:02Z

Lyang: /* Notebooks */

File:My-notebook.png

2020-12-16T14:32:19Z

Lyang:

File:Create-Notebooks.png

2020-12-16T14:28:53Z

Lyang: