Difference between revisions of "CWS"
(→Command Line Option) |
(→Spark Versions) |
||
| Line 30: | Line 30: | ||
=== Spark Versions === | === Spark Versions === | ||
After specifying the basic configuration, you can choose one of the Spark versions to deploy. Currently, the following Spark versions are available: | After specifying the basic configuration, you can choose one of the Spark versions to deploy. Currently, the following Spark versions are available: | ||
| − | * Spark 2.3 | + | * Spark 2.4.3 |
| − | * Spark 2. | + | * Spark 2.3.3 |
| − | |||
| − | |||
== Notebooks == | == Notebooks == | ||
Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. To [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/tutorial/t_create_share_notebook.html create notebook], go to Spark Instance Group -> Notebooks tab and click on Create Notebook for users. | Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. To [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/tutorial/t_create_share_notebook.html create notebook], go to Spark Instance Group -> Notebooks tab and click on Create Notebook for users. | ||
Revision as of 18:14, 10 December 2020
IBM Spectrum Conductor with Spark (CWS) enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.
Contents
Accessing CWS
In order to access the Spectrum Conductor and get started with Spark application you can either use the web interface management console or the command-line interface.
Management Console
The management console, which is the web interface to IBM Spectrum Conductor with Spark, provides a single point of access to key system components for cluster monitoring and control, configuration, and troubleshooting. The web interface to the DeepSense IBM CWS Management Console is at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Go to the url and log in using your DeepSense account information.
Command Line Option
Spectrum Conductor with Spark also includes a Command-Line Interface (CLI) for administration. You can launch the CLI by starting a command console and source the environment for your shell.
Steps to launch the command console:
- From the login node, ssh to ‘ds-cmhm-02.deepsense.cs.dal.ca’
ssh ds-cmhm-02.deepsense.cs.dal.ca
- Source the environment for your shell:
source /software/WMLA/wmla/profile.platform
- Login using your account:
egosh user logon -u ‘user_name’ -x ‘password’
- You can see the list of available resources:
egosh resource list
- View current activity:
egosh activity view
The complete list of the CLI commands with details is available at the IBM Knowledge Center.
Spark Workload
To create a Spark Instance Group (SIG), go to the management console and click Workload -> Spark -> Spark Instance Group. For the basic configuration you will need to specify the name, deployment directory and execution user.
Spark Versions
After specifying the basic configuration, you can choose one of the Spark versions to deploy. Currently, the following Spark versions are available:
- Spark 2.4.3
- Spark 2.3.3
Notebooks
Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. To create notebook, go to Spark Instance Group -> Notebooks tab and click on Create Notebook for users.
