CWS

IBM Spectrum Conductor with Spark (CWS) enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.

Accessing CWS

In order to access the Spectrum Conductor and get started with Spark application you can either use the web interface management console or the command-line interface.

Management Console

The management console, which is the web interface to IBM Spectrum Conductor with Spark, provides a single point of access to key system components for cluster monitoring and control, configuration, and troubleshooting. The web interface to the DeepSense IBM CWS Management Console is at https://ds-mgm-02.deepsense.cs.dal.ca:8443. Go to the url and log in using your DeepSense account information.

Command Line Option

Spectrum Conductor with Spark also includes a Command-Line Interface (CLI) for administration. You can launch the CLI by starting a command console and source the environment for your shell.

Steps to launch the command console:

• From the login node, ssh to ‘ds-cmhm-02.deepsense.cs.dal.ca’
  • ssh ds-cmhm-02.deepsense.cs.dal.ca
• Source the environment for your shell:
  • source /opt/ibm/spectrumcomputing/profile.platform
• Login using your account:
  • egosh user logon -u ‘user_name’ -x ‘password’
• You can see the list of available resources:
  • egosh resource list
• View current activity:
  • egosh activity view

The complete list of the CLI commands with details is available at the IBM Knowledge Center.

Spark Workload

To create a Spark Instance Group (SIG), go to the management console and click Workload -> Spark -> Spark Instance Group. For the basic configuration you will need to specify the name, deployment directory and execution user.

Spark Versions

After specifying the basic configuration, you can choose one of the Spark versions to deploy. Currently, the following Spark versions are available:

• Spark 2.3.1
• Spark 2.2.0
• Spark 2.1.1
• Spark 1.6.1

Notebooks

Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. To create notebook, go to Spark Instance Group -> Notebooks tab and click on Create Notebook for users.