CWS - Revision history

Lyang: /* Setup your own conda environment */

2020-12-16T15:12:14Z

‎Setup your own conda environment

Lyang at 14:57, 16 December 2020

2020-12-16T14:57:02Z

Lyang: /* Notebooks */

2020-12-16T14:54:59Z

‎Notebooks

Lyang: /* Notebooks */

2020-12-16T14:52:36Z

‎Notebooks

Lyang: /* Notebooks */

2020-12-16T14:39:19Z

‎Notebooks

Lyang: /* Notebooks */

2020-12-16T14:37:02Z

‎Notebooks

Lyang: /* Spark Workload */

2020-12-11T12:38:26Z

‎Spark Workload

Lyang at 12:36, 11 December 2020

2020-12-11T12:36:23Z

Lyang at 12:35, 11 December 2020

2020-12-11T12:35:31Z

Lyang: /* Spark Versions */

2020-12-10T18:14:09Z

‎Spark Versions

← Older revision		Revision as of 15:12, 16 December 2020
Line 34:		Line 34:

	== Setup your own conda environment ==		== Setup your own conda environment ==
		+	A conductor user can add his/her created conda environments to the conductor such that his/her ML scripts or notebooks can use of them. Below is the detailed instruction about how a user add his/her customized conda environments to the conductor.<br>
		+	On the GUI of the conductor, go to "Resources"-> "Frameworks" -> "Anaconda Management":
		+
		+	[[File:Conda-mgmt.png\|none\|500px\|Anaconda Management]]
		+
		+	On the "Anaconda Management" webpage, click on the "Add" button to add your conda environments. You will see a form asking you to type in the information required. Below is the summary of a created Anaconda instance. You can refer to this example to input the information of your conda environment:
		+
		+	[[File:sample-conda-instance.png\|none\|800px\|Anaconda instance example]]
		+
		+	You will have to finish the conda instance creation before you create your notebooks in the next step. During the creation of notebooks, you would be asked to input your conda environments.

	== Notebooks ==		== Notebooks ==

@@ Line 32: / Line 32: @@
 * Spark 2.4.3
 * Spark 2.3.3
 == Notebooks ==

@@ Line 52: / Line 52: @@
 After you click the Jupyter notebook you just created, you would be directed to a new web browser. Close the browser. You will need to go to the directory where your SIG is deployed to find the url with the token created and then copy and paste the url to a web browser to open the notebook. Here's an example:
-The test SIG is deployed in /dshome/faculty/luy/Lu-Jun-Test/. In the deploy directory, you will be able to find a directory similar to this: /dshome/faculty/luy/Lu-Jun-Test/Jupyter-5.4.0/Lu-Jun-Test/ad2a4cea-0279-4e30-8c99-b2d2e35a79ca/Jupyter-5-4-0-1/logs. In this directory, you are able to find a file named "ipython.log". Open this file, you would able to find the url similar to the following:
+The test SIG is deployed in /dshome/faculty/luy/Lu-Jun-Test/. In the deploy directory, you will be able to find a directory similar to this:
   https://127.0.0.1:8890/?token=8fa719b4e98e34e555f864f73113eca1acbe45ad4ec03618
 This would be the url you copy and paste to your web browser. However, before you copy and paste the url to your web browser, you need to forward the port to your local machine. This is the same as we open notebook when we submit jobs using LSF:
@@ Line 58: / Line 60: @@
 for example,
   ssh -l user1 login1.deepsense.ca -L 8890:ds-cmgpu-04:8890
-Note that you may need to use a different <local_port> than 8890 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8888 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8890 or another unused port.
+Note that you may need to use a different <local_port> than 8890 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8890 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8890 or another unused port.
 After the port forwarding is done, you can copy and paste the url to your web browser. The notebook would be opened for you.

← Older revision		Revision as of 14:52, 16 December 2020
Line 50:		Line 50:

	[[File:My-notebook.png\|none\|700px\|My created notebook]]		[[File:My-notebook.png\|none\|700px\|My created notebook]]
		+
		+	After you click the Jupyter notebook you just created, you would be directed to a new web browser. Close the browser. You will need to go to the directory where your SIG is deployed to find the url with the token created and then copy and paste the url to a web browser to open the notebook. Here's an example:
		+	The test SIG is deployed in /dshome/faculty/luy/Lu-Jun-Test/. In the deploy directory, you will be able to find a directory similar to this: /dshome/faculty/luy/Lu-Jun-Test/Jupyter-5.4.0/Lu-Jun-Test/ad2a4cea-0279-4e30-8c99-b2d2e35a79ca/Jupyter-5-4-0-1/logs. In this directory, you are able to find a file named "ipython.log". Open this file, you would able to find the url similar to the following:
		+	https://127.0.0.1:8890/?token=8fa719b4e98e34e555f864f73113eca1acbe45ad4ec03618
		+	This would be the url you copy and paste to your web browser. However, before you copy and paste the url to your web browser, you need to forward the port to your local machine. This is the same as we open notebook when we submit jobs using LSF:
		+	ssh -l <username> login1.deepsense.ca -L <local_port>:<remote_host>:<remote_port>
		+	for example,
		+	ssh -l user1 login1.deepsense.ca -L 8890:ds-cmgpu-04:8890
		+	Note that you may need to use a different <local_port> than 8890 if you have other web services running on your local computer. In particular, if you run a jupyter notebook locally then it will use port 8888 and you will try to connect to the local jupyter notebook instead of the cluster notebook. In this case close your port forwarding and try again with 8890 or another unused port.
		+
		+	After the port forwarding is done, you can copy and paste the url to your web browser. The notebook would be opened for you.

← Older revision		Revision as of 14:39, 16 December 2020
Line 49:		Line 49:
	After you create the Notebooks, you can click on the button "My Notebooks" and select your notebook:		After you create the Notebooks, you can click on the button "My Notebooks" and select your notebook:

−	[[File:My-~~Notebooks~~.png\|none\|~~500px~~\|My created notebook]]	+	[[File:My-notebook.png\|none\|700px\|My created notebook]]

← Older revision		Revision as of 14:37, 16 December 2020
Line 34:		Line 34:

	== Notebooks ==		== Notebooks ==
−	Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. To [~~https~~:~~//www~~.~~ibm~~.~~com/support/knowledgecenter/en/SSZU2E_2~~.3.~~0/tutorial/t_create_share_notebook~~.~~html create notebook~~], ~~go to Spark Instance Group~~ -> Notebooks ~~tab and~~ click on ~~Create Notebook for users~~.	+	Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. Below is an instruction how to setup and use notebook on the Conductor.<br>
		+	When you create your SIG, in the section "Enable notebooks", you would need to check the box of "Jupyter 5.4.0 and select your "Anaconda distribution instance" and "Conda environment". The following screenshot is an example:
		+
		+	[[File:SIG-Notebook.png\|none\|500px\|Notebook creation when creating a SIG]]
		+
		+	Finish the creation the SIG and then deploy and start it before you are able to use the notebook. The example SIG here is named "Lu-Jun-Test". Click on the SIG name and open it. We can see the Notebooks tab in the following screenshot:
		+
		+	[[File:Notebook-Tab.png\|none\|500px\|Notebook tab opening a SIG]]
		+
		+	Click the "Notebooks" tab, then click the green button "Create Notebooks for Users", select yourself as the the user, and click the "Create" button:
		+
		+	[[File:Create-Notebooks.png\|none\|500px\|Create notebooks]]
		+
		+	After you create the Notebooks, you can click on the button "My Notebooks" and select your notebook:
		+
		+	[[File:My-Notebooks.png\|none\|500px\|My created notebook]]

@@ Line 26: / Line 26: @@
 == Spark Workload ==
-To create a Spark Instance Group (SIG), go to the management console and click Workload -> Spark -> Spark Instance Group. For the basic configuration you will need to specify the name, deployment directory and execution user.
+To create a Spark Instance Group (SIG), go to the management console and click Workload -> Spark -> Spark Instance Group. For the basic configuration you will need to specify the name, deployment directory and execution user. For how to create and manage a SIG, please refer to the following [https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/developing_instances/sig.html instructions].
 === Spark Versions ===

← Older revision		Revision as of 12:36, 11 December 2020
Line 1:		Line 1:
−	[https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/~~developing_instances/sig~~.html IBM Spectrum Conductor with Spark (CWS)] enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.	+	[https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/conductorwithspark_kc_welcome.html IBM Spectrum Conductor with Spark (CWS)] enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.

	== Accessing CWS ==		== Accessing CWS ==

← Older revision		Revision as of 12:35, 11 December 2020
Line 1:		Line 1:
−	[https://www.ibm.com/support/knowledgecenter~~/en~~/SSZU2E_2.3.0/~~conductorwithspark_kc_welcome~~.html IBM Spectrum Conductor with Spark (CWS)] enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.	+	[https://www.ibm.com/support/knowledgecenter/SSZU2E_2.4.1/developing_instances/sig.html IBM Spectrum Conductor with Spark (CWS)] enables to efficiently deploy and manage multiple Spark deployments on DeepSense computing hardware. CWS supports multiple versions and instances of Spark, provides multitenancy through Spark instance groups, and maximizes usage of resources with increased performance and scale.

	== Accessing CWS ==		== Accessing CWS ==

@@ Line 30: / Line 30: @@
 === Spark Versions ===
 After specifying the basic configuration, you can choose one of the Spark versions to deploy. Currently, the following Spark versions are available:
-* Spark 2.3.1
+* Spark 2.4.3
-* Spark 2.2.0
+* Spark 2.3.3
 == Notebooks ==
 Notebooks provide an interactive environment for data analysis with visualization from a web browser. The Jupyter 5.4.0 notebook version is available. To [https://www.ibm.com/support/knowledgecenter/en/SSZU2E_2.3.0/tutorial/t_create_share_notebook.html create notebook], go to Spark Instance Group -> Notebooks tab and click on Create Notebook for users.