Using HPC on AWS and Azure - Revision history

PSuthar: Add benefits and restructure

2024-04-24T21:22:49Z

Add benefits and restructure

PSuthar: Add HPC Python Job file

2024-04-24T21:06:12Z

Add HPC Python Job file

PSuthar at 20:52, 24 April 2024

2024-04-24T20:52:55Z

PSuthar: /* Accessing Your HPC Cluster */

2024-04-24T20:00:13Z

‎Accessing Your HPC Cluster

PSuthar: /* DeepSense AWS ParallelCluster & Azure CycleCloud */

2024-04-24T19:59:32Z

‎DeepSense AWS ParallelCluster & Azure CycleCloud

PSuthar: Create initial HPC page

2024-04-24T19:52:03Z

Create initial HPC page

New page

To empower demanding workloads and specialized use cases with tailored compute resources, we leverage both managed HPC platforms on AWS and Azure as well as customizable virtual machines. This approach offers the best of both worlds.

== DeepSense AWS ParallelCluster & Azure CycleCloud ==
DeepSense leverages AWS ParallelCluster and Azure CycleCloud to provide robust and scalable HPC solutions for your most demanding computational workloads.

'''AWS ParallelCluster'''
*Simplified HPC Setup: AWS ParallelCluster handles much of the complexity involved in setting up and managing an HPC cluster on AWS.
*Flexible Configurations: Choose from a variety of EC2 instance types (CPU, GPU, high-memory, etc.), network options, and storage solutions.
*Integration: Easily connect your cluster with other AWS services for data processing, storage, and visualization.

'''Azure CycleCloud'''
*Orchestration & Automation: Azure CycleCloud simplifies the creation and management of HPC clusters on Azure, automating scaling and provisioning.
*Scheduler Support: Select your preferred HPC scheduler (Slurm, PBS Pro, GridEngine, etc.) for efficient job management.
*Azure Ecosystem: Seamlessly integrate with other Azure services like Azure Blob Storage and Azure Kubernetes Service.

== Accessing Your HPC Cluster ==
'''AWS & Azure Administrator Setup'''
*Account Creation: Your dedicated AWS or Azure administrator will provision the necessary accounts and permissions for our team to set up your HPC environment.
*Access & MFA: You'll receive secure credentials and instructions for setting up Multi-Factor Authentication (MFA) to enhance security.

'''Getting Started with HPC''
*Cluster Launch: Our team will configure and launch your cluster using either AWS ParallelCluster or Azure CycleCloud.
*Connecting: We'll provide detailed instructions on how to connect to your cluster's head node via secure methods like SSH.
*Job Submission: You'll utilize your chosen scheduler's interface (e.g., Slurm commands) to submit your computational jobs and scripts to the cluster's queue.
*Monitoring: Tools within AWS ParallelCluster or Azure CycleCloud dashboards will allow you to track job progress, resource usage, and costs.

Remember: It's crucial to stop or scale down your cluster when not actively in use to avoid unnecessary charges.

Our team has extensive experience with HPC on AWS and Azure. Contact us to explore how these powerful platforms can accelerate your computational workloads!

@@ Line 1: / Line 1: @@
 To empower demanding workloads and specialized use cases with tailored compute resources, we leverage both managed HPC platforms on AWS and Azure as well as customizable virtual machines. This approach offers the best of both worlds.
-== DeepSense AWS ParallelCluster & Azure CycleCloud ==
+== What is HPC? ==
 High-Performance Computing (HPC) harnesses the collective power of multiple connected computers (called a cluster) to tackle complex problems far beyond the capabilities of a single machine. HPC is essential for Large-scale simulations such as modeling weather patterns, fluid dynamics, or complex engineering systems. For massive data analysis such as finding patterns in vast datasets, accelerating scientific discovery, or driving AI-powered insights. And also training complex AI/ML models such as developing models that require immense computational resources for tasks like natural language processing or computer vision.
 Tools for efficient HPC are AWS ParallelCluster and Azure CycleCloud to streamline the management and deployment of HPC solutions.

@@ Line 25: / Line 25: @@
 *Connecting: We'll provide detailed instructions on how to connect to your cluster's head node via secure methods like SSH.
 *Job Submission: You'll utilize your chosen scheduler's interface (e.g., Slurm commands) to submit your computational jobs and scripts to the cluster's queue.
 *Monitoring: Tools within AWS ParallelCluster or Azure CycleCloud dashboards will allow you to track job progress, resource usage, and costs.

@@ Line 6: / Line 6: @@
 Tools for efficient HPC are AWS ParallelCluster and Azure CycleCloud to streamline the management and deployment of HPC solutions.
-'''AWS ParallelCluster'''
+===AWS ParallelCluster===
 *Simplified HPC Setup: AWS ParallelCluster handles much of the complexity involved in setting up and managing an HPC cluster on AWS.
 *Flexible Configurations: Choose from a variety of EC2 instance types (CPU, GPU, high-memory, etc.), network options, and storage solutions.
 *Integration: Easily connect your cluster with other AWS services for data processing, storage, and visualization.
-'''Azure CycleCloud'''
+===Azure CycleCloud===
 *Orchestration & Automation: Azure CycleCloud simplifies the creation and management of HPC clusters on Azure, automating scaling and provisioning.
 *Scheduler Support: Select your preferred HPC scheduler (Slurm, PBS Pro, GridEngine, etc.) for efficient job management.
@@ Line 17: / Line 17: @@
 == Accessing Your HPC Cluster ==
-'''AWS & Azure Administrator Setup'''
+===AWS & Azure Administrator Setup===
 *Account Creation: Your dedicated AWS or Azure administrator will provision the necessary accounts and permissions for our team to set up your HPC environment.
 *Access & MFA: You'll receive secure credentials and instructions for setting up Multi-Factor Authentication (MFA) to enhance security.
-'''Getting Started with HPC'''
+===Getting Started with HPC===
 *Cluster Launch: Our team will configure and launch your cluster using either AWS ParallelCluster or Azure CycleCloud.
 *Connecting: We'll provide detailed instructions on how to connect to your cluster's head node via secure methods like SSH.