Building an Automated Recovery Pipeline for GPU Clusters with Slurm on Azure Part1

Overview As GPU clusters grow in scale, failure recovery becomes a critical part of maintaining workload resiliency and maximizing compute resource utilization. In this article series, I’ll walk through how to build an automated recovery system for a Slurm-managed GPU cluster running on Microsoft Azure. This system detects job failures,…

21/05/2025Azure High Performance Computing (HPC) Blog

Share:

You may be interested in

New Linux VDI Solution: Deploy Cendio Thinlinc as a Standalone Image with Direct Connectivity
Azure High Performance Computing (HPC) Blog,
Prerequisites Microsoft Azure Subscription: Ensure you have an active Microsoft Azure subscription. Virtual Network: Ensure you have a VNet with connectivity to your corporate network (for example, over VPN) Resource…
29/04/2025
Reducing cost and time to solution for Simcenter STAR-CCM+ software on Azure’s HB-series VMs
Azure High Performance Computing (HPC) Blog,
Authors: Amirreza Rastegari, Jon Shelley Simcenter STAR-CCM+ is one of the most widely used computational fluid dynamics (CFD) simulation applications by high performance computing (HPC) customers in a variety…
03/03/2023
Massive Scaling of Barracuda Virtual Reactor on Azure-GPU
Azure High Performance Computing (HPC) Blog,
1. Introduction: We are excited to share with you the latest benchmarks for Barracuda Virtual Reactor on the Azure NDv2 ('Standard_ND40rs_v2') and ND A100 v4 ('Standard_ND96asr_v4') virtual machines (VMs), accelerated by NVIDIA…
12/06/2023
Deploy NDm_v4 (A100) Kubernetes Cluster
Azure High Performance Computing (HPC) Blog,
Introduction Today there is a lot of interest around generative AI, specifically training and inferencing large language models (OpenAI GPT4, DALL.E2), Git copilot, Azure OpenAI service). Training these large…
03/06/2023
GPU Slicing in CycleCloud Slurm with CUDA Multi-Process Service (MPS)
Azure High Performance Computing (HPC) Blog,
High-performance computing thrives on efficient GPU resource sharing, and integrating NVIDIA’s CUDA Multi-Process Service (MPS) with CycleCloud-managed Slurm clusters can revolutionize how teams optimize their workloads. CUDA MPS streamlines…
14/01/2025
Discover how Azure HPC can improve patient outcomes
Azure High Performance Computing (HPC) Blog,
Download the eBook 1McKinsey & Company, The next frontier of healthcare delivery, 2022.2Sesterce, Unleashing the Power of HPC in Healthcare: A Digital Odyssey, 2024. 3Accenture, Unleashing the power…
17/09/2024
Exploring CPU vs GPU Speed in AI Training: A Demonstration with TensorFlow
Azure High Performance Computing (HPC) Blog,
This marks my final blog post of 2023, serving as a fundamental demonstration highlighting the distinctions in speed between CPU and GPU in AI training. The primary aim of this…
21/12/2023
Daily schedule: Microsoft in-booth sessions at NVIDIA GTC
Azure High Performance Computing (HPC) Blog,
We look forward to seeing you at the NVIDIA GTC AI Conference March 17 - 21 in San Jose, CA or virtually. Visit Microsoft booth #514 for daily informative sessions…
06/03/2025
Pre-Job Health Checks on AKS: A Guide to Stable AI Workloads
Azure High Performance Computing (HPC) Blog,
Pre-Job Health Checks on AKS: A Guide to Stable AI Workloads Introduction In the realm of AI workloads, ensuring the health and stability of compute nodes is critical.…
19/07/2024
Running HPC and AI workloads in containers in Azure
Azure High Performance Computing (HPC) Blog,
Introduction Containers technologies are no longer something new in the industry. It all started focusing on how to deploy reproducible development environments but now you can find many other fields…
03/11/2023
Az-HOP in the Azure Marketplace
Azure High Performance Computing (HPC) Blog,
Az-HOP in the Azure Marketplace Introduction Azure HPC On-Demand Platform, or Az-HOP, provides an end-to-end deployment mechanism for a base HPC infrastructure on Azure. The platform includes a…
24/05/2023
Unlocking innovation with Altair Unlimited and Microsoft Azure
Azure High Performance Computing (HPC) Blog,
Image courtesy of Altair The world of high-performance computing (HPC) and simulation just got a major boost as Altair® Unlimited™, the state-of-the-art virtual private cloud appliance, has become available on…
07/08/2023