Linux HPC Cluster: Slurm, Ansible Automation, and MPI Operations

Concepts

• Linux cluster administration and deterministic networking
• Automation/configuration management with Ansible
• Batch scheduling with Slurm (controller/worker architecture)
• Authentication and distributed trust using MUNGE
• Validation of cluster health using MPI workloads
• Troubleshooting reliability issues (time sync drift and auth failures)
• Failure injection & recovery (simulated node outage)

Design

I created two nodes: the head node orchestrates, and the compute node executes. The idea is that compute nodes should be as consistent and reproducible as possible, and changes should be applied via automation rather than manual edits.

• Head node (head) (1 CPU, 2 GB RAM, Rocky Linux 9)
  • Slurm controller (slurmctld)
  • Ansible control node
  • Login and orchestration point
• Compute node (cone) (1 CPU, 2 GB RAM, Rocky Linux 9)
  • Slurm working daemon (slurmd)
  • Execute scheduled jobs

Networking Configuration

Clusters rely on internal networking where nodes can reach each other reliably, and hostnames resolve consistently. I set up a private host-only cluster network for node-to-node traffic and a NAT interface for package installs.

Each VM uses two adapters:

• Adapter 1: Host-Only (cluster network)
• Adapter 2: NAT (internet access)

IP Assignment (Host-Only)

Head node:

        
          sudo nmcli con mod hostonly \
            ipv4.method manual \
            ipv4.addresses 192.168.56.10/24
          
          sudo nmcli con up hostonly
        
      

Compute node:

        
          sudo nmcli con mod hostonly \
            ipv4.method manual \
            ipv4.addresses 192.168.56.11/24
          
          sudo nmcli con up hostonly
        
      

Host Identity and Name Resolution

I set hostnames and /etc/hosts so that both nodes can resolve each other consistently.

        
          sudo hostnamectl set-hostname head # on head node
          sudo hostnamectl set-hostname cone # on compute node (c-one)
        
      

Edited /etc/hosts on both nodes:

        
          192.168.56.10 head
          192.168.56.11 cone
        
      

Validation

        
          ping -c 3 cone # from head node
          ping -c 3 head # from compute node
        
      

Cluster Hygiene and Chrony

I installed baseline updates, essential utilities, and time synchronization. Distributed systems depend on time synchronization. For example, clock drift will break MUNGE authentication.

        
          sudo dnf -y update
          sudo dnf -y install vim git curl wget chrony
          sudo systemctl enable --now chronyd
          chronyc tracking
        
      

Passwordless SSH

Ansible will require a control point that can reach nodes. I configured passwordless SSH from the head node to the compute node so that automation can run unattended and repeatedly for my controller/worker management.

        
          sh-keygen -t ed25519 # generate key on head
          ssh-copy-id youruser@cone # copy to compute node
          ssh cone # tests
        
      

Locking SSH config:

        
          Host cone
            HostName cone
            User youruser
        
      

Ansible Automation Framework

Ansible will turn node configuration into versionable, repeatable infrastructure code. The head node acts as the controller, applying changes to the compute node(s) consistently.

Inventory Design

        
          [controllers]
          head ansible_host=192.168.56.10 ansible_connection=local

          [compute]
          cone ansible_host=192.168.56.11

          [all:vars]
          ansible_user=youruser
          ansible_become=true
        
      

Enabling passwordless sudo:

        
          sudo visudo
          youruser ALL=(ALL)NOPASSWD:ALL
        
      

Baseline Config via Ansible

Now, instead of installing common packages manually on each node, I can define a baseline configuration and push it via `ansible-playbook`. This automation ensures both nodes have consistent tooling (Python, Chrony, etc.), and re-running the playbook will be safe.

Slurm Installation and Authentication

Slurm provides scheduling: users request resources, and the scheduler decides where and when jobs run. MUNGE provides authentication between Slurm services. I used Ansible to install required packages across the cluster, then configure MUNGE so the Slurm controller and daemons can authenticate securely.

Installing Slurm and MUNGE via Ansible

Enabling CRB, EPEL, and installing: slurm, slurmctld, slurmd, munge

MUNGE Setup

MUNGE uses a shared key (/etc/munge/munge.key) across the cluster. This key must be identical and properly permissioned on every node.

• Generate key on head
• Copy securely to compute node
• Ensure ownership and permissions
• Start MUNGE

Slurm Configuration and Scheduling

Slurm is configured via slurm.conf, which defines cluster name, controller, authentication method, nodes, and partitions. Configuration lives in source control and is deployed to nodes via Ansible (push_slurm_conf.yml), not hand-edited on each node. Once services are started, I verify that the controller sees the compute node and can schedule jobs to it.

        
          sinfo
          srun hostname
        
      

Resource Scaling and Troubleshooting

I increased the compute node’s CPU allocation, updated slurm.conf, and validated that Slurm could schedule multi-task jobs. During this change, after I had stepped away from the project for the night, I encountered a downed node. I was able to diagnose the issue using logs and time sync tools and found that the problem was time drift, which caused MUNGE credentials to be rejected and Slurm to mark the node unavailable. I corrected the system time, restarted services, and revalidated scheduling.

CPU Scaling

• Increase compute node CPUs to 2 via VirtualBox
• Update slurm.conf
• Restart Slurm daemons

Troubleshooting: Time Drift

Encountering a downed node:

        
          sudo journalctl -u slurmd -xe --no-pager | tail -n 40
          timedatectl 
          chronyc tracking
          date
        
      

Logs indicated:

        
          Munge decode failed: expired credential
        
      

MUNGE credentials are time-sensitive. If node clocks drift far enough apart, credentials generated on one node can be rejected on another as “expired.”

Solution:

Re-sync time on both nodes:

        
          sudo systemctl restart chronyd
          sudo chronyc makestep
        
      

Restart MUNGE:

        sudo systemctl restart munge
      

Restart Slurm daemons:

        
          sudo systemctl restart slurmd # on compute node
          sudo systemctl restart slurmctld # on head
        
      

or use ansible playbook.

MPI Validation via Slurm

In this project, I’m using MPI as a health check. Running MPI under Slurm proves that the scheduler can allocate resources and launch a parallel workload reliably.

Automated MPI Validation Deployment via Ansible

I use ansible-playbook to install OpenMPI on all nodes, compile the validation binary on the controller, distribute it to all nodes, then run it through srun.

MPI Validation Program (C++)

• Uses MPI_Allreduce as a correctness check
• Prints rank and hostname
• Notified failure if communication breaks

Failure Injection and Recovery

I wanted to simulate a node failure by stopping the compute daemon, observing scheduler state changes and job failure behavior, then restoring services and re-running validation.

Small Simulated Node Failure

On compute node, stop slurmd:

        
          sudo systemctl stop slurmd
        
      

On head node, observe the impact:

        
          sinfo 
          scontrol show node cone
          srun --chdir=/tmp -n 2 /opt/mpi/mpi-validate/mpi_validate
        
      

Recovery

Bring the node back up and re-validate MPI:

        
          sudo systemctl start slurmd # on cone
          sinfo # on head
          scontrol show node cone

          srun --chdir=/tmp -n 2 /opt/mpi/mpi-validate/mpi_validate