RHDP MaaS — Infrastructure & Hardware

Cluster Overview

RHDP MaaS runs across two clusters — a primary AWS cluster for NVIDIA GPU workloads and an on-prem Intel Gaudi 3 cluster for bare-metal AI acceleration.

AWS Cluster — Primary (NVIDIA GPU)

Intel Gaudi Cluster — Rackspace Bare Metal

AWS — GPU Nodes

The AWS cluster has 9 GPU worker nodes across three AWS instance types, providing a mix of GPU VRAM capacities. All nodes have the label node-role.kubernetes.io/worker-gpu.

Instance Types in Use

GPU Hardware Reference

Intel Gaudi 3 — Hardware Details

The Rackspace cluster is a single Dell PowerEdge XE9680 server with 8 Intel Gaudi 3 AI accelerators managed by the Habana AI Operator on OpenShift.

Hardware Comparison

Gaudi-Specific Serving Runtimes

Gaudi Model Placement

oc get inferenceservice -n llm-hosting --context=gaudi

Model Placement — Current Allocation

Each model is pinned to a specific node via KServe's node selector. The table below shows where each model is currently running, how many GPUs it uses, and on which AWS instance type.

How Models are Deployed — OpenShift AI & KServe

All model servers are deployed as KServe InferenceServices via OpenShift AI (RHOAI). Each model is a separate Kubernetes custom resource that KServe translates into a Deployment, Service, and optional HorizontalPodAutoscaler.

Deployment Stack

InferenceService Structure

Each model is defined as an InferenceService custom resource. Here is an annotated example:

# Example: granite-3-2-8b-instruct InferenceService
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: granite-3-2-8b-instruct
  namespace: llm-hosting
spec:
  predictor:
    model:
      modelFormat:
        name: vLLM               # tells KServe which runtime to use
      runtime: granite-3-2-8b-instruct  # references a ServingRuntime CR
      storageUri: oci://...      # model weights location (OCI registry or S3)
    nodeSelector:
      node.kubernetes.io/instance-type: g6e.2xlarge  # pin to L40S node
    resources:
      limits:
        nvidia.com/gpu: "1"      # number of GPUs
        memory: 45Gi
        cpu: "6"
    
ServingRuntime
    
Each InferenceService references a ServingRuntime CR that defines the container image, environment variables, and default resource limits for that model family.
    
# List all serving runtimes in the namespace
oc get servingruntimes -n llm-hosting

# Inspect a specific runtime
oc get servingruntime granite-3-2-8b-instruct -n llm-hosting -o yaml

    
GitOps Management
    
All InferenceService and ServingRuntime manifests are managed via the rhpds/models-aas repository and synced automatically. See the GitOps section below for full details.
  

GitOps — Model Deployment Repository

Model servers are managed entirely through a dedicated GitOps repository, separate from the LiteMaaS platform repo. Any change committed to the repo is automatically applied to the cluster — no manual oc apply needed.

ArgoCD Application

How It Works

ArgoCD watches the model-serving/ directory at the pinned tag. When a commit changes a manifest, ArgoCD detects the drift and applies the changes automatically. The Validate=false annotation allows KServe CRDs to be applied even if CRD versions temporarily mismatch during cluster upgrades.

Checking Sync Status

# Check the model-serving app status
oc get applications.argoproj.io model-serving \
  -n openshift-gitops \
  -o custom-columns="NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status"

# See all managed resources (InferenceServices, ServingRuntimes, etc.)
oc get applications.argoproj.io model-serving -n openshift-gitops \
  -o jsonpath='{range .status.resources[*]}{.kind}{"\t"}{.name}{"\n"}{end}'

Adding or Updating a Model

# 1. Clone the models-aas repo
git clone https://github.com/rhpds/models-aas
cd models-aas

# 2. Add or edit the InferenceService in model-serving/

# 3. Commit and push — ArgoCD auto-syncs
git add model-serving/my-new-model.yaml
git commit -m "Add my-new-model InferenceService"
git push

# 4. Watch the sync
oc get applications.argoproj.io model-serving -n openshift-gitops -w

Forcing a Manual Sync

# Force reconciliation if the app shows OutOfSync
oc patch applications.argoproj.io model-serving -n openshift-gitops \
  --type merge -p '{"operation":{"sync":{"syncStrategy":{"apply":{"force":false}}}}}'

Operational Commands

Check all model pods and their nodes

oc get pods -n llm-hosting -o wide | grep predictor

Check GPU allocation per node

# Total GPU capacity across GPU nodes
oc get nodes -l 'node-role.kubernetes.io/worker-gpu' \
  -o custom-columns="NAME:.metadata.name,INSTANCE:.metadata.labels['node\.kubernetes\.io/instance-type'],GPU:.status.capacity['nvidia\.com/gpu']"

# GPU currently allocated (requested by pods)
oc describe node <node-name> | grep -A5 "Allocated resources"

Check InferenceService status

oc get inferenceservice -n llm-hosting

View model server logs

# Tail logs for a specific model
oc logs -n llm-hosting -l app=granite-3-2-8b-instruct --tail=50

# Check vLLM startup — shows model load time and VRAM usage
oc logs -n llm-hosting <predictor-pod-name> | grep -E "GPU|VRAM|loaded|error"

Scale a model (increase replicas)

# Scale via InferenceService (preferred — GitOps-managed)
oc patch inferenceservice llama-scout-17b -n llm-hosting \
  --type merge -p '{"spec":{"predictor":{"minReplicas":3}}}'

# Or scale the underlying deployment directly (not persisted in Git)
oc scale deployment llama-scout-17b-predictor -n llm-hosting --replicas=3