Module 1: Deploy a Model on Red Hat AI Enterprise

Overview

In this module you deploy a model for inference on Red Hat AI Enterprise. You select a validated open-source model, configure a model server backed by an NVIDIA GPU, and verify the deployment is accepting inference requests. The model and runtime you configure here are the foundation for the rest of the lab: they carry forward unchanged through every subsequent module.

Duration

10 minutes (3 min explore / 5 min deploy / 2 min verify)

Prerequisites

Module 0: Getting Connected — you are logged in to the Red Hat AI Enterprise console with your lab credentials and know how to reach the OpenShift AI dashboard.

Learning objectives

  • Deploy a model for inference on Red Hat AI Enterprise

  • Verify that the deployed model is running and accepting inference requests

Along the way you also review the available model server runtimes, select a model from the model catalog, configure model server parameters — runtime, replicas, and resource limits — and confirm the model responds to a test inference request.

Exercise 1: Explore the lab environment

Your lab environment is a Red Hat AI Enterprise cluster: OpenShift plus OpenShift AI, with an NVIDIA L4 24 GB GPU worker node reserved for your model server. No models are pre-deployed — you build the full serving lifecycle from scratch.

  1. Open the OpenShift web console at https://console-openshift-console.apps.cluster.example.com and log in with your lab credentials.

  2. Open the application navigation menu in the web console and select Red Hat OpenShift AI. The OpenShift AI dashboard opens.

  3. Explore the dashboard. The left navigation pane gives you access to Projects, Deployments, and the Gen AI Studio section (AI asset endpoints and API keys) that later modules use.

OpenShift AI dashboard home page with the left navigation pane

Check that the platform is healthy before deploying anything. Confirm the DataScienceCluster custom resource exists — it represents the OpenShift AI components installed on the cluster:

oc get dsc
NAME          READY   REASON
default-dsc   True

Next, confirm your GPU node is Ready and schedulable. List the cluster nodes, then filter for nodes that advertise NVIDIA GPUs:

oc get nodes -o wide
oc get nodes -L nvidia.com/gpu.product
NAME                                        STATUS   ROLES                         AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-0-10-166.us-east-2.compute.internal   Ready    worker,worker-gpu             22h   v1.35.8   10.0.10.166   <none>        Red Hat Enterprise Linux CoreOS 9.8.20261006-0 (Plow)   5.14.0-687.55.1.el9_8.x86_64   cri-o://1.35.9-12.rhaos4.22.git4c168db.el9
ip-10-0-23-196.us-east-2.compute.internal   Ready    control-plane,master,worker   23h   v1.35.8   10.0.23.196   <none>        Red Hat Enterprise Linux CoreOS 9.8.20261006-0 (Plow)   5.14.0-687.55.1.el9_8.x86_64   cri-o://1.35.9-12.rhaos4.22.git4c168db.el9

NAME                                        STATUS   ROLES                         AGE   VERSION   GPU.PRODUCT
ip-10-0-10-166.us-east-2.compute.internal   Ready    worker,worker-gpu             22h   v1.35.8   NVIDIA-L4
ip-10-0-23-196.us-east-2.compute.internal   Ready    control-plane,master,worker   23h   v1.35.8

The nvidia.com/gpu.product label confirms ip-10-0-10-166.us-east-2.compute.internal (roles worker,worker-gpu) is the NVIDIA L4 24 GB GPU node. Your model server in this module requests one GPU, so it schedules on this node. Your node ip address and versions may differ.

Project and namespace layout

The model you deploy in this module runs in your own project, named after your lab user — for example, lab-user-dev. Everything you create in the next sections lives in that project:

  • The model server deployment and its LLMInferenceService resource

  • The pod that runs the vLLM serving image on the GPU node

  • The token secret generated for inference requests

Platform-level components — the serving runtime templates, operators, and the DataScienceCluster configuration — are managed by the cluster administrator and run outside your project.

Your lab user has no project yet. Create it from the OpenShift AI dashboard:

  1. In the dashboard left menu, click Projects. The Projects page opens with no projects listed.

  2. Click Create project.

  3. In the Name field, enter lab-user-dev.

  4. Optional: enter a Description.

  5. Click Create. Your project opens in the dashboard.

Create the project from the dashboard rather than with oc new-project — the dashboard adds the OpenShift AI labels that make the project visible on the Projects page and selectable in the deploy wizard later in this module.

Projects page showing the lab-user-dev student project

Exercise 2: Deploy the model for inference

Now that you have confirmed the environment is ready, you will deploy the model for inference.

Select the model

Red Hat AI ships models at two support levels: validated and enabled. Red Hat tests validated models with GuideLLM performance benchmarking and Language Model Evaluation Harness accuracy evaluations across specific OpenShift Container Platform, Red Hat OpenShift AI, and Red Hat AI Inference version combinations, and ships them with quantized variants in formats such as FP8 and BF16.

For this lab you use GPT-OSS 20B (essential variant), a validated, open-weight large language model designed for enterprise deployment. The MXFP4 quantization keeps the memory footprint low: the model runs within roughly 16 GB of memory, so it fits on a single 24 GB NVIDIA L4 GPU node with plenty of headroom for the attention key-value cache. It delivers production-grade inference performance for coding assistance, reasoning tasks, and tool-calling workflows, which makes it ideal for Model-as-a-Service architectures where multiple teams share centrally-managed model infrastructure.

You can discover and select verified generative AI models in the OpenShift AI model catalog — a curated, user-facing directory for finding models appropriate for your business use case.

Deploy the model from the catalog

To deploy the model on the single-model serving platform (KServe), GPU support must be enabled on the cluster and the Node Feature Discovery Operator must be installed and configured. Both are already set up in the lab environment.

  1. In the OpenShift AI dashboard left menu, click AI hub, then Models. The Catalog tab opens, listing the models available to your organization.

  2. In the search field, enter gpt-oss and press Enter. The GPT-OSS model family appears.

Model catalog showing the GPT-OSS model search results
  1. Click the RedHatAI/gpt-oss-20b-essential model card. The model detail page opens with the model’s description, performance insights, and safety and security insights.

Model catalog page for gpt-oss-20b-essential with the Deploy model button
  1. Click Deploy model. The Deploy a model wizard opens.

  2. In the Preconfigure deployment step, select your project (lab-user-dev) and click Next.

  3. In the Model details step, the model location and model type are imported from the model catalog and shown read-only. The wizard points the deployment at the validated GPT-OSS 20B essential modelcar — an OCI-compliant registry image whose filesystem holds the model weights, so the serving engine loads it directly from the image. The lab environment pre-pulls the model and serving images, which keeps startup time short. Click Next.

Model details step with the model location imported from the model catalog
  1. In the Model deployment step, configure the deployment:

    1. Keep the pre-filled Model deployment name (RedHatAI/gpt-oss-20b-essential). The wizard derives the deployment name from the model in the catalog and fits it to Kubernetes naming rules.

    2. In the Description field, enter a short description of the deployment.

    3. Select the GPU serving (NVIDIA GPU) hardware profile - it may be selected automatically. The profile provides the CPU, memory, and GPU resource baseline for the deployment.

    4. From the Accelerator configuration list, select vLLM NVIDIA GPU Single Node. The configuration carries the GPU node toleration so the model server pod can schedule on the GPU node.

    5. Keep the Replica count at 1.

    6. Click Next.

Model deployment step with the GPU serving hardware profile and the accelerator configuration
  1. In the Advanced settings step:

    1. Optional: select Add as AI asset endpoint (tech preview) to publish the model’s endpoint on the AI asset endpoints page and make it testable in the Playground.

    2. Leave Publish as MaaS cleared — publishing the model through the MaaS gateway is covered in Module 3.

    3. Keep Token authentication enabled and enter model-tester in the Service account name field. The actual tokens are created and displayed when the model server is configured; you use the generated token in Exercise 3.

    4. Expand Additional runtime arguments. The wizard pre-populates the Validated arguments for Tool calling from the model catalog — --enable-auto-tool-choice --tool-call-parser openai. GPT-OSS handles reasoning and tool calls natively through its harmony response format, and the openai tool-call parser is built into the vLLM image, so no further arguments are needed:

--enable-auto-tool-choice
--tool-call-parser openai
  1. Click Next.

Advanced settings step with token authentication and runtime arguments
  1. On the Review step, check the summary and click Deploy model.

Review step summarizing the deployment configuration

Verification: the deployed model appears with a Ready status on the Gen AI studio → AI asset endpoints page (if you selected Add as AI asset endpoint), and the model server pod shows Running in your project. It will take several minutes for the model to be ready.

Watch the model server pod come up:

oc project lab-user-dev
oc get pods -w
NAME                                                    READY   STATUS    RESTARTS   AGE
redhataigpt-oss-20b-essential-kserve-78ccb7d7bf-m8bzp   2/2     Running   0          4m

The pod transitions from Pending to Running once the model server is up. On a first deployment the server also initializes the vLLM engine, so give it a few minutes. The lab environment pre-pulls the model and serving images, which keeps startup time short.

In the dashboard, the deployed model shows a Ready status on the Gen AI studio → AI asset endpoints page (if you selected Add as AI asset endpoint) once it finishes deploying.

AI asset endpoints page showing the deployed model with Ready status

What the wizard creates

Behind the scenes, the wizard creates an LLMInferenceService resource on the single-model serving platform (KServe), named after the model from the catalog. The resource shown here is the configuration used in this lab, simplified for readability:

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: redhataigpt-oss-20b-essential # <1>
  namespace: lab-user-dev
  annotations:
    opendatahub.io/hardware-profile-name: gpu-serving # <9>
    opendatahub.io/hardware-profile-namespace: redhat-ods-applications
  labels:
    kueue.x-k8s.io/queue-name: default # <2>
    opendatahub.io/dashboard: "true"
    opendatahub.io/genai-asset: "true"
spec:
  baseRefs:
    - name: redhataigpt-oss-20b-essential # <3>
  model:
    name: redhataigpt-oss-20b-essential
    uri: oci://registry.redhat.io/rhai/modelcar-gpt-oss-20b-essential:3.0 # <4>
  replicas: 1 # <5>
  router:
    gateway: {} # <6>
    route: {}
  template:
    containers:
      - name: main
        env:
          - name: VLLM_ADDITIONAL_ARGS # <7>
            value: >-
              --enable-auto-tool-choice
              --tool-call-parser openai
        resources: # <8>
          requests:
            cpu: "2"
            memory: 16Gi
            nvidia.com/gpu: "1"
          limits:
            cpu: "2"
            memory: 16Gi
            nvidia.com/gpu: "1"
1 Model-derived name — The wizard names the service after the model from the catalog, truncated to fit Kubernetes naming rules.
2 Kueue queue — Places the deployment in the project’s Kueue queue, so the model server is scheduled within the cluster’s GPU quota.
3 Accelerator configuration reference — Points at the LLMInferenceServiceConfig the wizard created from the accelerator configuration you selected (shown below).
4 Modelcar URI — The validated model ships as an OCI image; the serving engine loads the weights directly from the image filesystem.
5 Replica — One model server replica, holding one GPU. Scaling replicas out for shared consumption is covered in Module 3.
6 Gateway routing — The KServe controller attaches the model’s HTTPRoute to the openshift-ai-inference gateway, which terminates TLS and serves the model’s route over HTTPS.
7 Runtime arguments — The validated tool-calling arguments the wizard pre-populated from the model catalog, passed to vLLM through the VLLM_ADDITIONAL_ARGS environment variable. --enable-auto-tool-choice together with --tool-call-parser openai enables tool calling — GPT-OSS handles reasoning and tool calls natively through its harmony response format, and the openai tool-call parser is built into the vLLM image, so the model’s thinking is returned in a separate reasoning field and tool calls in the OpenAI-compatible format without any additional parsers.
8 Resource baseline — Requests and limits from the GPU serving hardware profile: 2 CPU, 16 Gi memory, and 1 GPU, sized for the GPT-OSS model on a single NVIDIA GPU node.
9 Hardware profile annotations — Record which hardware profile the wizard selected, by name and namespace (gpu-serving in redhat-ods-applications). The profile supplies the resource baseline and, through its Kueue queue, the project’s local queue.

The baseRefs entry points at an LLMInferenceServiceConfig the wizard creates alongside the service — it carries the GPU node toleration from the accelerator configuration you selected:

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceServiceConfig
metadata:
  name: redhataigpt-oss-20b-essential
spec:
  template:
    tolerations:
      - effect: NoSchedule
        key: nvidia.com/gpu (1)
        operator: Equal
        value: l4-gpu
1 GPU node toleration — Lets the model server pod schedule on the GPU node, which is tainted nvidia.com/gpu=l4-gpu:NoSchedule by the NVIDIA GPU operator.

The model name, serving runtime, and resource configuration you set in this module carry forward unchanged through all subsequent modules (Modules 2-6). Do not scale down, redeploy, or rename the model between modules.

Exercise 3: Verify the deployment

Now that the model is deployed, you will verify that it is running and accepting inference requests.

Check the pod and model status

Confirm the model server pod is Running and the LLMInferenceService reports ready:

oc get pods
NAME                                               READY   STATUS    RESTARTS   AGE
redhataigpt-oss-20b-essential-kserve-78ccb7d7bf-m8bzp  2/2  Running   0          4m
oc get llminferenceservice redhataigpt-oss-20b-essential
NAME                            URL                                                                              READY   AGE
redhataigpt-oss-20b-essential  https://inference-gateway.apps.cluster.example.com/lab-user-dev/redhataigpt-oss-20b-essential   True    5m

When the READY column shows True, the model server pod is Running and the model is ready to accept inference requests.

Locate the inference endpoint and token

  1. From the OpenShift AI dashboard, open Gen AI studio → AI asset endpoints. Locate the model’s row and click View in the Endpoints column — the dialog shows the external and internal API endpoint URLs. Copy the endpoint URL.

Endpoints dialog showing the external and internal API endpoint URLs for the model
  1. To authenticate requests, copy the token displayed on the model page: open AI hub → Models → Deployments, expand the model deployment row, and locate the Token authentication section. Copy the Token secret shown for the model-tester token name and save it. The deployment creates the token with the service account you named in Require token authentication. (Module 3 mints a subscription-scoped MaaS API key instead, with the same Authorization: Bearer header pattern.)

Token authentication section on the model deployment page showing the token name and token secret with the copy button

Send a test inference request

The vLLM runtime is compatible with the OpenAI REST API, so the model serves the /v1/chat/completions path. The model serving platform serves API requests over the HTTPS port of the inference gateway (usually port 443).

Store the endpoint and token as shell variables, then send a chat request. First, the endpoint URL from the LLMInferenceService:

MODEL_URL=$(oc get llminferenceservice redhataigpt-oss-20b-essential -o jsonpath='{.status.url}')
echo "MODEL_URL: $MODEL_URL"
The terminal showing the model endpoint URL retrieved from the LLMInferenceService

Then the token from the service account secret the deployment created:

TOKEN=$(oc get secret model-tester-redhataigpt-oss-20b-essential-sa -o jsonpath='{.data.token}' | base64 -d)
echo "TOKEN: ${TOKEN:0:12}..."
The terminal showing the model authentication token as a short prefix

Finally, send a chat request:

curl -s $MODEL_URL/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "model": "redhataigpt-oss-20b-essential",
    "messages": [
      {
        "role": "user",
        "content": "In one sentence, what is Model as a Service?"
      }
    ],
    "max_tokens": 3072
  }' | jq .

A successful response returns the assistant’s answer in the OpenAI-compatible chat completion format. GPT-OSS handles reasoning natively through its harmony response format: the model’s thinking is returned in a separate reasoning field, and content holds only the final answer:

The terminal showing the full chat completion response with the reasoning field and the final content answer
  1. Confirm the response contains a choices array with the assistant’s message in choices[0].message.content.

  2. Keep the endpoint URL and token saved — you will reuse them when you expose the model as a Model-as-a-Service in Module 3.

Your model is now deployed, running, and accepting inference requests — the foundation for every module that follows.

Module summary

What you accomplished:

  • Explored the lab environment and confirmed the platform and GPU node were ready.

  • Created your own Data Science project, lab-user-dev.

  • Deployed the model for inference with the vLLM NVIDIA GPU ServingRuntime for KServe.

  • Verified the deployment and sent a test inference request through the OpenAI-compatible endpoint.

Key takeaways:

  • Red Hat AI Enterprise provides a streamlined workflow for deploying models to inference endpoints.

  • A deployed model must be verified before it can be evaluated or exposed to consumers.

  • Model server configuration — runtime, replicas, and resource limits — determines how the model serves inference requests.

Next steps:

Now that the model is deployed and accepting inference requests, continue to Module 2 to benchmark its performance with EvalHub.