Module 1: Deploy a Model on Red Hat AI Enterprise
Overview
In this module you deploy a model for inference on Red Hat AI Enterprise. You select a validated open-source model, configure a model server backed by an NVIDIA GPU, and verify the deployment is accepting inference requests. The model and runtime you configure here are the foundation for the rest of the lab: they carry forward unchanged through every subsequent module.
10 minutes (3 min explore / 5 min deploy / 2 min verify)
Module 0: Getting Connected — you are logged in to the Red Hat AI Enterprise console with your lab credentials and know how to reach the OpenShift AI dashboard.
Learning objectives
-
Deploy a model for inference on Red Hat AI Enterprise
-
Verify that the deployed model is running and accepting inference requests
Along the way you also review the available model server runtimes, select a model from the model catalog, configure model server parameters — runtime, replicas, and resource limits — and confirm the model responds to a test inference request.
Exercise 1: Explore the lab environment
Your lab environment is a Red Hat AI Enterprise cluster: OpenShift plus OpenShift AI, with an NVIDIA L4 24 GB GPU worker node reserved for your model server. No models are pre-deployed — you build the full serving lifecycle from scratch.
-
Open the OpenShift web console at https://console-openshift-console.apps.cluster.example.com and log in with your lab credentials.
-
Open the application navigation menu in the web console and select Red Hat OpenShift AI. The OpenShift AI dashboard opens.
-
Explore the dashboard. The left navigation pane gives you access to Projects, Deployments, and the Gen AI Studio section (AI asset endpoints and API keys) that later modules use.
Check that the platform is healthy before deploying anything. Confirm the DataScienceCluster custom resource exists — it represents the OpenShift AI components installed on the cluster:
oc get dsc
NAME READY REASON default-dsc True
Next, confirm your GPU node is Ready and schedulable. List the cluster nodes, then filter for nodes that advertise NVIDIA GPUs:
oc get nodes -o wide
oc get nodes -L nvidia.com/gpu.product
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-10-166.us-east-2.compute.internal Ready worker,worker-gpu 22h v1.35.8 10.0.10.166 <none> Red Hat Enterprise Linux CoreOS 9.8.20261006-0 (Plow) 5.14.0-687.55.1.el9_8.x86_64 cri-o://1.35.9-12.rhaos4.22.git4c168db.el9 ip-10-0-23-196.us-east-2.compute.internal Ready control-plane,master,worker 23h v1.35.8 10.0.23.196 <none> Red Hat Enterprise Linux CoreOS 9.8.20261006-0 (Plow) 5.14.0-687.55.1.el9_8.x86_64 cri-o://1.35.9-12.rhaos4.22.git4c168db.el9 NAME STATUS ROLES AGE VERSION GPU.PRODUCT ip-10-0-10-166.us-east-2.compute.internal Ready worker,worker-gpu 22h v1.35.8 NVIDIA-L4 ip-10-0-23-196.us-east-2.compute.internal Ready control-plane,master,worker 23h v1.35.8
The nvidia.com/gpu.product label confirms
ip-10-0-10-166.us-east-2.compute.internal (roles worker,worker-gpu) is
the NVIDIA L4 24 GB GPU node. Your model server in this module requests one
GPU, so it schedules on this node. Your node ip address and versions may differ.
Project and namespace layout
The model you deploy in this module runs in your own project, named after
your lab user — for example, lab-user-dev. Everything you create in the
next sections lives in that project:
-
The model server deployment and its
LLMInferenceServiceresource -
The pod that runs the vLLM serving image on the GPU node
-
The token secret generated for inference requests
Platform-level components — the serving runtime templates, operators, and the DataScienceCluster configuration — are managed by the cluster administrator and run outside your project.
Your lab user has no project yet. Create it from the OpenShift AI dashboard:
-
In the dashboard left menu, click Projects. The Projects page opens with no projects listed.
-
Click Create project.
-
In the Name field, enter
lab-user-dev. -
Optional: enter a Description.
-
Click Create. Your project opens in the dashboard.
|
Create the project from the dashboard rather than with |
Exercise 2: Deploy the model for inference
Now that you have confirmed the environment is ready, you will deploy the model for inference.
Select the model
Red Hat AI ships models at two support levels: validated and enabled. Red Hat tests validated models with GuideLLM performance benchmarking and Language Model Evaluation Harness accuracy evaluations across specific OpenShift Container Platform, Red Hat OpenShift AI, and Red Hat AI Inference version combinations, and ships them with quantized variants in formats such as FP8 and BF16.
For this lab you use GPT-OSS 20B (essential variant), a validated, open-weight large language model designed for enterprise deployment. The MXFP4 quantization keeps the memory footprint low: the model runs within roughly 16 GB of memory, so it fits on a single 24 GB NVIDIA L4 GPU node with plenty of headroom for the attention key-value cache. It delivers production-grade inference performance for coding assistance, reasoning tasks, and tool-calling workflows, which makes it ideal for Model-as-a-Service architectures where multiple teams share centrally-managed model infrastructure.
You can discover and select verified generative AI models in the OpenShift AI model catalog — a curated, user-facing directory for finding models appropriate for your business use case.
Deploy the model from the catalog
To deploy the model on the single-model serving platform (KServe), GPU support must be enabled on the cluster and the Node Feature Discovery Operator must be installed and configured. Both are already set up in the lab environment.
-
In the OpenShift AI dashboard left menu, click AI hub, then Models. The Catalog tab opens, listing the models available to your organization.
-
In the search field, enter
gpt-ossand press Enter. The GPT-OSS model family appears.
-
Click the RedHatAI/gpt-oss-20b-essential model card. The model detail page opens with the model’s description, performance insights, and safety and security insights.
-
Click Deploy model. The Deploy a model wizard opens.
-
In the Preconfigure deployment step, select your project (
lab-user-dev) and click Next. -
In the Model details step, the model location and model type are imported from the model catalog and shown read-only. The wizard points the deployment at the validated GPT-OSS 20B essential modelcar — an OCI-compliant registry image whose filesystem holds the model weights, so the serving engine loads it directly from the image. The lab environment pre-pulls the model and serving images, which keeps startup time short. Click Next.
-
In the Model deployment step, configure the deployment:
-
Keep the pre-filled Model deployment name (
RedHatAI/gpt-oss-20b-essential). The wizard derives the deployment name from the model in the catalog and fits it to Kubernetes naming rules. -
In the Description field, enter a short description of the deployment.
-
Select the GPU serving (NVIDIA GPU) hardware profile - it may be selected automatically. The profile provides the CPU, memory, and GPU resource baseline for the deployment.
-
From the Accelerator configuration list, select vLLM NVIDIA GPU Single Node. The configuration carries the GPU node toleration so the model server pod can schedule on the GPU node.
-
Keep the Replica count at
1. -
Click Next.
-
-
In the Advanced settings step:
-
Optional: select Add as AI asset endpoint (tech preview) to publish the model’s endpoint on the AI asset endpoints page and make it testable in the Playground.
-
Leave Publish as MaaS cleared — publishing the model through the MaaS gateway is covered in Module 3.
-
Keep Token authentication enabled and enter
model-testerin the Service account name field. The actual tokens are created and displayed when the model server is configured; you use the generated token in Exercise 3. -
Expand Additional runtime arguments. The wizard pre-populates the Validated arguments for Tool calling from the model catalog —
--enable-auto-tool-choice --tool-call-parser openai. GPT-OSS handles reasoning and tool calls natively through its harmony response format, and theopenaitool-call parser is built into the vLLM image, so no further arguments are needed:
-
--enable-auto-tool-choice
--tool-call-parser openai
-
Click Next.
-
On the Review step, check the summary and click Deploy model.
Verification: the deployed model appears with a Ready status on the Gen AI studio → AI asset endpoints page (if you selected Add as AI asset endpoint), and the model server pod shows Running in your project. It will take several minutes for the model to be ready.
Watch the model server pod come up:
oc project lab-user-dev
oc get pods -w
NAME READY STATUS RESTARTS AGE redhataigpt-oss-20b-essential-kserve-78ccb7d7bf-m8bzp 2/2 Running 0 4m
The pod transitions from Pending to Running once the model server is
up. On a first deployment the server also initializes the vLLM engine, so
give it a few minutes. The lab environment pre-pulls the model and serving
images, which keeps startup time short.
In the dashboard, the deployed model shows a Ready status on the Gen AI studio → AI asset endpoints page (if you selected Add as AI asset endpoint) once it finishes deploying.
What the wizard creates
Behind the scenes, the wizard creates an LLMInferenceService resource on
the single-model serving platform (KServe), named after the model from the
catalog. The resource shown here is the configuration used in this lab, simplified
for readability:
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
name: redhataigpt-oss-20b-essential # <1>
namespace: lab-user-dev
annotations:
opendatahub.io/hardware-profile-name: gpu-serving # <9>
opendatahub.io/hardware-profile-namespace: redhat-ods-applications
labels:
kueue.x-k8s.io/queue-name: default # <2>
opendatahub.io/dashboard: "true"
opendatahub.io/genai-asset: "true"
spec:
baseRefs:
- name: redhataigpt-oss-20b-essential # <3>
model:
name: redhataigpt-oss-20b-essential
uri: oci://registry.redhat.io/rhai/modelcar-gpt-oss-20b-essential:3.0 # <4>
replicas: 1 # <5>
router:
gateway: {} # <6>
route: {}
template:
containers:
- name: main
env:
- name: VLLM_ADDITIONAL_ARGS # <7>
value: >-
--enable-auto-tool-choice
--tool-call-parser openai
resources: # <8>
requests:
cpu: "2"
memory: 16Gi
nvidia.com/gpu: "1"
limits:
cpu: "2"
memory: 16Gi
nvidia.com/gpu: "1"
| 1 | Model-derived name — The wizard names the service after the model from the catalog, truncated to fit Kubernetes naming rules. |
| 2 | Kueue queue — Places the deployment in the project’s Kueue queue, so the model server is scheduled within the cluster’s GPU quota. |
| 3 | Accelerator configuration reference — Points at the
LLMInferenceServiceConfig the wizard created from the accelerator
configuration you selected (shown below). |
| 4 | Modelcar URI — The validated model ships as an OCI image; the serving engine loads the weights directly from the image filesystem. |
| 5 | Replica — One model server replica, holding one GPU. Scaling replicas out for shared consumption is covered in Module 3. |
| 6 | Gateway routing — The KServe controller attaches the model’s HTTPRoute
to the openshift-ai-inference gateway, which terminates TLS and serves the
model’s route over HTTPS. |
| 7 | Runtime arguments — The validated tool-calling arguments the wizard
pre-populated from the model catalog, passed to vLLM through the
VLLM_ADDITIONAL_ARGS environment variable. --enable-auto-tool-choice
together with --tool-call-parser openai enables tool calling — GPT-OSS
handles reasoning and tool calls natively through its harmony response
format, and the openai tool-call parser is built into the vLLM image, so
the model’s thinking is returned in a separate reasoning field and tool
calls in the OpenAI-compatible format without any additional parsers. |
| 8 | Resource baseline — Requests and limits from the GPU serving hardware profile: 2 CPU, 16 Gi memory, and 1 GPU, sized for the GPT-OSS model on a single NVIDIA GPU node. |
| 9 | Hardware profile annotations — Record which hardware profile the
wizard selected, by name and namespace (gpu-serving in
redhat-ods-applications). The profile supplies the resource baseline and,
through its Kueue queue, the project’s local queue. |
The baseRefs entry points at an LLMInferenceServiceConfig the wizard
creates alongside the service — it carries the GPU node toleration from the
accelerator configuration you selected:
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceServiceConfig
metadata:
name: redhataigpt-oss-20b-essential
spec:
template:
tolerations:
- effect: NoSchedule
key: nvidia.com/gpu (1)
operator: Equal
value: l4-gpu
| 1 | GPU node toleration — Lets the model server pod schedule on the GPU
node, which is tainted nvidia.com/gpu=l4-gpu:NoSchedule by the NVIDIA GPU
operator. |
|
The model name, serving runtime, and resource configuration you set in this module carry forward unchanged through all subsequent modules (Modules 2-6). Do not scale down, redeploy, or rename the model between modules. |
Exercise 3: Verify the deployment
Now that the model is deployed, you will verify that it is running and accepting inference requests.
Check the pod and model status
Confirm the model server pod is Running and the LLMInferenceService
reports ready:
oc get pods
NAME READY STATUS RESTARTS AGE redhataigpt-oss-20b-essential-kserve-78ccb7d7bf-m8bzp 2/2 Running 0 4m
oc get llminferenceservice redhataigpt-oss-20b-essential
NAME URL READY AGE redhataigpt-oss-20b-essential https://inference-gateway.apps.cluster.example.com/lab-user-dev/redhataigpt-oss-20b-essential True 5m
When the READY column shows True, the model server pod is Running and
the model is ready to accept inference requests.
Locate the inference endpoint and token
-
From the OpenShift AI dashboard, open Gen AI studio → AI asset endpoints. Locate the model’s row and click View in the Endpoints column — the dialog shows the external and internal API endpoint URLs. Copy the endpoint URL.
-
To authenticate requests, copy the token displayed on the model page: open AI hub → Models → Deployments, expand the model deployment row, and locate the Token authentication section. Copy the Token secret shown for the
model-testertoken name and save it. The deployment creates the token with the service account you named in Require token authentication. (Module 3 mints a subscription-scoped MaaS API key instead, with the sameAuthorization: Bearerheader pattern.)
Send a test inference request
The vLLM runtime is compatible with the OpenAI REST API, so the model
serves the /v1/chat/completions path. The model serving platform serves
API requests over the HTTPS port of the inference gateway (usually
port 443).
Store the endpoint and token as shell variables, then send a chat request.
First, the endpoint URL from the LLMInferenceService:
MODEL_URL=$(oc get llminferenceservice redhataigpt-oss-20b-essential -o jsonpath='{.status.url}')
echo "MODEL_URL: $MODEL_URL"
Then the token from the service account secret the deployment created:
TOKEN=$(oc get secret model-tester-redhataigpt-oss-20b-essential-sa -o jsonpath='{.data.token}' | base64 -d)
echo "TOKEN: ${TOKEN:0:12}..."
Finally, send a chat request:
curl -s $MODEL_URL/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{
"model": "redhataigpt-oss-20b-essential",
"messages": [
{
"role": "user",
"content": "In one sentence, what is Model as a Service?"
}
],
"max_tokens": 3072
}' | jq .
A successful response returns the assistant’s answer in the
OpenAI-compatible chat completion format. GPT-OSS handles reasoning
natively through its harmony response format: the model’s thinking is
returned in a separate reasoning field, and content holds only the
final answer:
-
Confirm the response contains a
choicesarray with the assistant’s message inchoices[0].message.content. -
Keep the endpoint URL and token saved — you will reuse them when you expose the model as a Model-as-a-Service in Module 3.
Your model is now deployed, running, and accepting inference requests — the foundation for every module that follows.
Module summary
What you accomplished:
-
Explored the lab environment and confirmed the platform and GPU node were ready.
-
Created your own Data Science project,
lab-user-dev. -
Deployed the model for inference with the vLLM NVIDIA GPU ServingRuntime for KServe.
-
Verified the deployment and sent a test inference request through the OpenAI-compatible endpoint.
Key takeaways:
-
Red Hat AI Enterprise provides a streamlined workflow for deploying models to inference endpoints.
-
A deployed model must be verified before it can be evaluated or exposed to consumers.
-
Model server configuration — runtime, replicas, and resource limits — determines how the model serves inference requests.
Next steps:
Now that the model is deployed and accepting inference requests, continue to Module 2 to benchmark its performance with EvalHub.













