Resources

OpenShift Credentials

  • username: username
  • password: user-password

Dev Spaces

Developer Console

Matrix

Rocket.Chat

BGCreated with Sketch.

Tasks to complete

Get Started
BGCreated with Sketch.

Congratulations!

Return to the dashboard to continue your journey.

Go to home

Lab 5 - LLM Governance

Place a governance layer in front of the LLM endpoint using Connectivity Link. Enforce API key authentication and token-based rate limits across multiple service tiers.


The Scenario

In the previous labs, you built an AI agent that resolves customer support issues through natural language. The agent queries the database, updates records, and regenerates policy documents, all from within the chat interface.

Now you want to extend it with a new capability: customer profile analysis. When a support agent asks about a customer, the AI analyzes their data and returns a structured summary. Two tiers of analysis are available:

  • Gold — full policy history, claims history, and loyalty status.

  • Silver — recent policies and basic profile information.

flowchart LR
    AGT5 <--> AI5
    AI5 <--> GW

    GW <--> GWp
    %%GWp[" "] --> |"🥇 Rate Limit"| GP
    %%GWp --> |"🥈 Rate Limit"| SP
    GWp[" "] --- GP --- GWp2[" "] <--> LLM
    GWp --- SP --- GWp2

    AGT5@{ img: "../../_images/module-03/common/agent-icon.svg", label: "Support Agent", pos: "b", h: 80, w: 80, constraint: "on" }
    AI5@{ img: "../../_images/module-03/common/ai-icon.svg", label: "AI Assistant", pos: "b", h: 80, w: 80, constraint: "on" }
    GP@{ img: "../../_images/module-03/common/profile-icon.svg", label: "Gold Profile", pos: "b", h: 80, w: 70, constraint: "on" }
    SP@{ img: "../../_images/module-03/common/profile-silver-icon.svg", label: "Silver Profile", pos: "b", h: 80, w: 70, constraint: "on" }
    GW@{ img: "../../_images/module-03/common/guardrail-icon.svg", label: "Connectivity Link", pos: "b", h: 80, w: 80, constraint: "on" }
    LLM@{ img: "../../_images/module-03/common/ai-model-icon.svg", label: "LLM", pos: "b", h: 80, w: 80, constraint: "on" }

    style AGT5 fill:none,stroke:none
    style AI5 fill:none,stroke:none
    style GW fill:none,stroke:none
    style GWp fill:none,stroke:none,width:1px,height:1px
    style GWp2 fill:none,stroke:none,width:1px,height:1px
    style GP fill:none,stroke:none
    style SP fill:none,stroke:none
    style LLM fill:none,stroke:none

Gold consumes more tokens because the analysis is deeper and the response is richer. Silver is lighter. The system is designed with a fallback strategy: if the Gold tier’s token budget is exhausted, the service falls back to Silver. If Silver is also exhausted, the request is deferred.

But before any of this works, you need governance. Who can access the LLM endpoint? How many tokens can each tier consume? Without controls, there is nothing stopping unchecked consumption from burning through the organization’s API budget.

In this lab, you’ll place a governance layer in front of the LLM endpoint using Red Hat Connectivity Link. You’ll implement two policies:

  • AuthPolicy — Control who can access the endpoint using API key authentication and group-based authorization.

  • TokenRateLimitPolicy — Control how much each tier can consume by enforcing rate limits based on actual token usage.


This is exactly the kind of problem Red Hat Connectivity Link was designed for. Built on the Kuadrant project, it brings AI-aware governance to Kubernetes:

  • Token-aware rate limiting — limits are based on actual tokens consumed, not request counts. This is what makes tiered AI services viable: each tier gets its own token budget, enforced at the gateway.

  • Zero code changes — policies are applied at the infrastructure level. Your AI agent and integration code stay untouched.

  • Kubernetes-native — policies are standard CRDs, applied with oc apply, stored in Git, reviewed in PRs. No separate management console or vendor control plane.

  • Composable — one policy extracts identity, another uses it for per-tier token budgets. They layer on top of each other declaratively.

These capabilities come from how Connectivity Link builds on the Kubernetes Gateway API standard. Before you start configuring policies, let’s look at the Gateway API model.


The Gateway API Model

Connectivity Link layers on top of the Kubernetes Gateway API, the standard for managing traffic routing in Kubernetes. Understanding this model will make the exercises that follow intuitive.

Here are the building blocks:

  • GatewayClass + Gateway — owned by the platform team. Defines how traffic enters the cluster. Already provisioned for you.

  • HTTPRoute — owned by the platform or development team. Defines where your traffic goes (in this case, the LLM API).

---
title: The Gateway API Model
config:
    flowchart:
        titleTopMargin: -200
---
%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '18px'}}}%%
flowchart LR
    subgraph yoursLabel [" "]
        subgraph yours ["OpenShift"]
            GC[GatewayClass]
            GT[Gateway]
            HTTP[HTTPRoute]
        end
    end

    GC -->|defines| GT -->|routes via| HTTP

    style yoursLabel fill:none,stroke:none
    style yours fill:#e8eaf6,stroke:#7986cb
    style GC fill:#fff,stroke:#7986cb
    style GT fill:#fff,stroke:#7986cb
    style HTTP fill:#fff,stroke:#7986cb

GC | GatewayClass | Owned by the platform team. Defines which controller implementation manages the Gateway and its configuration (IP pools, resource limits).
GT | Gateway | The entry point to your cluster — a load balancer or service that directs traffic to the right destinations.
HTTP | HTTPRoute | Your routing rules — maps requests from the Gateway to backend services (LLMs) based on URL paths, headers, or hosts.

Hover over each component above in the diagram to learn what it does.


This is where Connectivity Link comes in. Rather than embedding governance in your application, it attaches policies directly to these Gateway API resources. Authentication Policy and TokenRateLimit Policy layer onto the HTTPRoute — enforced at the infrastructure level, invisible to your services:

---
config:
  markdownAutoWrap: false
---
flowchart LR
    Client([App])
    Gateway[Gateway]

    subgraph S1 ["HTTPRoute"]
        direction LR
        subgraph S2 [" "]
            direction LR
            Auth[Auth Policy]
            RateLimit[Token Rate Limit Policy]
            Auth ~~~ RateLimit
        end

        A@{ shape: braces, label: "<i>Connectivity Link policies attach to Gateway API resources</i>" }
    end

    Service[Service]
    Pod1[Pod]
    Pod2[Pod]

    Client -.->|HTTP request| Gateway --> S2 -->|routing rule| Service
    Service --> Pod1
    Service --> Pod2


    style S2 fill:#e8eaf6,stroke:#7986cb
    style S1 fill:none,stroke:none
    style Client fill:#f5f5f5,stroke:#ccc
    style Gateway fill:#fff,stroke:#7986cb
    style Auth fill:#fff3e0,stroke:#fb8c00
    style RateLimit fill:#fff3e0,stroke:#fb8c00
    style Service fill:#fff,stroke:#7986cb
    style Pod1 fill:#336,color:#fff
    style Pod2 fill:#336,color:#fff

End-to-End Request Flow

Now that you understand the model, here’s how it maps to your setup. The support agent interacts with the AI assistant, which sends requests through the gateway and governance layer before reaching the LLM:

flowchart LR
    Agent --> Bot
    Bot -->|api key| Gateway
    Gateway --> S2
    Auth -->|userid + group| Token
    S2 --> LLM

    subgraph S2 ["HTTPRoute"]
        subgraph S3 ["Connectivity Link"]
            Auth[Auth Policy]
            Token[Token Rate Limit]
        end
    end

    Agent@{ img: "../../_images/module-03/common/agent-icon.svg", label: "Support Agent", pos: "b", h: 80, w: 80, constraint: "on" }
    Bot@{ img: "../../_images/module-03/common/ai-icon.svg", label: "AI Assistant", pos: "b", h: 80, w: 80, constraint: "on" }
    LLM@{ img: "../../_images/module-03/common/ai-model-icon.svg", label: "LLM", pos: "b", h: 80, w: 80, constraint: "on" }

    style Agent fill:none,stroke:none
    style S2 fill:#e8eaf6,stroke:#7986cb
    style S3 fill:#e3f2fd,stroke:#7986cb
    style Bot fill:none,stroke:none
    style Gateway fill:#fff,stroke:#7986cb
    style LLM fill:none,stroke:none
    style Auth fill:#fff3e0,stroke:#fb8c00
    style Token fill:#fff3e0,stroke:#fb8c00

The same architecture applies regardless of how the support agent connects: through Rocket.Chat, an internal chatbot, or any other application.