AgentOps in Action: Evaluate, Secure, and Operate Enterprise AI Agents
Welcome to this hands-on AgentOps lab. You will take a single pre-deployed enterprise AI agent through the full observe–evaluate–contain–authorize–re-evaluate loop using Red Hat OpenShift AI.
What you will do
-
Meet the agent already deployed in your environment, try it out, and experience several attacks succeed against it while every legitimate use case also succeeds
-
Put the agent in an OpenShell sandbox and discover it’s now shielded but restricted to the point where you can’t complete your daily tasks
-
Make the agent more capable again, one control at a time — tools, file writes, and doc access — opening each back up only as far as it actually needs
-
Guard against injection attacks with NeMo Guardrails
-
Rely on MLflow traces and MLflow Evals throughout to see what’s working and what isn’t, instead of checking manually
Environment
The lab environment is a shared Red Hat OpenShift 4.22 cluster running Red Hat OpenShift AI 3.5. You have your dedicated project user-example-agentops including below components. You also have tabs on the right side to navigate between some of these components.
You do not need to memorize or access to these links. You’ll be given them when needed.
Your URLs
| What | Where |
|---|---|
Operator console |
water-plant-ui-user-example-agentops.apps.cluster-example.dyn.redhatworkshops.io |
OpenShift web console |
https://console-openshift-console.apps.cluster-example.dyn.redhatworkshops.io |
OpenShift API (for |
|
MLflow experiments (your traces) |
rh-ai.apps.cluster-example.dyn.redhatworkshops.io/develop-train/mlflow |
Keycloak admin console |
sso.apps.cluster-example.dyn.redhatworkshops.io/admin/waterplant-user-example/console |