Skip to content

System Design

System Overview

RCARS is a three-tier application (React SPA, FastAPI API, arq workers) that pulls from the RHDP catalog, analyzes lab content with an LLM, stores results in PostgreSQL with pgvector, and answers recommendation queries using vector similarity search and LLM ranking. The Advisor provides a multi-intent chat interface for catalog questions, performance metrics, and content overlap analysis.

Deployments

Component Image Queue Purpose
rcars-api rcars-api:latest — FastAPI JSON API, serves /api/v1/*
rcars-scan-worker rcars-api:latest arq:queue:scan Analysis, catalog refresh, stale checks
rcars-recommend-worker rcars-api:latest arq:queue:recommend Advisor recommendation queries + chat turns
rcars-frontend rcars-frontend:latest — React SPA (nginx), proxies /api/* to API

Workers are split into two deployments so bulk scans never block user-facing advisor queries. Both workers use the same container image with different arq entrypoints.

Supporting infrastructure: PostgreSQL 16 + pgvector, Redis 7, OAuth proxy.

The main pipelines — catalog sync, content analysis, recommendation, and reporting sync — run independently and can be triggered separately.

graph TB
    subgraph "Frontend"
        FE[React SPA<br/>nginx :80]
    end

    subgraph "OAuth Proxy"
        OA[oauth-proxy :4180]
    end

    subgraph "API Layer"
        API[FastAPI<br/>uvicorn :8080]
    end

    subgraph "Workers"
        SW[Scan Worker<br/>arq:queue:scan]
        RW[Recommend Worker<br/>arq:queue:recommend]
    end

    subgraph "Data Stores"
        PG[(PostgreSQL 16<br/>+ pgvector)]
        RD[(Redis 7)]
    end

    subgraph "External Systems"
        K8S[Babylon K8s API<br/>CatalogItem + AgnosticV CRDs]
        GH[GitHub<br/>Showroom repos]
        LLM[LiteMaaS / Vertex AI<br/>Claude Sonnet + Haiku]
        RPT[Reporting MCP Server<br/>provisions + sales + cost]
    end

    User -->|SSO| OA
    OA -->|X-Forwarded-Email| FE
    FE -->|/api/v1/*| API
    API -->|enqueue jobs| RD
    API -->|read/write| PG
    API -->|SSE relay| RD
    SW -->|process jobs| RD
    SW -->|read/write| PG
    SW -->|clone repos| GH
    SW -->|LLM calls| LLM
    SW -->|read CRDs| K8S
    SW -->|query| RPT
    RW -->|process jobs| RD
    RW -->|read| PG
    RW -->|LLM calls| LLM

Data Sources

Babylon Kubernetes CRDs

The RHDP catalog is defined as Kubernetes custom resources in the Babylon platform. RCARS reads two CRD types from the Babylon namespaces using a read-only kubeconfig:

  • AgnosticVComponent — the primary resource for each catalog item. Contains the display name, category, product, description, keywords, stage, and workload variable configuration (which includes Showroom URLs when present).
  • CatalogItem — the ordering layer resource. Used to resolve published Virtual CI identities and their relationship to underlying base components.

Three namespaces are synced:

  • babylon-catalog-prod — live production catalog items
  • babylon-catalog-dev — items in development or testing
  • babylon-catalog-event — event-specific items

All three are synced on every catalog refresh. The stage (prod/dev/event) is derived from the namespace and stored on each catalog item, allowing stage-scoped queries and filtering throughout the system.

CI Hierarchy

Catalog items in RHDP are not all the same kind of thing. There are broadly three tiers:

  • Published Virtual CIs — ordering entry points visible on catalog.demo.redhat.com. Some catalog items are published this way: the Published VCI is what a user orders, and it references an underlying Base CI for the actual content and provisioning.
  • Base CIs — the actual lab definitions, containing the Showroom content link, full description, and workload configuration. Many Base CIs are ordered directly — they don't have a Published VCI in front of them. This is actually the more common pattern.
  • Infrastructure CIs — the underlying provisioning layer. RCARS does not interact with these.

For content analysis and recommendations, what matters is whether a CI has a Showroom URL — that is where the lab content lives and what gets analyzed by the LLM. For infrastructure-aware queries, what matters is whether the CI uses AgnosticD v2 — those items have their workload roles extracted and mapped to products regardless of whether they have Showroom content. RCARS tracks the Published VCI ↔ Base CI relationship when it exists to avoid recommending the same underlying content twice.

RHDP Reporting Database

RCARS imports usage, sales, and cost data from the RHDP reporting database via an MCP server. This is the same data source that powers the SuperSet management dashboard. See Performance Analysis for full details on the data import, scoring methodology, and join approach.

Catalog Reader (services/catalog.py)

The catalog reader connects to the Babylon Kubernetes API using the configured kubeconfig and lists all CatalogItem and AgnosticVComponent resources.

For each component, it extracts:

  • Display name, category, product, description, keywords, stage — from CatalogItem CRD metadata and labels
  • Showroom URL and ref — extracted from the AgnosticVComponent using a two-path extraction strategy (see below)
  • Published/base CI relationship — derived from __meta__.components[].item references
  • Infrastructure metadata (AgnosticD v2 items only) — config type, cloud provider, OCP version, OS image, workload roles, ACL groups. See Infrastructure Metadata Extraction below.

The catalog reader is stateless. Each call to rcars refresh performs a full read and upsert. Items removed from Babylon are soft-deleted (retired_at = NOW()) rather than purged — all associated data (analysis, embeddings, tags, workload mappings) is preserved. Items that reappear in a future scan are automatically un-retired.

Showroom URL Extraction

Showroom URLs are not stored in a single consistent field. RCARS uses two extraction paths, checked in priority order:

Path 1. Top-level spec.definition — the most common pattern. URL variables checked (in order): ocp4_workload_showroom_content_git_repo, showroom_git_repo, bookbag_git_repo. Ref variables: ocp4_workload_showroom_content_git_repo_ref, ocp4_workload_showroom_content_git_ref, showroom_git_ref.

Template variable resolution: some CIs use Jinja2 templates for the ref (e.g., {{ showroom_repo_revision }}). RCARS resolves these by looking up the variable name in spec.definition, with catalog parameter defaults taking precedence per stage.

Path 2. Component parameter_values — Zero Touch items have deployer.type: null and delegate to a base component, passing the showroom URL as a parameter override in __meta__.components[].parameter_values. This covers ~254 CIs (entire zt-rhelbu and most zt-ansiblebu).

Template repos skipped: URLs containing showroom_template_default, showroom_template_nookbag, or showroom_template_zero are filtered out — these are placeholder defaults from shared includes, not real content.

Infrastructure Metadata Extraction (AgnosticD v2)

RCARS extracts infrastructure metadata from AgnosticD v2 component CRDs. This enables querying by infrastructure characteristics — "give me a cluster with OpenShift AI and Pipelines installed" — using faceted filters rather than vector search.

Scope: Only items using the canonical AgnosticD v2 deployer (__meta__.deployer.scm_url == https://github.com/agnosticd/agnosticd-v2).

What's extracted: config type (agd_config), cloud provider, OCP version, OS image, cluster sizing, VM topology, workloads (Ansible roles in FQCN format), and ACL groups.

Workload extraction: Workload role names are extracted from the CRD spec.definition during catalog refresh. These come from multiple sources — workloads, software_workloads, openshift_workload_deployer_workloads, and other stage-specific fields — and include roles from any Ansible collection, not just the agnosticd organization. All discovered roles are stored in babylon_item_workloads.

Workload mapping: Extracted role names are mapped to human-readable product names via a curated workload_mapping table. Product aliases allow queries using common names (e.g. "RHOAI", "ACS", "KubeVirt"). Only mapped workloads are surfaced in queries; unmapped roles are stored but invisible until curated.

Workload scanner: To help build the mapping table, RCARS scans the public agDv2 collection repos (github.com/agnosticd/*), reads the Ansible code (defaults, tasks, templates), and uses Haiku to determine the product name for each role. This covers the agnosticd.* roles but not roles from other collections — those must be mapped manually via the Admin UI. The scanner runs daily as part of the nightly pipeline, using git ls-remote change detection to skip unchanged repos.

Faceted search API: GET /catalog/search/infrastructure supports AND-semantics workload queries, config/cloud/OCP version/OS image filters, and automatic alias resolution.

PostgreSQL and Vector Embeddings

RCARS uses PostgreSQL with the pgvector extension as its sole data store. Schema is managed via SCHEMA_SQL in database.py — CREATE TABLE IF NOT EXISTS for fresh installs, ALTER TABLE ADD COLUMN IF NOT EXISTS for additions. No Alembic. For the full table list and column-level details, see the Data Design.

Catalog items use a soft-delete pattern: when items disappear from the Babylon CRDs, they receive a retired_at timestamp instead of being deleted. All dependent data (analysis, embeddings, workload mappings) is preserved. Reporting metrics (performance_scores) are removed during orphan cleanup since they are re-derivable from the MCP server. Active-item queries filter on retired_at IS NULL. See Performance Analysis — Soft-Delete for details.

The pgvector extension is central to how RCARS works. During the scan pipeline, every analyzed Showroom lab gets a vector embedding — a list of 768 numbers produced by nomic-embed-text-v1.5, served by a dedicated vLLM embedding server. These numbers represent the meaning of the lab content in a high-dimensional space where semantically similar content clusters together. The key property: texts that mean similar things produce similar vectors, even if they use completely different words.

For example, "hands-on OpenShift workshop for platform engineers" and "practical lab teaching Kubernetes cluster management to infrastructure teams" would produce similar vectors because they describe the same kind of thing. A keyword search would not connect them.

These embeddings power two core features:

  • Recommendations — a user's query is embedded with the same model, then pgvector's cosine similarity search (<=> operator) finds the labs whose embeddings are closest to the query. This replaces keyword matching with semantic understanding.
  • Content overlap detection — lab embeddings are compared against each other to find catalog items that teach the same material under different names.

Cosine similarity measures the angle between two vectors. A score of 1.0 means identical meaning; 0.0 means unrelated. pgvector returns cosine distance (1 minus similarity), so lower is better. An IVFFlat index makes this search fast even across thousands of embeddings.

The embedding model runs on a dedicated vLLM server (RCARS_EMBEDDING_URL). It provides an OpenAI-compatible /v1/embeddings API and supports batch generation.

Worker Architecture

Why Workers Are Split

All LLM operations run in background workers, not in the API process. This keeps the API responsive — it accepts requests, creates job records, enqueues tasks to Redis, and returns immediately with a job_id.

Workers are split into two separate deployments:

rcars-scan-worker — listens on arq:queue:scan. Handles all batch operations:

  • Content analysis (LLM scan of Showroom repos)
  • Catalog refresh (CRD sync from Babylon)
  • Stale content detection (git ls-remote checks)
  • Workload scanning (agDv2 collection repo analysis)
  • Reporting sync (MCP server data import)
  • Nightly pipeline (chains all of the above sequentially)

rcars-recommend-worker — listens on arq:queue:recommend. Handles advisor recommendation queries and multi-intent chat turns (via the run_chat_turn task). These are user-facing and must respond in 30–60 seconds. See Advisor Chat for the chat architecture.

The split exists because of a starvation problem: with a single worker, a bulk scan (400+ items at ~1 minute each) would monopolize all slots for hours, making the advisor completely unresponsive.

Job Lifecycle

  1. API receives a request and creates a job record in PostgreSQL (status: queued)
  2. API enqueues the task to the appropriate Redis queue
  3. Worker picks up the task, updates status to running
  4. Worker executes the task and writes results to PostgreSQL
  5. Worker atomically updates both babylon_items.scan_status and jobs.status in a single transaction (via complete_scan())
  6. For recommendation jobs: progress is published to Redis pub/sub, relayed to the browser via SSE

Configuration

Setting Scan Worker Recommend Worker
Concurrent jobs per pod 5 15
Default job timeout 600s 120s
CPU request/limit 500m / 2 250m / 1
Memory request/limit 1Gi / 4Gi 1Gi / 2Gi

Per-pod concurrency is configurable via RCARS_SCAN_MAX_JOBS and RCARS_RECOMMEND_MAX_JOBS environment variables. Resource limits and replica counts are configured via Ansible vars. See Operations Guide for details.

Some tasks override the default timeout: stale check (3600s), workload scan (3600s), nightly pipeline (7200s).

Nightly Pipeline

The scan worker runs a nightly maintenance pipeline at 04:00 UTC via arq cron:

  1. Catalog refresh — pull latest CRDs from Babylon
  2. Stale check — git ls-remote to detect changed Showroom repos
  3. Re-analysis — enqueue analysis jobs for stale items
  4. Workload scan — scan agDv2 collection repos for new/changed roles
  5. Reporting sync — pull reporting data from MCP server
  6. Compute similarity — recompute pairwise content overlap scores

LLM Provider Routing

RCARS supports two LLM providers with automatic failover:

  • LiteMaaS (preferred) — OpenAI-compatible API hosted internally. Models are discovered at startup from /v1/models.
  • Vertex AI (fallback) — Google Cloud Vertex AI with Anthropic SDK. Used when LiteMaaS is unavailable or doesn't have the requested model.

The unified call_llm() function routes each call to the appropriate provider. If LiteMaaS is available and has the requested model, it is used; otherwise the call falls back to Vertex AI automatically. Provider is tracked per LLM call in the token_usage table.

Two models are used: Sonnet for content analysis, rationale generation, and chat answers; Haiku for triage, workload scanning, and chat routing.

Frontend (src/frontend/)

The frontend is a React SPA built with Vite and TypeScript, using PatternFly 6 components with a custom theme supporting light and dark modes. It is served by nginx and communicates with the FastAPI backend via JSON API calls under /api/v1/.

Pages

  • Advisor — Two-pane layout: chat transcript on the left, evidence blocks on the right. Supports multi-intent queries (recommendations, performance metrics, content overlap, item details) with typed envelope responses, follow-up chips, and session continuity.
  • Browse — Filterable catalog view with collapsible filter panel (Cloud Provider, Workloads multi-select, AgnosticD Config), server-side filtering, numbered pagination. Expandable detail panels show summary, topics, products, duration, and similar content. Curator-only filter panel for unanalyzed/failures/stale items.
  • Content Analysis — Overlap (pairwise similarity within a stage) and Performance (scored dashboard with Prod/Without Prod tabs, retirement workflow for low performers).
  • Admin — Status (stat cards, scheduled maintenance, LLM provider, reporting sync), Sync & Analysis (catalog sync, content analysis, jobs), Workloads (workload scan, mapping management).

Authentication and Roles

An OAuth proxy authenticates users against Red Hat SSO and injects X-Forwarded-Email. The API reads this header on every request:

  • Admin — email in RCARS_ADMIN_EMAILS_STR. Full access including catalog sync, scan, and worker controls.
  • Curator — email in RCARS_CURATOR_EMAILS_STR. Can trigger single-item analysis and manage enrichment tags.
  • Viewer — authenticated but not in either list. Can use the advisor and browse.

ServiceAccount tokens (used by Publishing House and other platform services) are validated via K8s TokenReview API against a configurable allowlist.

Deployment

RCARS runs on OpenShift, managed by an Ansible playbook (ansible/deploy.yml) with tagged execution:

Tag What it does
full Full deploy: namespace, infra + app manifests, builds, schema setup, smoke test
api Apply manifests → build API → schema setup → smoke test
frontend Apply manifests → build frontend → smoke test
apply-config Apply manifests only (config changes, secrets, env vars — no builds)
mgmt-rbac Bootstrap management ServiceAccount, ClusterRole, and kubeconfig

Schema setup (rcars init-db) runs automatically after every API build. It executes on the new pod, so code and schema are always in sync.

See Deployment Guide for full setup instructions.