System Design

System Overview

RCARS is a three-tier application (React SPA, FastAPI API, arq workers) that pulls from two content sources — the RHDP Babylon catalog and the Red Hat Architecture Center (OSSPA) — analyzes content with an LLM, stores results in PostgreSQL with pgvector, and answers recommendation queries using vector similarity search and LLM ranking. The Advisor provides a multi-intent chat interface for catalog questions, performance metrics, and content overlap analysis.

Deployments

Component	Image	Queue	Purpose
rcars-api	rcars-api:latest	—	FastAPI JSON API, serves /api/v1/*
rcars-scan-worker	rcars-api:latest	arq:queue:scan	Analysis, catalog refresh, stale checks, retirement sweeps
rcars-recommend-worker	rcars-api:latest	arq:queue:recommend	Advisor recommendation queries + chat turns
rcars-frontend	rcars-frontend:latest	—	React SPA (nginx), proxies /api/* to API

Workers are split into two deployments so bulk scans never block user-facing advisor queries. Both workers use the same container image with different arq entrypoints.

Supporting infrastructure: PostgreSQL 16 + pgvector, Redis 7, OAuth proxy.

The main pipelines — catalog sync, content analysis, recommendation, and reporting sync — run independently and can be triggered separately.

graph TB
    subgraph "Frontend"
        FE[React SPA<br/>nginx :80]
    end

    subgraph "OAuth Proxy"
        OA[oauth-proxy :4180]
    end

    subgraph "API Layer"
        API[FastAPI<br/>uvicorn :8080]
    end

    subgraph "Workers"
        SW[Scan Worker<br/>arq:queue:scan]
        RW[Recommend Worker<br/>arq:queue:recommend]
    end

    subgraph "Data Stores"
        PG[(PostgreSQL 16<br/>+ pgvector)]
        RD[(Redis 7)]
    end

    subgraph "External Systems"
        K8S[Babylon K8s API<br/>CatalogItem + AgnosticV CRDs]
        GH[GitHub<br/>Showroom repos and catalog PRs]
        Jira[Jira<br/>retirement tickets]
        GL[GitLab<br/>OSSPA portfolio architectures]
        LLM[LiteMaaS<br/>Claude Sonnet + Haiku]
        RPT[Reporting MCP Server<br/>provisions + sales + cost]
    end

    User -->|SSO| OA
    OA -->|X-Forwarded-Email| FE
    FE -->|/api/v1/*| API
    API -->|enqueue jobs| RD
    API -->|read/write| PG
    API -->|SSE relay| RD
    API -->|retirement PR actions| GH
    API -->|retirement tickets and status| Jira
    SW -->|process jobs| RD
    SW -->|read/write| PG
    SW -->|clone repos| GH
    SW -->|clone repos| GL
    SW -->|LLM calls| LLM
    SW -->|read CRDs| K8S
    SW -->|query| RPT
    SW -->|retirement sweep| GH
    RW -->|process jobs| RD
    RW -->|read| PG
    RW -->|LLM calls| LLM

---

Data Sources

Babylon Kubernetes CRDs

The RHDP catalog is defined as Kubernetes custom resources in the Babylon platform. RCARS reads two CRD types from the Babylon namespaces using a read-only kubeconfig:

• AgnosticVComponent — the primary resource for each catalog item. Contains the display name, category, product, description, keywords, stage, and workload variable configuration (which includes Showroom URLs when present).
• CatalogItem — the ordering layer resource. Used to resolve published Virtual CI identities and their relationship to underlying base components.

Three namespaces are synced:

• babylon-catalog-prod — live production catalog items
• babylon-catalog-dev — items in development or testing
• babylon-catalog-event — event-specific items

All three are synced on every catalog refresh. The stage (prod/dev/event) is derived from the namespace and stored on each catalog item, allowing stage-scoped queries and filtering throughout the system.

CI Hierarchy

Catalog items in RHDP are not all the same kind of thing. There are broadly three tiers:

• Published Virtual CIs — ordering entry points visible on catalog.demo.redhat.com. Some catalog items are published this way: the Published VCI is what a user orders, and it references an underlying Base CI for the actual content and provisioning.
• Base CIs — the actual lab definitions, containing the Showroom content link, full description, and workload configuration. Many Base CIs are ordered directly — they don't have a Published VCI in front of them. This is actually the more common pattern.
• Infrastructure CIs — the underlying provisioning layer. RCARS does not interact with these.

For content analysis and recommendations, what matters is whether a CI has a Showroom URL — that is where the lab content lives and what gets analyzed by the LLM. For infrastructure-aware queries, what matters is whether the CI uses AgnosticD v2 — those items have their workload roles extracted and mapped to products regardless of whether they have Showroom content. RCARS tracks the Published VCI ↔ Base CI relationship when it exists to avoid recommending the same underlying content twice.

OSSPA Portfolio Architectures

RCARS ingests Red Hat Architecture Center portfolio architectures from two public OSSPA GitLab repositories. These are conceptual architecture assets — not provisioned environments — that complement Babylon labs when a sales team or learner needs an architectural overview rather than a hands-on lab.

• osspa-site — provides PAList.csv, the inventory of all architecture assets with metadata (display name, status, solutions, verticals, products, and the path to each item's AsciiDoc content)
• portfolio-architecture-examples — provides the AsciiDoc content files analyzed by the LLM

Three asset types are ingested: Portfolio Architectures (PA), Validated Patterns (VP, which appear as PA,VP), and Solution Patterns (SP). All map to content_type='architecture' and source='portfolio_arch'. Interactive Experiences and standalone Demos are out of scope for Phase 1.

Status is derived from the CSV: items where both islive and showInCatalog are true receive status='prod' and are visible in Advisor and Browse by default. Items where only one or neither flag is set receive status='dev' and are visible to curators only — the same gating pattern used for Babylon dev/event items.

See Portfolio Architectures for full details on the ingest pipeline, LLM analysis, and Browse integration.

RHDP Reporting Database

RCARS imports usage, sales, and cost data from the RHDP reporting database via an MCP server. This is the same data source that powers the SuperSet management dashboard. The reporting sync also pulls field source content provisions — Git repository URLs that field teams provide when ordering OCP and RHEL items. See Performance Analysis for scoring and metrics, and Field Source Content for the field source visibility report.

---

Catalog Reader (services/catalog.py)

The catalog reader connects to the Babylon Kubernetes API using the configured kubeconfig and lists all CatalogItem and AgnosticVComponent resources.

For each component, it extracts:

• Display name, category, product, description, keywords, stage — from CatalogItem CRD metadata and labels
• Showroom URL and ref — extracted from the AgnosticVComponent using a two-path extraction strategy (see below)
• Published/base CI relationship — derived from __meta__.components[].item references
• Infrastructure metadata (AgnosticD v2 items only) — config type, cloud provider, OCP version, OS image, workload roles, ACL groups. See Infrastructure Metadata Extraction below.

The catalog reader is stateless. Each call to rcars refresh performs a full read and upsert. Items removed from Babylon are soft-deleted (retired_at = NOW()) rather than purged — all associated data (analysis, embeddings, tags, workload mappings) is preserved. Items that reappear in a future scan are automatically un-retired.

Showroom URL Extraction

Showroom URLs are not stored in a single consistent field. RCARS uses two extraction paths, checked in priority order:

Path 1. Top-level spec.definition — the most common pattern. URL variables checked (in order): ocp4_workload_showroom_content_git_repo, showroom_git_repo, bookbag_git_repo. Ref variables: ocp4_workload_showroom_content_git_repo_ref, ocp4_workload_showroom_content_git_ref, showroom_git_ref.

Template variable resolution: some CIs use Jinja2 templates for the ref (e.g., {{ showroom_repo_revision }}). RCARS resolves these by looking up the variable name in spec.definition, with catalog parameter defaults taking precedence per stage.

Path 2. Component parameter_values — Zero Touch items have deployer.type: null and delegate to a base component, passing the showroom URL as a parameter override in __meta__.components[].parameter_values. This covers ~254 CIs (entire zt-rhelbu and most zt-ansiblebu).

Template repos skipped: URLs containing showroom_template_default, showroom_template_nookbag, or showroom_template_zero are filtered out — these are placeholder defaults from shared includes, not real content.

Infrastructure Metadata Extraction (AgnosticD v2)

RCARS extracts infrastructure metadata from AgnosticD v2 component CRDs. This links catalog items to the workload roles and base configs they use.

Scope: Only items using the canonical AgnosticD v2 deployer (__meta__.deployer.scm_url == https://github.com/agnosticd/agnosticd-v2).

What's extracted: config type (agd_config), cloud provider, OCP version, OS image, cluster sizing, VM topology, workloads (Ansible roles in FQCN format), and ACL groups.

Workload extraction: Workload role names are extracted from the CRD spec.definition during catalog refresh. These come from multiple sources — workloads, software_workloads, openshift_workload_deployer_workloads, and other stage-specific fields — and include roles from any Ansible collection, not just the agnosticd organization. All discovered roles are stored in babylon_item_workloads.

Infrastructure Catalog

The infrastructure catalog is a unified registry of workload roles and base configs, stored in the infrastructure table. It replaces the old curated workload_mapping table with an LLM-scanned catalog that understands what each role and config actually does.

Two types of entries:

• Workload roles — Ansible roles that install products on existing clusters (e.g. ocp4_workload_openshift_ai, ocp4_workload_acs). Scanned from agDv2 collection repos.
• Base configs — Environment configurations that provision cloud infrastructure (e.g. ocp4-cluster, cloud-vms-base). Scanned from the agDv2 base config directories.

How scanning works: RCARS clones the agDv2 collection repos, reads key files from each role or config (defaults, tasks, templates), and sends them to the LLM for structured analysis. The LLM determines what the role installs, its products, capabilities, category, and prerequisites. Results are stored in the infrastructure table with vector embeddings for semantic search.

Linking to catalog items: Workload roles link to catalog items through babylon_item_workloads (extracted from CRDs during catalog refresh). Base configs link through babylon_items.agd_config. The Workloads & Automation page shows these connections.

Semantic search: Infrastructure embeddings enable natural-language queries through the Advisor chat (e.g. "What workload deploys OpenShift AI?"). The chat's infrastructure intent runs vector similarity search against infrastructure embeddings, then returns matching entries with their linked catalog items.

For full details on scanning, embedding search, and the UI, see Infrastructure Catalog.

---

PostgreSQL and Vector Embeddings

RCARS uses PostgreSQL with the pgvector extension as its sole data store. Schema is managed via SCHEMA_SQL in database.py — CREATE TABLE IF NOT EXISTS for fresh installs, ALTER TABLE ADD COLUMN IF NOT EXISTS for additions. No Alembic. For the full table list and column-level details, see the Data Design.

Catalog items use a soft-delete pattern: when items disappear from the Babylon CRDs, they receive a retired_at timestamp instead of being deleted. All dependent data (analysis, embeddings, workload mappings) is preserved. Reporting metrics (performance_scores) can be removed during orphan cleanup since they are re-derivable from the MCP server. Active-item queries normally filter on retired_at IS NULL. See Retirement System — Catalog Soft-Delete for details and the distinction between catalog visibility and workflow completion.

The pgvector extension is central to how RCARS works. During the scan pipeline, every analyzed Showroom lab gets a vector embedding — a list of 768 numbers produced by nomic-embed-text-v1.5, served by a dedicated vLLM embedding server. These numbers represent the meaning of the lab content in a high-dimensional space where semantically similar content clusters together. The key property: texts that mean similar things produce similar vectors, even if they use completely different words.

For example, "hands-on OpenShift workshop for platform engineers" and "practical lab teaching Kubernetes cluster management to infrastructure teams" would produce similar vectors because they describe the same kind of thing. A keyword search would not connect them.

These embeddings power two core features:

• Recommendations — a user's query is embedded with the same model, then pgvector's cosine similarity search (<=> operator) finds the labs whose embeddings are closest to the query. This replaces keyword matching with semantic understanding.
• Content overlap detection — lab embeddings are compared against each other to find catalog items that teach the same material under different names.

Cosine similarity measures the angle between two vectors. A score of 1.0 means identical meaning; 0.0 means unrelated. pgvector returns cosine distance (1 minus similarity), so lower is better. An IVFFlat index makes this search fast even across thousands of embeddings.

The embedding model runs on a dedicated vLLM server (RCARS_EMBEDDING_URL). It provides an OpenAI-compatible /v1/embeddings API and supports batch generation.

---

Worker Architecture

Why Workers Are Split

All LLM operations run in background workers, not in the API process. This keeps the API responsive — it accepts requests, creates job records, enqueues tasks to Redis, and returns immediately with a job_id.

Workers are split into two separate deployments:

rcars-scan-worker — listens on arq:queue:scan. Handles all batch operations:

• Content analysis (LLM scan of Showroom repos)
• Catalog refresh (CRD sync from Babylon)
• Stale content detection (git ls-remote checks)
• Infrastructure scanning (agDv2 workload roles and base configs)
• Reporting sync (MCP server data import)
• Retirement sweeps (due-removal PR creation and merged removal/revert reconciliation)
• Nightly pipeline (chains all of the above sequentially)

rcars-recommend-worker — listens on arq:queue:recommend. Handles advisor recommendation queries and multi-intent chat turns (via the run_chat_turn task). These are user-facing and must respond in 30–60 seconds. See Advisor Chat for the chat architecture.

The split exists because of a starvation problem: with a single worker, a bulk scan (400+ items at ~1 minute each) would monopolize all slots for hours, making the advisor completely unresponsive.

Job Lifecycle

1. API receives a request and creates a job record in PostgreSQL (status: queued)
2. API enqueues the task to the appropriate Redis queue
3. Worker picks up the task, updates status to running
4. Worker executes the task and writes results to PostgreSQL
5. Worker atomically updates both babylon_items.scan_status and jobs.status in a single transaction (via complete_scan())
6. For recommendation jobs: progress is published to Redis pub/sub, relayed to the browser via SSE

Configuration

Setting	Scan Worker	Recommend Worker
Concurrent jobs per pod	5	15
Default job timeout	600s	120s
CPU request/limit	500m / 2	250m / 1
Memory request/limit	1Gi / 4Gi	1Gi / 2Gi

Per-pod concurrency is configurable via RCARS_SCAN_MAX_JOBS and RCARS_RECOMMEND_MAX_JOBS environment variables. Resource limits and replica counts are configured via Ansible vars. See Operations Guide for details.

Some tasks override the default timeout: stale check (3600s), workload scan (3600s), nightly pipeline (7200s).

Nightly Pipeline

The scan worker runs a nightly maintenance pipeline at 04:00 UTC via arq cron. The Babylon and OSSPA sub-pipelines run sequentially, followed by the retirement sweep:

Babylon sub-pipeline:

1. Catalog refresh — pull latest CRDs from Babylon
2. Stale check — git ls-remote to detect changed Showroom repos
3. Re-analysis — enqueue analysis jobs for stale items
4. Infrastructure scan — scan agDv2 collection repos for workload roles and base configs
5. Reporting sync — pull reporting data from MCP server
6. Compute similarity — recompute pairwise content overlap scores

OSSPA sub-pipeline:

1. OSSPA sync — fetch PAList.csv, clone portfolio-architecture-examples, analyze changed items, retire removed items

Retirement sweep:

1. Retirement reconciliation — when GitHub App integration is configured, open due removal PRs and reconcile merged removal/cancellation-revert PRs. This uses the shared AgV access layer. Humans still review and merge every catalog-change PR.

The sub-pipelines and retirement sweep can also be triggered independently from the Sync & Analysis page. The direct rcars retirement sweep CLI command opens due removal PRs but does not perform the worker's merge reconciliation; see Retirement System.

LLM Provider Routing

LiteMaaS is the only LLM provider:

• LiteMaaS — OpenAI-compatible API hosted internally. Models are discovered at startup from /v1/models.

The unified call_llm() function routes every call to LiteMaaS. A model that is not served by LiteMaaS fails loudly instead of falling back, and LiteMaaS errors propagate to the caller. Provider is tracked per LLM call in the token_usage table.

The former Vertex AI / direct Anthropic fallback was removed on 2026-11-03 (RHDPCD-2226): we lost direct Vertex access and will not use it again. If a fallback provider is ever needed, it will be built fresh.

Two models are used: Sonnet (claude-sonnet-5-5-a46b) for content analysis, rationale generation, OSSPA analysis, and chat answers; Haiku (claude-haiku-5-5-a8b) for triage, workload scanning, overlap assessment, and chat routing.

---

Frontend (src/frontend/)

The frontend is a React SPA built with Vite and TypeScript, using PatternFly 6 components with a custom theme supporting light and dark modes. It is served by nginx and communicates with the FastAPI backend via JSON API calls under /api/v1/.

Pages

• Advisor — Two-pane layout: chat transcript on the left, evidence blocks on the right. Supports multi-intent queries (recommendations, performance metrics, content overlap, item details, infrastructure search) with typed envelope responses, follow-up chips, and session continuity.
• Browse — Filterable catalog view with collapsible filter panel. Supports both Babylon labs (Cloud Provider, Workloads, AgnosticD Config filters) and OSSPA architecture items (Solutions, Verticals, Audience filters). A Content Format filter switches between content types. Server-side filtering, numbered pagination. Expandable detail panels show summary, topics, products, duration (labs) or use cases and key components (architectures), and similar content. Curator-only filter panel for unanalyzed/failures/stale items.
• Workloads & Automation — Searchable catalog of infrastructure entries (workload roles and base configs) with type, category, and collection filters. Shows which catalog items use each entry. Accessible to all authenticated users.
• Content Analysis — Overlap (pairwise similarity within a stage), Performance (scored dashboard with Prod/Without Prod tabs), and Retirement (curator/admin lifecycle report). The shared workflow drawer is accessible from Performance, Non-Prod Items, and Retirement. See Retirement System.
• Admin — Status (stat cards, scheduled maintenance, LLM provider, reporting sync), Sync & Analysis (catalog sync, content analysis, jobs).

Authentication and Roles

An OAuth proxy authenticates users against Red Hat SSO and injects X-Forwarded-Email. The API reads this header on every request:

• Admin — email in RCARS_ADMIN_EMAILS_STR. Full access including catalog sync, scan, and worker controls.
• Curator — email in RCARS_CURATOR_EMAILS_STR. Can trigger single-item analysis and manage enrichment tags.
• Viewer — authenticated but not in either list. Can use the advisor and browse.

ServiceAccount tokens (used by Publishing House and other platform services) are validated via K8s TokenReview API against a configurable allowlist.

---

Deployment

RCARS runs on OpenShift, managed by an Ansible playbook (ansible/deploy.yml) with tagged execution:

Tag	What it does
full	Full deploy: namespace, infra + app manifests, builds, schema setup, smoke test
api	Apply manifests → build API → schema setup → smoke test
frontend	Apply manifests → build frontend → smoke test
apply-config	Apply manifests only (config changes, secrets, env vars — no builds)
mgmt-rbac	Bootstrap management ServiceAccount, ClusterRole, and kubeconfig

Schema setup (rcars init-db) runs automatically after every API build. It executes on the new pod, so code and schema are always in sync.

See Deployment Guide for full setup instructions.