RAG & Knowledge Grounding

From generic to grounded.

In the previous section, you saw that system prompts can inject domain knowledge but do not scale. Retrieval-Augmented Generation (RAG) solves this by retrieving relevant passages from uploaded documents at query time and injecting them into the model’s context automatically.

Estimated time: 15 minutes

Learning Objectives

By the end of this section, you will be able to:

Enable RAG in the Playground and upload a private document
Compare model responses across four configurations: no prompt, system prompt only, RAG-enabled, and RAG + global system prompt
Describe how OGX and pgvector provide document retrieval under the hood

Exercise 2: RAG in the Playground

First, clear the system prompt. Click Reset to remove the domain prompt you set in the previous exercise. You want to see what RAG does on its own, without the hardcoded product data.
Now download the Fed Aura Capital mortgage product definitions. This is the document you will upload as the knowledge source. It contains product names, eligibility criteria, rate details, and policy rules that the model has no knowledge of from training.

Download mortgage-products.pdf
Open Settings and select the Knowledge tab. Toggle Knowledge on and select Use uploaded documents:
Click Upload and select the file you just downloaded (mortgage-products.pdf). The upload dialog shows the embedding settings - leave the defaults (chunk length 800, overlap 400) and click Upload:
Wait until the document finishes processing. You will see a "Source uploaded" notification and the file listed under Uploaded files:
Send the same question you have been asking all along:
```
What mortgage products do you offer? I'd like to compare my options.
```
The model responds with product details grounded in the uploaded document. It should reference the five products, their rates, credit score requirements, and down payment minimums - all pulled from the catalog, not from training data or a hardcoded prompt.
For a more specific test, ask a follow-up that requires precise retrieval:
```
What are the underwriting requirements for a Jumbo Loan? Include the reserve requirements and DTI limits.
```
The model should return that jumbo loans require a minimum credit score of 700, a maximum back-end debt-to-income (DTI) of 38%, and at least 12 months of reserves covering the proposed housing payment. It may also mention that jumbo loans require a senior underwriter and a second review, and that loan amounts exceeding $1.5 million need a second comparable analysis. These are specific Fed Aura Capital underwriting standards from the document - not general knowledge a model would have from training data.

Because the application uses generative AI, responses are non-deterministic. The exact wording and formatting may differ, but the factual content should match the uploaded document.

How RAG works under the hood (click to expand)

RAG is, at its core, about sending relevant information into the prompt so the model answers based on real data instead of hallucinating. But you cannot send the entire knowledge base - the less context you send, the faster and more accurate the reply. So the system needs a way to find the most relevant pieces automatically.

That is where embeddings come in. An embedding model converts text into a numerical vector - a list of numbers that captures its meaning. Similar texts produce similar vectors. This lets the system search semantically rather than by keyword: "minimum down payment for first-time buyers" matches "3.5% for FHA borrowers with 580+ credit score" even though they share no words.

RAG pipeline: documents are parsed by Docling then embedded and stored in a vector database. User queries are embedded and matched against stored vectors. Retrieved chunks are combined with the query into a context prompt sent to the LLM.

Here is what happened when you uploaded mortgage-products.pdf and asked your question:

Chunking - The PDF was split into smaller passages. The chunk settings you saw during upload (length 800, overlap 400) control how big each piece is and how much consecutive chunks overlap. Overlap ensures that information spanning two chunks is not lost at the boundary.
Embedding - Each chunk was converted into a vector using an embedding model. The result is a numerical fingerprint of the chunk’s meaning.
Storage - The vectors were stored in pgvector, a PostgreSQL extension optimized for similarity search across millions of vectors.
Retrieval - When you sent your question, it was also embedded into a vector. pgvector found the stored chunks with the most similar vectors - the ones most semantically relevant to your question.
Augmentation - The retrieved chunks were injected into the model’s context alongside your question, giving it access to the actual document content.
Generation - The model composed its response using both its language capabilities and the retrieved facts. It cited specific rates, credit scores, and reserve requirements because those were in the retrieved chunks - not in its training data.

Fed Aura Capital’s production application uses the same pipeline. OGX provides the unified API layer that coordinates retrieval and generation, and pgvector provides the vector storage. The Playground’s RAG toggle wraps this into a simple upload-and-query experience, but the same chunking, embedding, and retrieval steps run under the hood.

Exercise 3: The Global System Prompt

The RAG response is factually correct. But compare it to what Lenora said in the Overview when you clicked "Explore Products". We asked you to remember that answer for exactly this moment. The content is similar, but the tone, formatting, and persona are different. Lenora in the live application is warm, concise, and speaks as a named mortgage assistant. The Playground response is informative but impersonal.

The difference is the system prompt. Lenora in the Fed Aura Capital application uses a carefully crafted system prompt that defines her persona, tone, and behavioral boundaries. Your Playground has RAG enabled but no persona prompt.

Load the platform prompt

In a production platform, development teams building on top of Lenora need to work from the same baseline, otherwise each team ends up with a slightly different version of her persona. Beyond that, the Prompt Registry can also ship global prompts for common use cases: generic kickstarters that any team can load and adapt rather than writing from scratch. The platform admins can maintain these so teams spend less time on boilerplate and more time on what makes their use case specific. You do not need to configure anything, just load it.

With RAG still enabled, go to the Prompt tab. You should see the default prompt:
Click Load Prompt. In the dialog, select the Global prompts tab. You will see fed-aura-capital-system-prompt marked as Read-only - this is the production prompt deployed by the platform admins:
Select it. The prompt loads into the Playground with a Global badge. Notice it defines capabilities, tool usage rules, response style, formatting constraints, and safety rules:
Send the thread prompt one last time:
```
What mortgage products do you offer? I'd like to compare my options.
```
The response now combines RAG-retrieved facts with the production persona. The model lists all seven products with specific rates, credit scores, and down payment requirements from the uploaded document - but presents them in the concise, numbered format that the system prompt defines. The facts come from RAG. The style comes from the global prompt. Together, they reproduce what Lenora does in the Fed Aura Capital app.

How global prompts work on the platform (click to expand)

The platform admins deployed the global prompt in the global-prompts namespace. The opendatahub.io/global-mlflow-workspace: mlflow label on the namespace tells the Prompt Registry to make all prompts in that namespace available globally. Any project in the cluster can load these prompts, but only admins can modify them. This is how the platform enforces consistent behavior across teams without requiring each engineer to copy-paste prompts manually.

apiVersion: v1
kind: Namespace
metadata:
  name: global-prompts
  labels:
    opendatahub.io/global-mlflow-workspace: mlflow

RAG provides the facts. The system prompt provides the personality. Production AI assistants need both. The GenAI Playground lets you iterate on each independently.

What’s next

Your Playground now matches what Lenora does in the live application. But the document you uploaded was a clean, well-structured PDF. In the next section, you will learn how more complex enterprise documents with tables, charts, and mixed layouts are preprocessed for RAG pipelines.