2.5 Importing the Data Science Pipeline

Now that the mock API and Milvus database are running, we need to import the pipeline that connects them. A Kubeflow Pipeline can be defined in Python and then compiled into a static YAML file. This file contains the complete, portable definition of the pipeline’s components and logic.

For this lab, the pipeline has already been compiled for you and is available within your pre-created workbench. Your task is to download this file and then import it into the Red Hat OpenShift AI Pipelines server.

Pipeline Import Procedure

Step 1: Download the Pipeline File from Your Workbench

First, we need to get the pipeline definition file from your Jupyter environment.

  1. Navigate back to the Data Science Projects page in the Red Hat OpenShift AI Dashboard and open your project.

    Opening your dedicated user project

  2. Go to the Workbenches tab and launch your workbench (named "My Workbench") by clicking on its name, or switch to it if it is already open.

    Launching the Jupyter Workbench from the OpenShift AI dashboard

  3. Once JupyterLab opens, use the file browser on the left to navigate to the following path:

    hello-chris-rag-pipeline/lab-content/3.5/

  4. Inside this directory, you will find the file api_to_milvus_pipeline.yaml. Right-click on the file and select Download from the context menu. This will save the file to your local computer (likely in your "Downloads" folder).

    Downloading the pipeline YAML file from the JupyterLab interface.

  5. You can now close the JupyterLab browser tab.

Step 2: Import the Pipeline into OpenShift AI

Now, let’s upload the file you just downloaded.

  1. In the Red Hat OpenShift AI Dashboard, navigate to the Pipelines section within your Data Science Project.

    Navigating to the Pipelines section of the Data Science Project.

  2. Click the Import pipeline button.

In the "Import pipeline" dialog, ensure the Upload a file option is selected.

  1. Give your pipeline a name. For consistency, please use the name below:

    ServiceNow Ticket Ingestion

  2. (Optional) You can add a description, such as:

    A pipeline to fetch closed ServiceNow incidents and ingest them into a Milvus vector database.

  3. Click the Upload box and select the api_to_milvus_pipeline.yaml file that you just downloaded to your local machine.

  4. After selecting the file, your dialog should look like this. Click the Import pipeline button to finish.

    The Import pipeline dialog with the file selected and ready to import.

Once imported, the pipeline details view will show the graph of the workflow:

The pipeline details view showing the graph of the imported ServiceNow Ticket Ingestion pipeline.

With the pipeline successfully imported, we are now ready to create a run and process the data.

Summary

  • Downloaded a compiled pipeline YAML from your workbench — a self-contained, portable definition of the two-component workflow

  • Imported it into the Red Hat OpenShift AI pipeline server so it is registered and runnable from both the dashboard and the API

  • The pipeline bridges the mock API and Milvus: it will fetch incidents, generate vector embeddings, and insert them into a collection