Model Deployment and Configuration

As you are cluster admin, you currently see a lot of projects. The one we will be using in the workshop is the LLM Host project. To access it, click on Go to Data Science Projects at the bottom of the Data Science Projects sections.

In the list of projects, you should see the LLM Host project. Click on it to access the project.

Workbench: This is where you can create and manage various development environments like JupyterLab, VSCode, or other custom Workbenches. It provides a user-friendly interface for data scientists to work with notebooks, libraries, and datasets. There is already a Custom Workbench deployed in the project to explore the storage space where our models are. We will come to it later.

Pipelines: We won’t use Pipelines in this workshop, but you can use them to automate the process of processing date or training and deploying machine learning models.

Models: This is where you can manage and deploy machine learning models. You can create, update, and delete models, as well as monitor their performance and usage. You can see that a model is already deployed as an example. We will come to it later.

Cluster storage: This is where you can manage the storage resources used by your models and workbenches. You can create, update, and delete storage resources, as well as monitor their usage. At the moment you can see the storage used by the workbench we deployed.

Connections: This is where you can manage the connections between your workbenches or model runtimes and other services, such as storage (S3), databases or APIs. You can create, update, and delete connections, as well as see which environment is using them.

Permissions: This is where you can manage the permissions for project. You can create, update, and delete permissions, as well as see which users or groups have access to which resources.

As we mentioned before, a model is already deployed in the LLM Host project. To access it, click on the Models tab in the project. The model deployed is the Granite-3.2 model. It is designed to generate human-like text and can be used for various natural language processing tasks.

If you click on the expand button at the left of the model, you will see more information about its configuration, such as the resources allocated to it.

If you click on Internal and external endpoint details, you will see the different endpoints available for the model. The Internal endpoint is used to access the model from within the OpenShift cluster, while the External endpoint is used to access the model from outside the cluster (in this case the model is only accessible internally, so the external endpoint address is a placeholder one).

Switching to the Connections tab, you can see there is already a connection called models configured in the project.

As we will reuse the same connection for the model we will deploy later, let’s take a look at its configuration. Click on the three dots on the right of the connection and select Edit.

You can see that this is an S3 connection to object storage provided by Minio. Scrolling down, you can see the different parameters configured for this connection.

As we said, the model is stored in Minio, which is an S3 compatible object storage. To see those files we are going to use the ODH-TEC workbench that is already deployed in the project. To access it, click on the Workbench tab in the project, then on Open.

This will open a new tab. After logging in and accepting the license, you will access ODH-TEC. It is a simple tool used to manage and view S3 storage.

To do so, we will first create a new Workbench based on VSCode. On the Workbench tab, click on Create workbench.

Create your workbench with the following parameters:

See screenshot below for reference:

Then click on Create workbench at the bottom.

Your new workbench should appear in the list of workbenches in Starting mode. After a few moments, you can click on Open to access it.

After logging in, you will access the VSCode interface. You can close the Welcome tab if you want.

Open a Terminal in VSCode by clicking on the Terminal menu, then New Terminal. You can also use the shortcut Ctrl + Shift + \` (the backtick key).

In the terminal, you can use curl to test the model. The command to use is:

curl -X POST -H "Content-Type: application/json" -d '{"model":"ibm-granite/granite-3.2-8b-instruct" ,"prompt": "Hello, how are you?"}' http://granite-3dot2-8b-instruct-predictor.llm-hosting.svc.cluster.local:8080/v1/completions | jq

The answer should look like this:

{
    "id":"cmpl-061bbe3b71394ef98765f44e93c0df9b",
    "object":"text_completion",
    "created":1746042889,
    "model":"ibm-granite/granite-3.2-8b-instruct",
    "choices":[
        {
            "index":0,
            "text":"\",\"assistant_response\": \"Hello! I'm an artificial intelligence,",
            "logprobs":null,
            "finish_reason":"length",
            "stop_reason":null,"prompt_logprobs":null
        }
        ],
    "usage":{
        "prompt_tokens":6,
        "total_tokens":22,
        "completion_tokens":16
        }
}

In the Dashboard, navigate to the "Models" tab.

Click on the Deploy model button to start the model deployment process.

In the deployment form, fill in the following details:

Then click on Deploy to start the deployment process. This will take a few minutes to complete. At any time, you can check the status of the deployment by opening the OpenShift console and navigating to the llm-hosting project. You should see a new Pod called tinyllama-predictor-…​ being deployed. Looking at its logs, you should see the model being loaded and the server starting.

Once the deployment is complete, click on Internal and external endpoint details to see the different endpoints available for the model.

Please go back to the OpenShift AI Dashboard.

In the Model-Test workbench, open a terminal.