3Scale API Gateway

From a Terminal (in your VSCode workbench) log into OpenShift.

oc login -u userX -p openshift --server=https://api.cluster-guid.guid.sandbox.opentlc.com:6443

Now that you are logged in, you can get the secret values with the following commands:

oc get secret system-seed -n 3scale -o jsonpath='{.data.ADMIN_USER}' | base64 --decode

oc get secret system-seed -n 3scale -o jsonpath='{.data.ADMIN_PASSWORD}' | base64 --decode

Want to do it all in one command? Here you go:

oc get secret system-seed -n 3scale -o template='{{range $key, $value := .data}}{{if or (eq $key "ADMIN_USER") (eq $key "ADMIN_PASSWORD")}}{{printf "%s: " $key}}{{ $value | base64decode }}{{"\n"}}{{end}}{{end}}'

Audience: this gives you information about the users, registrations, etc.

In the APIs section, you have an overview of:

On the Audience section, got to Developer Portal→Settings→Domains & Access.

The Developer Portal Access Code is used to control the access the developer portal. You can use the default value in the next section to connect to it, or simply delete it and click on Update Account. This will remove the access code and allow anyone to access the developer portal, although they will still need to authenticate. This is what we will do in this workshop.

On the Audience section, got to Accounts→Listing and click on userX.

In the Service Subscriptions tab, you can see that the user is not subscribed to the granite-3dot2-8b-instruct service. Click on the Subscribe button at the right.

On the window that opens, select the Default Plan and click on Create subscription.

Your username: userX

Your password: openshift

Click on the Sign In button in the top right corner of the screen

Click on Private login and enter your credentials.

Click on the See your Applications and their credentials link on the front page.

You will be taken to the Apps and API Keys section, where you can see that there is an application that was already created for you (the 3Scale default Echo API). Start creating a new application by clicking on the Create new application button.

Select the Service you want to use. In this case, we will use the granite-3dot2-8b-instruct service that has been pre-deployed.

Give a name to your application, for example Granite application. Click on Create Application.

Your application has been created. You can see the Endpoint URL you can use to connect to the API, the name of the model you must use in your requests, and the API key that has been generated for you. You can now use this key to access the API.

Open a terminal in your VSCode environment and run the following command, replacing the placeholder values with the ones you got from the previous step:

curl -X 'POST' \
    '___Endpoint_URL___/v1/completions' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer ___API_KEY___' \
    -d '{
    "model": "___MODEL_NAME___",
    "prompt": "San Francisco is a",
    "max_tokens": 15,
    "temperature": 0
}'

In the OpenShift Web Console, click on the Operators menu on the left side of the screen, then select Installed Operators. On the top Project dropdown, select the 3scale project.

Click on Red Hat Integration - 3scale operator from the list.

You will be taken to the operator details page.

Click on the 3scale Backend tab. You will see that there is already a backend created for the granite-3dot2-8b-instruct model.

To deploy the new backend for TinyLlama, click on the Create Backend button.

In the YAML view, replace the default content with the following YAML and click on create. This will create a new backend for the TinyLlama model.

kind: Backend
apiVersion: capabilities.3scale.net/v1beta1
metadata:
  name: tinyllama
  namespace: 3scale
spec:
  name: TinyLlama
  privateBaseURL: 'https://tinyllama-llm-hosting.apps.cluster-guid.guid.sandbox.opentlc.com'
  systemName: tinyllama

After a few seconds, you should see the new backend in the list of backends.

You can also switch to the 3Scale Admin Portal and check that the new backend is listed there.

In the OpenShift Console, on the 3Scale operator, click on the 3scale Product tab. You will see that there is already a product created for the granite-3dot2-8b-instruct model.

To deploy the new product for TinyLlama, click on the Create Product button.

In the YAML view, replace the default content with the following YAML and click on create. This will create a new product for the TinyLlama model. You can see that the configuration is more complex than the backend, as it includes the mapping rules, policies, and methods that will be used by the product.

apiVersion: capabilities.3scale.net/v1beta1
kind: Product
metadata:
  name: tinyllama
  namespace: 3scale
spec:
  name: TinyLlama
  systemName: tinyllama
  metrics:
    hits:
      description: Number of API hits
      friendlyName: Hits
      unit: hit
  deployment:
    apicastHosted:
      authentication:
        userkey:
          authUserKey: Authorization
          credentials: headers
  backendUsages:
    tinyllama:
      path: /
  mappingRules:
    - httpMethod: GET
      increment: 1
      metricMethodRef: health
      pattern: /health
    - httpMethod: POST
      increment: 1
      metricMethodRef: tokenize
      pattern: /tokenize
    - httpMethod: POST
      increment: 1
      metricMethodRef: detokenize
      pattern: /detokenize
    - httpMethod: GET
      increment: 1
      metricMethodRef: models
      pattern: /v1/models
    - httpMethod: GET
      increment: 1
      metricMethodRef: version
      pattern: /version
    - httpMethod: POST
      increment: 1
      metricMethodRef: chat/completions
      pattern: /v1/chat/completions
    - httpMethod: POST
      increment: 1
      metricMethodRef: completions
      pattern: /v1/completions
    - httpMethod: POST
      increment: 1
      metricMethodRef: embeddings
      pattern: /v1/embeddings
  policies:
    - configuration:
        allow_credentials: true
        allow_headers:
          - Authorization
          - Content-type
          - Accept
        allow_methods: []
        allow_origin: '*'
      enabled: true
      name: cors
      version: builtin
    - configuration: {}
      enabled: true
      name: remove-bearer
      version: '0.1'
    - configuration: {}
      enabled: true
      name: apicast
      version: builtin
    - configuration:
        connect_timeout: 180
        read_timeout: 180
        send_timeout: 180
      enabled: true
      name: upstream_connection
      version: builtin
  methods:
    chat/completions:
      friendlyName: Chat Completions
    completions:
      friendlyName: Completions
    detokenize:
      friendlyName: Detokenize
    embeddings:
      friendlyName: Embeddings
    health:
      friendlyName: Health
    models:
      friendlyName: Models
    tokenize:
      friendlyName: Tokenize
    version:
      friendlyName: Version
  applicationPlans:
    standard:
      appsRequireApproval: false
      name: Standard Plan
      published: true

After a few seconds, you should see the new product in the list of products.

You can also switch to the 3Scale Admin Portal and check that the new product is listed there.

When a new Product is created, it is only available in a "staging" environment. This means that it is not yet available to the users. You need to publish it to make it available. This can be done in the 3Scale Admin Portal, or using the operator. In the operator view, click on the ProxyConfig Promote tab.

Click on the Create ProxyConfigPromote button.

In the YAML view, replace the default content with the following YAML and click on create. This will publish the product to the production environment.

kind: ProxyConfigPromote
apiVersion: capabilities.3scale.net/v1beta1
metadata:
  name: tinyllama
  namespace: 3scale
spec:
  productCRName: tinyllama
  production: true

In the 3Scale Admin Portal, you can see that the product is now published to production. Got to Products, select the TinyLlama product and go to Integration→Configuration. You can see that the Production APIcast environment is now available.

We must now create the API Documentation for the new product. We will again use the 3Scale operator to do this. Click on the ActiveDoc tab and click on the Create ActiveDoc button.

In the YAML view, replace the default content with the following YAML and click on Create. This will create the API documentation for the product. As the APIDoc can be complex, we already created a JSON file that contains the API documentation for the TinyLlama model and we will only need to reference it in the YAML.

apiVersion: capabilities.3scale.net/v1beta1
kind: ActiveDoc
metadata:
  name: tinyllama
  namespace: 3scale
  labels:
    app.kubernetes.io/instance: 3scale
spec:
  activeDocOpenAPIRef:
    url: 'https://raw.githubusercontent.com/redhat-gpte-devopsautomation/lb1816-summit-llm-aas/refs/heads/main/bootstrap/3scale/api_definitions/tinyllama-1dash1b-chat-v1dot0.json'
  name: tinyllama
  productSystemName: tinyllama
  published: true
  skipSwaggerValidations: true
  systemName: tinyllama

Finally, we must subscribe our user to this new product, as we did previously for Granite. Again, this is normally something you would automate as part of a deployment of a new product, but here we are going to do it in the 3Scale Admin Portal. Go to Audience→Listing, select user1 account.

In the account section, select the Service Subscriptions tab at the right.

At the bottom right of the page, click on the Subscribe button on the tinyLlama item.

Select the Default Plan and click on Create subscription.

user1 is now subscribed to the TinyLlama product. You can see that the subscription is active and the plan is set to Default Plan.

Connect to the Developer Portal at https://maas.apps.cluster-guid.guid.sandbox.opentlc.com and log in using your credentials.

Click on the See your Applications and their credentials link on the front page.

Click on the Create new application button.

Select the TinyLlama service.

Enter a name for your application, for example TinyLlama application. Click on Create Application.

Your application has been created. You can see the Endpoint URL you can use to connect to the API, the name of the model you must use in your requests, and the API key that has been generated for you. You can now use this key to access the API.

You can now test the API using the curl command in your terminal. Open a terminal in your VSCode environment (or on your laptop) and run the following command, replacing the placeholder values with the ones you got from the previous step:

curl -X 'POST' \
    '___ENDPOINT_URL___/v1/completions' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer ___API_KEY___' \
    -d '{
    "model": "___MODEL_NAME___",
    "prompt": "San Francisco is a",
    "max_tokens": 15,
    "temperature": 0
}'