Backup Operations

Setup Automated Monthly Backup

# Requires S3 bucket with IAM role attached to bastion EC2 instance
./setup-litemaas-backup-cronjob.sh litellm-rhpds maas-db-backup

Manual Database Backup

# Backup LiteMaaS + LiteLLM shared database
oc exec -n litellm-rhpds litellm-postgres-0 -- \
  pg_dump -U litellm litellm | gzip > litellm-backup-$(date +%Y%m%d).sql.gz

# Upload to S3
aws s3 cp litellm-backup-$(date +%Y%m%d).sql.gz \
  s3://maas-db-backup/litemaas-backups/

List and Verify Backups

aws s3 ls s3://maas-db-backup/litemaas-backups/ --human-readable | sort -r

Backup & Restore

LiteMaaS v0.4.0 includes a built-in backup and restore capability available directly from the admin portal. It covers the LiteMaaS database — all users, API keys, subscriptions, models, branding settings, and usage data.

What Gets Backed Up

Data	Included
Users and roles	Yes
Virtual API keys	Yes
Model subscriptions	Yes
Registered models and capabilities	Yes
Branding settings (logos, title, subtitle)	Yes
Usage analytics and spend logs	Yes
System settings and defaults	Yes
LiteLLM proxy config (pass-through endpoints, etc.)	Separate — use LiteLLM admin UI

Taking a Backup — via UI

1. Log in to the LiteMaaS portal as admin: litellm-prod-frontend.apps.maas.redhatworkshops.io
2. Navigate to Admin → Tools → Backup
3. Click Create Backup — a .sql.gz file is generated
4. Click Download to save it locally

Taking a Backup — via API

# Create a backup
curl -X POST https://litellm-prod-admin.apps.maas.redhatworkshops.io/api/v1/admin/backup/create \
  -H "Authorization: Bearer <admin-jwt>"

# List available backups
curl https://litellm-prod-admin.apps.maas.redhatworkshops.io/api/v1/admin/backup/list \
  -H "Authorization: Bearer <admin-jwt>"

# Download a backup file
curl https://litellm-prod-admin.apps.maas.redhatworkshops.io/api/v1/admin/backup/download/<filename> \
  -H "Authorization: Bearer <admin-jwt>" \
  -o backup.sql.gz

Restoring from Backup — via UI

1. Navigate to Admin → Tools → Backup
2. Select a backup from the list and click Restore, or upload a backup file
3. Confirm the restore — the backend restarts automatically when complete

Automated Backup — Bastion Cronjob (S3)

For scheduled backups to S3, the rhpds.litemaas collection includes a setup script. This runs monthly on the bastion host and uploads compressed PostgreSQL dumps to an S3 bucket, keeping the last 12 (1 year of monthly backups).

# Run on bastion — installs monthly backup cronjob to S3
./setup-litemaas-backup-cronjob.sh litellm-rhpds my-s3-bucket

# Prerequisites: AWS CLI + IAM instance profile on bastion with S3 write access

Manual Database Backup

# Dump the full database (litemaas_db + litellm_db share one PostgreSQL instance)
oc exec -n litellm-rhpds litellm-postgres-0 -- \
  pg_dump -U litellm litellm | gzip > litemaas-backup-$(date +%Y%m%d).sql.gz

# Upload to S3
aws s3 cp litemaas-backup-$(date +%Y%m%d).sql.gz s3://my-bucket/backups/

Usage Analytics

There are two ways to query usage statistics, each with different trade-offs:

• LiteLLM API — no cluster access needed, returns spend data. Works for anyone with the master key.
• PostgreSQL direct — requires oc exec on the postgres pod, but returns token counts which the API does not expose without an Enterprise licence.

Unique users & spend — last 30 days (API)

Use this when you need a quick user count or want to see who is spending the most. No cluster access required.

# Set your master key first
export LITELLM_URL=https://litellm-prod.apps.maas.redhatworkshops.io
export MASTER_KEY=sk-...

curl -sk "$LITELLM_URL/spend/users?start_date=$(date -v-30d '+%Y-%m-%d')&end_date=$(date '+%Y-%m-%d')" \
  -H "Authorization: Bearer $MASTER_KEY" | python3 -c "
import sys, json
users = json.load(sys.stdin)
users = users if isinstance(users, list) else users.get('data', [])
print(f'Unique users (last 30d): {len(users)}')
for u in sorted(users, key=lambda x: x.get('spend', 0), reverse=True)[:10]:
    print(f'  {u.get(\"user_email\", u.get(\"user_id\",\"?\"))} -- \${u.get(\"spend\",0):.2f}')
"

Spend by model — last 30 days (API)

Shows cost breakdown per model. Useful for understanding which models are driving spend.

curl -sk "$LITELLM_URL/global/spend/models?start_date=$(date -v-30d '+%Y-%m-%d')&end_date=$(date '+%Y-%m-%d')" \
  -H "Authorization: Bearer $MASTER_KEY" | python3 -c "
import sys, json
for m in sorted(json.load(sys.stdin), key=lambda x: x.get('total_spend',0), reverse=True):
    print(f'  {m[\"model\"]}: \${m[\"total_spend\"]:.2f}')
"

Total spend (API)

curl -sk "$LITELLM_URL/global/spend?start_date=$(date -v-30d '+%Y-%m-%d')&end_date=$(date '+%Y-%m-%d')" \
  -H "Authorization: Bearer $MASTER_KEY" | python3 -m json.tool

Unique users & tokens — monthly breakdown (DB)

Use this when you need token counts. The LiteLLM API does not expose tokens without an Enterprise licence — query the database directly instead.

# Requires oc exec access to the postgres pod
oc exec -n litellm-rhpds litellm-postgres-0 -- \
  psql -U litellm -d litellm -t -c "
SELECT
  TO_CHAR(DATE_TRUNC('month', \"startTime\"), 'YYYY-MM') as month,
  COUNT(DISTINCT \"user\")  as unique_users,
  SUM(total_tokens)        as total_tokens,
  COUNT(*)                 as requests
FROM \"LiteLLM_SpendLogs\"
WHERE \"startTime\" >= NOW() - INTERVAL '3 months'
GROUP BY 1
ORDER BY 1 DESC;"

Top models by token consumption (DB)

oc exec -n litellm-rhpds litellm-postgres-0 -- \
  psql -U litellm -d litellm -t -c "
SELECT
  model,
  COUNT(*)               as requests,
  SUM(total_tokens)      as total_tokens,
  SUM(prompt_tokens)     as prompt_tokens,
  SUM(completion_tokens) as completion_tokens
FROM \"LiteLLM_SpendLogs\"
WHERE \"startTime\" >= NOW() - INTERVAL '30 days'
GROUP BY model
ORDER BY total_tokens DESC
LIMIT 10;"

Daily active users — last 7 days (DB)

oc exec -n litellm-rhpds litellm-postgres-0 -- \
  psql -U litellm -d litellm -t -c "
SELECT
  TO_CHAR(DATE_TRUNC('day', \"startTime\"), 'YYYY-MM-DD') as day,
  COUNT(DISTINCT \"user\") as unique_users,
  COUNT(*)                as requests,
  SUM(total_tokens)       as tokens
FROM \"LiteLLM_SpendLogs\"
WHERE \"startTime\" >= NOW() - INTERVAL '7 days'
GROUP BY 1
ORDER BY 1 DESC;"