CI smoke test

The job below needs only the langwatch CLI, curl and jq, and it exercises the management API, the data plane and one provider in a single run. Run it on a schedule so a broken credential or an expired secret shows up before a caller finds it.

Runtime: typically 3 to 8 seconds wall-clock. One gateway request + two REST round-trips.

What it checks

The management API at /api/gateway/v1/* answers with your CI token.
A virtual key is created at the scope you name and its secret is returned once.
The gateway resolves that secret and dispatches to a provider.
The response carries X-LangWatch-Gateway-Request-Id, so the request is traceable.
The key is revoked when the job ends, whether it passed or failed.

Prerequisites

An API key for the calling project, in CI_LANGWATCH_TOKEN. It needs the virtualKeys:create and virtualKeys:delete permissions.
The scope the CI key lives at, in CI_SCOPE, written as type:id. The types are org, team and project, for example org:acme.
At least one model provider credential visible from that scope, configured under Settings → Model Providers.
jq on the runner.

The script

#!/usr/bin/env bash
# ci-smoke-test.sh
set -euo pipefail

export LANGWATCH_API_KEY="$CI_LANGWATCH_TOKEN"
GATEWAY_URL="${LANGWATCH_GATEWAY_URL:-https://gateway.langwatch.ai}"
MODEL="${SMOKE_MODEL:-openai/gpt-5-mini}"
STAMP=$(date -u +%Y%m%dT%H%M%S)

cleanup() {
  if [ -n "${VK_ID:-}" ]; then
    echo "Revoking $VK_ID"
    langwatch virtual-keys revoke "$VK_ID" --format json >/dev/null || true
  fi
}
trap cleanup EXIT

# 1. Create the key.
echo "Creating a virtual key at $CI_SCOPE"
CREATE=$(langwatch virtual-keys create \
  --name "ci-smoke-$STAMP" \
  --scope "$CI_SCOPE" \
  --format json)

VK_ID=$(echo "$CREATE" | jq -r '.virtual_key.id')
VK_SECRET=$(echo "$CREATE" | jq -r '.secret')

[ -n "$VK_ID" ] && [ "$VK_ID" != "null" ] || { echo "FAIL: no virtual key id"; exit 1; }
[ -n "$VK_SECRET" ] && [ "$VK_SECRET" != "null" ] || { echo "FAIL: no secret"; exit 1; }

# 2. Report what the key enumerates. Discovery can be empty or partial while
# dispatch still works, so this is a diagnostic, not a gate. Step 3 is the check.
MODELS=$(curl -sS --fail-with-body --max-time 15 \
  -H "Authorization: Bearer $VK_SECRET" \
  "$GATEWAY_URL/v1/models" | jq -r '.data | length')

echo "Models the key enumerates: $MODELS"

# 3. Send a completion.
BODY=$(mktemp)
HEADERS=$(mktemp)
curl -sS --fail-with-body --max-time 60 \
  -H "Authorization: Bearer $VK_SECRET" \
  -H "Content-Type: application/json" \
  -D "$HEADERS" -o "$BODY" \
  -X POST "$GATEWAY_URL/v1/chat/completions" \
  -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}],\"max_tokens\":4}"

# 4. Check the response.
REQUEST_ID=$(grep -i '^x-langwatch-gateway-request-id:' "$HEADERS" \
  | awk '{print $2}' | tr -d '\r')
[ -n "$REQUEST_ID" ] || { echo "FAIL: no X-LangWatch-Gateway-Request-Id"; exit 1; }

CONTENT=$(jq -r '.choices[0].message.content' "$BODY")
[ -n "$CONTENT" ] && [ "$CONTENT" != "null" ] || { echo "FAIL: empty content"; exit 1; }

echo "OK: request_id=$REQUEST_ID"

Set SMOKE_MODEL to a model your own providers serve. Write it as provider/model, for example anthropic/claude-sonnet-4-5 or azure/<your deployment name>. See Model naming.

GitHub Actions

# .github/workflows/gateway-smoke.yml
name: Gateway smoke test
on:
  schedule:
    - cron: '*/15 * * * *'
  workflow_dispatch:

jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm i -g langwatch
      - run: bash ci-smoke-test.sh
        env:
          CI_LANGWATCH_TOKEN: ${{ secrets.CI_LANGWATCH_TOKEN }}
          CI_SCOPE: ${{ vars.CI_SCOPE }}
          LANGWATCH_GATEWAY_URL: https://gateway.langwatch.ai

Wall-clock time is one provider completion plus three short API calls, so it is dominated by the model you pick. Give the job a timeout rather than an expected duration.

Reading a failure

Failure	Cause	Where to look
`FAIL: no virtual key id`	The token lacks `virtualKeys:create`, or the scope id is wrong	Run `langwatch virtual-keys list` with the same token
`curl` exits 7	The gateway hostname does not resolve or does not answer	DNS and the load balancer. See DNS and TLS
`401 invalid_api_key`	The gateway could not resolve the secret	Troubleshooting
`FAIL: the key can route to no models`	No provider credential is visible from `CI_SCOPE`	Add a credential at that scope under Settings → Model Providers
`400 model_not_recognized`	`SMOKE_MODEL` is not a model any visible credential serves	Troubleshooting
`502 provider_error`	The gateway is healthy and the provider is not	The provider’s status page, and the body `curl` printed
`FAIL: empty content`	The provider answered with no message content	Read `$BODY` in the job log

Against a self-hosted install

export LANGWATCH_ENDPOINT="https://langwatch.acme.com"
export LANGWATCH_GATEWAY_URL="https://gateway.acme.com"
bash ci-smoke-test.sh

The two hostnames can be the same if your load balancer routes /api/* to the control plane and /v1/* to the gateway.

Alerting

Fire your own alert from the job’s failure step:

      - name: Alert on failure
        if: failure()
        run: |
          curl -X POST https://events.pagerduty.com/v2/enqueue \
            -d '{"routing_key":"${{ secrets.PD_KEY }}","event_action":"trigger","payload":{"summary":"Gateway smoke test failed","severity":"error","source":"github-actions"}}'

Run this more often than you read /readyz. The probe reports process state only, while this exercises authentication, dispatch, the provider and the spend record.

Get Started

SDK Integration

Virtual Keys & Budgets

Billing & Metering

Governance

Providers

Features

API Reference

Self-Hosting

Cookbooks

What it checks

Prerequisites

The script

GitHub Actions

Reading a failure

Against a self-hosted install

Alerting

See also

​What it checks

​Prerequisites

​The script

​GitHub Actions

​Reading a failure

​Against a self-hosted install

​Alerting

​See also

What it checks

Prerequisites

The script

GitHub Actions

Reading a failure

Against a self-hosted install

Alerting

See also