Connect Your Agent

Paste this prompt into your coding agent (Claude Code, Cursor, or any agent with shell access) and it performs the whole setup, from finding the endpoint to running the first suite:

Connect this codebase's AI agent to LangWatch agent simulations, so scenario suites run against it over HTTP.

Setup: the `langwatch` CLI reads LANGWATCH_API_KEY (and LANGWATCH_ENDPOINT for self-hosted installs) from the environment or a .env file. Install it with `npm install -g langwatch` if it is missing. If there is no API key, ask me for one; I can copy it from my LangWatch project settings.

1. Inspect the codebase and identify the HTTP endpoint that takes a user message and returns the agent's reply. If none exists, add one: accept a JSON body carrying the conversation messages, run the agent, return the reply text in a JSON field. Ask me for the URL where this service is deployed; prefer staging. If it only runs locally, plan to use `langwatch agent dev --port <port>` at the end instead of a public URL.

2. Wire authentication for scenario traffic. If the endpoint accepts a fixed token in a header, use that. If its normal authentication is built for human users (sessions, OAuth redirects), add a dedicated API key check for scenario traffic: read the expected value from an environment variable such as SCENARIO_API_KEY and check it against an Authorization: Bearer header.

3. Make the endpoint adopt the W3C traceparent header that LangWatch sends on every call, so the judge can read the agent's own traces:
   - If the service uses OpenTelemetry HTTP auto-instrumentation, this already happens; change nothing.
   - Python otherwise: ctx = propagate.extract(dict(request.headers)), then open the handler's root span with context=ctx.
   - TypeScript otherwise: context.with(propagation.extract(context.active(), req.headers), () => handler()).
   Confirm the agent reports its traces to the same LangWatch project that runs the scenarios (same LANGWATCH_API_KEY project).

4. Register the endpoint as an HTTP agent. Adjust bodyTemplate to the request shape the endpoint expects ({{ messages }} is the conversation as a JSON array, {{ input }} the last user message, {{ threadId }} a stable conversation id) and outputPath to the JSONPath of the reply text:

   langwatch agent create 'My Agent' --type http --config '{
     "url": "https://staging.example.com/chat",
     "bodyTemplate": "{\"thread_id\": \"{{ threadId }}\", \"messages\": {{ messages }}}",
     "outputPath": "$.reply",
     "auth": {"type": "bearer", "token": "{{ secrets.SCENARIO_API_KEY }}"}
   }'

   For the secret reference to resolve, the credential must exist as a project secret. Ask me to create SCENARIO_API_KEY under Settings > Secrets in LangWatch, or run `langwatch secret create SCENARIO_API_KEY --value "<key>"` if I hand you a test-only value.

5. Create one scenario about something this agent really handles, and a suite pairing it with the agent, then run it:

   langwatch scenario create 'Order status question' --situation "A customer asks about the status of a recent order" --criteria "The agent looks up the order before answering,The agent gives a concrete delivery estimate"
   langwatch suite create 'Smoke' --scenarios <scenario-id> --targets http:<agent-id>
   langwatch suite run <suite-id> --wait

   Write the situation and criteria from the agent's real behavior in this codebase, and include at least one criterion about a tool call or a lookup, which the judge verifies against the traces from step 3.

6. Report: the simulations page URL of my LangWatch project, what you changed in the codebase, and the result of the first run.

The steps below are the same setup by hand, and what to check when the prompt’s run needs a correction.

1. Expose an endpoint LangWatch can reach

Scenario runs call your agent from the LangWatch backend, so the agent needs a URL that backend can reach. A staging deployment is the recommended target: it exercises the real system without touching production data. Any reachable URL works. For the agent running on your own machine, langwatch agent dev opens a tunnel to a local port and points the registered agent at it for the session. See Local development. The endpoint receives one HTTP request per conversation turn and returns the agent’s reply. It does not need to know anything about LangWatch: you configure the request body and the response parsing on the LangWatch side, in step 3.

2. Authenticate the scenario traffic

Your endpoint’s existing authentication passes through the HTTP agent’s header rows and its auth block (bearer, api_key, or basic). Store the credential as a project secret and reference it as {{ secrets.NAME }}, so it stays encrypted at rest instead of readable in the agent’s configuration. When the normal authentication is built for human users (sessions, OAuth redirects), add a dedicated API key for scenario traffic instead: your server reads the expected key from an environment variable and checks it on each request, and the HTTP agent sends it from a secret. A dedicated key is also the one you revoke to close the testing path.

Also check: Testing agents behind authentication lists exactly which fields resolve secret references, and covers OAuth2 client-credentials exchange with a code agent.

3. Define the request and the response contract

The body template renders as a Liquid template on every turn, and outputPath is a JSONPath expression that picks the reply text out of the response.

Body template

{
  "thread_id": "{{ threadId }}",
  "messages": {{ messages }}
}

Variable	Value
`{{ messages }}`	The whole conversation as a raw JSON array of `{role, content}` messages
`{{ input }}`	The text of the last user message
`{{ threadId }}`	A conversation id, the same on every turn of a run
`{{ params.NAME }}`	A run parameter
`{{ traceId }}`, `{{ traceparent }}`	The turn’s trace identifiers, see Remote traces

The URL and the header values render the same variables. For the response, set outputPath to where the reply text lives. If the endpoint answers {"reply": "It ships on Tuesday."}, set outputPath to $.reply. The platform reads the reply text at that path; the response needs nothing else.

4. Adopt the trace context

The platform sends a W3C traceparent header on every call, one trace per conversation turn. When your server adopts it, the spans your agent produces land in that same trace, and the judge reads them before its verdict: tool calls, database writes, retrievals. A criterion like “the agent looked up the order before answering” then passes on evidence instead of on the reply’s wording. With standard OpenTelemetry HTTP auto-instrumentation, adoption already happens and you change nothing. Without it, extract the context from the request headers and open your handler’s root span inside it:

Python
TypeScript

from opentelemetry import propagate, trace

tracer = trace.get_tracer("my-agent")

def handle_chat(request):
    ctx = propagate.extract(dict(request.headers))
    with tracer.start_as_current_span("chat", context=ctx):
        return run_agent(request)

import { context, propagation } from "@opentelemetry/api";

app.post("/chat", (req, res) => {
  const ctx = propagation.extract(context.active(), req.headers);
  context.with(ctx, () => handleChat(req, res));
});

The agent must report its traces to the same LangWatch project that runs the scenarios. Traces sent to another project, or to another observability backend only, are invisible to the judge.

Also check: Remote traces explains what the judge sees, how long it waits for traces, and what happens when they do not arrive.

5. Register the agent and run the first scenario

In the platform, create the agent on the Agents page: type HTTP, then the URL, authentication, body template and output path from the steps above. The same registration is available as POST /api/agents. With the CLI:

langwatch agent create 'My Agent' --type http --config '{
  "url": "https://staging.example.com/chat",
  "bodyTemplate": "{\"thread_id\": \"{{ threadId }}\", \"messages\": {{ messages }}}",
  "outputPath": "$.reply",
  "auth": {"type": "bearer", "token": "{{ secrets.SCENARIO_API_KEY }}"}
}'

Then create a scenario, pair it with the agent in a suite, and run it:

langwatch scenario create 'Order status question' \
  --situation "A customer asks about the status of a recent order" \
  --criteria "The agent looks up the order before answering,The agent gives a concrete delivery estimate"

langwatch suite create 'Smoke' --scenarios <scenario-id> --targets http:<agent-id>

langwatch suite run <suite-id> --wait

6. Verify the run

Open the run in the Simulations section of your project. A connected setup shows:

The conversation transcript, with the reply text your outputPath extracted.
A trace link on each turn in the run detail, opening the agent’s own spans for that turn.
Judge reasoning that cites spans, naming the tool call or lookup that satisfied a criterion.

A trace-dependent criterion that comes back inconclusive means the traces did not arrive; see the failures below.

Common failures

Symptom	Cause	Fix
The run fails with a connection error	The URL is not reachable from the LangWatch backend: an internal hostname, a firewall, or a stopped service.	Deploy the endpoint to a reachable URL, or use `langwatch agent dev` for a local process.
Every turn fails with 401 or 403	The credential is missing or wrong: no header row or `auth` block, or the `{{ secrets.NAME }}` reference names a secret the project does not have.	Add the header row or `auth` block, and check the secret’s name and project in Settings → Secrets.
The transcript shows empty replies or raw JSON	`outputPath` does not match the response shape, so no reply text is found.	Set `outputPath` to the JSONPath of the reply text in your endpoint’s real response.
Trace-dependent criteria come back inconclusive, and turns have no trace link	The server does not adopt the incoming `traceparent`, or it reports traces to a different LangWatch project.	Adopt the context as in step 4, and point the agent’s tracing at the same project’s API key.

Get Started

Agent Simulations

Observability

Evaluations

Prompt Management

Platform

Examples & Cookbooks

Help

Connect Your Agent

Connect Your Agent

1. Expose an endpoint LangWatch can reach

2. Authenticate the scenario traffic

3. Define the request and the response contract

4. Adopt the trace context

5. Register the agent and run the first scenario

6. Verify the run

Common failures

​Connect Your Agent

​1. Expose an endpoint LangWatch can reach

​2. Authenticate the scenario traffic

​3. Define the request and the response contract

​4. Adopt the trace context

​5. Register the agent and run the first scenario

​6. Verify the run

​Common failures

Connect Your Agent

1. Expose an endpoint LangWatch can reach

2. Authenticate the scenario traffic

3. Define the request and the response contract

4. Adopt the trace context

5. Register the agent and run the first scenario

6. Verify the run

Common failures