What are setup and teardown hooks
Setup and teardown hooks are two optional functions you declare on a connected agent, which LangWatch calls around each scenario run. Setup runs before the first turn and returns the state the run starts with, such as the id of a test customer with a loyalty tier. Teardown runs after the last turn and gets that state, so it can delete what setup made. Both hooks run in your own process, the one that already answers the agent’s turns, so they can use your database, your secrets and your internal APIs the same way the agent does. LangWatch calls them over the connection the SDK already holds open. A run goes in this order: setup, then the turns, then teardown.Hooks work on agents you connect with
connectAgent (TypeScript) or @langwatch.connect_agent (Python), and only on scenario runs. HTTP agents, code agents, experiments, the Test button and langwatch agent run never call them.Declare the hooks
Declare the hooks on the agent thatconnectAgent or @langwatch.connect_agent returns. See Connect your agent for the connect function itself.
A turn can update the state by returning
{ output, state } in TypeScript or langwatch.AgentReply(output, state=...) in Python. LangWatch keeps the newest state, and a turn that returns none leaves it unchanged. A setup that returns no value starts the run with an empty state.
Teardown’s return value is optional. If it returns an object, the run shows it under Returned by teardown.
In Python, a hook’s keyword arguments join the agent’s run parameters. If the agent and a hook declare the same name with a different type or default, the SDK logs a parameters_invalid warning and does not register the agent. In TypeScript, the hooks receive the parameters you declare in connectAgent.
In the Python SDK,
state is a reserved turn field. Do not name a run parameter state: the argument receives the scenario state, and the name is not registered as a parameter.What do the simulated user and the judge see
The simulated user and the judge get the current state on every turn, as a system message that starts withCurrent test state:, so they can check the agent’s answers against the real test data. That message is part of their prompts, which means the state costs tokens and is sent to your model provider.
What happens when a hook fails
Each hook has a 60 second timeout. The agent’s per-turn timeout does not apply to hooks, and you cannot change the hook timeout per scenario.
A failed setup is not scored because a fault in your test data is not the agent failing, so it should not lower the pass rate. Teardown never changes the score, because the score is about the agent’s conversation.
Where to see what a hook did
Open the run in Agent Testing or in Simulations. For each hook, the run shows its status (ok, failed, timed_out, skipped or not_run), duration, state or returned object, and error, with a Trace button for the hook’s own trace. The run keeps the first 1,000 characters of an error, and keeps no copy of a state over 64 kilobytes.
Rerun and keep the test data
To inspect the data a run made, open the run in Simulations and choose Rerun and keep data. The rerun skips teardown, and the run shows “Teardown skipped”. The action appears only for a run that recorded a teardown. “Teardown not run” means LangWatch called teardown but no instance of the agent was online to take the call, so teardown did not happen even though no one asked to skip it.Limits
- Not yet available: teardown for a run whose scenario process is killed. Name your test data with the run id, so you can find what is left behind.
- Not yet available: teardown for a run you cancel during its turns.
Best practices
Create test data in setup and remove it in teardown. Data made anywhere else, such as inside a turn, is not in the state, so teardown cannot know about it. Return references, not data. Put ids and URLs in the state, never whole records or secrets. The state goes into every call, into the prompts of the simulated user and the judge, and into the run record, and it is capped at 64 kilobytes. Name all test data with the run id. Userun.scenarioRunId (run.scenario_run_id in Python) in a name or a tag, as the setup above does with label. Runs of a suite can overlap, so a unique name keeps one run’s data apart from another’s, and lets you find leftovers by name if a teardown is missed.
Make teardown safe to run twice and safe on partial state. Teardown runs after a failed setup too, so the state it gets can be missing or incomplete. Check that each id exists before you delete it, treat “already gone” as success, and remove everything you can.
Return what teardown did. Return what you deleted and what you left behind, so the run shows whether anyone needs to clean up by hand. If setup made something you have no way to delete, say so in the returned object.
Keep hooks fast. A hook that runs past 60 seconds counts as failed. Seed slow data once, ahead of time, and have setup pick from it instead of building it on every run.
Throw in setup when the data cannot be made. The run is then marked Not scored and stays out of the pass rate. If setup catches the error and returns an empty state instead, the agent is graded against data that does not exist.
Keep secrets out of the state and the returned object. Read credentials from your own process’s environment, because the state and the returned object are stored on the run and sent to the model provider.
Use run parameters to pick the test data. For example, one scenario can cover several loyalty tiers when setup reads the loyalty_tier parameter and the scenario text reads the same value as {{ params.loyalty_tier }}. See Run parameters.
Also check: Connect your agent, Run parameters, Scenarios, Results, TypeScript agent SDK reference, Python agent SDK reference.