Find my context sweet spot

Install via CLI

npx skills add langwatch/skills/context-sweet-spot

Skill Usage

/context-sweet-spot

Copy Full Prompt Run skill without installing

````text theme={null} Find my context sweet spot You are using LangWatch for your AI agent project. Follow these instructions. IMPORTANT: You will need a LangWatch API key. Check whether LANGWATCH_API_KEY is already set: in the process environment, which is where CI injects it, and otherwise in the project's .env file. Use that key instead of asking for a new one. Read LANGWATCH_ENDPOINT from the same places, and nothing else out of .env: if the endpoint is set, the project is on a self-hosted instance, and the CLI works against that endpoint instead of app.langwatch.ai. Use the `langwatch` CLI for everything: documentation (`langwatch docs ...`, `langwatch scenario-docs ...`) and platform operations (prompts, scenarios, evaluators, datasets, monitors, traces, analytics). Install it once with `npm install -g langwatch`, then run the `langwatch` binary directly; an unpinned `npx langwatch` re-resolves the package from the registry on every run. # Find the Context Size Where Your Sessions Stop Paying This skill answers one question with the user's own data: at what context size do their coding-agent sessions become a bad deal? Long context is not free even when it fits the window: every cache rebuild re-bills the whole context at write rates, compactions burn turns, and models degrade before their window ends. The sweet spot is where those costs start outrunning the value of the carried context. It is read-only on the platform. Locally it writes a trace export while it works and deletes it again, and leaves one report file behind. ## Step 1: Set up the LangWatch CLI Use the `langwatch` CLI for everything: documentation (`langwatch docs ...`, `langwatch scenario-docs ...`) and platform operations (prompts, scenarios, evaluators, datasets, monitors, traces, analytics). Install it once with `npm install -g langwatch`, then run the `langwatch` binary directly; an unpinned `npx langwatch` re-resolves the package from the registry on every run. Coding-agent sessions live in the user's personal LangWatch workspace by default. `langwatch login --device` signs this machine in; add `--project ` on the read commands when the sessions live in a team project instead. Use `langwatch docs ` to read documentation as Markdown. Some useful entry points: ```bash langwatch docs # Docs index langwatch docs integration/python/guide # Python integration langwatch docs integration/typescript/guide # TypeScript integration langwatch docs prompt-management/cli # Prompts CLI langwatch scenario-docs # Scenario docs index ``` Discover commands with `langwatch --help` and `langwatch --help`. List and get commands accept `--format json` for machine-readable output. Every list command takes `--limit ` to cap the rows and `--jq ` to read part of the answer. A paginated list answers with an envelope, so count its rows through the row array (`--jq '.traces | length'`), and read how many there are in all at `.pagination.total`. Bare `--jq length` counts the fields of the envelope, not the rows. Read the docs first instead of guessing SDK APIs or CLI flags. If no shell is available, fetch the same Markdown over plain HTTP. Append `.md` to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run `npx --yes langwatch report --user-approved` with a `--title` and `--summary` (or `--session `) to send it to the LangWatch team, and it directly shapes what gets fixed. No login or API key needed. Nothing is sent without `--user-approved`, and `--dry-run` prints the exact payload without sending anything. The title, summary and transcript are scrubbed locally first, by pattern: secrets and API keys, plus email addresses, phone numbers, card numbers and public IPv4 addresses. Anything no pattern matches is sent as written, including a contact address passed with `--email`. With `--session`, always run `--dry-run` first and let the user read the payload, because a transcript carries content they never reviewed. `npx --yes langwatch report --help` explains the options. ## Step 2: Collect the Sessions Pick one window and use it everywhere, because `trace export` defaults to the last 7 days. Compute a start and an end date once (30 days back to now is a good default) and pass both: ```bash langwatch trace export --origin coding_agent --format jsonl --limit 20000 \ --start-date --end-date -o coding-traces.jsonl ``` `--limit` caps the whole export, not one page, so a window with more matches than the limit gives a partial file and the sessions it drops are the ones missing from the buckets. The command reports both counts when it truncates, for example `Exported 20000 traces (48213 total)`. Raise `--limit` until the two agree, or say in the report that the buckets come from a sample of N of M traces. Report the window you used in the report, and delete `coding-traces.jsonl` once the analysis is done. Each trace carries `metadata.thread_id` (the session id) and `metadata."langwatch.source"` (which agent). Collect the distinct session ids, then for each session with enough turns to mean anything (5 or more model calls): ```bash langwatch session events --format json ``` The events are the raw material: every model call with its input, output, cache-read and cache-creation tokens, its cost, its model, plus explicit `compaction` and `rate_limit` events. ## Step 3: Compute the Economics Write a small local script (python3 or jq) over the events. Per session, compute: 1. **Peak context**: the largest (input + cache-read) of any model call, and its share of the model's context window. 2. **Cache rebuilds**: model calls whose cache-creation tokens are the bulk of their input, counted only once the session already holds cached context. The first cache-creating call of a session builds the cache rather than rebuilding it, and it pays for the context once, so it is a setup cost and not a rebuild. Count a write from the first call that follows a call with cache-read tokens, or that follows an explicit cache miss. Each rebuild re-paid for context that was already paid for, at the provider's cache-write rate. That rate is specific to the provider and the model: some price a write above fresh input, some price it the same, and some charge for cache storage by time instead. Take it from the price card of the model in question, never from a rule of thumb. 3. **Compaction count and where they landed**: a compaction late in a session marks the point where the carried context stopped fitting. 4. **Cost per model call over session lifetime**: split each session into thirds by call order and compare the average cost per call between the first and last third. 5. **Waiting time around rebuilds**: rebuilt context is also re-uploaded and re-processed, so rebuild-heavy sessions are slower per turn. Then aggregate across sessions: bucket by peak-context share (for example under 25%, 25 to 50%, 50 to 75%, over 75% of the window) and compare cost per call, rebuild rate and compaction rate between buckets. The sweet spot is the highest bucket where those three stay flat. ## Step 4: Report the Finding Write a single self-contained `context-sweet-spot-report.html` in the project root (inline CSS, no external assets) with: - **The number**: the context share where this user's sessions start degrading, stated in the first line ("your sessions stay economical up to about 55% of the window; past that, cost per turn doubles") - The bucket comparison table with cost per call, rebuild rate and compaction rate per bucket - The three most expensive sessions dissected: where the context grew, where it rebuilt, what one rebuild cost - **Concrete habits**, each tied to the evidence: when to start a fresh session instead of pushing through, what to offload to sub-agents (the sub-agent keeps its context out of the main session), whether the user's compactions happen late, after the context already crossed the sweet spot - Links to example sessions in LangWatch for every claim Also state the top finding directly in the conversation, leading with the number. The LangWatch session detail shows the same cache-health stats per session (`/me/sessions`), so name it as the place to watch the habit change. ## Common Mistakes - Do NOT judge context by peak share alone; a fat context that never rebuilds is cheap, and a modest one that rebuilds every turn is expensive. The rebuild rate carries the finding. - Do NOT compare sessions across different models as one population; window sizes and cache pricing differ. Bucket per model, then compare. - Do NOT count cache-read tokens as cost the way input tokens are; they bill at a fraction. The split is in the event rows, use it. - Do NOT count the first cache-creating call of a session as a rebuild. It builds the cache, and counting it makes every short session look rebuild-heavy and drags the whole bucket with it. - Do NOT report a threshold without the sessions behind it; every bucket statistic needs 2 or 3 example session ids. - Do NOT include sessions with fewer than 5 model calls; they carry no lifetime signal and flatten the buckets. - If the CLI returns an error, report the user-facing consequence, not the raw error text. ````

Download SKILL.md Manual installation