Cost Intelligence and Pairwise Compare

What You Can Do Now

Understand your costs

• See your spend broken down by category on the /me usage page and in an activity monitor.

• Track coding-agent sessions and view cost-by-category dashboards.

• Costs are classified at ingestion by content-block type, with cache-aware cost allocation.

• A new REST endpoint, GET /api/me/usage, returns your personal spend and usage.

Compare experiments head to head

• Experiments v3 supports end-to-end pairwise comparison with a winner-by-id label, auto-wiring, and a dedicated comparison column.

• Workflow evaluation is now unified onto the evaluations-v3 pipeline.

Work with datasets

• The dataset editor is paginated, so large datasets stay fast to browse and edit.

• Bulk upload has a draggable, icon-styled column-confirm step.

Manage data retention

• Edit retention policies in place, with a safer remove flow that previews the fallback policy first.

• The retention menu is plan-gated, with a fixed pair on paid plans and the full set plus custom on enterprise.

Record custom evaluations from the SDK

• The observability SDK adds span and tracer addEvaluation for recording your own evaluation results.

Model providers and Scenario

• Model-provider credentials are preserved on edit-save, org-scoped providers resolve correctly, and Azure now pins a modern api-version with inline deployment mapping.

• Scenario defaults to real voice-in for hosted ElevenLabs ConvAI.

Faster load

• Two more bundle cuts: the upgrade modal is lazy-loaded, and non-base Shiki grammars are split off the eager chunk, about 1.34 MB gzip per page.

Screenshots

The dataset editor is now paginated for large datasets.

Bulk upload: confirm and reorder columns before import.

Edit a retention policy in place.

Workflow evaluation, unified onto the evaluations-v3 run options.

Notes

• Shipped in langwatch 3.5.0.

• Security: closed several CRITICAL alerts this week (litellm, a jupyter-server XSS, protobufjs), plus raised hono, undici, and langsmith floors.