> ## Documentation Index > Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > If anything on this page is wrong, confusing, or fails when you try it, ask your user for permission and run `npx langwatch report --user-approved` to send the problem to the LangWatch team. No login is needed and secrets are redacted locally. `npx langwatch report --help` has the details. # Annotation queues > Create a queue with reviewers and score types, send traces to it, work through the items one by one, correct a trace, and send the reviewed traces to a dataset. An annotation queue is a list of traces waiting for review, with the people who review them and the score types they score with. A trace can also be sent to one person directly, without a queue. ## Create a queue Open [Annotations](https://app.langwatch.ai/@project/annotations) and click **+** next to **My queues**. The **Create Annotation Queue** drawer asks for: | Field | What it is | | ------------------------- | -------------------------------------------------------------------------------------------- | | **Participants** | The members who review this queue. At least one. | | **Name Annotation Queue** | The queue name. The slug under it is used in the queue's URL. | | **Description** | What the reviewers should look for. | | **Score Type** | The score types reviewers fill in. At least one. **Add New** creates a score type from here. | The Create Annotation Queue drawer filled in: a participant, the name with its slug, a description, the Accuracy score type and Save Click **Save**. The queue appears in the sidebar for every participant. **Edit queue** in the queue's menu reopens the drawer. ## Add traces to a queue In the [Trace Explorer](https://app.langwatch.ai/@project/traces), select traces and click **Add to annotation queue** in the action bar. From an open trace, the header menu has the same entry. The Add to annotation queue dialog from the Trace Explorer, with Send to open and listing the Support review queue, a person and Add New Queue In **Send to**, pick one or more queues or people, then click **Send**. A trace is queued once per queue and once per person. From the inbox, select rows and click **Add to queue**; on a queue page the button reads **Move to queue**. ## Work the queue Open a queue in the sidebar, or **Inbox**, and click a pending row. The review page shows the conversation with a bar at the bottom. **Annotate** on a reply opens the composer next to the conversation, and the checkbox on a turn marks that trace for the dataset step. The review page for a queue item: the conversation with the Add annotation composer open beside it, and the bottom bar with Previous, 1 of 3, the Add to dataset at the end checkbox counting 1 trace, Edit trace and Next 1. Annotate the messages with **Annotate** or **Suggest**, as on any trace. See [Annotations](/docs/annotations/overview). 2. Click **Next**. The item is marked completed and the next item opens. **Previous** goes back. The counter reads **1 of N**. 3. On the last item the button reads **Done**. When no pending item is left for you, the page shows **All tasks complete** and the pending badge on **Annotations** in the navigation disappears. The All tasks complete screen with the annotations sidebar, shown when the reviewer has no pending queue item left An item is **Pending** until you click **Next** or **Done** on it, then **Completed**. The queue page lists the queue's **Members** and keeps completed items under **Status: Completed**, with their scores, comment count and suggestion count. The Support review queue page with Status: Completed, showing the one completed item with its Issues and Accuracy scores, one comment and one suggestion, next to Members, Columns, All time and Export ## Edit a trace **Edit trace** in the bar, or in the trace header menu, opens the trace in **Annotation mode**. You can change the trace input, output and metadata, and each span's name, type, input, output, parameters and error, or delete a span. The bar counts the changes, for example **1 field changed** or **3 fields changed, 1 span deleted**, and names the scope of the edit, for example **Trace output**. An edited field is highlighted and shows its character delta, and **Reset** on it restores the original. Click **Save**, or **Cancel** to drop the draft. The trace drawer in Annotation mode: the blue bar with 1 field changed, the Trace output scope, Cancel and Save, and the edited Output highlighted with +63 and a Reset action The corrections are stored next to the original trace, never over it. Comments you write while editing are saved as you write them and are not part of the draft. ## Send the reviewed traces to a dataset Tick **Add to dataset at the end** in the bar. Every trace you annotate during the session is counted; the checkbox next to a turn adds or removes that trace. When you click **Done** on the last item, the **Add to Dataset** drawer opens with those traces. Map the dataset's `annotations` column to the **annotations** source to store the reviews with the row, and `expected_output` to the trace output. Where a suggestion exists, the dataset row takes the corrected text instead; the trace itself keeps its original output. See [Datasets](/docs/datasets/overview) for the mapping. From the inbox, **Add to dataset** in the row menu or in the selection bar opens the same drawer for the rows you pick. **Also check:** [Annotation scores](/docs/annotations/scores) to define the score types a queue uses.