Skip to main content
An annotation queue is a list of traces waiting for review, with the people who review them and the score types they score with. A trace can also be sent to one person directly, without a queue.

​
Create a queue

Open Annotations and click + next to My queues. The Create Annotation Queue drawer asks for:
Click Save. The queue appears in the sidebar for every participant. Edit queue in the queue’s menu reopens the drawer.

​
Add traces to a queue

In the Trace Explorer, select traces and click Add to annotation queue in the action bar. From an open trace, the header menu has the same entry.
In Send to, pick one or more queues or people, then click Send. A trace is queued once per queue and once per person. From the inbox, select rows and click Add to queue; on a queue page the button reads Move to queue.

​
Work the queue

Open a queue in the sidebar, or Inbox, and click a pending row. The review page shows the conversation with a bar at the bottom. Annotate on a reply opens the composer next to the conversation, and the checkbox on a turn marks that trace for the dataset step.
  1. Annotate the messages with Annotate or Suggest, as on any trace. See Annotations.
  2. Click Next. The item is marked completed and the next item opens. Previous goes back. The counter reads 1 of N.
  3. On the last item the button reads Done. When no pending item is left for you, the page shows All tasks complete and the pending badge on Annotations in the navigation disappears.
An item is Pending until you click Next or Done on it, then Completed. The queue page lists the queue’s Members and keeps completed items under Status: Completed, with their scores, comment count and suggestion count.

​
Edit a trace

Edit trace in the bar, or in the trace header menu, opens the trace in Annotation mode. You can change the trace input, output and metadata, and each span’s name, type, input, output, parameters and error, or delete a span. The bar counts the changes, for example 1 field changed or 3 fields changed, 1 span deleted, and names the scope of the edit, for example Trace output. An edited field is highlighted and shows its character delta, and Reset on it restores the original. Click Save, or Cancel to drop the draft.
The corrections are stored next to the original trace, never over it. Comments you write while editing are saved as you write them and are not part of the draft.

​
Send the reviewed traces to a dataset

Tick Add to dataset at the end in the bar. Every trace you annotate during the session is counted; the checkbox next to a turn adds or removes that trace. When you click Done on the last item, the Add to Dataset drawer opens with those traces. Map the dataset’s annotations column to the annotations source to store the reviews with the row, and expected_output to the trace output. Where a suggestion exists, the dataset row takes the corrected text instead; the trace itself keeps its original output. See Datasets for the mapping. From the inbox, Add to dataset in the row menu or in the selection bar opens the same drawer for the rows you pick. Also check: Annotation scores to define the score types a queue uses.
Last modified on September 6, 2026