Improve a Function
Each iteration surfaces a small number of cases for expert review to gather
additional context on how the task should be carried out.
Review three kinds of examples
- Low confidence: samples that are ambiguous and require expert annotation.
- High confidence: samples where there is majority consensus, but which are
worth auditing to ensure the task is well understood.
- Held-out set (optional): preserves a fixed set to ensure the task
generalizes. These samples are not seen by the optimizer while learning the task.
Correct the label or extracted fields. Add a rationale when the prompt needs a
rule that is not obvious from the label alone.
If your rationale differs from the model’s, explain your decision rule directly
to guide its task understanding. Rationales are highly valuable for transferring
the nuances of your task preferences to the model.
Run the optimizer
After review, run the optimizer. Sutro abstracts your annotations, corrections, and
rationales into generalized rules for the next iteration, helping the Function
apply your feedback to new cases rather than memorize individual examples.
Complete an iteration
Use metrics to track progress
Model consensus measures task-level consistency across various models.
User/model agreement measures how often reviewed predictions match your annotations.
Optimization metrics measure how well a model can learn the task from the annotated data.
Review the prompt diff
Review the prompt diff and validation results before starting the next
iteration.
Sutro automatically indicates when a Function appears to have learned the task
and is ready for deployment. From there, we recommend running a model sweep.
Select a deployment model
Use Model Sweep to compare supported models on your annotated rows. Sutro
reports quality and estimated cost, then lets you deploy the selected
prompt/model combination. This allows you to better understand cost and performance
trade-offs for your task, or choose between open-source and proprietary models.
Continue improving
You can return later with new production data and continue from the same
annotation history, or choose a new model to optimize for your task.
