" can significantly boost performance and recall for nuanced tasks
3. **Structured outputs**: Use schemas when you need the final answer in a specific format while preserving a "thinking canvas" to explore the problem space
# Sampling Parameters
Source: https://docs.sutro.sh/concepts/sampling-parameters
Sampling parameters control the token sampling process, allowing for fine-grained customization of the model's output.
## How to Use Sampling Parameters
To use custom sampling parameters, pass a dictionary of your desired parameters to the `sampling_params` argument in your SDK call.
Each model has a set of default sampling parameters that are recommended by the model creator for best performance. When you provide your own dictionary, **it is merged with these defaults**, and any values you specify will **always take precedence**.
### Example of Overriding Defaults
Let's assume the model's default parameters are:
* `temperature`: 0.6
* `top_k`: 20
* `max_tokens`: 32768
If you want a more creative response (higher temperature) and a shorter output, you can provide just those specific overrides:
```python theme={null}
import sutro as so
# We only specify the parameters we want to change.
sampling_overrides = {
"temperature": 0.9,
"max_tokens": 512
}
results = so.infer(
inputs=...,
sampling_params=sampling_overrides
)
```
The final parameters used by the model for this request will be a combination of your overrides and the defaults:
* `temperature`: **0.9**
* `max_tokens`: **512**
* `top_k`: **20**
## Parameter Reference
We support any vLLM compatible sampling parameters. Please reference [their sampling parameters class](https://docs.vllm.ai/en/stable/api/vllm/sampling_params.html#vllm.sampling_params.SamplingParams) for a complete list of valid parameters.
Note: we do not set defaults for every parameter in `vllm.SamplingParams`, in this case the value used falls back to the vLLM default.
## Default Parameters by Model Family
Different model families have different recommended default parameters. Here is a reference for the base configurations. Each configuration is set based off the given lab's recommended settings, e.g. [Qwen3 14B](https://huggingface.co/Qwen/Qwen3-14B#best-practices) > Sampling Parameters.
### [Llama Family](https://huggingface.co/meta-llama)
| Parameter | Default Value |
| -------------------- | ------------- |
| `temperature` | `0.75` |
| `top_p` | `1` |
| `max_tokens` | `4096` |
| `repetition_penalty` | `1.0` |
### [Qwen 3 Family](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f)
The Qwen family has different defaults depending on whether the model is used for standard generation (`Non-Thinking`) or for tasks that require reasoning (`Thinking`).
**Non-Thinking Defaults**
| Parameter | Default Value |
| -------------------- | ------------- |
| `temperature` | `0.7` |
| `top_p` | `0.8` |
| `top_k` | `20` |
| `max_tokens` | `4096` |
| `repetition_penalty` | `1.0` |
**Thinking Defaults**
| Parameter | Default Value |
| -------------------- | ------------- |
| `temperature` | `0.6` |
| `top_p` | `0.95` |
| `top_k` | `20` |
| `max_tokens` | `4096` |
| `repetition_penalty` | `1.0` |
Certain Qwen Mixture-of-Experts (MoE) models use a higher `max_tokens` default. They are as follows:
Defaults to 16,384:
* `qwen-3-30b-a3b`
* `qwen-3-235b-a22b`
Defaults to 32,768:
* `qwen-3-30b-a3b-thinking`
* `qwen-3-235b-a22b-thinking`
This allows for sufficient length and robustness in the model's reasoning process.
### [Gemma Family](https://huggingface.co/collections/google/gemma-3-release-67c6c6f89c4f76621268bb6d)
| Parameter | Default Value |
| -------------------- | ------------- |
| `temperature` | `0.95` |
| `top_p` | `0.95` |
| `top_k` | `64` |
| `max_tokens` | `4096` |
| `repetition_penalty` | `1.0` |
# Structured Outputs
Source: https://docs.sutro.sh/concepts/structured-outputs
Reference documentation for Structured Outputs.
# Structured Outputs
Sutro's SDK allows you to enforce a JSON schema for the outputs of your inference job. This can be useful if you want to ensure that the outputs of your inference job are in a specific format, or if you want to extract specific information from the outputs.
To specify a schema for the outputs of your inference job, you can pass a Pydantic Model or a JSON schema to the `output_schema` parameter in the SDK. If you pass a JSON schema, it must follow the json-schema.org specification.
## Example with Pydantic Model
```python theme={null}
import sutro as so
from pydantic import BaseModel
texts = [
"The airplane is flying at an altitude of 30,000 feet.",
"The capital of France is Paris.",
"The best way to cook a steak is on a grill."
]
class AviationOutput(BaseModel):
is_aviation_related: bool
answer_justification: str
results = so.infer(texts, system_prompt="Is the following sentence aviation related? Please provide a justification for your answer.", output_schema=AviationOutput)
print(results)
```
## Example with JSON Schema
```python theme={null}
import sutro as so
texts = [
"The airplane is flying at an altitude of 30,000 feet.",
"The capital of France is Paris.",
"The best way to cook a steak is on a grill."
]
json_schema = {
"type": "object",
"properties": {
"is_aviation_related": {
"type": "boolean"
},
"answer_justification": {
"type": "string"
}
},
"required": ["is_aviation_related", "answer_justification"]
}
results = so.infer(texts, system_prompt="Is the following sentence aviation related? Please provide a justification for your answer.", output_schema=json_schema)
print(results)
```
# System Prompts
Source: https://docs.sutro.sh/concepts/system-prompts
Reference documentation for System Prompts.
# System Prompts
System prompts are used to provide consistent, task-specific instructions to the model. They are used to ensure that the model is oriented towards the task at hand.
This is useful in batch inference jobs, where the task is typically uniform across all inputs.
## Basic System Prompt Example
For example, let's say you are classifying customer support tickets.
You can provide a consistent system prompt to the model when you run the batch inference job, like so:
```python theme={null}
import sutro as so
customer_support_conversations = ... # (code to retrieve customer support conversations from a database)
json_schema = {
"type": "object",
"properties": {
"was_handled_properly": {
"type": "boolean"
}
}
}
system_prompt = """
You are a overseeing customer support agents.
Your job is to review dialogues between customers and customer support agents, and ensure that they were handled properly.
You will be provided a dialogue, and you should respond with True if the dialogue was handled properly, and False otherwise."
"""
results = so.infer(
inputs=dialogues,
system_prompt=system_prompt,
output_schema=json_schema
)
print(results)
```
# Large Scale Embedding Generation with Qwen3 0.6B
Source: https://docs.sutro.sh/examples/large-scale-embeddings
Easily (and inexpensively) create a semantic search index of over 4M document chunks from Apple's patent literature, using Sutro
10 min read
\~1-2 hour project
\~15
Beginner
## Overview
In this example, we're going to demonstrate how to easily embed over 4M document chunks to create a searchable index. The documents we'll be embedding is the entire corpus of Apple's patent literature. By the end of this guide we'll be able to search for things like "wireless charging technology", "biometric authentication methods", or even complex queries like "patents related to reducing battery consumption in mobile devices" - and get relevant patent results in milliseconds.
### Why Embeddings and Vector Search Matter
Traditional keyword search breaks down when users don't know the exact terminology - searching for "making phones last longer" won't find documents about "battery optimization." Embeddings solve this by converting text into vectors that better capture semantic meaning, enabling search that actually understands intent rather than just matching strings. What once required teams of search quality experts can now be implemented in an afternoon for around \$15, as we'll demonstrate by making 30,000 Apple patents (split into over 4M+ document chunks) semantically searchable.
## Data Source
The source for the corpus of patent documents will be Google BigQuery, where we can query for the full text and all relevant metadata with the below query. The results for this query (the base for our embeddings) can be found at this [HuggingFace Dataset](https://huggingface.co/datasets/sutro/apple-patents-bigquery)
```sql theme={null}
SELECT DISTINCT
p.publication_number,
p.application_number,
p.country_code,
p.kind_code,
title.text as patent_title,
abstract.text as patent_abstract,
claims.text as patent_claims,
description.text as patent_description,
p.filing_date,
p.publication_date,
p.grant_date,
p.priority_date,
p.inventor,
assignee.name as assignee_name,
p.cpc,
p.family_id,
FROM
`patents-public-data.patents.publications` AS p,
UNNEST(title_localized) AS title,
UNNEST(abstract_localized) AS abstract,
UNNEST(claims_localized) AS claims,
UNNEST(description_localized) AS description,
UNNEST(assignee_harmonized) as assignee
WHERE
LOWER(assignee.name) LIKE '%apple inc%'
AND p.kind_code IN ('B1', 'B2')
AND p.country_code = 'US'
AND p.grant_date IS NOT NULL
AND title.language = 'en'
AND abstract.language = 'en'
AND claims.language = 'en'
AND description.language = 'en'
ORDER BY
p.grant_date DESC
```
We'll store the results from that query into a Polars `DataFrame` using the below snippet. Polars is great for efficiently manipulating large datasets.
```python theme={null}
# Authenticate
credentials = service_account.Credentials.from_service_account_info(
json.loads(os.environ['SERVICE_ACCOUNT_JSON'])
)
project_id = 'xxxxx'
client = bigquery.Client(project=project_id, credentials=credentials)
# Run query
query_job = client.query(query_full_text)
results = query_job.result()
patents_df = pl.from_arrow(results.to_arrow())
```
## Document Chunking
After retrieving and storing the patent results, we'll need to chunk all the relevant sections into smaller sections of text.
### Why Chunking is Essential
Chunking is essential when working with embeddings mainly because of **semantic precision**, essentially meaning it is important for the reader of the results to be able to interpret them with the correct contextual meaning:
Smaller chunks allow for more precise retrieval of relevant information, however there is a critical balance to strike:
**Too small chunks (e.g., 50-100 tokens):**
* ✅ High precision - returns exactly what matches
* ❌ Loss of contextual meaning - "battery life" might miss "wasn't that good" in the next sentence
* ❌ Fragmented results - you might need to retrieve multiple chunks to get a complete idea
* ❌ More vectors to store and search (higher costs)
**Too large chunks (e.g., 2000+ tokens):**
* ✅ Rich context preserved - full patent claims or entire technical descriptions
* ✅ Fewer vectors to manage (lower costs)
* ❌ Diluted relevance - a chunk about "display technology" might rank poorly for "OLED" even if it contains relevant OLED information buried within
* ❌ Multiple concepts per chunk - retrieval becomes less precise
**The sweet spot (typically 200-800 tokens) depends on:**
* Your content type (patent claims are self-contained; descriptions are narrative)
* Search intent (looking for specific facts vs. understanding concepts)
* Embedding model characteristics (some models better preserve semantics in longer sequences)
### Chunking Strategy
For this example, we're using a simple fixed-size chunking strategy with overlap, but there are many approaches to consider:
* **Fixed-size chunking**: Simple and predictable, splits text every N characters/tokens
* **Semantic chunking**: Uses NLP to find natural boundaries (sentences, paragraphs, sections)
* **Recursive chunking**: Hierarchically splits documents while preserving structure
* **Corpus-specific chunking**: For patents, you might chunk by claims, abstract, description sections; for code, you might designate chunks by function boundaries
You can view the implementation of our chunking strategy in the collapsable below. We chose to write this ourselves for simplicity, but we know teams like [Unstructured](https://unstructured.io/) also do a great job here!
```python theme={null}
def chunk_text(text: str, chunk_size: int = 2000, overlap: int = 400) -> List[str]:
"""Split text into overlapping chunks for embedding"""
if not text:
return []
chunks = []
start = 0
text_length = len(text)
while start < text_length:
end = min(start + chunk_size, text_length)
chunk = text[start:end]
chunks.append(chunk)
start += chunk_size - overlap
return chunks
def prepare_patent_for_embedding(row: Dict) -> List[Dict]:
"""Convert patent row to embedding-ready documents"""
patent_id = row['publication_number']
base_metadata = {
'patent_id': patent_id,
'title': row['patent_title'],
'assignee': row['assignee_name'],
'grant_date': str(row['grant_date']),
'inventors': row['inventor'],
'cpc_codes': row['cpc'],
'family_id': row['family_id'],
}
documents = []
# Title as separate document
documents.append({
'doc_id': hashlib.md5(f"{patent_id}_title".encode()).hexdigest(),
'patent_id': patent_id,
'section': 'title',
'text': row['patent_title'],
'metadata': base_metadata
})
# Abstract as separate document
if row['patent_abstract']:
documents.append({
'doc_id': hashlib.md5(f"{patent_id}_abstract".encode()).hexdigest(),
'patent_id': patent_id,
'section': 'abstract',
'text': row['patent_abstract'],
'metadata': base_metadata
})
# Claims - chunk if long
if row['patent_claims']:
claims_chunks = chunk_text(row['patent_claims'])
for i, chunk in enumerate(claims_chunks):
documents.append({
'doc_id': hashlib.md5(f"{patent_id}_claims_{i}".encode()).hexdigest(),
'patent_id': patent_id,
'section': 'claims',
'chunk_index': i,
'text': chunk,
'metadata': base_metadata
})
# Description - chunk into smaller pieces
if row['patent_description']:
desc_chunks = chunk_text(row['patent_description'])
for i, chunk in enumerate(desc_chunks):
documents.append({
'doc_id': hashlib.md5(f"{patent_id}_description_{i}".encode()).hexdigest(),
'patent_id': patent_id,
'section': 'description',
'chunk_index': i,
'text': chunk,
'metadata': base_metadata
})
return documents
```
```python theme={null}
# Process patents for embedding
all_documents = []
for row in patents_df.iter_rows(named=True):
all_documents.extend(prepare_patent_for_embedding(row))
```
## Preparing Data for Sutro
Once we have our patent documents split into chunked sections, we can then pass them to Sutro to transform them into embeddings!
Since we're working with such a large amount of data, we'll write everything to a single Parquet object, upload it to our own object storage (Amazon S3 in this example), and hand Sutro a presigned HTTPS GET URL. This keeps the submission request small and avoids loading the full corpus into the SDK process. See [Presigned S3 Inputs](/python-sdk/presigned-s3-inputs) for the full contract.
```python theme={null}
docs_df = pl.DataFrame({
'doc_id': [doc['doc_id'] for doc in all_documents],
'patent_id': [doc['patent_id'] for doc in all_documents],
'section': [doc['section'] for doc in all_documents],
'text': [doc['text'] for doc in all_documents],
'metadata': [json.dumps(doc['metadata']) for doc in all_documents]
})
local_file = "apple_patents_documents.parquet"
docs_df.write_parquet(local_file, compression='snappy')
print(f"Saved {len(docs_df)} documents to {local_file}")
```
Now we'll upload the Parquet file to S3 and generate a presigned GET URL:
```python theme={null}
import boto3
s3 = boto3.client("s3")
bucket = "your-bucket"
key = "apple-patents/apple_patents_documents.parquet"
s3.upload_file(local_file, bucket, key)
presigned_url = s3.generate_presigned_url(
ClientMethod="get_object",
Params={"Bucket": bucket, "Key": key},
ExpiresIn=6 * 60 * 60, # leave headroom for the worker's download
)
```
## Running the Embedding Job
Once our data is uploaded, we can use it to run our embedding job. For this example, we chose to use **Qwen3-Embedding-0.6B**; it has a great balance of performance due to only having 595M parameters, but still performs very well on relevant tasks like retrieval and re-ranking. See the [MTEB leaderboard](https://huggingface.co/spaces/mteb/leaderboard) for a more in-depth and numerical comparison.
Now we'll kick off our embedding job with the code below:
```python theme={null}
job_id = sutro.infer(
presigned_url,
column="text",
model="qwen-3-embedding-0.6b",
job_priority=1,
)
sutro.await_job_completion(job_id, obtain_results=False)
```
> **Note:** We're using `job_priority=1` here. Sutro currently has a notion of job priorities, which is essentially how we designate job SLAs. Currently, we have two priorities:- **Priority 0**: Prototyping jobs (for small scale testing, targeted at \<= 10m completion time)- **Priority 1**: Production oriented jobs (large scale jobs, targeted at 1hr completion time)More details about job priorities can be [found here](https://docs.sutro.sh/concepts/job-priority).
Once we have our job started, we wait for it to complete using `await_job_completion(...)`. This will periodically poll for the job's status until its complete; alternatively, we can monitor via [Sutro's web UI](https://app.sutro.sh).
We also disabled automatically fetching the results. The default `get_job_results()` helper materializes the entire result set as JSON, which isn't viable for 4M embeddings. Instead, once the job succeeds, we use `download_job_results()` to pull the results down as a single Parquet artifact:
```python theme={null}
local_results = sutro.download_job_results(job_id, include_inputs=True)
job_results_df = pl.read_parquet(local_results)
print(f"Total patents: {len(job_results_df)}")
print(f"Columns: {job_results_df.columns}")
# Total patents: 4039988
# Columns: ['inputs', 'job-76844041-b2bf-4248-9603-b7f750231b34']
```
The download streams to disk with a progress bar and, if interrupted, resumes where it left off when rerun. If you need the presigned URLs themselves (e.g. to download from a different machine), use `results_download_url()` — see [Presigned Results URLs](/batch-api-reference/job-results-url) for details.
If you want to play around with the embeddings yourself, the entire set can be found [here on HuggingFace](https://huggingface.co/datasets/sutro/apple-patents-embeddings)!
## Loading into the Vector Database
Now we get to upload the 4M embeddings we just pulled down from Sutro into a vector database, so that we can search over the entire corpus. When uploading, we want to preserve all the metadata associated with each chunk, so that we can correctly attribute a retrieved vector to the right patent and section within the patent.
```python theme={null}
# The embeddings column in the results is named after the job
vector_col_name = job_id
# Combine the embeddings with docs_df containing attributing IDs and metadata
combined_df = docs_df.with_columns(
job_results_df[vector_col_name]
)
```
We chose **Qdrant** as our vector DB of choice for this example; its performant and easy enough to get started with. However, there are many options out there that are well adapted to different use cases, notable ones being:
* **TurboPuffer** - Great for multi-tenant architectures with many tenants
* **Chroma** - Simple and developer-friendly
* **pgvector** - If you're already using PostgreSQL
Since we have such a large dataset, we want to upload using batches:
```python theme={null}
# We're using an im-memory DB here for convenience, but Qdrant Cloud
# can be faster to upload to and will persist the embeddings as well
client = QdrantClient(":memory:")
collection_name = "apple_patents_collection"
# Create the collection, inferring vector size from the first
# row of the DataFrame
client.recreate_collection(
collection_name=collection_name,
vectors_config=models.VectorParams(
size=len(combined_df[vector_col_name][0]),
distance=models.Distance.COSINE
),
)
BATCH_SIZE = 8192
# Loop through the DataFrame in batches
for i in tqdm(range(0, len(combined_df), BATCH_SIZE), desc="Uploading to Qdrant"):
batch_df = combined_df.slice(i, BATCH_SIZE)
points = [
models.PointStruct(
id=row['doc_id'],
vector=row[vector_col_name],
payload={
'patent_id': row['patent_id'],
'section': row['section'],
'text': row['text'],
**json.loads(row['metadata'])
}
) for row in batch_df.to_dicts()
]
client.upload_points(
collection_name=collection_name,
points=points,
wait=True,
parallel=4
)
print(f"Finished uploading all {len(combined_df)} points.")
```
## Searching Your Patents
Now that we have our embeddings loaded, we can search over them!
```python theme={null}
def search_patents(query_text, top_k=5):
# Pick your real time provider of choice, we have heard great
# things (latency & consistency wise) about Vertex, but Baseten
# is the easier choice
# https://www.baseten.co/library/qwen3-06b-embedding/
query_embedding = real_time_api(
text=query_text,
model="qwen-3-embedding-0.6b"
)
# Search in Qdrant
results = client.search(
collection_name=collection_name,
query_vector=query_embedding,
limit=top_k
)
for result in results:
print(f"Patent: {result.payload['patent_id']}")
print(f"Section: {result.payload['section']}")
print(f"Score: {result.score:.3f}")
print(f"Text: {result.payload['text'][:200]}...")
print("-" * 80)
# Let's test it out!
search_patents("wireless charging efficiency improvements")
```
## Example Queries We Tried
### `wireless charging efficiency improvements`
### 1. Patent US-11994681-B2 - description
**Score:** 0.946
into waveguide 26 exhibits a relatively wide effective field of view 70 . Switchable reflective layer 56 may be
switched between the first and second states at a speed greater than the response speed of the human eye (e.g.,
greater than 60 Hz, greater than 120 Hz, greater than 240 Hz, greater than 1 kHz, greater than 10 kHz, etc.) so that
a user at eye box 24 ( FIG. 2 ) is unable to separately perceive each state and instead perceives a single effective
field of view 70 . In this way, image light 22 may be coupled into waveguide 26 and provided to the eye box with a
wider effective field of view than would otherwise be provided to the eye box. As an example, fields of view 62 and
66 may each be 30 degrees, 25 degrees, between 25 and 35 degrees, less than 45 degrees, etc., whereas field of view
70 is 60 degrees, between 55 and 65 degrees, greater than 45 degrees, or any other desired angle greater than field
of view 62 or field of view 66 .
FIG. 6 sh
***
### 2. Patent US-11054882-B2 - description
**Score:** 0.874
ments thereof are shown by way of example in the drawings and will herein be described in detail. It should be
understood, however, that the drawings and detailed description thereto are not intended to limit the embodiments to
the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and
alternatives falling within the spirit and scope of the appended claims. The headings used herein are for
organizational purposes only and are not meant to be used to limit the scope of the description. As used throughout
this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than
the mandatory sense (i.e., meaning must). Similarly, the words “include”, “including”, and “includes” mean
“including, but not limited to.” As used herein, the terms “first,” “second,” etc. are used as labels for nouns that
they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unle
***
### 3. Patent US-11380077-B2 - description
**Score:** 0.874
ed in units of pressure). Using the intensity of a contact as an attribute of a user input allows for user access to
additional device functionality that may otherwise not be accessible by the user on a reduced-size device with
limited real estate for displaying affordances (e.g., on a touch-sensitive display) and/or receiving user input
(e.g., via a touch-sensitive display, a touch-sensitive surface, or a physical/mechanical control such as a knob or
a button).
As used in the specification and claims, the term “tactile output” refers to physical displacement of a device
relative to a previous position of the device, physical displacement of a component (e.g., a touch-sensitive
surface) of a device relative to another component (e.g., housing) of the device, or displacement of the component
relative to a center of mass of the device that will be detected by a user with the user's sense of touch. For
example, in situations where the device or the component of the device is in
***
### 4. Patent US-11128157-B2 - description
**Score:** 0.869
scan for communications (e.g., Bluetooth communications from secondary power receiving device 24 B) at a first rate.
In response to the notification, the primary power receiving device 24 A may use the antenna to scan for
communications at a second rate that is faster than the first rate. By increasing the rate of scanning for
communications, the primary power receiving device 24 A may receive any communications from secondary power
receiving device 24 B at an earlier time than if the rate was not increased. In the event that the newly added
object is not a supported wireless power receiving device, primary power receiving device 24 A will not actually
receive the expected wireless communication. However, in this case, the faster scan rate may time-out after a
predetermined length of time (e.g., after the predetermined length of time the scan rate will revert back to the
first scan rate) without any adverse effects.
Additional action may be taken by primary power receiving d
***
### 5. Patent US-10742297-B2 - description
**Score:** 0.869
, where for a period larger than 1.60 ms, 240 occasions can be configured. Note that, as above, 60 subframes may be
used, thus I CQI/PMI has incrementations of 60 in Tables 4 and 5.
TABLE 4
mframe
Value of
Value of
offset
I CQI/PMI
N pd,frame
N OFFSET,CQI
N OFFSET,mframe
### `biometric authentication using facial recognition`
### 1. Patent US-10928697-B1 - description
**Score:** 0.907
tween the first transparent layer and the second transparent layer, the transparent light-producing layer having
light-emitting diodes that are arranged in an array, and a controller for causing the transparent light-producing
layer to display information using the light-emitting diodes.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a side-view representation of an example configurable transparent structure for lighting and/or display.
FIG. 2A is a side-view representation of an example configurable transparent structure for lighting and/or display
that employs an edge-lit light guide plate.
FIG. 2B is a side-view representation of an example configurable transparent structure for lighting and/or display
that employs an organic light-emitting diode (OLED) display layer.
FIG. 2C is a side-view representation of an example configurable transparent structure for lighting and/or display
that employs a micro-light-emitting di
***
### 2. Patent US-11150734-B2 - description
**Score:** 0.907
ion mechanism 210 and may couple the deflection mechanism 210 to a surface. The surface may be a cover glass of an
electronic device, a housing of the electronic device, and so on. Because the surface is coupled to the deflection
mechanism 210 , as the deflection mechanism 210 deflects, the surface may also deflect and provide a haptic output.
Although the haptic structure 200 is specifically discussed with respect to an electronic device, the haptic
structure 200 may be used with other devices including mechanical devices and electrical devices, as well as
non-mechanical and non-electrical devices such as described herein.
FIG. 3A illustrates another example haptic structure 300 for an electronic device. The haptic structure 300 may be
referred to as a cantilevered beam structure as one end of the deflection mechanism 310 is coupled to, machined
from, or otherwise integrated with a substrate of the haptic structure 300 while the other end of the defle
***
### 3. Patent US-11868258-B2 - description
**Score:** 0.907
he bytes in a cache block. Thus, the coherency controller 24 may cause other agents to invalidate the cache block.
If an agent has the cache block modified, the agent may supply the modified cache block to the request agent.
Otherwise, the agents may not supply the cache block.
The coherency controller 24 may be configured to read the directory entry for the address of the request (block 220
). Based on the cache states in the directory entry, the coherency controller 24 may be configured to generate
snoops. More particularly, if a given agent may have a modified copy of the cache block (e.g., the given agent has
the cache block in exclusive or primary state) (block 222 , “yes” leg), the coherency controller 24 may generate a
snoop forward-Dirty only (SnpFwdDonly) to the agent to transmit the cache block to the request agent (block 224 ).
As mentioned above, the SnpFwdDonly request may cause the receiving agent to transmit the cache block if the data is
modified, but o
***
### 4. Patent US-11393258-B2 - description
**Score:** 0.906
more activations required to initiate biometric authentication such that the electronic device is enabled to
implement the respective function.
In some examples, the electronic device (e.g., 2300 , 2400 ) displays, on the display, the prompt to provide the one
or more activations of the button (e.g., 2304 , 2404 ) at a first position in the biometric authentication interface
(e.g., 2322 , 2420 ). Outputting a prompt requesting that one or more activations of the button be provided provides
the user with feedback about the current state of the device and provides visual feedback to the user indicating
what steps the user must take in order to proceed with a particular function using the device. Providing improved
visual feedback to the user enhances the operability of the device and makes the user-device interface more
efficient (e.g., by helping the user to provide proper inputs and reducing user mistakes when operating/interacting
with the device) which, additionally, red
***
### 5. Patent US-11764907-B2 - description
**Score:** 0.906
henever the product of residual SINRs is below 1, the error probability for both use cases is reduced and
consequently, achieving a certain target rate requires less power.
FIG. 8 , which is motivated by the finding above, shows a graph 800 of the achievable outage probability as a
function of the transmit power when a single (so-called one-shot) transmission is performed with low rate.
Similarly, as before, we are interested in the performance of the single device using a dedicated resource, and of
two devices sharing the slot. To achieve fairness, in the case of two users, their transmit powers are adjusted so
that the sum matches that of a single user. The particular values are found by solving a min-max problem. For
example, the maximum of the tuple (P er 1 , er 2 ) is minimized according to the following set of equations:
### `battery thermal management`
### 1. Patent US-11700035-B2 - description
**Score:** 0.838
nating elements 68 (e.g., as shown by arrow 162 of FIG. 16 ). Manufacturing equipment 148 may, for example, use
lasers to activate or create a seed layer on dielectric resonating elements 68 . Manufacturing equipment 148 may
then deposit conductive material over the activated portions of dielectric resonating elements 68 . The conductive
material may form conductive structures 86 V and 86 H (e.g., for feed probes 100 V and 100 H of FIG. 6 ) and/or
parasitic elements for the antennas.
At step 178 , manufacturing equipment 148 may surface-mount connectors 123 onto the connector contact pads 168 of
substrate 72 (e.g., as shown by arrow 166 of FIG. 17 ).
At step 180 , manufacturing equipment 148 may dice substrate 180 into individual antenna modules 120 and may add
corresponding shielding structures to the antenna modules (e.g., as shown by arrow 170 of FIG. 17 ). The shielding
may serve to isolate electronic components 150 fr
***
### 2. Patent US-11297732-B2 - description
**Score:** 0.838
illustrate in top plan views various stages of a partially assembled exemplary outer housing foot with integrated
fan assembly according to various embodiments of the present disclosure.
FIG. 6 illustrates in side cross-sectional view an exemplary electronic device having a low profile thermal flow
assembly according to various embodiments of the present disclosure.
FIG. 7 illustrates in top perspective view an exemplary impeller and fin stack arrangement for an integrated fan
assembly according to various embodiments of the present disclosure.
FIGS. 8A and 8B illustrate in bottom plan views exemplary foot and scroll geometries for an integrated fan assembly
according to various embodiments of the present disclosure.
FIG. 9 illustrates a flowchart of an exemplary method of cooling an electronic device according to various
embodiments of the present disclosure.
FIG. 10 illustrates in block diagram format an exemplary computing devi
***
### 3. Patent US-11605274-B2 - description
**Score:** 0.838
ement 212 , but has to still select “yes” to end the call. In this way, a user cannot accidentally end the call with
the emergency service. In some cases, block 214 is an alternate option for block 206 , where the layout of the text
is different and the call time is shown.
In some examples, the audio messages can be configured to continue looping as long as the call with the emergency is
active or until the user selects the “Stop Recorded Message” UI element. If the emergency service responder hangs
up, the call will end. In some instances, the workflow may begin to reestablish the call (or make another call) if
the user remains nonresponsive. Alternatively, if the call ends, and the emergency service responder calls back, the
device 102 may answer the call and begin playing the already generated audio message. This audio message could also
be looped until the user selects the “Stop Recorded Message” UI element. Further, in some cases, the device 102 may
detect an indicat
***
### 4. Patent US-11994681-B2 - title
**Score:** 0.838
Optical systems with reflective prism input couplers
***
### 5. Patent US-10719225-B2 - description
**Score:** 0.838
ronic device (e.g., the first software application), such as a background application, a suspended application, or a
hibernated application. Thus, the user can perform operations that are not provided by the application currently
displayed on the display of the electronic device (e.g., the second software application) but are provided by one of
the currently open applications (e.g., displaying a home screen or switching to a next software application using
gestures for a hidden application launcher software application).
In some embodiments, the first software application is ( 804 ) an application launcher (e.g., a springboard). For
example, as shown in FIG. 7A , the application launcher displays a plurality of application icons 5002 that
correspond to a plurality of applications. The application launcher receives a user-selection of an application icon
5002 (e.g., based on a finger gesture on touch screen 156 ), and in response to receiving the user-selection,
launches an
### `haptic feedback for touch interfaces`
### 1. Patent US-12193062-B2 - abstract
**Score:** 0.897
Disclosed are techniques for reducing likelihood of Random Access Channel (RACH) transmission blockages and thereby
facilitate an initial access procedure for new radio (NR) unlicensed spectrum (NR-U) operation in a fifth generation
(5G) wireless communication system including an NR node. In some embodiments, a parameter generated by a gNB and
received by a UE indicates that, from among a set of consecutive RACH Occasions (ROs), a gap is available for
performing a listen-before-talk (LBT) procedure before commencing a RACH transmission.
***
### 2. Patent US-11468890-B2 - description
**Score:** 0.897
mediums), memory controller 122 , one or more processing units (CPUs) 120 , peripherals interface 118 , RF circuitry
108 , audio circuitry 110 , speaker 111 , microphone 113 , input/output (I/O) subsystem 106 , other input control
devices 116 , and external port 124 . Device 100 optionally includes one or more optical sensors 164 . Device 100
optionally includes one or more contact intensity sensors 165 for detecting intensity of contacts on device 100
(e.g., a touch-sensitive surface such as touch-sensitive display system 112 of device 100 ). Device 100 optionally
includes one or more tactile output generators 167 for generating tactile outputs on device 100 (e.g., generating
tactile outputs on a touch-sensitive surface such as touch-sensitive display system 112 of device 100 or touchpad
355 of device 300 ). These components optionally communicate over one or more communication buses or signal lines
103 .
As used in the specification and claim
***
### 3. Patent US-11733656-B2 - description
**Score:** 0.897
rt of the second user interface screen; detect (e.g., with detecting unit 2516 ) a contact on the touch-sensitive
surface unit (e.g., touch-sensitive surface unit 2510 ) at the affordance for revealing an edit option; and in
response to detecting the contact at the affordance for revealing an edit option, enable display (e.g., with display
enabling unit 2508 ), on the display unit (e.g., display unit 2502 ), of a delete affordance in association with the
first user interface preview image as part of the second user interface screen.
In some embodiments, displaying the delete affordance comprises translating the first user interface preview image
on-screen.
In some embodiments, the processing unit 2506 is further configured to: after displaying the delete affordance as
part of the second user interface screen, detect (e.g., with detecting unit 2516 ) a contact on the touch-sensitive
surface unit (e.g., touch-sensitive surface unit 2510 ) at the delete affordance disp
***
### 4. Patent US-11379113-B2 - title
**Score:** 0.897
Techniques for selecting text
***
### 5. Patent US-11039417-B2 - description
**Score:** 0.897
at least SIB2 from the system information.
At 1832 , the UE may compute a paging frame identifier I PF and a paging occasion identifier I PO based on the DRX
cycle T, the parameter nB and the Range\_UE\_ID.
At 1835 , the base station transmits a paging message 1840 for the link-budget-limited UE. The paging message is
included in a paging frame and paging occasion consistent with the previously transmitted values of DRX cycle T,
parameter nB and Range\_UE\_ID.
At 1845 , the UE wakes up for every subframe consistent with the computed paging frame identifier and computed
paging occasion identifier, and checks the PDCCH of the subframe for the presence of P-RNTI.
At 1850 , if the UE determines that P-RNTI is present in the PDCCH, the UE decodes resource allocation information
from the PDCCH, and checks PDSCH resource block(s) identified by the allocation information, e.g., PDSCH resource
blocks in the same subframe as the PDCCH.
At 1855 , if the paging
Interestingly, none of the results for our queries seem very good! We imagine that this is mainly due to a few things.
1. The language used in our queries is *very* different from the langauge used in the patent documents, so the similarity between the query-document pairs is generally not great. There are well known fixes to this problem, commonly [HyDE](https://arxiv.org/abs/2212.10496) is used to generate queries that are more similar to langauge in the real document, and thus retrieve better results for the same source query.
2. Under retrieving: we're currently only retrieving the first 5 documents, which is not very many; if we retrieved more documents, its likely we'd have more relevant snippets in our results.
3. Not reranking: Combining a higher top\_k with a re-ranking step can lead to the finding the most relevant set documents. These two techniques used together can prove to be very powerful and is common with many folks we talk to who use vector search in production.
## Scale, Cost & Speed Breakdown
### Scale
* **Chunk count**: 4.04M
* **Input token count**: 879.5M
### Cost Breakdown
* **BigQuery query**: \~\$6
* **Sutro embedding generation**: \$8.80
* **Total: \~\$14.80**
### Time
* **Job completion time**: 44 minutes
## Conclusion
In this guide, we've demonstrated how Sutro makes it trivial to:
* Go from source documents to a searchable index in under 2 hours
* Generate high-quality embeddings using state-of-the-art models
* Build a semantic search system that can easily be productionized
The entire pipeline - from data extraction to searchable index - can be run using a Jupyter notebook and cost under \$20. Sutro handled all the worker fan out, inference, and fault tolerance automatically.
### Next Steps
* Try different embedding models for your case
* Experiment with different techniques to improve retrieval quality
* Hybrid search (combining embeddings with keyword search)
* HyDE
* Over retrieval and re-ranking
* Productionize this workflow as part of an event driven pipeline that creates new indices for every X event (say a new user signing up)
### Resources
* [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard)
* [Apple Patents Dataset](https://huggingface.co/datasets/sutro/apple-patents-bigquery)
* [Embeddings Dataset](https://huggingface.co/datasets/sutro/apple-patents-embeddings)
* [Sutro Documentation](https://docs.sutro.sh)
# LLM-as-a-Judge - Iteratively Improve Models, AI Apps, and Agents Offline
Source: https://docs.sutro.sh/examples/llm-as-a-judge
Learn three LLM-as-a-judge techniques to improve models and agents without human feedback
25 min read
3 hour project
\~\$100
Medium
### Overview
If you're training LLMs, building AI apps, or developing agents - you'll need some way to evaluate their performance. This can come at different points in the development lifecycle: during initial development, as it's in production to detect regressions or drift, or as you seek to upgrade to newer models and/or configurations.
Because the outputs of LLMs, AI apps, and agents are typically subjective or open-ended, there are essentially only three ways performance can be measured:
1. Human data labeling
2. Real user feedback
3. LLM-as-a-judge
4. ✨ *Vibes* ✨
Human data labeling is the most time-consuming and extremely expensive. It's also not reproducible - different human labelers can have different interpretations and preferences for the same data.
Real user feedback is often the most helpful, but often not something that's available until you have a lot of users. You'll likely want a way to bootstrap evaluate your application before it hits production and scale.
[Much](https://x.com/justinstorre/status/1964029634796015685) [has](https://x.com/swyx/status/1963725773355057249) [been](https://x.com/lennysan/status/1963688207280955839) [written](https://x.com/sh_reya/status/1963988545057456138) about vibes-based evals - a debate we don't intend to get into here.
This leaves us with option 2: LLM-as-a-judge, a surprisingly practical, effective, and rigorous method1 when done correctly.
In this guide, we'll show you a few techniques for using LLM-as-a-judge to iteratively improve your models, apps, and agents without the need for human feedback. Using our ensemble approaches, you'll also see how we can avoid the biases that are present in human labelers or any one evalautor model.
We'll implement them using the [Sutro SDK](/python-sdk/setup) which will make experiments fast, cheap, scalable, collaborative, easy to run - and dare we say fun?!
### The Goal
We're really going to *nerd out* on this one: our task today is to determine which open-source model family is best at creating ELI5 explanations of paper abstracts from the pre-print server [Arxiv](https://arxiv.org/). ELI5 stands for "Explain Like I'm 5", popularized by the now-defunct dataset [ELI5](https://research.facebook.com/publications/eli5-long-form-question-answering/) from Facebook.
This task may seem trivial, but it's a great demonstration of LLM-as-a-judge for a few reasons:
1. It requires high-level world-knowledge and understanding of complex technical concepts
2. It requires the ability to map a high-level concept to a low-level explanation and communicate it effectively
3. It's extremely subjective, so there is no realistic way a ground-truth label set can be created
But don't let the specificity of the task mislead you - **the following techniques can be applied to almost any scenario where you're evaluating model, AI app, or agent performance**.
### Getting the data
We'll grab the current snapshot of the [Arxiv metadata from Kaggle](https://www.kaggle.com/datasets/Cornell-University/arxiv) and sample 100,000 rows.
```python theme={null}
import sutro as so
import polars as pl
import json
rows = []
with open('arxiv-metadata-oai-snapshot.json', 'r') as f:
for line in f:
data = json.loads(line)
id, categories, abstract = data['id'], data['categories'], data['abstract']
if categories and abstract:
rows.append({'id': id, 'categories': categories, 'abstract': abstract})
df = pl.DataFrame(rows)
df.write_parquet('arxiv-metadata-id-categories-abstract-100000.snappy.parquet', compression='snappy')
```
Let's briefly inspect the data to get a sense of what we're working with.
```python theme={null}
import matplotlib.pyplot as plt
import seaborn as sns
# read the parquet file
df = pl.read_parquet('arxiv-metadata-id-categories-abstract-100000.snappy.parquet')
# cast categories to list by separating on space
df = df.with_columns(pl.col('categories').str.split(' ').alias('categories_list'))
# create high-level categories list by splitting on '.' for each item in categories_list
def get_high_level_categories(categories_list):
return list(set([item.split('.')[0] for item in categories_list]))
df = df.with_columns(pl.col('categories_list').map_elements(get_high_level_categories).alias('high_level_categories_list'))
# explode the high_level_categories_list
df = df.explode('high_level_categories_list')
# aggregate by high_level_categories_list and count the number of papers
df = df.group_by('high_level_categories_list').agg(pl.count()).sort('count', descending=True)
# create a better looking bar chart of the high_level_categories_list
plt.figure(figsize=(12, 8))
sns.set_theme(style="whitegrid")
# Convert to pandas for seaborn
df_pandas = df.to_pandas()
# Create a more attractive bar plot
ax = sns.barplot(
data=df_pandas,
x='high_level_categories_list',
y='count',
palette='viridis',
order=df_pandas.sort_values('count', ascending=False)['high_level_categories_list']
)
# Improve the styling
plt.xticks(rotation=45, ha='right')
plt.title('ArXiv High-Level Categories Distribution', fontsize=16, fontweight='bold', pad=20)
plt.xlabel('High-Level Categories', fontsize=12, fontweight='bold')
plt.ylabel('Number of Papers', fontsize=12, fontweight='bold')
# Improve layout
plt.tight_layout()
# Save with higher DPI for better quality
plt.savefig('high_level_categories_distribution.png', dpi=300, bbox_inches='tight')
```
As you can see most of the high-level categories are computer science, math, and physics - with a long-tail of other more esoteric categories.
### Generating the explanations
As of this writing, we serve 4 model "families" on the Sutro platform: Llama, Qwen, Gemma, and GPT-OSS.
As a general rule of thumb, it's better to use smaller models for applications and larger models for evaluations of the applications. Generally speaking, any of the text models on the Sutro platform should be able to generate the ELI5 explanations, but the larger models should be more adept at evaluating them. Larger models generally contain more world knowledge and are better at understanding complex technical concepts.
So, we'll use smaller or medium-sized variants of each model family for the explanations, and their larger counterparts for the evaluations.
It's incredibly simple to generate the explanations using the Sutro SDK. For a sanity check, we'll run the first 1000 rows through each model using protoptying (`p0`) jobs.
```python theme={null}
import sutro as so
df = pl.read_parquet('arxiv-metadata-id-categories-abstract-100000.snappy.parquet')
system_prompt = """
Explain the abstract to me like I'm a 5 year old.
Return the explanation, and nothing else.
"""
results = so.infer(
df[0:1000],
column='abstract',
model=["llama-3.1-8b", "qwen-3-14b", "gemma-3-12b-it", "gpt-oss-20b"],
system_prompt=system_prompt,
)
```
This kicks of four jobs in parallel each of which take just a few minutes to complete. We can inspect the results quickly and easily in the Sutro Web UI:
An example result from GPT-OSS 20B looks like this:
**Input Arxiv Abstract:**
`We report an experimental design where the position and resonance frequency
of the Nitrogen Vacancy (NV) in a diamond are correlated with the room temperature.
A simple model trained on the interpolated correlation data predicts both quantities.
The predictive tracking of the NV's location enables continuous operation of the NV
quantum computer under ambient conditions for a week without recalibration.`
**Output ELI5 Explanation:**
`We did a test to see how a tiny thing in a diamond moves and makes a special
sound when the room is warm. We made a smart guess that tells us where it is and
how it sounds. This lets a tiny computer in the diamond keep working all week
without having to fix it.`
Scaling to the 100,000 row sample is as simple as removing the slicing of the dataframe, and setting the job priority to 1.
```python theme={null}
... previous code ...
results = so.infer(
df,
column='abstract',
model=["llama-3.1-8b", "qwen-3-14b", "gemma-3-12b-it", "gpt-oss-20b"],
system_prompt=system_prompt,
job_priority=1,
)
```
This will kick off four jobs in parallel, each of which should take less than an hour to complete.
In our case, here were the run stats:
| Model | Duration | Input Tokens | Output Tokens | Cost |
| -------------- | ---------- | ------------ | ------------- | ------ |
| Llama 3.1 8B | 35 minutes | 27M | 16.3M | \$1.03 |
| Qwen 3 14B | 48 minutes | 25.4M | 11.9M | \$5.76 |
| Gemma 3 12B IT | 35 minutes | 24.1M | 18.8M | \$9.16 |
| GPT-OSS 20B | 30 minutes | 29.5M | 25.8M | \$1.03 |
Note this is *much* faster and less expensive compared to more latency-focused or closed-model providers. We won't try to provide exact numbers or apples-to-apples comparisons here, but you can generally expect Sutro to be about 20x faster, and up to 10x cheaper than alternatives - which is extremely important when dealing with large scale data processing and evaluation.
Let's pull them down the explanations and append them to our original dataframe.
```python theme={null}
jobs = {
"llama-3.1-8b": "job-c5227a15-3928-479e-a988-bef4231a9f5b",
"qwen-3-14b": "job-a921b731-5c24-46e5-9433-c363a25777b6",
"gemma-3-12b-it": "job-c6925843-97c8-45f8-a10a-48b17be6fa86",
"gpt-oss-20b": "job-2b9b70f4-fd27-40d4-9f79-276b5b34df9e",
}
for model, job_id in jobs.items():
results = so.get_job_results(job_id)
if model == 'gpt-oss-20b': # automatically unpacks final response to content field b/c it's a reasoning model, so we handle this one differently
results = results.with_columns(pl.col('content').alias(model))
results = results.drop(['content', 'reasoning_content'])
else:
results = results.with_columns(pl.col('inference_result').alias(model))
results = results.drop(['inference_result'])
df = pl.concat([df, results], how='horizontal')
df.write_parquet('arxiv-metadata-id-categories-abstract-100000-explanations.parquet')
```
### Evaluating the explanations
As mentioned earlier, it's likely that larger models will be better at evaluating the explanations. This is because they contain more world knowledge and have more free parameters with which to reason.
In some cases, you may have a "trusted" model that you want to use for all evaluations. But in this, case how do we know which model is the best judge? And how do we know the larger model in a specific family won't be biased towards its smaller sibling due to similarities in the underlying training data?
To combat these biases, we'll use a larger model from each of the four families to evaluate the explanations from the three smaller models. Consequentially, this means that each smaller model will be evaluated by three larger models from other families. In traditional ML, this is known as ensemble modeling - using the responses of multiple models to make a single prediction.
```python theme={null}
from pydantic import BaseModel
df = pl.read_parquet('arxiv-metadata-id-categories-abstract-100000-explanations.parquet')
eval_pairs = {
"llama-3.1-8b": ["qwen-3-32b", "gemma-3-27b-it", "gpt-oss-120b"],
"qwen-3-14b": ["llama-3.3-70b", "gemma-3-27b-it", "gpt-oss-120b"],
"gemma-3-12b-it": ["llama-3.3-70b", "qwen-3-32b", "gpt-oss-120b"],
"gpt-oss-20b": ["llama-3.3-70b", "qwen-3-32b", "gemma-3-27b-it"],
}
system_prompt = """
You are a judge. You will be shown an arXiv paper abstract and an explanation of the abstract intended for a 5 year old.
Your job is to evaluate the explanation. You should evaluate according to the following criteria:
- technical accuracy
- conceptual accuracy
- clarity
- effectiveness of communication
- overall quality
Return a score between 0 and 100, and nothing else.
"""
class Evaluation(BaseModel):
score: int
for model, eval_models in eval_pairs.items():
so.infer(
df,
column=["Abstract: ", "abstract", " Explanation: ", model],
model=eval_models,
system_prompt=system_prompt,
output_schema=Evaluation,
name=[eval_model + '_evals_' + model for eval_model in eval_models], # name the jobs so we can easily identify them
job_priority=1,
)
```
This is a very tight way to kick off the evaluation pairs! One thing you'll notice is that we're creating names for the jobs so we can easily identify them later.
As our experimental setup grows, accurately attaching relevant metadata to each job becomes increasingly important for historical analysis, tracking, and collaboration. Sutro makes this easy.
We can use the same eval mapping to pull down the results and append them to our original dataframe.
```python theme={null}
... previous code ...
jobs = so.list_jobs()
for model, eval_models in eval_pairs.items():
for eval_model in eval_models:
job_name = eval_model + '_evals_' + model
job = next((job for job in jobs if job['name'] == job_name), None)
if job:
job_id = job['job_id']
results = so.get_job_results(job_id)
if eval_model == 'gpt-oss-120b': # automatically unpacks final response to content field b/c it's a reasoning model, so we handle this one differently
results = results.with_columns(pl.col('content').alias('score'))
results = results.with_columns(pl.col('score').struct.field('score').alias(job_name))
results = results.drop(['content', 'reasoning_content', 'score'])
else:
results = results.with_columns(pl.col('score').alias(job_name))
# drop all columns except job_name
results = results.drop([col for col in results.columns if col != job_name])
df = pl.concat([df, results], how='horizontal')
```
Each of the jobs took around an hour to complete, and cost between 4-10 dollars each. This comes out to around 50-80 dollars for all 1.2M evals spread across the 12 runs (100k rows each).
We can now see how each model performed against each of its three evaluators. Let's plot the results in a heatmap.
```python theme={null}
import pandas as pd
df = pl.read_parquet('arxiv-metadata-id-categories-abstract-100000-explanations-evals.parquet')
i_vals = sorted(list(set([eval_model for eval_models in eval_pairs.values() for eval_model in eval_models])))
j_vals = sorted(list(eval_pairs.keys()))
matrix = np.zeros((len(i_vals), len(j_vals)))
for model, eval_models in eval_pairs.items():
for eval_model in eval_models:
job_name = eval_model + '_evals_' + model
scores = df[job_name]
matrix[i_vals.index(eval_model), j_vals.index(model)] = scores.mean()
# wrap in DataFrame with labels
df_matrix = pd.DataFrame(matrix, index=i_vals, columns=j_vals)
plt.figure(figsize=(10, 8))
sns.heatmap(df_matrix, annot=True, fmt=".2f", cmap="RdYlGn", cbar=True, square=True)
plt.title("Score Matrix")
plt.xlabel("ELI5 Model")
plt.ylabel("Evaluator Model")
plt.savefig("llm-as-a-judge-score-matrix.png", dpi=300, bbox_inches='tight')
plt.show()
```
These are interesting and revealing results. We can see that the GPT-OSS 20B model has the strongest overall performance as reviewed by the three larger models in other families. The GPT-OSS 120B model is the also the harshest evaluator, giving the lowest scores to the other model families.
The situation is inverted for the Llama models, where Llama 3.1 8B is the weakest performing model as reviewed by the three larger models in other families, yet Llama 3.3 70B is the most generous evaluator of the other model families.
We already likely have our answer: GPT-OSS 20B is the best model for this task of the models we evaluated. At this point, we could move onto another set of evals to optimize our prompt, sampling parameters, or structured output schema. Instead, we'll run one more set of evals to confirm our hypothesis.
### Using relative (ranking-based) evaluations
In our previous evals, we used larger models to evaluate the ELI5 explanations in isolation: just showing the abstract and the explanation to the evaluator model and asking it to score the explanation on a scale of 0 to 100.
But since we're trying to understand which model is best for our task, it makes more sense to compare relative performance directly. This is where relative evaluations come in.
We'll now show each evaluator all three ELI5 explanations at once, and ask it to rank them from best to worst.
```python theme={null}
df = pl.read_parquet('arxiv-metadata-id-categories-abstract-100000-explanations-evals.parquet')
inverse_eval_pairs = {
"qwen-3-32b": ["llama-3.1-8b", "gemma-3-12b-it", "gpt-oss-20b"],
"gemma-3-27b-it": ["llama-3.1-8b", "qwen-3-14b", "gpt-oss-20b"],
"gpt-oss-120b": ["llama-3.1-8b", "qwen-3-14b", "gemma-3-27b-it"],
"llama-3.3-70b": ["qwen-3-14b", "gemma-3-12b-it", "gpt-oss-20b"],
}
system_prompt = """
You are a judge. You will be shown an arXiv paper abstract and three explanations of the abstract intended for a 5 year old. They will be labeled as A, B, and C.
Your job is to rank the explanations from best to worst. You should evaluate according to the following criteria:
- technical accuracy
- conceptual accuracy
- clarity
- effectiveness of communication
- overall quality
You should return the ranking as a list of the labels A, B, and C, and nothing else.
The first item in the list should be the label of the best explanation, the second item in the list should be the label of the second best explanation, and the third item in the list should be the label of the worst explanation.
"""
class Ranking(BaseModel):
ranking: list[str]
for eval_model, eli5_models in inverse_eval_pairs.items():
so.infer(
df,
column=["Abstract: ", "abstract", " Explanation A: ", eli5_models[0], " Explanation B: ", eli5_models[1], " Explanation C: ", eli5_models[2]],
model=eval_model,
system_prompt=system_prompt,
output_schema=Ranking,
name=[eval_model + '_relative_evals'],
job_priority=1,
)
```
These jobs were a bit more token heavy, but there were only four of them. However, in total all four only jobs cost around \$35.
We can pull down the results and append them to our original dataframe.
```python theme={null}
df = pl.read_parquet('arxiv-metadata-id-categories-abstract-100000-explanations-evals.parquet')
relative_eval_jobs = {
"qwen-3-32b": 'job-8f460128-6426-4078-b355-7790a00c2dfa',
"gemma-3-27b-it": 'job-5a1eed4a-44b2-4d74-a6cd-d2424e0df113',
"gpt-oss-120b": 'job-64b2a381-5f25-4917-b2c9-78140da3c475',
"llama-3.3-70b": 'job-2860b900-31e5-4160-b8f1-4162767215c0',
}
jobs = so.list_jobs()
for eval_model, job_id in relative_eval_jobs.items():
eval_job_name = eval_model + '_relative_evals'
results = so.get_job_results(job_id)
if eval_job_name == 'gpt-oss-120b_relative_evals': # automatically unpacks final response to content field b/c it's a reasoning model, so we handle this one differently
results = results.with_columns(pl.col('content').struct.field('ranking').alias(eval_job_name))
results = results.drop(['content', 'reasoning_content'])
elif eval_job_name == 'llama-3.3-70b_relative_evals': # 2 of the results had problematic json formatting, so we handle this one differently
results = results.with_columns(
pl.when(pl.col("inference_result").str.contains('"C", "B", "A'))
.then(pl.lit('{"ranking": ["C", "B", "A"]}'))
.otherwise(pl.col("inference_result"))
.alias("inference_result")
)
results = results.with_columns(pl.col('inference_result').str.json_decode().alias('inference_result'))
results = results.with_columns(pl.col('inference_result').struct.field('ranking').alias(eval_job_name))
results = results.drop(['inference_result'])
else:
results = results.drop([col for col in results.columns if col != 'ranking'])
results = results.with_columns(pl.col('ranking').alias(eval_job_name))
results = results.drop(['ranking'])
eli5_map = {"A": inverse_eval_pairs[eval_model][0], "B": inverse_eval_pairs[eval_model][1], "C": inverse_eval_pairs[eval_model][2]}
results = results.with_columns(pl.col(eval_job_name).map_elements(lambda x: [eli5_map[y] for y in x]).alias(eval_job_name))
df = pl.concat([df, results], how='horizontal')
df.write_parquet('arxiv-metadata-id-categories-abstract-100000-explanations-relative-evals.parquet')
```
A little more data wrangling, and we can determine the win rate percentage matrix.
```python theme={null}
df = pl.read_parquet('arxiv-metadata-id-categories-abstract-100000-explanations-relative-evals.parquet')
eval_models = inverse_eval_pairs.keys()
eli5_models = list(set([model for eval_models in inverse_eval_pairs.values() for model in eval_models]))
eli5_model_matrix = {}
for i in eli5_models:
for j in eli5_models:
if i != j:
eli5_model_matrix[(i, j)] = 0
for eval_model in eval_models:
eval_job_name = eval_model + '_relative_evals'
eval_results = df[eval_job_name].to_list()
for eval_result in eval_results:
if eval_result is not None:
eli5_model_matrix[(eval_result[0], eval_result[1])] += 1
eli5_model_matrix[(eval_result[1], eval_result[2])] += 1
eli5_model_matrix[(eval_result[0], eval_result[2])] += 1
win_rate_pct_matrix = {}
for i in eli5_models:
for j in eli5_models:
if i != j:
num = eli5_model_matrix[(i, j)]
den = num + eli5_model_matrix[(j, i)]
win_rate_pct_matrix[(i, j)] = num / den if den else float('nan')
df = pd.DataFrame(np.nan, index=eli5_models, columns=eli5_models, dtype=float)
for (i, j), v in win_rate_pct_matrix.items():
df.loc[i, j] = v
np.fill_diagonal(df.values, np.nan) # optional: blank diagonal
plt.figure(figsize=(10, 8))
sns.heatmap(df, annot=True, fmt=".2f", cmap="RdYlGn", cbar=True, square=True)
plt.title("Win Rate Percentage Matrix (cell = P(row beats column))")
plt.xlabel("ELI5 Model (column)")
plt.ylabel("ELI5 Model (row)")
plt.savefig("llm-as-a-judge-win-rate-percentage-matrix.png", dpi=300, bbox_inches='tight')
plt.tight_layout()
plt.show()
```
This time, we're pooling the results of the three evaluators to determine the win rate percentage matrix. We only care about the head-to-head comparisons of the ELI5 models.
Once again, even with the context of the three evaluators, we can see that GPT-OSS 20B is hands down the best model, beating all the others in head-to-head comparisons. Llama is by far the worst, getting beaten down by the others, often by a wide margin. Almost 9/10 times the GPT-OSS 20B model beats it.
We like to think we're *fancy mathematicians* here at Sutro, so we'll finally use the Elo rating system to determine the best model. This is often used for human skill ratings in games like chess, and more recently to rank responses from LLMs on websites like the [LM Arena Leaderboard](https://lmarena.ai/leaderboard).
```python theme={null}
import math
import numpy as np
import pandas as pd
def bt_elo_from_pair_counts(
pair_counts: dict[tuple[str, str], int],
ties: dict[tuple[str, str], int] | None = None,
laplace: float = 0.5,
max_iter: int = 1000,
tol: float = 1e-8,
elo_mean: float = 1500.0,
):
"""
pair_counts: { (winner, loser): wins } for all observed directed pairs.
ties: optional { (a, b): tie_count } counted once per unordered pair (a,b).
If provided, each tie contributes 0.5 win to both directions.
laplace: additive smoothing to each *directed* count (prevents zeros).
"""
# ---- Build model list ----
models = sorted(set([k[0] for k in pair_counts] + [k[1] for k in pair_counts]))
m = len(models)
idx = {name: i for i, name in enumerate(models)}
# ---- Build directed wins matrix W[i,j] = times i beat j ----
W = np.zeros((m, m), dtype=float)
for (w, l), c in pair_counts.items():
if w == l:
continue
W[idx[w], idx[l]] += float(c)
# ---- Optional ties: add 0.5 to both directions for each tie ----
if ties:
for (a, b), t in ties.items():
if a == b:
continue
i, j = idx[a], idx[b]
W[i, j] += 0.5 * t
W[j, i] += 0.5 * t
# ---- Laplace smoothing on directed edges (excluding diagonal) ----
if laplace and laplace > 0:
W += laplace
np.fill_diagonal(W, 0.0)
# Unordered totals N_ij = W_ij + W_ji
N = W + W.T
np.fill_diagonal(N, 0.0)
# Guard: drop models with zero matches
active = (N.sum(axis=1) > 0)
if not np.all(active):
keep = np.where(active)[0]
models = [models[i] for i in keep]
idx = {name: i for i, name in enumerate(models)}
W = W[np.ix_(keep, keep)]
N = N[np.ix_(keep, keep)]
m = len(models)
# ---- Bradley–Terry via MM updates (Hunter 2004) ----
s = np.ones(m, dtype=float) # abilities (positive)
for _ in range(max_iter):
s_old = s.copy()
w_i = W.sum(axis=1) # total (smoothed) wins per model
# denom_i = sum_j N_ij / (s_i + s_j)
denom = (N / (s.reshape(-1,1) + s.reshape(1,-1) + 1e-12)).sum(axis=1)
upd = denom > 0
s[upd] = w_i[upd] / denom[upd]
# normalize to keep scale stable (geometric mean = 1)
s /= np.prod(s) ** (1.0 / m)
if np.max(np.abs(np.log(s + 1e-12) - np.log(s_old + 1e-12))) < tol:
break
# ---- Convert to beta and Elo-like ratings ----
beta = np.log(s + 1e-12)
elo = (400.0 / math.log(10.0)) * beta
elo = elo - np.mean(elo) + elo_mean # center
# ---- Summaries and expected probabilities ----
wins = W.sum(axis=1)
losses = W.sum(axis=0)
matches = N.sum(axis=1) # unordered total vs all opponents
ratings = pd.DataFrame({
"ability": s,
"beta": beta,
"elo": elo,
"wins": wins,
"losses": losses,
"matches": matches,
}, index=models).sort_values("elo", ascending=False)
P = s.reshape(-1,1) / (s.reshape(-1,1) + s.reshape(1,-1))
np.fill_diagonal(P, np.nan)
p_matrix = pd.DataFrame(P, index=models, columns=models)
return ratings, p_matrix
ratings, p_exp = bt_elo_from_pair_counts(eli5_model_matrix, ties=None)
print(ratings[["elo","wins","losses","matches"]].to_markdown())
```
We get the following results:
| | elo | wins | losses | matches |
| :------------- | ----------: | -----: | -----: | ------: |
| gpt-oss-20b | **1629.77** | 425974 | 174028 | 600001 |
| qwen-3-14b | 1554.52 | 351422 | 248580 | 600001 |
| gemma-3-12b-it | 1523.23 | 319396 | 280606 | 600003 |
| llama-3.1-8b | **1292.47** | 103212 | 496790 | 600001 |
The results are clear - GPT-OSS 20B is the clear winner, followed by Qwen 3 14B, and then Gemma 3 12B. Llama 3.1 8B is the worst performer.
### Conclusion
We just demonstrated how you can use Sutro to run model task evals.
Our approach avoided the need for human feedback altogether, instead using an ensemble of LLMs to evaluate our task and remove the biases of any single model. This avoided the need for human labelers, or any online human feedback. We bootstrapped the evaluation process entirely offline:
* in just a **few hours**
* on **100,000 samples, across 4 models and 16 evaluation jobs** (but could be scaled to millions of samples and billions of tokens)
* for around **100 dollars**
* without the need for any custom infrastructure or GPU setup
* using **only the Sutro SDK and open-source Python tools**
### Next Steps
While we identified a good base model to start with, a good next step might be to use our winning model, and run further offline evals to improve prompts, sampling parameters, or structured output schemas - taking our task specification to a truly optimized state.
And to reiterate once more - this iterative, offline eval process can be used to improve nearly any AI application or agent, including but not limited to:
#### New model development and benchmarking:
You can continuously run large scale evals as you post-train, fine-tune, and improve your models without human feedback.
#### Task specialization:
When trying to optimize an LLM task such as summarization, classification, or code-generation, you can evaluate your choice of model, prompt, sampling parameters, and more to optimize selections and maximize performance.
#### Application and agent development/tuning:
As you create more complex, compound, and agentic tasks, you can evaluate entire reasoning traces, workflow outputs, and application logs to tune performance without the need for human feedback.
Hopefully this helps get you started with LLM-as-a-judge methods on the Sutro platform. We encourage you to bring your own approaches, and get creative with the task you're trying to evaluate. If you need any help getting started, contact us at [team@sutro.sh](mailto:team@sutro.sh).
You can review the full, resulting MIT-licensed dataset associated with this guide here:
### References
1. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: [https://arxiv.org/abs/2306.05685](https://arxiv.org/abs/2306.05685)
# Synthetic Data For Privacy Preservation
Source: https://docs.sutro.sh/examples/synthetic-data-privacy
Learn how to create useful synthetic data from the relevant characteristics of another dataset while reducing privacy concerns.
15 min read
\~2-3 hour project
\~\$40
Medium
It's common for organizations to sit on vast amounts of sensitive data that can't be analyzed, shared, or used to train AI models because of privacy concerns and regulatory constraints.
However, in certain cases, synthetic data can unlock the ability to make use of otherwise off-limits data by creating derived versions that capture the intent, signal, and other important characteristics of the original dataset while simultaneously reducing privacy concerns associated with the original dataset.
For example, consider the use of healthcare or financial data such as clinical notes and financial transactions. Such records are often extremely sensitive, but can be useful for training models to help with patient care or financial planning. Such data is often off-limits for sharing, analyzing, training, or even moving off of physical servers. But with synthetic data, many of these concerns can be mitigated.
**Note: it's important to consult with legal and privacy professionals before using synthetic data for privacy purposes. This example is for educational purposes only.**
## The Goal
We want to **create a synthetic dataset that captures the relevant characteristics and signals of another, sensitive dataset**.
By the end, we'll have a final, derived synthetic dataset based on the original one that **retains over 95% of the signal of the original dataset** and **reduces PII occurrences by over 97%** - as well as a multi-step pipeline written using the Sutro Python SDK to generate it easily, quickly, and inexpensively.
Let's get started!
## Choosing our Original Dataset
In our [previous example](/examples/synthetic-data-zero-to-hero), we created a heterogeneous, 20,000 row synthetic dataset of product reviews. Typically product reviews would be public facing, and often would not be a major issue with regards to privacy.
However, we can use this dataset to create simulated customer support questions and dialogues, which are often sensitive, internal, and private. Because this will be a synthetic dataset itself, we're relieved of privacy concerns for this demo.
This step is relatively straightforward, and we can use a similar pipeline to the one we used in the previous example.
```python theme={null}
import sutro as so
import polars as pl
from pydantic import BaseModel
from typing import List
# Login using e.g. `huggingface-cli login` to access this dataset
df = pl.read_parquet('hf://datasets/sutro/synthetic-product-reviews-20k/results.parquet')
system_prompt = """
You will be shown a product review. It contains the product name, product description, review title, review, product rating, and product reviewer's name.
Your job is to write a realistic customer support dialogue between the product reviewer and a customer support agent.
The dialogue should start with a realistic question about the product from the reviewer.
You should return the dialogue as a list of strings.
"""
class CustomerSupportQuestion(BaseModel):
dialogue: List[str]
results = so.infer(
df,
column=["Product Name: ", "product_name", " Product Description: ", "product_description", " Review Title: ", "review_title", " Review: ", "review_text", " Product Rating: ", "rating_out_of_5", " Product Reviewer: ", "review_author"],
system_prompt=system_prompt,
model="qwen-3-32b",
output_schema=CustomerSupportQuestion,
job_priority=1
)
```
We opted to use a 32B model in this case, a good middle ground between speed and quality to create a nuanced, realistic customer support dialogue. We iterated using prototyping (`job_priority=0`) jobs on a few different models before settling on this one. Despite the larger model it only **cost about \$8 to generate**.
We're also using Sutro's helpful column concatenation feature to assembly a single input (string) using multiple others.
Once it finished, we easily grabbed the resulting data, appending it to our original product review dataset, and saved the relevant columns to create a new, synthetic dataset.
```python theme={null}
# Grab the results from the job
results = so.get_job_results('job-b1e75fa1-821e-4479-8009-5b6b22616513')
dialogues = results['dialogue'].to_list()
# Append the dialogues to the original dataset
df = df.with_columns(pl.Series(dialogues).alias('customer_support_dialogue'))
df = df.select(['product_name', 'product_description', 'customer_support_dialogue'])
# Save the new dataset
df.write_parquet('20k-customer-support-dialogues.parquet')
```
You can view the dataset on Hugging Face here: [https://huggingface.co/datasets/sutro/synthetic-customer-support-dialogues-20k](https://huggingface.co/datasets/sutro/synthetic-customer-support-dialogues-20k)
## Creating a Feature Map
Here's where the mad science begins! The goal overall is to **create a synthetic dataset that captures the relevant characteristics and signals of the original dataset**. So to do that, we need to decide what those important characteristics and signals are.
Once we have a good understanding of what the important, distinct features are that we want to preserve, we can extract each of them from each record in the original dataset, and use this as a "feature map" to guide the generation of the final synthetic dataset.
In this case, we're dealing with customer support dialogues. And let's say for the sake of this example, our goal is to train a helpful, downstream assistant chatbot to help answer the questions about the products in our catalog. So, what are the important signals we want to preserve from each record?
For this example, we'll preserve:
* product name
* product description
* issue type (open-set label)
* issue severity (low, medium, high)
* issue description
* resolution path description
* outcome (success, failure)
* customer sentiment (positive, negative, neutral)
* customer satisfaction (low, medium, high)
Things we'll want to filter out or avoid:
* customer name and other personally identifiable information
* sensitive personal details that aren't relevant to the issue
* any other information that isn't relevant to the issue
If we're able to effectively extract out these features and avoid the privacy-compromising information, we'll be retaining what's actually useful about the original dataset. Using this feature map, we'll be able to reconstruct new, synthetic dialogues that are just as useful for training a helpful assistant chatbot, while simultaneously avoiding privacy concerns.
Let's do that now!
```python theme={null}
import sutro as so
import polars as pl
from pydantic import BaseModel
df = pl.read_parquet('hf://datasets/sutro/synthetic-customer-support-dialogues-20k/20k-customer-support-dialogues.parquet')
# join the customer support dialogue into a single string
df = df.with_columns(
pl.col("customer_support_dialogue").list.join("\n"),
)
system_prompt = """
You will be shown a customer support dialogue about a product ordered by a customer. It contains the product name, product description, and customer support dialogue.
Your goal is to extract out the following important features from the dialogue:
- issue type (open-set label)
- issue severity (low, medium, high)
- issue description
- resolution path description
- outcome (success, failure)
- customer sentiment (positive, negative, neutral)
- customer satisfaction (low medium, high)
You should avoid extracting any personally identifiable information or sensitive personal details that aren't relevant to the issue.
"""
class CustomerSupportFeatureExtraction(BaseModel):
issue_type: str
issue_severity: str
issue_description: str
resolution_path_description: str
outcome: str
customer_sentiment: str
customer_satisfaction: str
results = so.infer(
df,
column=["Product Name: ", "product_name", " Product Description: ", "product_description", " Customer Support Dialogue: ", "customer_support_dialogue"],
system_prompt=system_prompt,
model="qwen-3-32b",
output_schema=CustomerSupportFeatureExtraction,
job_priority=1
)
```
You can easily explore the results in the Sutro Web UI:
These results look good! There is a lot of variation in issue type, issue description, and resolution path description. In a real dataset, we'd likely see more variation in customer sentiment, customer satisfaction, and outcome. But for our purposes, this should be sufficient for our demo.
And despite using a 32B model, it only cost **\$3.11 to run this feature extraction job**. Just for numbers sake - this 20,000 row job processed 10.8M input tokens, and generated 2.3M output tokens.
## Let the Map Lead the Way
Now that we've extracted the relevant features from the original dataset, we can use it guide the generation of a new, synthetic dataset.
```python theme={null}
df = pl.read_parquet('20k-customer-support-dialogues.parquet')
results = so.get_job_results('job-63e3cd5a-0e32-4583-bf1e-b6cc72c844ad')
# horizontally concatenate df with results
df = df.with_columns(results.select('issue_type', 'issue_severity', 'issue_description', 'resolution_path_description', 'outcome', 'customer_sentiment', 'customer_satisfaction'))
system_prompt = """
You will be shown features extracted from a customer dialogue about a product.
It contains the product name, product description, issue type, issue severity, issue description, resolution path description, outcome, customer sentiment, and customer satisfaction.
Your goal is to generate a new, multi-turn, realistic customer dialogue that captures the same features as the original dialogue between the customer and the customer support agent.
It should start with a question about the product from the customer.
You should return the dialogue as a list of strings.
"""
class CustomerSupportDialogue(BaseModel):
dialogue: List[str]
results = so.infer(
df,
column=["Product Name: ", "product_name", " Product Description: ", "product_description", " Issue Type: ", "issue_type", " Issue Severity: ", "issue_severity", " Issue Description: ", "issue_description", " Resolution Path Description: ", "resolution_path_description", " Outcome: ", "outcome", " Customer Sentiment: ", "customer_sentiment", " Customer Satisfaction: ", "customer_satisfaction"],
system_prompt=system_prompt,
model="qwen-3-32b",
output_schema=CustomerSupportDialogue,
job_priority=1
)
```
This produced a new, synthetic dataset of 20,000 customer support dialogues that captures the same features as the original dataset. The total **cost of this job was \$5.97**.
## Evaluating the Results
We claim to have created a dataset that's just as useful, but without privacy concerns. Let's see if that's the case!
To do so, we'll use a lightweight LLM-as-a-judge method to evaluate
1. The extent to which the synthetic dataset captures the same features as the original dataset
2. The drop in PII and sensitive personal details between the two datasets
First, we'll evaluate the extent to which the synthetic dataset captures the same features as the original dataset.
```python theme={null}
df = pl.read_parquet('20k-customer-support-dialogues.parquet')
new_dialogues = so.get_job_results('job-16f9f7f2-1247-4fed-bbb0-46742e77f2a8')
df = df.with_columns(new_dialogues['dialogue'].alias('new_customer_support_dialogue'))
system_prompt = """
You will be shown two customer support dialogues about a product.
Your goal is to evaluate the similarity between the two dialogues, with respect to the following features:
- product name
- product description
- issue type
- issue severity
- issue description
- resolution path description
- outcome
- customer sentiment
- customer satisfaction
You should return a score between 0 and 100, where 100 means the two dialogues capture the same features.
"""
class CustomerSupportDialogueEvaluation(BaseModel):
score: int
results = so.infer(
df,
column=["Customer Support Dialogue 1: ", "customer_support_dialogue", " Customer Support Dialogue 2: ", "new_customer_support_dialogue"],
system_prompt=system_prompt,
model="qwen-3-4b-thinking",
output_schema=CustomerSupportDialogueEvaluation,
job_priority=1
)
```
This job was a bit token-heavy comparing the dialogues and reasoning about them (15.2M input tokens, and 55M output tokens). Generally speaking, LLMs are strong at comparing text, so we opted to use a 4B model for this evaluation. Despite the heavy token usage, it only cost **\$11.81** to run. Most proprietary models would cost significantly more - likely 5-10x more for this job - showing the cost-saving potential of using small, open-source models at scale.
Once the job finishes, we can grab the results and evaluate the similarity score.
```python theme={null}
similarity_results = so.get_job_results('job-38b880e0-41bf-41aa-83b7-277668086e5f')
# extract out score from content field
df = df.with_columns(similarity_results['content'].struct.field('score').alias('similarity_score'))
print(df['similarity_score'].describe())
```
| statistic | value |
| ----------- | -------- |
| count | 20000.0 |
| null\_count | 0.0 |
| mean | 95.93085 |
| std | 3.517469 |
| min | 20.0 |
| 25% | 95.0 |
| 50% | 95.0 |
| 75% | 98.0 |
| max | 100.0 |
If our LLM-as-a-judge is to be trusted, this is a good result! We're averaging a 95.93% similarity score between the two dialogues.
Next we'll evaluate for PII and sensitive personal details.
```python theme={null}
df = pl.read_parquet('20k-customer-support-dialogues.parquet')
new_dialogues = so.get_job_results('job-16f9f7f2-1247-4fed-bbb0-46742e77f2a8')
df = df.with_columns(new_dialogues['dialogue'].alias('new_customer_support_dialogue'))
df = df.with_columns(
pl.col("customer_support_dialogue").list.join("\n")
.alias('customer_support_dialogue_prompt')
)
df = df.with_columns(
pl.col("new_customer_support_dialogue").list.join("\n")
.alias('new_customer_support_dialogue_prompt')
)
system_prompt = """
You will be shown a customer support dialogue about a product.
Your goal is to review the dialogue for any personally identifiable information (PII) or sensitive personal details.
Return the number of PII or sensitive personal details in the dialogue.
"""
class CustomerSupportDialoguePIIReview(BaseModel):
pii_count: int
results = so.infer(
df,
column="customer_support_dialogue_prompt",
system_prompt=system_prompt,
model="qwen-3-14b-thinking",
output_schema=CustomerSupportDialoguePIIReview,
job_priority=1
)
results = so.infer(
df,
column="new_customer_support_dialogue_prompt",
system_prompt=system_prompt,
model="qwen-3-14b-thinking",
output_schema=CustomerSupportDialoguePIIReview,
job_priority=1
)
```
Each of these jobs were lighter weight, and cost less than **\$6** for both.
We'll gather the results and evaluate the PII count.
```
old_pii_results = so.get_job_results('job-ebfec0d1-1674-43ef-8eeb-38a2534614f0')
df = df.with_columns(old_pii_results['content'].struct.field('pii_count').alias('old_pii_count'))
new_pii_results = so.get_job_results('job-be5291e8-930f-474b-bf3f-0ac642391cc7')
df = df.with_columns(new_pii_results['content'].struct.field('pii_count').alias('new_pii_count'))
print(df['old_pii_count'].describe())
print(df['new_pii_count'].describe())
```
In the original dataset, we get the following statistics:
| statistic | value |
| ----------- | -------- |
| count | 19999.0 |
| null\_count | 1.0 |
| mean | 1.131107 |
| std | 0.534079 |
| min | 0.0 |
| 25% | 1.0 |
| 50% | 1.0 |
| 75% | 1.0 |
| max | 5.0 |
In the (new) synthetic dataset, we get the following statistics:
| statistic | value |
| ----------- | -------- |
| count | 20000.0 |
| null\_count | 0.0 |
| mean | 0.0314 |
| std | 0.183618 |
| min | 0.0 |
| 25% | 0.0 |
| 50% | 0.0 |
| 75% | 0.0 |
| max | 3.0 |
As you can see, the average PII count drops from 1.13 to 0.03 per dialogue. The results aren't perfect, but it's a significant improvement.
## Conclusion
We've demonstrated that using LLMs alone, we can create synthetic datasets that capture signal, intent, and other important characteristics of the original dataset, while simultaneously minimizing privacy concerns. This can be a powerful tool for organizations who want to unlock the value of their sensitive data without compromising privacy. Again - using such methods aren't perfect, so it's important to make sure such methods are appropriate for your use case.
In summary:
✅ We created a synthetic dataset that captures the same features as the original dataset (95.93% similarity score).
✅ We decreased the PII count from 1.13 to 0.03 per dialogue (a 97% reduction).
✅ We did this using a handful of small Python scripts and no infrastructure setup.
✅ We did all of this in a few hours.
✅ For less than \$40 worth of tokens!
The final dataset is available on HuggingFace here:
## Note on Limitations
It's worth noting that this method can **reduce** privacy concerns, but **does not contain the same mathematical gaurantees** as other methods such as differential privacy or k-anonymity. Similarly, LLM-as-a-judge methods are fast, cheap, and scalable proxies for human judgement, but are not a perfect substitute in critical applications.
Make sure to use the appropriate tools for your use case.
# Generating Representative Synthetic Data with LLMs - Zero to Hero
Source: https://docs.sutro.sh/examples/synthetic-data-zero-to-hero
Get up and running with synthetic data generation using the Sutro Python SDK.
10 min read
\~1.5 hour project
\~\$2
Beginner
This example demonstrates how you can can quickly, easily, and inexpensively generate synthetic data using the Sutro Python SDK.
## The Goal
Our goal today will be to **generate a dataset of 20,000 high-quality, synthetic product reviews**. This could be useful for:
* Training/evaluating sentiment analysis models, recommendation systems, spam classifiers, and other machine learning models
* Market research simulations, A/B testing, customer segmentation
* ... and more!
We'll use a crawl, walk, run approach:
1. Start by generating a basic dataset of 100 product reviews.
2. Add structure and randomness (diversity) to the reviews.
3. Add representation to the reviews, so that they are representative of underlying real-world data.
4. Scale up to 20,000 reviews, seamlessly and inexpensively.
Let's get started!
## Baby Steps
First, make sure you have the Sutro Python SDK [installed](/installation). This will include all dependencies required for the examples.
Let's start by creating a basic dataset of 100 product reviews.
```python theme={null}
import sutro as so
import polars as pl
system_prompt = "Generate a novel product review."
inputs = [""] * 100 # <--- Generate 100 reviews.
results = so.infer(
inputs,
model="qwen-3-4b",
system_prompt=system_prompt
)
```
And that's it! In five lines of code, we've generated 100 product reviews. As a prototyping (`p0`) job, this should take a few minutes to run. You should see something like the following when you run the job. It should take a a couple of minutes to run (GIF sped up for brevity):
Once it's done running, we can inspect the results in the Sutro Web UI:
As you can see, we have data - but it's not very useful (at least not yet)! We have far too much commonality between the product reviews including the type of product being reviewed, the rating, sentiment, and more.
This can generally be expected from an LLM until we introduce further steps for obtaining structure, heterogeneity and representativeness. So, let's do that next.
## Let's Walk - Adding Structure & Randomness
To begin, let's see if we can introduce some more diversity and structure into our reviews. We'll do this in a few ways:
1. Update the system prompt to include specific fields.
2. Add a random numerical seed to the each of the inputs to increase diversity.
3. Modify the `temperature` sampling parameter to increase randomness.
4. Add a Pydantic model to enforce a schema structure of the reviews we want to generate.
5. Increase the model size from `qwen-3-4b` to `qwen-3-14b` to sample from more world knowledge.
Let's implement these changes now.
```python theme={null}
import sutro as so
import polars as pl
from pydantic import BaseModel
system_prompt = """
Generate a novel product review.
Include a title, text, author, product name,
product description, product category,
and rating out of 5.
"""
inputs = [""] * 100
sampling_params = {
"temperature": 1.1,
}
class ProductReview(BaseModel):
review_title: str
review_text: str
review_author: str
product_name: str
product_description: str
product_category: str
rating_out_of_5: int
results = so.infer(
inputs,
model="qwen-3-14b",
system_prompt=system_prompt,
output_schema=ProductReview,
random_seed_per_input=True # <-- uses a random seed for each input
)
```
This is a significant improvement! We now have more product diversity overall, and a consistent schema for the reviews we've generated.
However, we can still do better. The review titles, product names/descriptions, author names, and ratings are still very similar across the reviews we've generated.
## Time to Run - Adding Representation
For most valuable use cases, we want a final dataset that is representative of real-world data. To achieve this, we not only want diversity itself, but rather diversity that adheres to the real-world distribution of the data we're trying to represent.
There are various levels of complexity to achieve this, but for this example we'll take a simple approach by using two other "seed" datasets to produce the representation we're looking for.
In our example, we're creating product reviews, so we probably need a set of products to review, right? In the real world, perhaps if you're running an e-commerce business you'd want to use your own product dataset for this. However, for our toy example, we'll use an [Amazon Products sample dataset](https://huggingface.co/datasets/ckandemir/amazon-products) from Hugging Face.
It works well for our purposes - it contains 33,000 products with associated product names, descriptions, and prices.
This will certainly help us get more product representation, but how about reviewer representation? For that, we can use a personas dataset, in this case our very own [Synthetic Humans 50k dataset](https://huggingface.co/datasets/sutro/synthetic-humans-50k).
This dataset contains 50,000 personas sampled from actual US demographics, and contains qualitative descriptions of each persona.
For this example, let's say we want to generate product reviews from 18-30 year olds living in popular US cities over all of the products in the Amazon Products dataset.
To do this, we'll need to merge the two datasets, sampling a random persona for each product. We'll then run our previous inference job over the merged dataset. Let's do that now.
```python theme={null}
import sutro as so
import polars as pl
from pydantic import BaseModel
from random import randint
products_df = pl.read_parquet('hf://datasets/ckandemir/amazon-products/data/train-00000-of-00001.parquet')[0:20000]
personas_df = pl.read_parquet('hf://datasets/sutro/synthetic-humans-50k/chunk_0.parquet')
personas_df = personas_df.filter(
(pl.col('age') >= 22) &
(pl.col('age') <= 30) &
(pl.col('location').is_in([
'New York, New York',
'Los Angeles, California',
'Chicago, Illinois',
'Houston, Texas',
'Miami, Florida',
'Seattle, Washington',
'Boston, Massachusetts',
'San Francisco, California',
'Washington, D.C.',
'Atlanta, Georgia',
'Philadelphia, Pennsylvania',
'Phoenix, Arizona',
'San Diego, California',
'San Jose, California',
'Austin, Texas',
]))
)
def get_random_persona_demographic_summary():
row = personas_df.sample(1, seed=randint(0, 1000000))
return row['demographic_summary'][0]
random_personas = [get_random_persona_demographic_summary() for _ in range(len(products_df))]
products_df = products_df.with_columns(
pl.Series("persona", random_personas)
)
system_prompt = """You will be given a product name, description, and price.
You will also be given a reviewer persona.
Your task is to generate a novel product review from the reviewer persona's perspective.
Include a title, text, author, product name, product
description, product category, and rating out of 5.
"""
class ProductReview(BaseModel):
review_title: str
review_text: str
review_author: str
product_name: str
product_description: str
product_category: str
rating_out_of_5: int
results = so.infer(
products_df[0:100],
column=["Product Name: ", "Product Name", " Product Description: ", "Description", " Price: ", "Selling Price", " Reviewer Persona: ", "persona"],
model="qwen-3-4b",
system_prompt=system_prompt,
output_schema=ProductReview
)
```
As you can see, we actually went back to the qwen-3-4b model and removed our other diversity-boosting techniques, as we can gain sufficient representation with just the two seed datasets.
We're also using Sutro's helpful column concatenation feature to assembly a single input (string) using multiple others.
Much, much better! We now have a wide array of products and reviewers, faithful to our underlying real-world data. We even see cases where the rating is lower because of the mismatch between the product and reviewer demographics.
Now, let's scale up our example to 20,000 product reviews!
## Scaling Up
To scale up to our 20,000 product reviews, it's dead simple! We just need to make a couple of changes to our code above.
```python theme={null}
... # previous code
results = so.infer(
products_df, # <-- we're now using the full dataset
column="product_info",
model="qwen-3-4b",
system_prompt=system_prompt,
output_schema=ProductReview,
job_priority=1 # <-- we're now setting the job priority to 1
)
```
All we need to do is remove the slicing of the products\_df, and set the job priority to 1. Previously, our jobs were running as priority 0 (p0), which is the default for small-scale testing (see [Job Priority](/concepts/job-priority) for more details).
This should take less than an hour to run, and generate all 20,000 product reviews. You can inspect the progress and sample results in the Sutro Web UI, cancelling the job when the samples don't look promising.
In this case, it took **29 minutes to run, and cost only \$1.88** using Sutro!
Once it's done running, you can grab the results using the SDK, or download the results directly from the Web UI.
```python theme={null}
results = so.get_job_results('job-0daebfba-ce27-462e-9d1c-0bc566238a50', include_inputs=True)
results.write_parquet('results.parquet')
```
Sutro will automatically unpack the JSON fields in the results into separate columns, so you can access them like any other column.
You can view the resulting dataset directly on Hugging Face:
## Recap
In this example, we demonstrated how you can easily create synthetic data with LLMs using the Sutro Python SDK.
Our final 20,000 product review dataset:
✅ Created using a few dozen lines of code.
✅ Representative of our underlying real-world data.
✅ Required zero infrastructure setup.
✅ In less than an hour.
✅ For less than \$2!
With the Sutro Python SDK, you can easily create synthetic data with LLMs for your own use cases. Try it out today by [requesting access](https://sutro.sh/request-access) to Sutro!
## Addendum
If you want to generate even more variations, you can set the `n` sampling parameter, which will produce `n` samples for each input.
```python theme={null}
sampling_params = {
"n": 5
}
results = so.infer(
products_df[0:100],
column="product_info",
model="qwen-3-4b",
system_prompt=system_prompt,
output_schema=ProductReview,
sampling_params=sampling_params
)
```
This will produce 5 samples for each input. If we applied this to the full dataset, we would generate 100,000 reviews.
As you can see, this now has 5 distinct outputs, with slight variations on each review.
# Installation
Source: https://docs.sutro.sh/installation/index
Reference documentation for Installation.
# Installation
You can use Sutro via the API, or using the Python SDK and CLI. There is no installation required to use the API directly, but you will need to install the SDK and CLI to use the Sutro CLI.
To install the Sutro Python SDK and CLI, install via pip from [PyPI](https://pypi.org/project/sutro/):
```
[uv] pip install sutro
```
If you experience issues with installation, please reach out to us at [team@sutro.sh](mailto:team@sutro.sh) or join our Slack community.
## Setting up your API key
Open the Sutro UI and create a deployment key from the **API Keys** panel.
Configure both the Sutro deployment base URL and the one-time key reveal:
If the panel is not visible, contact the Sutro team at
[team@sutro.sh](mailto:team@sutro.sh) to create a key for you.
```bash theme={null}
export SUTRO_API_URL="https://your-sutro-deployment.example.com"
export SUTRO_API_KEY="sk_..."
```
Both the Sutro deployment base URL and the same URL with `/v1` appended are
accepted. You can instead store both settings locally with the CLI:
```
sutro login
```
This prompts for the Sutro deployment URL and API key, validates them
against that deployment, and saves them to `~/.sutro/config.json` for future SDK
and CLI commands with owner-only file permissions.
# Introduction
Source: https://docs.sutro.sh/introduction/index
Reference documentation for Sutro.
## What is Sutro?
Sutro helps you build reliable AI decision systems for repeated tasks. Instead of hand-tuning prompts and hoping they hold up in production, Sutro gives you a structured workflow to align AI behavior with your team's actual preferences and then deploy the result at scale.
Sutro has two core components:
* **Sutro Functions** — Build task-specific judges, along with classifiers and extractors, that reflect your team's decision preferences. With an evolving record book of impactful annotations, it helps you build an optimized, deployable function you can invoke by name.
* **Batch Inference** — Run large-scale offline inference across thousands to millions of inputs. Easily execute your Sutro Functions at any scale, or run OSS LLMs directly for analytical and generation workloads.
Functions and Batch work best together, but can be used independently.
## Sutro Functions
With Sutro Functions, you can expect:
* **Speed:** Maximizes prompt quality per unit of your time; you spend minutes labeling, not hours of testing and rewriting.
* **Stability** Create a consistent foundation of expertise to measure & optimize against
* **Maintainability:** Swap models, add new data, and re-optimize without regressing on past failures.
* **Adaptability:** Compress tasks into the right model for the job.
Describe your task, upload a representative sample of your data, and label the highest-impact cases Sutro surfaces. Sutro then optimizes a prompt that matches your preferences — and improves further with each iteration.
### Key use cases
* Evals for agents and single call LLMs
* User intent analysis
* Data filtering and transformation (multimodal and text-based)
* Data tagging or labelling
* Classical ML decisioning (lead generation, fraud detection, compliance, KYC, etc)
Learn how Functions work and what you can build with them.
## Batch Inference
Sutro's batch inference platform is the production runtime for Sutro Functions, built to process millions of rows at once. It also handles standalone large-scale offline workloads — synthetic data generation, embeddings, LLM-as-a-judge evaluations, and more.
With batch inference, you can expect:
* **Speed:** Large-scale jobs finish in an hour or less, not a day from now.
* **Scale:** From a handful of inputs to billions of tokens per job.
* **Cost:** Less than 25% the cost of real-time inference providers.
* **Security:** Custom data retention policies and optional bring-your-own-storage.
Run your first batch job.
## When to use Sutro
**Quickly build an expert-aligned judge -> Run it at scale**
Most users start with **Sutro Functions** to build a judge or other decision model that is aligned with their preferences, then use **Batch Inference** to run that function across production data at scale.
We think Functions and Batch work well used together, as the types of tasks that Functions is best at formalizing are often latency insensitive and high volume in nature, thus well suited for batch inference.
Batch Inference also works standalone for workloads that don't need strong preference alignment — synthetic data generation, embeddings, and more.
## When *not* to use Sutro
### Functions
If your model will produce lengthy, abstractive, or otherwise "open-world" text - rather than a specific decision - Sutro Functions won't be a great fit today.
In some cases, you may be able to optimize these types of tasks indirectly using a Sutro built judge, if it can be evaluated in a verifiable manner. See our [task design section](/sutro-functions/designing-your-task) section for more info on best practices.
### Batch Inference
You're building a user-facing application with real-time, one-off inference calls (e.g. a chatbot) where latency is critical. For those use cases, we recommend an inference provider that optimizes for latency.
## Not sure where to start?
We'd love to hear about your use case and help you figure out the right approach. We also offer custom solutions for enterprise customers. Contact us at [team@sutro.sh](mailto:team@sutro.sh).
# Batch Inference
Source: https://docs.sutro.sh/python-sdk/batch-inference
Reference documentation for Batch Inference.
### Running batch inference
```Python theme={null}
infer(self, data, model='gpt-oss-20b', name=None, description=None, column=None, output_column='inference_result', job_priority=0, output_schema=None, sampling_params=None, system_prompt=None, dry_run=False, stay_attached=None, random_seed_per_input=False, truncate_rows=True, id_column=None)
```
Run LLM inference on a large list, table, dataframe, local file, or HTTP(S) download URL.
#### Parameters:
* `data` (Union\[List, pd.DataFrame, pl.DataFrame, str]): The data to run inference on. A string can be a local file path or HTTP(S) presigned download URL for a CSV or Parquet object.
* `model` (str, optional): The model or published Function to use for inference. Defaults to `"gpt-oss-20b"`. To run the same data across multiple models, use `infer_per_model(models=[...])`.
* `name` (str, optional): A job name for experiment and metadata tracking. Defaults to `None`.
* `description` (str, optional): A job description for experiment and metadata tracking. Defaults to `None`.
* `column` (Union\[str, List\[str]], optional): The column name to use for standalone-model inference. It is required for a DataFrame or local CSV/Parquet file. For a standalone-model download URL, it selects the input column; if omitted, the first column is used. If a list is supplied for an in-memory DataFrame, it concatenates the named columns and literal separator strings. Omit it for a published Function URL, which reads the Function's declared input fields.
* `id_column` (str, optional): For an HTTP(S) CSV or Parquet download URL, the column containing user-provided row IDs. The IDs are carried into results so they can be joined back to the source table. This parameter is not supported for lists, DataFrames, or local files.
* `output_column` (str, optional): The output column name when the SDK retrieves attached results. Defaults to `"inference_result"`. For a detached job, pass the same value when later calling `await_job_completion()` or `get_job_results()`.
* `job_priority` (int, optional): The [priority](/concepts/job-priority) of the job. Default is 0.
* `output_schema` (Union\[Dict\[str, Any], BaseModel], optional): A structured schema for the output. Can be either a dictionary representing a JSON schema or a pydantic BaseModel. Defaults to None.
* `system_prompt` (str, optional): A system prompt to add to all inputs. This allows you to define the behavior of the model. Defaults to None.
* `sampling_params` (dict, optional): A dictionary of sampling parameters to use for the inference. Defaults to None, which uses the default sampling parameters.
* `random_seed_per_input` (bool, optional): If True, a random seed will be generated for each input. This is useful for diversity in outputs. Defaults to False.
* `dry_run` (bool, optional): If `True`, submit an estimate job, wait for its estimate, print the estimate, and return the estimate job ID. This does not launch the normal full job, but sufficiently large priority-1 estimates run inference on an approximately 1-million-token prefix sample. Defaults to `False`.
* `stay_attached` (bool, optional): If True, the SDK will stay attached to the job and update you on the status and results as they become available. Default behavior is True for priority 0 jobs, and False for priority 1 jobs.
* `truncate_rows` (bool, optional): If True, any rows that have a token count exceeding the context window length of the selected model will be truncated to the max length that will fit within the context window. Defaults to True.
**Returns:**
str: The ID of the inference job.
### Production inputs from Amazon S3
For large production jobs, pass a presigned S3 GET URL instead of loading the file into the SDK process:
```python theme={null}
import sutro as so
job_id = so.infer(
data=presigned_url,
column="prompt",
id_column="row_id",
model="gpt-oss-20b",
system_prompt="Classify each input.",
job_priority=1,
stay_attached=False,
)
results = so.await_job_completion(job_id)
# Results include row_id alongside inference_result.
```
Use an unwrapped Parquet object for the most memory-efficient priority-1 tokenization path. Give the URL enough lifetime for Sutro to begin its one full-object download, with a scheduling safety buffer, and treat the complete URL as a credential. See [Presigned S3 Inputs](/python-sdk/presigned-s3-inputs) for file schemas, URL generation, security guidance, and production troubleshooting.
The ID column is returned even when inputs are not requested. Sutro preserves ID values but does not guarantee preservation of the source file's physical integer type.
### Monitoring job status
```Python theme={null}
attach(self, job_id: str)
```
Attach to an existing job and stream its progress in real-time. This has the equivalent behavior of setting `stay_attached=True` when calling `infer(...)`
>
This method connects to a running job and displays live progress updates, including the number of rows processed and token statistics. It shows a progress bar with real-time updates until the job completes.
>
**Parameters:**
* `job_id` (str): The ID of the job to attach to
>
**Returns:** None
>
**Job Status Behavior:**
* `RUNNING`: Streams progress updates with a live progress bar and job statistics
* `SUCCEEDED`: Notifies that the job already completed and suggests using `sutro jobs results`
* `FAILED`: Displays failure message and exits
* `CANCELLED`: Displays cancellation message and exits
>
**Example:**
```python theme={null}
# Attach to a running job to monitor its progress
sutro.attach("job_12345")
# Progress bar will display:
# Progress: 45%[████████████████ ] 450/1000 [00:32\<00:45] Input tokens processed: 12500, Tokens generated: 8300, Total tokens/s: 325.4
```
>
**Note:** This method is ideal for monitoring long-running jobs interactively. For programmatic use cases where you don't want live progress updates, use the simpler `await_job_completion()` instead.
## Await Job Completion
```Python theme={null}
await_job_completion(self, job_id: str, timeout: int = 7200, obtain_results: bool = True, output_column: str = 'inference_result', is_cost_estimate: bool = False) → pl.DataFrame | None
```
When deployed as part of a pipeline (Dagster, Airflow, etc) you might not be interested in seeing the progress of the job as it happens. `await_job_completion` is best for this use case, and should only be used when not using the `stay_attached` parameter of `infer(...)`, or the `attach(...)` function.
>
Waits for a job to reach a terminal state. By default, a successful job is retrieved through the SDK's JSON results path and returned as a Polars DataFrame.
>
This method polls the job status every 5 seconds (and prints it out) until the job completes, fails, is cancelled, or the timeout is reached.
>
**Parameters:**
* `job_id` (str): The ID of the job to await.
* `timeout` (int): Maximum time in seconds to wait for job completion. Defaults to 7200 (2 hours). Passing `None` is not supported.
* `obtain_results` (bool): Whether to retrieve and materialize results after the job succeeds. Defaults to `True`. Set this to `False` for large production results, then use a Parquet [results download URL](/batch-api-reference/job-results-url).
* `output_column` (str): Name of the output column in the returned DataFrame. Defaults to `"inference_result"`.
* `is_cost_estimate` (bool): Suppresses normal job-progress messaging for the SDK's cost-estimate workflow. Most callers should leave this as `False`.
>
**Returns:** `pl.DataFrame | None`: A Polars DataFrame when the job succeeds and `obtain_results=True`; otherwise `None`.
>
**Job Status Outcomes:**
* `SUCCEEDED`: Returns a Polars DataFrame when `obtain_results=True`; otherwise returns `None`
* `FAILED`: Returns `None`
* `CANCELLED`: Returns `None`
* Timeout reached: Returns `None`
>
**Example:**
```python theme={null}
results = so.await_job_completion("job_12345", timeout=3600)
# Job status is RUNNING for job-f9102252-ae2f-4d61-a879-a657e314f2e0
if results is not None:
print(f"Job completed with {len(results)} results")
```
For a large production job, wait without materializing JSON:
```python theme={null}
so.await_job_completion(job_id, timeout=6 * 60 * 60, obtain_results=False)
if so.get_job_status(job_id) != "SUCCEEDED":
raise RuntimeError("Job did not complete successfully")
```
Because `obtain_results=False` returns `None` for every terminal outcome and timeout, confirm `SUCCEEDED` before requesting and downloading the unified Parquet artifact through [Getting a Results Download URL](/batch-api-reference/job-results-url).
## Getting Quotas
```Python theme={null}
get_quotas(self)
```
Get your current quotas.
**Returns:** list: A list of quotas, one for each priority level. Contains row\_quota and token\_quota for each priority level.
# Functions
Source: https://docs.sutro.sh/python-sdk/functions
Run published Sutro Functions through the Python SDK.
Use `batch_run_function()` to execute a published Function over one or more rows.
Call the Function by name and pass the input fields it expects. Sutro uses the Function's published setup automatically.
Use the Function name only. Do not include a namespace, owner, or revision. The authenticated API key determines which customer namespace is searched, and the currently published revision is resolved automatically.
Only text Functions are supported today. Image, PDF, and other multimodal Functions are not yet runnable through `batch_run_function()`.
Synchronous `run_function()` is temporarily unavailable. Sutro deployments
do not yet expose a synchronous serving proxy, and the SDK will not send your
deployment key to the retired centralized serving endpoint. Use
`batch_run_function()` for all Function execution.
## Running a Function
```python theme={null}
batch_run_function(
name: str,
data: List[dict] | pd.DataFrame | pl.DataFrame | str,
job_priority: int | None = 0,
output_column: str = "inference_result",
dry_run: bool = False,
stay_attached: bool = False,
job_name: Optional[str] = None,
description: Optional[str] = None,
langsmith_metadata: Optional[Dict[str, Any]] = None,
langsmith_tags: Optional[List[str]] = None,
id_column: Optional[str] = None,
)
```
`batch_run_function()` submits many rows to the same Function.
### Parameters
* `name` — Function name to execute.
* `data` — One of:
* `List[dict]`
* `pandas.DataFrame`
* `polars.DataFrame`
* local `.csv` or `.parquet` file path
* `job_priority` — Batch priority level. Defaults to `0`.
* `output_column` — Output column name when results are retrieved while the SDK stays attached. For a detached job, pass the same value to `await_job_completion()` or `get_job_results()`.
* `dry_run` — If `True`, submit an estimate job, print its estimate, and return the estimate job ID. This does not launch the normal full job, but sufficiently large priority-1 estimates run inference on an approximately 1-million-token prefix sample. The SDK maps this to the Batch API's `cost_estimate` request field.
* `stay_attached` — If `True`, stay attached to the job and stream progress.
* `job_name` — Optional job name.
* `description` — Optional job description.
* `langsmith_metadata` — Optional trace metadata when LangSmith tracing is enabled.
* `langsmith_tags` — Optional trace tags when LangSmith tracing is enabled.
* `id_column` — For a pre-signed HTTP(S) CSV or Parquet URL, the column containing user-provided row IDs to carry into results.
When `data` is a local CSV or Parquet path, the SDK reads the file locally and submits rows whose column names must match the Function inputs.
When `data` is a pre-signed URL, Sutro reads the file remotely. `id_column` is metadata and must not duplicate a Function input field. Per-row LangSmith traces are not created because the SDK does not download the source rows.
### Examples
Using a list of dictionaries:
```python theme={null}
import sutro as so
job_id = so.batch_run_function(
name="lead-qualifier",
data=[
{
"query": "Find cybersecurity leaders evaluating AI vendors.",
"region": "APAC",
},
{
"query": "Find sales operations leaders replacing manual enrichment.",
"region": "EMEA",
},
],
)
print(job_id)
```
Using a Polars DataFrame:
```python theme={null}
import polars as pl
import sutro as so
df = pl.DataFrame(
{
"query": [
"Find cybersecurity leaders evaluating AI vendors.",
"Find sales operations leaders replacing manual enrichment.",
],
"region": ["APAC", "EMEA"],
}
)
job_id = so.batch_run_function(
name="lead-qualifier",
data=df,
)
print(job_id)
```
Using a local file path:
```python theme={null}
import sutro as so
job_id = so.batch_run_function(
name="lead-qualifier",
data="./lead_inputs.csv",
)
print(job_id)
```
Using a pre-signed URL with row IDs:
```python theme={null}
import sutro as so
job_id = so.batch_run_function(
name="lead-qualifier",
data="https://your-bucket.s3.amazonaws.com/leads.parquet?X-Amz-Algorithm=...",
id_column="lead_id",
job_priority=1,
)
results = so.await_job_completion(job_id)
# Results include lead_id alongside the Function output.
```
The CSV or Parquet file must contain every required Function input column. Optional columns may be absent, extra columns are ignored, and input values are converted to strings. Do not pass `column`, `system_prompt`, or `output_schema`; those values come from the published Function.
Because the SDK does not download presigned input rows, this path cannot create per-row LangSmith traces. See [Presigned S3 Inputs](/python-sdk/presigned-s3-inputs) for the complete production workflow.
### Working with batch results
Batch jobs run asynchronously. Use `await_job_completion()` to wait for completion and retrieve results:
```python theme={null}
import sutro as so
job_id = so.batch_run_function(
name="lead-qualifier",
data=[
{
"query": "Find cybersecurity leaders evaluating AI vendors.",
"region": "APAC",
}
],
)
results_df = so.await_job_completion(job_id)
print(results_df)
```
The default retrieval path materializes the complete result as JSON and is intended for manageable result sets. For large production jobs, wait with `obtain_results=False` and use a [Parquet results download URL](/batch-api-reference/job-results-url):
```python theme={null}
so.await_job_completion(
job_id,
timeout=6 * 60 * 60,
obtain_results=False,
)
if so.get_job_status(job_id) != "SUCCEEDED":
raise RuntimeError("Job did not complete successfully")
```
`obtain_results=False` returns `None` for success, failure, cancellation, and timeout, so confirm the final status before requesting the Parquet artifact.
By default, `get_job_results()` and `await_job_completion()` unpack JSON outputs when possible. For a Function that returns `label` and `rationale`, the result DataFrame may contain columns such as:
```text theme={null}
shape: (1, 3)
┌───────────┬─────────────────────────────────────────────┬──────────────────┐
│ label ┆ rationale ┆ confidence_score │
│ --- ┆ --- ┆ --- │
│ str ┆ str ┆ f64 │
╞═══════════╪═════════════════════════════════════════════╪══════════════════╡
│ qualified ┆ The lead matches the qualification rubric. ┆ 1.0 │
└───────────┴─────────────────────────────────────────────┴──────────────────┘
```
To inspect the raw JSON string instead, disable JSON unpacking:
```python theme={null}
import json
import sutro as so
raw_results_df = so.get_job_results(job_id, unpack_json=False)
first_result = json.loads(raw_results_df["inference_result"][0])
content = first_result.get("content", first_result)
print(content)
print(first_result.get("reasoning_content"))
print(raw_results_df["confidence_score"][0])
```
# Job Methods
Source: https://docs.sutro.sh/python-sdk/job-methods
Reference documentation for Job Methods.
### Listing jobs
```Python theme={null}
list_jobs(self)
```
List all jobs associated with the API key.
>
**Returns:** list: A list of job details.
### Getting job status
```Python theme={null}
get_job_status(self, job_id: str)
```
Get the status of a job by its ID.
>
**Parameters:**
* `job_id` (str): The ID of the job to retrieve the status for.
>
**Returns:** dict: The status of the job.
### Getting job results
```Python theme={null}
get_job_results(self, job_id: str, include_inputs: bool = False, include_cumulative_logprobs: bool = False, with_original_df: pl.DataFrame | pd.DataFrame = None, output_column: str = 'inference_result')
```
Get the results of a job by its ID.
>
**Parameters:**
* `job_id` (str): The ID of the job to retrieve the results for.
* `include_inputs` (bool, optional): Whether to include the inputs in the results. Defaults to False.
* `include_cumulative_logprobs` (bool, optional): Whether to include the cumulative logprobs in the results. Defaults to False.
* `with_original_df` (Union\[pl.DataFrame, pd.DataFrame], optional): Original DataFrame to join results with. Defaults to None.
* `output_column` (str, optional): Name of the column containing results. Defaults to "inference\_result".
* `disable_cache` (bool, optional): Whether to disable reading from or writing to the local job results cache. Defaults to False.
* `unpack_json` (bool, optional): If the output\_column is formatted as a JSON string, decides whether to unpack the top level JSON fields in the results into separate columns. Defaults to True.
>
**Returns:** Union\[pl.DataFrame, pd.DataFrame]: Results as a DataFrame.
>
* If `with_original_df` is provided: Returns the same type as the input DataFrame with results added as a new column
* If `with_original_df` is None: Returns a polars DataFrame by default
>
The DataFrame will contain:
>
* `inputs` column (if `include_inputs=True`). Each cell contains the input string given to the model.
* The user-provided ID column when the job was submitted with `id_column`. This column is returned even when `include_inputs=False`.
* `inference_result` column (or custom name via `output_column`)
* `cumulative_logprobs` column (if `include_cumulative_logprobs=True`)
>
**Example:**
>
```python theme={null}
# Get just the results
results = sutro.get_job_results(job_id)
# Returns: pl.DataFrame with one column 'inference_result'
# Get results with inputs
results = sutro.get_job_results(job_id, include_inputs=True)
# Returns: pl.DataFrame with columns ['inputs', 'inference_result']
# Add results back to original DataFrame
df_with_results = sutro.get_job_results(job_id, with_original_df=original_df)
# Returns: Same type as original_df with 'inference_result' column added. Matches the return shape of .infer(...) when stay_attached=True.
```
>
**Choosing between `get_job_results()` and `download_job_results()`:**
>
* Use **`get_job_results()`** for smaller jobs — as a rule of thumb, up to the thousands of rows. One call returns a ready-to-use DataFrame: the output column is named `inference_result`, structured outputs are unpacked into columns, and results are cached locally. The whole result set is held in memory.
* Use **[`download_job_results()`](#downloading-large-results)** for anything larger, or when you want the results as a durable Parquet file on disk rather than a DataFrame. It streams with a progress bar and resumes interrupted downloads, but hands you the raw artifact: the output column is named after the job ID, and structured outputs stay as JSON strings.
### Downloading large results
```Python theme={null}
download_job_results(self, job_id: str, output_path: Optional[str] = None, include_inputs: bool = False, include_cumulative_logprobs: bool = False, resume: bool = True, expires_in_seconds: int = 3600)
```
Download a job's results as a single Parquet file on local disk.
This is the recommended way to retrieve results for production-scale jobs — anything beyond the thousands of rows, embedding jobs especially. For small jobs, [`get_job_results()`](#getting-job-results) is more convenient: it returns a ready DataFrame with friendly column names and unpacked structured outputs, but holds everything in memory. This method instead streams the job's unified Parquet artifact to disk with a progress bar and returns the local path, leaving the file exactly as the server produced it.
An interrupted download leaves a `.part` file behind. When `resume` is True, rerunning the call picks up where it left off using an HTTP Range request, validated against the artifact's ETag — if the artifact changed server-side (or the ETag isn't available), the download restarts from scratch instead of resuming.
>
**Parameters:**
* `job_id` (str): The ID of the job to download results for.
* `output_path` (str, optional): Where to write the Parquet file. May be a directory (the server-provided artifact filename is used) or a full file path. Defaults to the artifact filename in the current directory.
* `include_inputs` (bool, optional): Whether to include the inputs in the results. Defaults to False.
* `include_cumulative_logprobs` (bool, optional): Whether to include the cumulative logprobs in the results. Defaults to False.
* `resume` (bool, optional): Whether to resume a partial download if one exists. Defaults to True.
* `expires_in_seconds` (int, optional): TTL for the underlying presigned URLs, up to 604800 (7 days). Defaults to 3600.
>
**Returns:** str: The local path of the downloaded Parquet file. Returns None if the request fails.
>
The job must have succeeded before results can be downloaded — the backend returns **409** for jobs that are still running, since the results artifact is cached permanently once materialized. Use `await_job_completion()` first.
>
The Parquet file's columns differ from `get_job_results()` in one important way: **the output column is named after the job ID**, not `inference_result`. The file contains:
>
* `inputs` column (if `include_inputs=True`)
* The user-provided ID column when the job was submitted with `id_column`. This column is included even when `include_inputs=False`.
* An output column named after the job ID (e.g. `job-76844041-b2bf-4248-9603-b7f750231b34`)
* `cumulative_logprobs` column (if `include_cumulative_logprobs=True`)
* `confidence_score` column (when the model provides one)
>
Structured outputs are kept as raw JSON strings; nothing is unpacked into separate columns (unlike `get_job_results(unpack_json=True)`).
>
**Example:**
>
```python theme={null}
import polars as pl
local_path = sutro.download_job_results(job_id, include_inputs=True)
results_df = pl.read_parquet(local_path) # or pl.scan_parquet to read lazily
# The output column is named after the job ID
outputs = results_df[job_id]
# Interrupted? Run the same call again and it resumes from the .part file
local_path = sutro.download_job_results(job_id, include_inputs=True)
```
### Getting a results download URL
```Python theme={null}
results_download_url(self, job_id: str, include_inputs: bool = False, include_cumulative_logprobs: bool = False, expires_in_seconds: int = 3600)
```
Get presigned download URLs for a job's results artifact, without downloading anything.
This is the escape hatch underneath `download_job_results()`: use it when you want the URLs themselves — to hand the download off to another system, fetch from a different machine, or issue partial `Range` reads. It wraps the [results download URL endpoint](/batch-api-reference/job-results-url); see that page for the full payload reference and raw-HTTP download patterns.
>
**Parameters:**
* `job_id` (str): The ID of the job to retrieve results for.
* `include_inputs` (bool, optional): Whether to include the inputs in the results. Defaults to False.
* `include_cumulative_logprobs` (bool, optional): Whether to include the cumulative logprobs in the results. Defaults to False.
* `expires_in_seconds` (int, optional): TTL for the returned presigned URLs, up to 604800 (7 days). Defaults to 3600.
>
**Returns:** dict: The endpoint payload, containing `artifact` metadata (`filename`, `size_bytes`, ...) and presigned `urls` — `urls.get` for downloading (supports HTTP `Range` requests) and `urls.head` for metadata only. Returns None if the request fails.
>
Treat presigned URLs like credentials: anyone with the URL can download the results until it expires. Do not send your Sutro `Authorization` header when using them — the URL already contains the credentials.
>
**Example:**
>
```python theme={null}
import requests
meta = sutro.results_download_url(job_id)
# Check the artifact size without downloading
head = requests.head(meta["urls"]["head"])
print(int(head.headers["Content-Length"]))
# Hand meta["urls"]["get"] to whatever does the actual download,
# e.g. a data pipeline running on another machine
```
### Cancelling jobs
```Python theme={null}
cancel_job(self, job_id: str)
```
Cancel a job by its ID.
>
**Parameters:**
* `job_id` (str): The ID of the job to cancel.
>
**Returns:** dict: The status of the job cancellation.
# Presigned S3 Inputs
Source: https://docs.sutro.sh/python-sdk/presigned-s3-inputs
Run production batch inference from private CSV or Parquet objects in S3, R2, Google Cloud Storage, and compatible stores.
For production batch jobs, place the input file in private object storage and give Sutro a time-limited HTTPS download URL. The SDK forwards the URL instead of loading the file or including every row in the submission request.
Amazon S3, Cloudflare R2, Google Cloud Storage, and other S3-compatible stores can all work. Sutro does not use your cloud credentials or storage API; it performs an ordinary HTTPS GET with the signed URL.
## Input contract
| Requirement | Guidance |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| URL | Provide an `https://` URL that authorizes a **GET** without additional headers. Cloud URIs such as `s3://` and `gs://` are not accepted. |
| File format | CSV and Parquet are supported. Use an object key ending in `.csv` or `.parquet`. |
| Large production jobs | Prefer an unwrapped Parquet file with normal Parquet compression, such as Snappy or Zstandard. For standalone-model priority-1 jobs, Sutro tokenizes this format in bounded chunks after downloading and mirroring the complete source object. |
| Standalone model schema | Put the text sent to the model in one column and pass its name with `column`. |
| Sutro Function schema | Use one column per Function input field. Required Function columns must be present. Do not pass `column`. |
| URL lifetime | Use the shortest lifetime that covers worst-case asynchronous pickup before the worker starts its GET, with a safety buffer. Provider policies or temporary credentials can shorten the effective lifetime. |
Treat a presigned URL like a temporary credential. It is reusable until it expires or is otherwise invalidated. Do not commit it, put it in a job name or description, or write the complete URL to application logs. Redact the entire query string in logs and error-reporting tools.
Sutro currently makes one full-object GET and does not resume or retry a failed input download. For Amazon S3, expiration is checked when the request starts, and a URL can expire earlier than `ExpiresIn` when its temporary credentials or a bucket-policy restriction expires. See [AWS's expiration guidance](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html).
## Prepare a Parquet input
For standalone model inference, create a prompt column. Other columns are not sent to the model or copied into results. Sutro converts the selected cells to strings; null and NaN values become empty prompts, and embedded line breaks become the literal `\n` sequence.
```python theme={null}
import polars as pl
inputs = pl.DataFrame(
{
"record_id": ["review-001", "review-002"],
"prompt": [
"The product was easy to use and worked well.",
"The package arrived damaged.",
],
}
)
inputs.write_parquet(
"reviews-2026-07-29.parquet",
compression="zstd",
)
```
Keep the exact source object until you reconcile the results. Sutro preserves input order, so join results by row position. A stable identifier such as `record_id` makes your pipeline easier to audit, although unused columns are not copied into standalone-model results.
For a Sutro Function, columns must instead match the Function's declared inputs. Required columns must exist; missing optional columns and extra columns are allowed. Values become strings and nulls become empty strings, so validate domain-specific values before submission.
## Sign an Amazon S3 input
In production, use separate AWS identities for producing the object and signing read access:
* The producer needs `s3:PutObject`. For a customer-managed AWS KMS key, single-part upload also needs `kms:GenerateDataKey`; multipart upload needs both `kms:GenerateDataKey` and `kms:Decrypt`.
* The read-only signer needs `s3:GetObject` or, when signing a specific version, `s3:GetObjectVersion`. For a customer-managed AWS KMS key, it also needs `kms:Decrypt`. Prefer short-lived role credentials whose session remains valid for the URL lifetime.
The producer should upload to a unique key and record the version ID when bucket versioning is enabled. The following signing step assumes a versioned bucket and a signer authenticated with the read-only identity:
```python theme={null}
import boto3
bucket = "my-production-inputs"
key = "sutro/review-sentiment/2026-07-29/reviews.parquet"
bucket_region = "us-west-2" # Replace with the bucket's actual AWS region.
version_id = "VERSION_ID_RECORDED_BY_THE_UPLOAD_PIPELINE"
signer = boto3.client("s3", region_name=bucket_region)
input_url = signer.generate_presigned_url(
ClientMethod="get_object",
Params={
"Bucket": bucket,
"Key": key,
"VersionId": version_id,
},
# One hour is illustrative. Choose the shortest practical lifetime for
# your worst-case scheduling and pickup window, with a safety buffer.
ExpiresIn=60 * 60,
)
```
Including `VersionId` pins the URL to the uploaded version. Otherwise, use a unique key for every run and never overwrite it. Keep all four S3 Block Public Access settings enabled.
## Use R2, Google Cloud Storage, or another provider
The same input contract applies to other object stores:
| Provider | How to create the URL |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Cloudflare R2 | Generate a presigned `GetObject` URL with an S3-compatible client. With Boto3, set `endpoint_url` to `https://.r2.cloudflarestorage.com` and `region_name` to `auto`, then call `generate_presigned_url` as above. R2 presigned URLs must use the R2 S3 API domain, not a custom domain. See [R2 presigned URLs](https://developers.cloudflare.com/r2/api/s3/presigned-urls/). |
| Google Cloud Storage | Generate a native V4 signed download URL with a Cloud Storage client library or `gcloud storage sign-url gs://BUCKET/OBJECT --duration=1h`. Submit the returned `https://storage.googleapis.com/...` XML API URL. See [Cloud Storage signed URLs](https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers). |
| Other S3-compatible stores | Configure the vendor's S3 endpoint and signing region in an AWS SDK, then generate a presigned GET. Verify that the resulting HTTPS URL downloads the object from outside your private network without extra authentication headers. |
Provider-specific expiration, permission, encryption, and versioning rules still apply. Keep the object private and immutable, give the signing identity read access only, and test the exact URL before submitting the full job.
## Submit a standalone model job
Pass the URL as `data`. The SDK forwards it to Sutro instead of downloading the object locally.
```python theme={null}
import sutro as so
job_id = so.infer(
data=input_url,
column="prompt",
model="gpt-oss-20b",
system_prompt="Classify each review as positive, neutral, or negative.",
job_priority=1,
stay_attached=False,
name="review-sentiment-2026-07-29",
)
print(job_id)
```
Always specify `column` for standalone-model production jobs. Otherwise Sutro uses the first column, which can silently change when an upstream schema is reordered.
The `POST /batch-inference` response only confirms that the job was created. Download and quota validation happen asynchronously. Keep access valid while the worker begins and performs its one full-object GET.
## Submit a Sutro Function job
When the object contains columns matching a published Function's inputs, call the same batch runtime through `infer()` and use the Function name as `model`:
```python theme={null}
import sutro as so
job_id = so.infer(
data=input_url,
model="lead-qualifier",
job_priority=1,
stay_attached=False,
truncate_rows=False,
name="lead-qualifier-2026-07-29",
)
```
Do not pass `column`, `system_prompt`, or `output_schema` for this form. Sutro reads the Function input columns and uses the prompt and output schema from the published Function.
`infer()` defaults to `truncate_rows=True`; set it to `False` if an oversized rendered prompt should fail instead. This URL-based path cannot create the SDK's per-row LangSmith traces because the SDK never downloads the rows.
The bounded Parquet tokenization path currently applies only to standalone-model jobs. Presigned URLs keep Function submissions and SDK memory small, but Function ingestion materializes the input after download. Split very large Function inputs into independently submitted objects.
## Estimate, monitor, and retrieve results
Check the priority-1 row and token quotas, run a small canary, and create an estimate before the full workload:
```python theme={null}
print(so.get_quotas())
estimate_job_id = so.infer(
data=input_url,
column="prompt",
model="gpt-oss-20b",
system_prompt="Classify each review as positive, neutral, or negative.",
job_priority=1,
dry_run=True,
)
print(estimate_job_id)
```
`dry_run=True` creates an estimate job, prints its estimate, and returns its ID without launching the full job. Priority-1 estimates at or above the sampling threshold run inference on a prefix of approximately 1 million input tokens.
Persist the full job's ID before doing other work. Job creation is not idempotent. The SDK deliberately does not retry `POST /batch-inference` after an HTTP 524 because the job might already have been created. Do not add a blind outer retry loop; after an ambiguous response, inspect recent jobs before resubmitting.
```python theme={null}
status = so.get_job_status(job_id)
print(status)
so.await_job_completion(
job_id,
timeout=6 * 60 * 60,
obtain_results=False,
)
if so.get_job_status(job_id) != "SUCCEEDED":
raise RuntimeError("Job did not complete successfully")
```
`obtain_results=False` avoids materializing results as JSON but returns `None` regardless of outcome. Confirm `SUCCEEDED` before requesting `GET /jobs/{job_id}/results-url?format=parquet` and downloading `urls.get`. That endpoint supports resumable range requests; see [Presigned Results URLs](/batch-api-reference/job-results-url). For smaller jobs, see [Retrieving Results](/batch-api-reference/job-results).
Results preserve the original input order. Keep the source object available until you have downloaded and reconciled the result artifact.
## Troubleshooting
| Symptom | Likely cause |
| ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The provider returns `403 Forbidden`, or Sutro cannot start the download | The URL or its signing credentials expired; it does not authorize GET; it requires signed headers Sutro does not send; or a storage/network policy blocks Sutro's request. |
| Unsupported file type | The object is not CSV or Parquet, or its key does not make the format clear. Use a `.csv` or `.parquet` suffix. |
| Column not found | `column` does not exactly match the CSV/Parquet column name. Column names are case-sensitive. |
| Function input validation failure | A required Function input column is missing. Validate values upstream because Sutro converts values to strings and nulls to empty strings. |
| Job remains pending after a download or file-preparation error | Some early ingestion failures may occur before a terminal status is recorded. Keep the job ID and contact Sutro support rather than blindly submitting duplicates. |
| A very large standalone-model CSV job fails during preparation | Convert the input to unwrapped Parquet so priority-1 tokenization can process it in bounded chunks after the complete source download. |
If the full-object GET fails, inspect the original job before generating a fresh URL and submitting a replacement. Standalone-model URL jobs mirror the raw source into Sutro-managed object storage before tokenization; the mirror and derived artifacts follow your configured [data retention policy](/concepts/data-retention). The original object remains under your provider's lifecycle policy.
# Setting Up
Source: https://docs.sutro.sh/python-sdk/setup
Getting started with our Python SDK.
# Python SDK
The Python SDK provides a Pythonic way to interact with the API. In many prototyping scenarios, you may find it most convenient to use the Python SDK and CLI to interact with Sutro.
See the [installation](/installation) guide to install the SDK.
## Basic Methods
### Configure your Sutro deployment
Create a deployment key from the **API Keys** panel in your Sutro UI. Then
set the deployment URL and key before importing or using the SDK:
If the panel is not visible, contact the Sutro team at
[team@sutro.sh](mailto:team@sutro.sh) to create a key for you.
```bash theme={null}
export SUTRO_API_URL="https://your-sutro-deployment.example.com"
export SUTRO_API_KEY="sk_..."
```
`SUTRO_API_URL` accepts either the Sutro deployment base URL or the same URL
with `/v1` appended. All SDK requests are routed through this deployment.
You can also persist both settings interactively:
```bash theme={null}
sutro login
```
Or set them in Python:
```python theme={null}
import sutro as so
so.set_api_url("https://your-sutro-deployment.example.com")
so.set_api_key("sk_...")
```
The SDK resolves configuration as a URL-and-key pair: explicit constructor
arguments or setters first, then `SUTRO_API_URL` plus `SUTRO_API_KEY`, then
`~/.sutro/config.json`. `Sutro(api_key="...")` may use `SUTRO_API_URL`, but it
does not borrow a saved deployment URL that may belong to a different saved
key. An explicit URL reuses a fallback key only when the normalized URLs match
exactly. A partial environment pair never borrows its missing value from saved
configuration. When using setters to change deployments, set the URL first and
its key second.
# Quickstart
Source: https://docs.sutro.sh/quickstart/index
Quickly get up and running with the Sutro Python SDK.
# Quickstart
Create a deployment key in the **API Keys** panel of your Sutro UI, then
configure the deployment before running these examples:
If the panel is not visible, contact the Sutro team at
[team@sutro.sh](mailto:team@sutro.sh) to create a key for you.
```bash theme={null}
export SUTRO_API_URL="https://your-sutro-deployment.example.com"
export SUTRO_API_KEY="sk_..."
```
The SDK accepts either the Sutro deployment base URL or the same URL with `/v1`
appended. You can persist the same values with `sutro login` instead.
### Using the Python SDK
Here's the simplest way to run a job with Sutro - just pass a list of inputs and a system prompt.
```python theme={null}
import sutro as so
user_reviews = [
"I loved the product! It was easy to use and had a great user interface.",
"The product was okay, but the customer support could be better.",
"I had a terrible experience with the product. It didn't work as advertised and customer service was unhelpful.",
]
results = so.infer(
user_reviews,
system_prompt="Classify the review as positive, neutral, or negative."
)
print(results)
```
This outputs a list preserving the original ordering:
```python theme={null}
[
'This review is positive.',
'This review is neutral.',
'This review is negative.'
]
```
Below are some more complex examples that show different ways to use Sutro!
## Structuring outputs
In the above example, we're trying to perform a simple classification task. In such cases, we may want structured outputs. We can accomplish this by passing in a Pydantic model or JSON schema using the `output_schema` parameter. The model will strictly adhere to this schema in its output content.
```python theme={null}
import sutro as so
from pydantic import BaseModel
user_reviews = [
"I loved the product! It was easy to use and had a great user interface.",
"The product was okay, but the customer support could be better.",
"I had a terrible experience with the product. It didn't work as advertised and customer service was unhelpful."
]
class ReviewClassification(BaseModel):
classification: str
results = so.infer(
user_reviews,
system_prompt="Classify the review as positive, neutral, or negative.",
output_schema=ReviewClassification
)
print(results)
```
Now we should obtain the following output:
```python theme={null}
[
{"classification": "positive"},
{"classification": "neutral"},
{"classification": "negative"}
]
```
Structured outputs also work well with reasoning models, since the model has the token space to go through a reasoning process before its outputs get constrained to the output schema.
```python theme={null}
import sutro as so
from pydantic import BaseModel
from typing import List
reviews = [...]
class ReviewAnalysis(BaseModel):
sentiment: str
rating: int
key_aspects: List[str]
would_recommend: bool
system_prompt = """Analyze the review and extract structured insights.
Reflect and conisider the implications of what the customer is stating
and how that may affect your analysis. Rate from 1-5."""
results = so.infer(
data=reviews,
system_prompt=system_prompt,
output_schema=ReviewAnalysis,
model='qwen-3-30b-a3b-thinking'
)
# Note the `content` and `reasoning_content` fields
print(results[0])
# >> {"content": {"sentiment": ... }, "reasoning_content": "Ok, our tasks is to..."}
```
## Working with DataFrames and Sampling Parameters
This example shows how to work with DataFrames, customize sampling parameters, and wait for job completion.
```python theme={null}
import sutro as so
import polars as pl
# Load your data
df = pl.read_csv('customer_feedback.csv')
# Run inference with custom sampling parameters
results_df = so.infer(
data=df,
column='feedback_text',
output_column='sentiment_analysis',
model='llama-3.1-70b',
system_prompt='Analyze sentiment and extract key themes',
sampling_params={
'temperature': 0.3,
'top_p': 0.9,
'max_tokens': 200
},
)
print(results_df)
```
## Multi-Model Comparison
Run the same inputs across multiple models to compare outputs and quality.
```python theme={null}
import sutro as so
prompts = [
"Explain quantum computing in simple terms",
"What are the benefits of renewable energy?",
"How does photosynthesis work?"
]
# Run same inputs across multiple models for comparison
job_ids = so.infer_per_model(
data=prompts,
models=['gemma-3-27b-it', 'qwen-2.5-32b-instruct', 'gpt-oss-20b'],
names=['gemma-27b-run', 'qwen-32b-run', 'gpt-oss-run'],
system_prompt='Provide a concise, accurate explanation',
)
# Retrieve results from each model
for model_name, job_id in zip(['gemma-27b', 'qwen-32b', 'gpt-oss'], job_ids):
results = so.await_job_completion(job_id)
print(f"\n{model_name} results:")
print(results)
```
## Cost Estimation
Before submitting production work, inspect your row and token quotas with `so.get_quotas()` or `sutro quotas`. See [Increasing Quotas](/concepts/increasing-quotas) if the input exceeds your priority-1 limits.
You can request an estimate with `dry_run=True`. The SDK prints the estimate and returns the estimate job ID; it does not return the estimate value.
```python theme={null}
import sutro as so
import polars as pl
# Load a large dataset
df = pl.read_csv('large_dataset.csv')
# The estimate is displayed automatically; retain its job ID for audit/support.
estimate_job_id = so.infer(
data=df,
column='text_column',
model='gemma-3-27b-it',
system_prompt='Summarize this text',
job_priority=1,
dry_run=True,
)
print(estimate_job_id)
```
## Using Files
You can also use files to pass in data. We currently support CSV, Parquet, and TXT files. If you're using a TXT file, each line should represent a single input. If you're using a CSV or Parquet file, you must specify the column name that contains the inputs using the `column` parameter.
```python theme={null}
import sutro as so
job_id = so.infer(
data='my_file.csv',
column='reviews',
system_prompt='Classify the review as positive, neutral, or negative.',
)
print(job_id)
```
You can view the full details of the SDK at [here](../python-sdk/setup).
## Moving to Production
So far we've shown prototyping jobs (priority 0, the default). For a production job, use priority 1 and give Sutro a presigned HTTPS GET URL for a private Parquet object in Amazon S3. This keeps the submission request small and avoids loading the full production dataset into the SDK process.
Install Boto3 if your signing environment does not already include it:
```bash theme={null}
python -m pip install boto3
```
Have your producer upload to a unique, immutable key. In production, use a separate read-only signing identity with `s3:GetObject`; the producer identity needs `s3:PutObject`. If bucket versioning is enabled, capture the uploaded version ID and follow the version-pinning example in [Presigned S3 Inputs](/python-sdk/presigned-s3-inputs).
```python theme={null}
import boto3
import sutro as so
bucket = "my-production-inputs"
key = "sutro/reviews/2026-07-29/reviews.parquet"
bucket_region = "us-west-2" # Replace with the bucket's actual AWS region.
signer = boto3.client("s3", region_name=bucket_region)
input_url = signer.generate_presigned_url(
ClientMethod="get_object",
Params={
"Bucket": bucket,
"Key": key,
},
# One hour is illustrative. Use the shortest lifetime that covers your
# worst-case scheduling and pickup window, with a safety buffer.
ExpiresIn=60 * 60,
)
job_id = so.infer(
data=input_url,
column="reviews",
system_prompt="Classify the review as positive, neutral, or negative.",
job_priority=1,
stay_attached=False,
)
```
The job ID is returned immediately. Persist it before continuing. For a production-sized result set, poll without loading the complete JSON result into the SDK process:
```python theme={null}
import sutro as so
so.await_job_completion(
job_id,
timeout=6 * 60 * 60,
obtain_results=False,
)
if so.get_job_status(job_id) != "SUCCEEDED":
raise RuntimeError("Job did not complete successfully")
```
`obtain_results=False` returns `None` for success, failure, cancellation, and timeout, so confirm `SUCCEEDED` as shown. Then request a Parquet download URL from `GET /jobs/{job_id}/results-url?format=parquet` and download `urls.get`; see [Presigned Results URLs](/batch-api-reference/job-results-url).
S3 evaluates expiration when a request starts, and a download started before expiration can finish afterward. Sutro currently issues one full-object GET and does not resume a failed input download; inspect the original job before creating a fresh URL and replacement job. Treat the full URL as a temporary credential, keep it out of logs and source control, and retain the immutable source object until results are reconciled. For version pinning, KMS permissions, retention, and troubleshooting guidance, see [Presigned S3 Inputs](/python-sdk/presigned-s3-inputs).
## Using the CLI to view job progress and results
Once you've submitted jobs via the SDK or API, you can use the CLI to view the status of the job and retrieve the results.
Viewing current and past jobs:
```
sutro jobs list
```
Retrieving job status:
```
sutro jobs status
```
Retrieving job results:
```
sutro jobs results
```
You can view the full details of the CLI at [here](/command-line-interface/account-management).
## API Usage Example
You can accomplish the same tasks using the API directly. You'll need to preprocess data as a list or array and pass it in via the parameters in the JSON body.
```python theme={null}
import requests
import json
import os
api_url = os.environ["SUTRO_API_URL"].rstrip("/")
if not api_url.endswith("/v1"):
api_url = f"{api_url}/v1"
user_reviews = [
"I loved the product! It was easy to use and had a great user interface.",
"The product was okay, but the customer support could be better.",
"I had a terrible experience with the product. It didn't work as advertised and customer service was unhelpful."
]
params = {
"model": "llama-3.1-8b",
"inputs": user_reviews,
"system_prompt": "Classify the review as positive, neutral, or negative.",
}
headers = {
"Authorization": f"Key {os.environ['SUTRO_API_KEY']}",
"Content-Type": "application/json"
}
response = requests.post(f"{api_url}/batch-inference", json=params, headers=headers)
results = response.json()
```
For more details on using the API directly, refer to the [Batch API Reference](/batch-api-reference/running-batch-inference).
# Core Workflow
Source: https://docs.sutro.sh/sutro-functions/core-workflow
The iterative loop that builds a Sutro Function.
# Core Workflow
A Sutro Function is built through an iterative loop: you create the function, Sutro generates predictions, you label the highest-impact cases, and Sutro optimizes the prompt to match your preferences. Each iteration refines the function further.
## 1. Create the function
You provide:
* **A task definition** — a plain-language description of what you want the function to do. This becomes the prompt the model receives, so be specific about the decision criteria. You write this yourself; it should reflect how you'd explain the task to a new team member.
* **A dataset** — a representative sample of your production data (CSV, JSONL, JSON, or Parquet). We find around 1,000 rows works for many tasks, but depending on how large or diverse your dataset is, more examples may be needed (i.e. 5-10k).
* **A task type** — LLM-as-a-judge, binary classification, single-label classification, multi-label classification, or structured extraction.
* **Labels or schema** — for judge and classification tasks, the set of labels to choose from. For extraction, the output schema defining which fields to extract. We recommend using a Pydantic or Zod based schema, but JSON is accepted as well.
You can also configure optional settings, for example: which model to target during optimization or whether to enable web search.
## 2. Predict
Sutro runs your task definition against the dataset. An ensemble of models processes every row independently, and Sutro analyzes where they agree and disagree. Where they disagree, those rows represent the cases where your preferences matter most.
## 3. Label
Sutro surfaces two sets of items for your review:
* **Low-confidence items** — the cases with the most model disagreement. These are the highest-value labels you can provide, because they're the ones where there is the most ambiguity on how to make the given decision.
* **High-confidence items** — cases where the models strongly agree. These are shown so you can catch cases where the consensus algorithm is confidently wrong.
For each item, you have the ability to see the input (text, image, or PDF), the consensus-chosen prediction, and can select your label.
You can optionally write a justification explaining your reasoning. This is especially valuable since it helps inform the model through nuance, opinions, or motivations that might not be captured by just the label selection.
You can also configure a **held-out set** — a fixed slice of your data that's evaluated every iteration so you can track accuracy over time.
## 4. Optimize
Once you've finished labeling, Sutro searches for a better prompt. The prompt optimizer tries different variations of your task definition, scores each one against a subset of your accumulated labels, and keeps the version that best matches your preferences. It also ensures the chosen prompt scores well across a diverse set of different edge cases.
After optimization completes, you'll see:
* **The new prompt** — the optimized task definition, which you can review and compare against the previous version via a diff view.
* **A validation score** — how well the optimized prompt agreed with your labels on a held-out validation split.
You approve the prompt (or edit it if needed), and then start the next iteration.
## The iteration loop
Each iteration builds on the last. Your labels accumulate across iterations, so the optimizer always has the full history of your preferences to work with. This means:
* **Early iterations** tend to produce the biggest gains, as the function learns your core preferences.
* **Later iterations** refine edge cases and improve consistency on harder examples.
* **You can stop whenever the function meets your needs.** Most tasks converge in 2-4 iterations.
Because labels persist, you can also come back later — add new production data that's drifted in, swap to a different model, or re-optimize — without losing the work you've already done.
## What's next
Once the function is performing well, you can [deploy it](/sutro-functions/index#executing-a-function) and invoke it by name through the batch inference SDK or API.
# Designing Your Task
Source: https://docs.sutro.sh/sutro-functions/designing-your-task
How to architect your goal into well-scoped Sutro Functions.
# Designing Your Task
The most common question when getting started with Sutro Functions is: "how should I structure this?" The right task design makes the difference between a function that converges quickly and one that struggles. This page covers the key decisions and common patterns.
## Start narrow
A Sutro Function works best when it does one thing well. If your business problem involves multiple decisions, break it into separate functions rather than forcing a single function to handle everything.
**Example:** You want to process inbound support tickets — categorize them, assess urgency, and extract key details. That's three functions:
1. A **single-label classifier** for ticket category
2. A **judge** for urgency (e.g. low / medium / high / critical)
3. A **structured extractor** for key details (customer name, product, issue summary)
Each function gets its own labels, its own optimization cycle, and its own deployment. When one needs updating, you don't risk breaking the others.
## Choosing a task type
### LLM-as-a-judge
Use for evaluative judgments where you're assessing quality, correctness, or adherence to criteria.
* "How well did the agent answer this question?" → good | acceptable | poor
* "Does this summary accurately reflect the source?" → pass | fail
Judge tasks are a natural fit for building evals for agents and LLM pipelines. The function encodes your evaluation criteria so you can run consistent evals at scale.
### Binary classification
Use when the decision is yes/no, true/false, or pass/fail.
* "Is this lead qualified?"
* "Does this document contain PII?"
* "Did the agent follow the escalation policy?"
Binary classification is the simplest task type and tends to converge fastest. When in doubt, start here.
### Single-label classification
Use when each input gets exactly one label from a set you define.
* "What category is this support ticket?" → billing / technical / account / feature\_request
* "What is the sentiment of this review?" → positive / neutral / negative
For classification tasks, smaller label sets are easier for the model to learn, but larger sets (15+) can work well too. What matters most is that your labels are clearly defined, mutually exclusive, and complete — every possible outcome should be covered, including null-type outcomes like "unknown" or "not applicable."
### Multi-label classification
Use when each input can be assigned multiple labels simultaneously.
* "What topics does this article cover?" → politics, technology, and healthcare
* "What compliance issues are present?" → data\_retention, access\_control, and encryption
* "What actions is the user attempting to perform in this turn?" -> appointment, prescription, lab\_test, referral, insurance, billing, symptom\_check
### Structured extraction
Use when you need to pull specific fields out of unstructured text.
* Extract invoice fields: vendor, amount, date, line items
* Extract contact info: name, title, company, email
* Extract legal clause details: clause type, parties, obligations
Extractive tasks work best when the answer exists in the input text. Abstractive tasks (where the model needs to generate new or lengthy text) are a weaker fit.
## Writing a good task definition
Your task definition is the seed prompt the model receives. Describe the task and its details in a simple but complete way — don't overthink it. The iteration process will back into a strong ruleset from your labels.
## Useful patterns
### DAG of Functions
Use a broad classifier to route inputs, then apply specialized functions to each category.
**Example:** Classify documents into type (contract / invoice / correspondence), then run separate extraction functions for each type. The contract extractor pulls parties and obligations; the invoice extractor pulls amounts and dates.
### Binary decomposition
When a multi-label problem is hard to optimize, break it into independent binary classifiers — one per label. Each function asks a single yes/no question and converges independently.
## Common mistakes
* **Task too broad.** "Analyze this customer interaction and determine next steps" is doing too many things. Break it down.
* **Ambiguous labels.** If your labelers would disagree on what "medium priority" means, the function will too. Define your labels precisely in the task definition.
* **Too many labels.** Every additional label increases the labeling burden and slows convergence. Start with fewer labels when possible.
* **Abstractive extraction.** Asking the function to "summarize the key points" is abstractive. Asking it to "extract the stated deadline and responsible party" is extractive. The latter works much better in Sutro.
# Overview
Source: https://docs.sutro.sh/sutro-functions/index
What Sutro Functions are and when to use them.
# Sutro Functions
Sutro Functions are task-specific judges, classifiers, and extractors that are robustly aligned with your decision preferences, easy to build, and easy to maintain..
You describe a task in plain language, upload a representative sample of your data, and Sutro surfaces the inputs where the right answer is most ambiguous. You label those cases, explain your reasoning when the decision is subjective, and Sutro optimizes a prompt that matches your preferences. The result is a deployable function you can invoke by name.
However, its also more than that: every model built in Sutro also builds its own record book of ground truth values that can continuously be used to measure future iterations against. This also means that prompt regressions and fragility is no longer a concern. Each Function's record book can be continuously added to, and quickly re-optimized against; leading to a model that has the context to make the right decision every time, even as your data changes.
## "Eval hell"
When trying to build LLM systems, we often find teams in a place we like to call "eval hell". What it usually looks like is someone has a prompt (they might've hand written, had Claude write it,, etc), and it sort of works. However, they keep running into new edges cases - either in testing or production - where it fails. They then enter a painful spiral of prompt engineering whack-a-mole, where they try to fix one issue, only to see another pop up.
With Sutro Functions we want to build a better way.
## Supported task types
Currently, Functions is mainly focused on helping people improve *decision models*. Which we support in the following ways:
* **LLM-as-a-judge** — evaluative judgments like `pass, fail` or `highly_relevant, relevant, not_relevant`
* **Binary classification** — true/false decisions (e.g. "is this lead qualified?")
* **Single-label classification** — one label from a set you define (e.g. support ticket category)
* **Multi-label classification** — multiple labels per input (e.g. content tags)
* **Structured extraction** — pull named fields from text into a schema
## Executing a Function
Once a Function is deployed, invoke it by name:
* [Python SDK](/python-sdk/functions) for `batch_run_function()` and result helpers
* [Batch API](/batch-api-reference/running-batch-inference) for asynchronous jobs over many rows
## Next steps
Understand the create → predict → label → optimize loop.
How to break down your goal into well-scoped tasks.
## FAQ
Existing labels can seed the process, but we still recommend labeling the cases Sutro surfaces and providing justifications where the task is subjective. This ensures the function learns your preferences, not just your historical labels.
Most functions converge in 2-4 iterations. Each iteration takes a few minutes of compute plus your labeling time. The total time depends on how many items you label per iteration and how subjective the task is.
Yes. Return to the function at any time, add new data or labels, re-optimize, and redeploy.
Sutro Functions work on unstructured input data (text, images, etc.) and don't require a pre-labeled training set. You start with zero labels and build them iteratively on the cases that matter most.
Keep tasks narrow. If the real business problem is complex, decompose it into several smaller functions rather than forcing one function to do everything. See [Designing Your Task](/sutro-functions/designing-your-task) for guidance.