> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sutro.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Batch Inference

> Reference documentation for Batch Inference.

### Running batch inference

```Python theme={null}
infer(self, data, model='gpt-oss-20b', name=None, description=None, column=None, output_column='inference_result', job_priority=0, output_schema=None, sampling_params=None, system_prompt=None, dry_run=False, stay_attached=None, random_seed_per_input=False, truncate_rows=True, id_column=None)
```

Run LLM inference on a large list, table, dataframe, local file, dataset, or HTTP(S) download URL.

#### Parameters:

* `data` (Union\[List, pd.DataFrame, pl.DataFrame, str]): The data to run inference on. A string can be a local file path, dataset ID, or HTTP(S) presigned download URL for a CSV or Parquet object.

* `model` (str, optional): The model or published Function to use for inference. Defaults to `"gpt-oss-20b"`. To run the same data across multiple models, use `infer_per_model(models=[...])`.

* `name` (str, optional): A job name for experiment and metadata tracking. Defaults to `None`.

* `description` (str, optional): A job description for experiment and metadata tracking. Defaults to `None`.

* `column` (Union\[str, List\[str]], optional): The column name to use for standalone-model inference. It is required for a DataFrame, local CSV/Parquet file, or dataset. For a standalone-model download URL, it selects the input column; if omitted, the first column is used. If a list is supplied for an in-memory DataFrame, it concatenates the named columns and literal separator strings. Omit it for a published Function URL, which reads the Function's declared input fields.

* `id_column` (str, optional): For an HTTP(S) CSV or Parquet download URL, the column containing user-provided row IDs. The IDs are carried into results so they can be joined back to the source table. This parameter is not supported for lists, DataFrames, local files, or dataset IDs.

* `output_column` (str, optional): The output column name when the SDK retrieves attached results. Defaults to `"inference_result"`. For a detached job, pass the same value when later calling `await_job_completion()` or `get_job_results()`.

* `job_priority` (int, optional): The [priority](/concepts/job-priority) of the job. Default is 0.

* `output_schema` (Union\[Dict\[str, Any], BaseModel], optional): A structured schema for the output. Can be either a dictionary representing a JSON schema or a pydantic BaseModel. Defaults to None.

* `system_prompt` (str, optional): A system prompt to add to all inputs. This allows you to define the behavior of the model. Defaults to None.

* `sampling_params` (dict, optional): A dictionary of sampling parameters to use for the inference. Defaults to None, which uses the default sampling parameters.

* `random_seed_per_input` (bool, optional): If True, a random seed will be generated for each input. This is useful for diversity in outputs. Defaults to False.

* `dry_run` (bool, optional): If `True`, submit an estimate job, wait for its estimate, print the estimate, and return the estimate job ID. This does not launch the normal full job, but sufficiently large priority-1 estimates run inference on an approximately 1-million-token prefix sample. Defaults to `False`.

* `stay_attached` (bool, optional): If True, the SDK will stay attached to the job and update you on the status and results as they become available. Default behavior is True for priority 0 jobs, and False for priority 1 jobs.

* `truncate_rows` (bool, optional): If True, any rows that have a token count exceeding the context window length of the selected model will be truncated to the max length that will fit within the context window. Defaults to True.

**Returns:**

str: The ID of the inference job.

### Production inputs from Amazon S3

For large production jobs, pass a presigned S3 GET URL instead of loading the file into the SDK process:

```python theme={null}
import sutro as so

job_id = so.infer(
    data=presigned_url,
    column="prompt",
    id_column="row_id",
    model="gpt-oss-20b",
    system_prompt="Classify each input.",
    job_priority=1,
    stay_attached=False,
)

results = so.await_job_completion(job_id)
# Results include row_id alongside inference_result.
```

Use an unwrapped Parquet object for the most memory-efficient priority-1 tokenization path. Give the URL enough lifetime for Sutro to begin its one full-object download, with a scheduling safety buffer, and treat the complete URL as a credential. See [Presigned S3 Inputs](/python-sdk/presigned-s3-inputs) for file schemas, URL generation, security guidance, and production troubleshooting.

The ID column is returned even when inputs are not requested. Sutro preserves ID values but does not guarantee preservation of the source file's physical integer type.

### Monitoring job status

```Python theme={null}
attach(self, job_id: str)
```

Attach to an existing job and stream its progress in real-time. This has the equivalent behavior of setting `stay_attached=True` when calling `infer(...)`

>

This method connects to a running job and displays live progress updates, including the number of rows processed and token statistics. It shows a progress bar with real-time updates until the job completes.

>

**Parameters:**

* `job_id` (str): The ID of the job to attach to

>

**Returns:** None

>

**Job Status Behavior:**

* `RUNNING`: Streams progress updates with a live progress bar and job statistics
* `SUCCEEDED`: Notifies that the job already completed and suggests using `sutro jobs results`
* `FAILED`: Displays failure message and exits
* `CANCELLED`: Displays cancellation message and exits

>

**Example:**

```python theme={null}
# Attach to a running job to monitor its progress
sutro.attach("job_12345")
# Progress bar will display:
# Progress: 45%[████████████████                ] 450/1000 [00:32\<00:45] Input tokens processed: 12500, Tokens generated: 8300, Total tokens/s: 325.4
```

>

**Note:** This method is ideal for monitoring long-running jobs interactively. For programmatic use cases where you don't want live progress updates, use the simpler `await_job_completion()` instead.

## Await Job Completion

```Python theme={null}
await_job_completion(self, job_id: str, timeout: int = 7200, obtain_results: bool = True, output_column: str = 'inference_result', is_cost_estimate: bool = False) → pl.DataFrame | None
```

When deployed as part of a pipeline (Dagster, Airflow, etc) you might not be interested in seeing the progress of the job as it happens. `await_job_completion` is best for this use case, and should only be used when not using the `stay_attached` parameter of `infer(...)`, or the `attach(...)` function.

>

Waits for a job to reach a terminal state. By default, a successful job is retrieved through the SDK's JSON results path and returned as a Polars DataFrame.

>

This method polls the job status every 5 seconds (and prints it out) until the job completes, fails, is cancelled, or the timeout is reached.

>

**Parameters:**

* `job_id` (str): The ID of the job to await.
* `timeout` (int): Maximum time in seconds to wait for job completion. Defaults to 7200 (2 hours). Passing `None` is not supported.
* `obtain_results` (bool): Whether to retrieve and materialize results after the job succeeds. Defaults to `True`. Set this to `False` for large production results, then use a Parquet [results download URL](/batch-api-reference/job-results-url).
* `output_column` (str): Name of the output column in the returned DataFrame. Defaults to `"inference_result"`.
* `is_cost_estimate` (bool): Suppresses normal job-progress messaging for the SDK's cost-estimate workflow. Most callers should leave this as `False`.

>

**Returns:** `pl.DataFrame | None`: A Polars DataFrame when the job succeeds and `obtain_results=True`; otherwise `None`.

>

**Job Status Outcomes:**

* `SUCCEEDED`: Returns a Polars DataFrame when `obtain_results=True`; otherwise returns `None`
* `FAILED`: Returns `None`
* `CANCELLED`: Returns `None`
* Timeout reached: Returns `None`

>

**Example:**

```python theme={null}
results = so.await_job_completion("job_12345", timeout=3600)
# Job status is RUNNING for job-f9102252-ae2f-4d61-a879-a657e314f2e0
if results is not None:
    print(f"Job completed with {len(results)} results")
```

For a large production job, wait without materializing JSON:

```python theme={null}
so.await_job_completion(job_id, timeout=6 * 60 * 60, obtain_results=False)

if so.get_job_status(job_id) != "SUCCEEDED":
    raise RuntimeError("Job did not complete successfully")
```

Because `obtain_results=False` returns `None` for every terminal outcome and timeout, confirm `SUCCEEDED` before requesting and downloading the unified Parquet artifact through [Getting a Results Download URL](/batch-api-reference/job-results-url).

## Getting Quotas

```Python theme={null}
get_quotas(self)
```

Get your current quotas.

**Returns:** list: A list of quotas, one for each priority level. Contains row\_quota and token\_quota for each priority level.