Batch Inference

​

Running batch inference

infer(self, data, model='gpt-oss-20b', name=None, description=None, column=None, output_column='inference_result', job_priority=0, output_schema=None, sampling_params=None, system_prompt=None, dry_run=False, stay_attached=None, random_seed_per_input=False, truncate_rows=True, id_column=None)

​

Parameters:

• data (Union[List, pd.DataFrame, pl.DataFrame, str]): The data to run inference on. A string can be a local file path, dataset ID, or HTTP(S) presigned download URL for a CSV or Parquet object.
• model (str, optional): The model or published Function to use for inference. Defaults to "gpt-oss-20b". To run the same data across multiple models, use infer_per_model(models=[...]).
• name (str, optional): A job name for experiment and metadata tracking. Defaults to None.
• description (str, optional): A job description for experiment and metadata tracking. Defaults to None.
• column (Union[str, List[str]], optional): The column name to use for standalone-model inference. It is required for a DataFrame, local CSV/Parquet file, or dataset. For a standalone-model download URL, it selects the input column; if omitted, the first column is used. If a list is supplied for an in-memory DataFrame, it concatenates the named columns and literal separator strings. Omit it for a published Function URL, which reads the Function’s declared input fields.
• id_column (str, optional): For an HTTP(S) CSV or Parquet download URL, the column containing user-provided row IDs. The IDs are carried into results so they can be joined back to the source table. This parameter is not supported for lists, DataFrames, local files, or dataset IDs.
• output_column (str, optional): The output column name when the SDK retrieves attached results. Defaults to "inference_result". For a detached job, pass the same value when later calling await_job_completion() or get_job_results().
• job_priority (int, optional): The priority of the job. Default is 0.
• output_schema (Union[Dict[str, Any], BaseModel], optional): A structured schema for the output. Can be either a dictionary representing a JSON schema or a pydantic BaseModel. Defaults to None.
• system_prompt (str, optional): A system prompt to add to all inputs. This allows you to define the behavior of the model. Defaults to None.
• sampling_params (dict, optional): A dictionary of sampling parameters to use for the inference. Defaults to None, which uses the default sampling parameters.
• random_seed_per_input (bool, optional): If True, a random seed will be generated for each input. This is useful for diversity in outputs. Defaults to False.
• dry_run (bool, optional): If True, submit an estimate job, wait for its estimate, print the estimate, and return the estimate job ID. This does not launch the normal full job, but sufficiently large priority-1 estimates run inference on an approximately 1-million-token prefix sample. Defaults to False.
• stay_attached (bool, optional): If True, the SDK will stay attached to the job and update you on the status and results as they become available. Default behavior is True for priority 0 jobs, and False for priority 1 jobs.
• truncate_rows (bool, optional): If True, any rows that have a token count exceeding the context window length of the selected model will be truncated to the max length that will fit within the context window. Defaults to True.

​

Production inputs from Amazon S3

import sutro as so

job_id = so.infer(
    data=presigned_url,
    column="prompt",
    id_column="row_id",
    model="gpt-oss-20b",
    system_prompt="Classify each input.",
    job_priority=1,
    stay_attached=False,
)

results = so.await_job_completion(job_id)
# Results include row_id alongside inference_result.

​

Monitoring job status

attach(self, job_id: str)

• job_id (str): The ID of the job to attach to

• RUNNING: Streams progress updates with a live progress bar and job statistics
• SUCCEEDED: Notifies that the job already completed and suggests using sutro jobs results
• FAILED: Displays failure message and exits
• CANCELLED: Displays cancellation message and exits

# Attach to a running job to monitor its progress
sutro.attach("job_12345")
# Progress bar will display:
# Progress: 45%[████████████████                ] 450/1000 [00:32\<00:45] Input tokens processed: 12500, Tokens generated: 8300, Total tokens/s: 325.4

​

Await Job Completion

await_job_completion(self, job_id: str, timeout: int = 7200, obtain_results: bool = True, output_column: str = 'inference_result', is_cost_estimate: bool = False) → pl.DataFrame | None

• job_id (str): The ID of the job to await.
• timeout (int): Maximum time in seconds to wait for job completion. Defaults to 7200 (2 hours). Passing None is not supported.
• obtain_results (bool): Whether to retrieve and materialize results after the job succeeds. Defaults to True. Set this to False for large production results, then use a Parquet results download URL.
• output_column (str): Name of the output column in the returned DataFrame. Defaults to "inference_result".
• is_cost_estimate (bool): Suppresses normal job-progress messaging for the SDK’s cost-estimate workflow. Most callers should leave this as False.

• SUCCEEDED: Returns a Polars DataFrame when obtain_results=True; otherwise returns None
• FAILED: Returns None
• CANCELLED: Returns None
• Timeout reached: Returns None

results = so.await_job_completion("job_12345", timeout=3600)
# Job status is RUNNING for job-f9102252-ae2f-4d61-a879-a657e314f2e0
if results is not None:
    print(f"Job completed with {len(results)} results")

so.await_job_completion(job_id, timeout=6 * 60 * 60, obtain_results=False)

if so.get_job_status(job_id) != "SUCCEEDED":
    raise RuntimeError("Job did not complete successfully")

​

Getting Quotas

get_quotas(self)