Sampling Parameters

Sampling parameters are a dictionary of parameters that are used to control the sampling process and allow for more customization of outputs. They are passed to the model as a JSON object.

By default, we set sampling parameters appropriate for the selected model.

You can override the default sampling parameters by passing a dictionary with overrides to the sampling_params parameter in the SDK.

For example:

import sutro as so

sampling_params = {
    "temperature": 0.75,
    "max_tokens": 1024,
    "repetition_penalty": 1.15
}

results = so.infer(
    inputs=...,
    sampling_params=sampling_params
)