Workers in the Spice runtime represent configurable units of compute that help coordinate and manage interactions between models and tools. Currently, workers define how one or more llms can be combined into a logically single model.
workersThe workers section in your configuration specifies one or more workers.
Example:
nameA unique identifier for this worker component.
descriptionAdditional details about the worker, useful for displaying to users and providing to LLM context.
cronSpecifies a cron schedule to automatically run the worker at the specified times. The worker action controls the behavior of the schedule. See the cron schedule reference for more information on cron schedules.
cron with a load_balance actionWhen a load_balance action is specified with a cron schedule, the params.prompt parameter is used to automatically request a chat completion. When no params.prompt parameter is specified, the cron schedule is not activated.
cron with a sql actionWhen a sql action is specified with a cron schedule, the worker runs the SQL at the specified scheduled times.
load_balanceSpecifies the configuration for a load_balance worker. When a load_balance section is present, other worker actions cannot be specified (e.g. sql).
load_balance.routingA list of model configurations that define how the load balancing behaves.
The elements' structure uniquely determine the model worker algorithm. List elements should be of consistent type.
| Key name | Key type | Description |
|---|---|---|
| from | String | The model.name of a defined model spicepod component. |
| order | Integer, positive | The priority of the model in order. The lowest value is used first, followed by increasing order. The ordering of models with equal order is undefined. |
Example
The worker selects each model in turn for subsequent requests.
Example
The worker uses the models in increasing order, returning the first result that is not an error.
Example
The worker routes traffic to the models in accordance to the weighting (i.e. 80% to llama3_2, 20% to gpt4_1).
sqlSpecifies an SQL query action to run for this worker. When specified without a cron parameter, the worker does nothing.
When this parameter is present, other worker actions cannot be specified (e.g. load_balance)
paramsOptional, additional parameters for the specified worker action.
params.promptValid only when the load_balance worker action is specified with a cron schedule, otherwise ignored. The value specified by this parameter is used as the input to a new chat completion request on the specified cron schedule for the worker.