title: 'Language Model Overrides'
sidebar_label: 'Parameter Overrides'
description: 'Learn how to override default LLM hyperparameters in Spice.'
sidebar_position: 5
pagination_prev: null
pagination_next: null
tags:
- models
- parameters
- overrides
- configuration
Chat Completion Parameter Overrides
The v1/chat/completion (opens in a new tab) endpoint is compatible with OpenAI's API. It supports a subset of request body parameters defined in the OpenAI reference documentation (opens in a new tab). Spice helps configure different defaults for these request parameters.
Supported parameters:
Example: Setting Default Overrides
:::warning[Deprecated Default Overrides Parameters]
The openai_ prefix is deprecated for non-OpenAI model providers. Use the model provider prefix (opens in a new tab) instead.
:::
To specify a default override for a parameter, use the model provider prefix (opens in a new tab) followed by the parameter name. For example, to set the temperature parameter to 0.1 for all requests with this model for Hugging Face model, use hf_temperature: 0.1. A temperature parameter in the request body will still override the default.
models:
- name: pirate-haikus
from: openai:gpt-4o
params:
openai_temperature: 0.1
openai_response_format: { 'type': 'json_object' }
models:
- name: pirate-haikus
from: openai:gpt-4o
params:
openai_temperature: 0.1
openai_response_format: { 'type': 'json_object' }
When sending this payload to spice /v1/chat/completions:
{
"model": "pirate-haikus",
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
],
"temperature": 0.5
}
{
"model": "pirate-haikus",
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
],
"temperature": 0.5
}
Will be passed to the OpenAI API as:
{
"model": "gpt-4",
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
],
"temperature": 0.5, // temperature overriden by value in request body
"response_format": { "type": "json_object" } // default response format from model configuration
}
{
"model": "gpt-4",
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
],
"temperature": 0.5, // temperature overriden by value in request body
"response_format": { "type": "json_object" } // default response format from model configuration
}
System Prompt
In addition to any system prompts provided in message dialogue, or added by model providers, Spice can configure an additional system prompt.
models:
- name: pirate-haikus
from: openai:gpt-4o
params:
system_prompt: |
Write everything in Haiku like a pirate
models:
- name: pirate-haikus
from: openai:gpt-4o
params:
system_prompt: |
Write everything in Haiku like a pirate
Any request to HTTP v1/chat/completion (opens in a new tab) or v1/responses will include the configured system prompt. For v1/responses, the system prompt is set as the instructions field. If client-provided instructions are also included in the request, both are combined: the configured system prompt appears first, followed by the client instructions.
Example: Enforcing default structured output and using system prompt
This example demonstrates how to create a specialized math tutoring model by combining system prompts with structured JSON output.
The configuration ensures consistent, step-by-step mathematical solutions in a machine-readable format.
models:
- name: math-tutor
from: openai:gpt-4o
params:
system_prompt: |
You are a helpful math tutor. Guide the user through the solution step by step.
openai_response_format:
type: json_schema
json_schema:
name: math_reasoning
schema:
type: object
properties:
steps:
type: array
items:
type: object
properties:
explanation:
type: string
output:
type: string
required:
- explanation
- output
additionalProperties: false
final_answer:
type: string
required:
- steps
- final_answer
additionalProperties: false
strict: true
models:
- name: math-tutor
from: openai:gpt-4o
params:
system_prompt: |
You are a helpful math tutor. Guide the user through the solution step by step.
openai_response_format:
type: json_schema
json_schema:
name: math_reasoning
schema:
type: object
properties:
steps:
type: array
items:
type: object
properties:
explanation:
type: string
output:
type: string
required:
- explanation
- output
additionalProperties: false
final_answer:
type: string
required:
- steps
- final_answer
additionalProperties: false
strict: true
To use the configured math tutor, send a simple request to the chat completions endpoint:
curl -s -XPOST http://localhost:8090/v1/chat/completions -H "Content-Type: application/json" -d \
'{
"model": "math-tutor",
"messages": [{
"role": "user",
"content" :"how can I solve 8x + 7 = -23"
}]
}' \
| jq '.choices[0].message.content | fromjson'
curl -s -XPOST http://localhost:8090/v1/chat/completions -H "Content-Type: application/json" -d \
'{
"model": "math-tutor",
"messages": [{
"role": "user",
"content" :"how can I solve 8x + 7 = -23"
}]
}' \
| jq '.choices[0].message.content | fromjson'
Example response:
{
"final_answer": "x = -3.75",
"steps": [
{
"explanation": "We start with the given equation that we need to solve.",
"output": "8x + 7 = -23"
},
{
"explanation": "Our goal is to solve for x. We can start by isolating the term with x on one side of the equation. To do this, we need to eliminate the constant term (7) on the left side. We subtract 7 from both sides of the equation in order to keep it balanced.",
"output": "8x + 7 - 7 = -23 - 7"
},
{
"explanation": "Subtracting 7 from both sides simplifies the equation. On the left side, the +7 and -7 cancel out, leaving just the term with the variable.",
"output": "8x = -30"
},
{
"explanation": "Now, we have 8 times x equals -30. To solve for x, we divide both sides of the equation by the coefficient of x, which is 8.",
"output": "8x / 8 = -30 / 8"
},
{
"explanation": "Dividing both sides results in x on the left side and simplifies the fraction on the right side. The fraction -30/8 can be simplified further by dividing both the numerator and the denominator by their greatest common divisor, which is 2.",
"output": "x = -3.75"
},
{
"explanation": "The solution has been simplified completely, giving us the value of x.",
"output": "x = -3.75"
}
]
}
{
"final_answer": "x = -3.75",
"steps": [
{
"explanation": "We start with the given equation that we need to solve.",
"output": "8x + 7 = -23"
},
{
"explanation": "Our goal is to solve for x. We can start by isolating the term with x on one side of the equation. To do this, we need to eliminate the constant term (7) on the left side. We subtract 7 from both sides of the equation in order to keep it balanced.",
"output": "8x + 7 - 7 = -23 - 7"
},
{
"explanation": "Subtracting 7 from both sides simplifies the equation. On the left side, the +7 and -7 cancel out, leaving just the term with the variable.",
"output": "8x = -30"
},
{
"explanation": "Now, we have 8 times x equals -30. To solve for x, we divide both sides of the equation by the coefficient of x, which is 8.",
"output": "8x / 8 = -30 / 8"
},
{
"explanation": "Dividing both sides results in x on the left side and simplifies the fraction on the right side. The fraction -30/8 can be simplified further by dividing both the numerator and the denominator by their greatest common divisor, which is 2.",
"output": "x = -3.75"
},
{
"explanation": "The solution has been simplified completely, giving us the value of x.",
"output": "x = -3.75"
}
]
}
Visit OpenAI Structured Outputs (opens in a new tab) for more information on how to use structured output formats.
Prompt Caching
Spice supports provider-aware prompt caching to reduce latency and cost for repeated prompts. Set prompt_cache_key on a model to enable the provider's native caching mechanism.
models:
- from: openai:gpt-4o
name: my_model
params:
prompt_cache_key: "schema-context"
models:
- from: openai:gpt-4o
name: my_model
params:
prompt_cache_key: "schema-context"
When prompt_cache_key is set as a model default, it is injected into every Chat API and Responses API request to that model (unless the request itself provides one). The key is mapped into the appropriate provider-native mechanism — for example, Anthropic's cache_control, xAI's x-grok-conv-id header, or Bedrock's CachePoint block. See the full provider mapping (opens in a new tab).
For the OpenAI Responses API, prompt_cache_retention can also be set to request a retention duration (e.g. "24h").
The prompt_cache_key can also be passed per-request in the /v1/nsql API (opens in a new tab) body to enable caching for text-to-SQL queries.
For local models using mistral-rs, paged-attention scheduling is requested automatically for KV-cache prefix reuse, but it is not used everywhere: it needs a CUDA build on Unix with a CUDA device actually available, and even there the engine serves dense attention for architectures that have no paged kernel, such as the Multi-head Latent Attention GGUFs. On a from: file: model, paged_attention: disabled forces dense attention with a contiguous KV cache; a huggingface: model always takes the automatic path.
