Invoke and run AI models anywhere

How does Spice serve AI models?

Spice serves AI models from the same runtime that federates and accelerates SQL queries. Local models run inside that runtime, in the same process as the accelerated data they read. Hosted models stay with their provider, and Spice sends each call to the provider's API. Applications reach both kinds through one OpenAI-compatible HTTP API and the ai() SQL function. A model call moves through four steps:

1. Declare: the models section of spicepod.yaml names each model. The from prefix selects the provider, such as openai:gpt-4o-mini for a hosted model or huggingface:huggingface.co/<org>/<repo> for a local one.
2. Load: at startup, Spice resolves any ${secrets:...} references in model parameters from a secret store. Some providers also read cloud credentials, such as IAM roles or application default credentials. Spice downloads a Hugging Face model's weights, or reads a file: model from disk, and loads local weights with the mistral.rs inference engine.
3. Ground: the tools in a model's tools parameter, such as sql, search, and table_schema, let the model query federated and accelerated datasets while it answers.
4. Serve: applications call the model at /v1/chat/completions, or from a SQL query with ai(). Task history, which is on by default, records each call in the spice.runtime.task_history table.

The following Spicepod defines one hosted model and one local model:

version: v1
kind: Spicepod
name: model-serving

models:
  - name: hosted
    from: openai:gpt-4o-mini
    params:
      openai_api_key: ${secrets:OPENAI_API_KEY}
      system_prompt: Answer questions about the sales datasets.
      tools: auto

  - name: local
    from: huggingface:huggingface.co/lmstudio-community/Qwen2.5-Coder-3B-Instruct-GGUF
    files:
      - path: Qwen2.5-Coder-3B-Instruct-Q3_K_L.gguf

Any OpenAI-compatible client can then call either model by its Spice name:

curl http://localhost:8090/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "local", "messages": [{"role": "user", "content": "What is a Spicepod?"}], "stream": true}'

The models documentation has the full configuration reference for each provider.

Which providers and model formats can Spice run?

Spice calls hosted models through each provider's API and runs local models in the runtime. The from prefix sets the provider:

Local models load from GGUF, GGML, or SafeTensors files. Spice rejects pickle-format weights (.pt, .pth, .bin, .ckpt) unless the model sets trust_pickle: true, because loading a pickle file can run code. Release builds of the open-source runtime run local models on the CPU. CUDA and Metal GPU builds compile from source with make install-cuda or make install-metal, and Spice Cloud and Enterprise ship prebuilt CUDA images. See the distributions reference for the build matrix.

How do applications call a model served by Spice?

Spice exposes each model through several interfaces, so one model can serve a chat application and a batch SQL job at the same time:

The ai() function sends one model call per row and runs the calls in parallel. It splits large inputs into batches of up to 1,000 rows, and each message can be up to 1 MB. LLM inference from SQL shows the function in larger workloads.

What controls does Spice apply to model calls?

Spice applies the same runtime controls to every model, hosted or local. Configure them per model in the Spicepod:

• Tool access: a model calls only the tools in its tools parameter. The MCP server gateway adds tools from external MCP servers to the same list.
• Rate limits: max_concurrency and requests_per_minute_limit cap the calls Spice sends to a model. Defaults follow the provider's published limits, and the cap also applies to the parallel calls from ai().
• Tool recursion: tool_recursion_limit, 10 by default, stops a chain of tool calls inside one request from looping.
• Secrets: ${secrets:...} references resolve API keys at startup, so keys stay out of the Spicepod file.
• Observability: Spice records chat and completion calls as ai_chat and ai_completion tasks with their token counts. It also exports llm_prompt_tokens_total and llm_completion_tokens_total metrics and accepts a traceparent header.

For isolation beyond these controls, see secure AI sandboxing. To run model serving as a managed service, compare Spice Cloud and Enterprise plans.