spiceai/docs

spiceai/

docs

Help Login

trunk

Edit on GitHub

Fork

/docs/website/versioned_docs/version-1.11.x/components/models/index.md

spiceai/docs | Spice Cloud Platform

trunk

Edit on GitHub

Fork

/docs/website/versioned_docs/version-1.11.x/components/models/index.md

spiceai/docs/README.md

title: 'Model Providers' sidebar_label: 'Model Providers' description: 'Overview of supported model providers for ML and LLMs in Spice.' image: /img/og/models.png

Spice supports various model providers for traditional machine learning (ML) models and large language models (LLMs).

Name	Description	Status	ML Format(s)	LLM Format(s)*
`openai`	OpenAI (or compatible) LLM endpoint	Stable	-	OpenAI-compatible HTTP endpoint
`bedrock`	Amazon Bedrock	Alpha	-	OpenAI-compatible HTTP endpoint
`xai`	Models hosted on xAI	Alpha	-	OpenAI-compatible HTTP endpoint
`file`	Local filesystem	Release Candidate	ONNX	GGUF, GGML, SafeTensor
`huggingface`	Models hosted on HuggingFace	Release Candidate	ONNX	GGUF, GGML, SafeTensor
`spice.ai`	Models hosted on the Spice.ai Cloud Platform	Release Candidate	ONNX	OpenAI-compatible HTTP endpoint
`azure`	Azure OpenAI	Alpha	-	OpenAI-compatible HTTP endpoint
`anthropic`	Models hosted on Anthropic	Alpha	-	OpenAI-compatible HTTP endpoint
`google`	Google AI language models	Alpha	-	OpenAI-compatible HTTP endpoint
`xai`	Models hosted on xAI	Alpha	-	OpenAI-compatible HTTP endpoint
`databricks`	Models deployed to Databricks Mosaic AI	Alpha	-	OpenAI-compatible HTTP endpoint

Spice also tests and evaluates common models and grades their ability to integrate with Spice. See the Models Grade Report.

*LLM Format(s) may require additional files (e.g., tokenizer_config.json).

The model type is inferred based on the model source and files. For more detail, refer to the model reference specification.

Features

Spice supports a variety of features for large language models (LLMs):

Custom Tools: Provide models with tools to interact with the Spice runtime. See Tools.
System Prompts: Declaratively define system prompts and default values for v1/chat/completion parameters. See Parameter Overrides. Use Jinja templating to parameterize system prompts per request. See Parameterized prompts.
Memory: Provide LLMs with memory persistence tools to store and retrieve information across conversations. See Memory.
Vector Search: Perform advanced vector-based searches using embeddings. See Vector Search.
Evals: Evaluate, track, compare, and improve language model performance for specific tasks. See Evals.
Local Models: Load and serve models locally from various sources, including local filesystems and Hugging Face. See Local Models.

For more details, refer to the Large Language Models documentation.

Model Provider Prefix

The model provider prefix identifies the source or provider of a model in Spice configuration files. This prefix is specified before the model identifier in the from field of a model definition, and is used in specifying model default parameter overrides. It helps the runtime determine how to load and interact with the model.

The following provider prefixes are supported:

Prefix	Description
`openai`	OpenAI or OpenAI-compatible endpoints
`azure`	Azure OpenAI
`xai`	xAI
`anthropic`	Anthropic
`google`	Google AI
`perplexity`	Perplexity
`hf`	Hugging Face
`file`	Local filesystem
`spiceai`	Spice.ai Cloud Platform
`databricks`	Databricks Mosaic AI
`bedrock`	Amazon Bedrock

Example usage in spicepod.yaml:

Model Examples

The following examples demonstrate how to configure and use various models or model features with Spice. Each example provides a specific use case to help understand the configuration options available.

Example: Configuring an OpenAI Model

To use a language model hosted on OpenAI (or compatible), specify the openai path and model ID in from. For more details, see OpenAI Model Provider.

Example spicepod.yml:

Example: Using an OpenAI Model with Tools

To specify tools for an OpenAI model, include them in the params.tools field. For more details, see the Tools documentation.

Example: Adding Memory to a Model

To enable memory tools for a model, define a store memory dataset and specify memory in the model's tools parameter. For more details, see the Memory documentation.

Example: Setting Default Parameter Overrides

To set default overrides for parameters, use the model provider prefix followed by the parameter name. For more details, see the Parameter Overrides documentation.

Example: Configuring a System Prompt

To configure an additional system prompt, use the system_prompt parameter. For more details, see the Parameter Overrides documentation.

Example: Serving a Local Model

To serve a model from the local filesystem, specify the from path as file and provide the local path. For more details, see Filesystem Model Provider.

Example: Analyzing GitHub Issues with a Chat Model

This example demonstrates how to pull GitHub issue data from the last 14 days, accelerate the data, create a chat model with memory and tools to access the accelerated data, and use Spice to ask the chat model about the general themes of new issues.

Step 1: Pull GitHub Issue Data

First, configure a dataset to pull GitHub issue data from the last 14 days.

Step 2: Create a Chat Model with Memory and Tools

Next, create a chat model that includes memory and tools to access the accelerated GitHub issue data.

Step 3: Query the Chat Model

At this step, the spicepod.yaml should look like:

Finally, use Spice to ask the chat model about the general themes of new issues in the last 14 days. The following curl command demonstrates how to make this request using the OpenAI-compatible API.

Refer to the Create Chat Completion API documentation for more details on making chat completion requests.

import DocCardList from '@theme/DocCardList';