# Model Providers

Overview of supported model providers for LLMs in Spice.

Spice supports various model providers for large language models (LLMs).

> **Note**: Support for traditional machine learning (ONNX) models was removed in v2.2. See [Machine Learning Models](https://spiceai.org/docs/features/machine-learning-models) for details.

| Name | Description | Status | LLM Format(s)\* |
| - | - | - | - |
| [`openai`](https://spice.ai/docs/building-blocks/model-providers/openai) | OpenAI (or compatible) LLM endpoint | Stable | OpenAI-compatible HTTP endpoint |
| [`bedrock`](https://spice.ai/docs/building-blocks/model-providers/bedrock) | Amazon Bedrock | Alpha | OpenAI-compatible HTTP endpoint |
| [`xai`](https://spice.ai/docs/building-blocks/model-providers/xai) | Models hosted on xAI | Alpha | OpenAI-compatible HTTP endpoint |
| [`file`](https://spice.ai/docs/building-blocks/model-providers/filesystem) | Local filesystem | Release Candidate | GGUF, GGML, SafeTensor |
| [`huggingface`](https://spice.ai/docs/building-blocks/model-providers/huggingface) | Models hosted on HuggingFace | Release Candidate | GGUF, GGML, SafeTensor |
| [`spice.ai`](https://spice.ai/docs/building-blocks/model-providers/spiceai) | Models hosted on the Spice Cloud Platform | Release Candidate | OpenAI-compatible HTTP endpoint |
| [`azure`](https://spice.ai/docs/building-blocks/model-providers/azure) | Azure OpenAI | Alpha | OpenAI-compatible HTTP endpoint |
| [`anthropic`](https://spice.ai/docs/building-blocks/model-providers/anthropic) | Models hosted on Anthropic | Alpha | OpenAI-compatible HTTP endpoint |
| [`google`](https://spice.ai/docs/building-blocks/model-providers/google) | Google Vertex AI language models | Alpha | Vertex AI HTTP endpoint |
| [`databricks`](https://spice.ai/docs/building-blocks/model-providers/databricks) | Models deployed to Databricks Mosaic AI | Alpha | OpenAI-compatible HTTP endpoint |
| ~~`perplexity`~~ | ~~Perplexity~~ ([Deprecated](https://spice.ai/docs/building-blocks/model-providers/perplexity)) | Deprecated | - |

Spice also serves decision models, which answer typed questions through the SQL decision functions and `POST /v1/decisions`. [TypeSafe](https://spice.ai/docs/building-blocks/model-providers/typesafe) Jev answers decisions only, rather than chat completions. OpenAI `gpt-6-luna` answers decisions through OpenAI's Decisions API and also serves chat completions like any other OpenAI model. Chat models answer decisions as well. See [Decisions](https://spiceai.org/docs/features/large-language-models/decisions).

Spice also tests and evaluates common models and grades their ability to integrate with Spice. See the [Models Grade Report](https://spiceai.org/docs/reference/models).

\*LLM Format(s) may require additional files (e.g., `tokenizer_config.json`).

The model type is inferred based on the model source and files. For more detail, refer to the `model` [reference specification](https://spiceai.org/docs/reference/spicepod/models).

## Features

Spice supports a variety of features for large language models (LLMs):

- **Custom Tools**: Provide models with tools to interact with the Spice runtime. See [Tools](https://spiceai.org/docs/features/large-language-models/tools).
- **System Prompts**: Declaratively define system prompts and default values for [`v1/chat/completion`](https://spiceai.org/docs/api/HTTP/post-chat-completions) parameters. See [Parameter Overrides](https://spiceai.org/docs/features/large-language-models/parameter_overrides). Use Jinja templating to parameterize system prompts per request. See [Parameterized prompts](https://spiceai.org/docs/features/large-language-models/parameterized_prompts).
- **Memory**: Provide LLMs with memory persistence tools to store and retrieve information across conversations. See [Memory](https://spiceai.org/docs/features/large-language-models/memory).
- **Vector Search**: Perform advanced vector-based searches using embeddings. See [Vector Search](https://spiceai.org/docs/features/search/vector-search).
- **Local Models**: Load and serve models locally from various sources, including local filesystems and Hugging Face. See [Local Models](https://spiceai.org/docs/features/large-language-models/serving).

For more details, refer to the [Large Language Models documentation](https://spiceai.org/docs/features/large-language-models).

## Model Provider Prefix

The model provider prefix identifies the source or provider of a model in Spice configuration files. This prefix is specified before the model identifier in the `from` field of a model definition, and is used in specifying model [default parameter overrides](https://spice.ai/docs/building-blocks/model-providers#example-setting-default-parameter-overrides). It helps the runtime determine how to load and interact with the model.

The following provider prefixes are supported:

| Prefix | Description |
| - | - |
| `openai` | OpenAI or OpenAI-compatible endpoints |
| `azure` | Azure OpenAI |
| `xai` | xAI |
| `anthropic` | Anthropic |
| `google` | Google Vertex AI |
| `hf` | Hugging Face |
| `file` | Local filesystem |
| `spiceai` | Spice Cloud |
| `databricks` | Databricks Mosaic AI |
| `bedrock` | Amazon Bedrock |
| `typesafe` | TypeSafe |

**Example usage in `spicepod.yaml`:**

```yaml
models:
  - from: openai:gpt-4o
    name: openai-model

  - from: hf:meta-llama/Llama-3-8B-Instruct
    name: llama3-hf

  - from: file://absolute/path/to/model.gguf
    name: local-model
```

## Model Examples

The following examples demonstrate how to configure and use various models or model features with Spice. Each example provides a specific use case to help understand the configuration options available.

### Example: Configuring an OpenAI Model

To use a language model hosted on OpenAI (or compatible), specify the `openai` path and model ID in `from`. For more details, see [OpenAI Model Provider](https://spice.ai/docs/building-blocks/model-providers/openai).

Example `spicepod.yml`:

```yaml
models:
  - from: openai:gpt-4o-mini
    name: openai
    params:
      openai_api_key: ${ secrets:SPICE_OPENAI_API_KEY }

  - from: openai:llama3-groq-70b-8192-tool-use-preview
    name: groq-llama
    params:
      endpoint: https://api.groq.com/openai/v1
      openai_api_key: ${ secrets:SPICE_GROQ_API_KEY }
```

### Example: Using an OpenAI Model with Tools

To specify tools for an OpenAI model, include them in the `params.tools` field. For more details, see the [Tools documentation](https://spiceai.org/docs/features/large-language-models/tools).

```yaml
models:
  - name: sql-model
    from: openai:gpt-4o
    params:
      tools: list_datasets, sql, table_schema
```

### Example: Adding Memory to a Model

To enable memory tools for a model, define a `store` memory dataset and specify `memory` in the model's `tools` parameter. For more details, see the [Memory documentation](https://spiceai.org/docs/features/large-language-models/memory).

```yaml
datasets:
  - from: memory:store
    name: llm_memory
    access: read_write

models:
  - name: memory-enabled-model
    from: openai:gpt-4o
    params:
      tools: memory, sql
```

### Example: Setting Default Parameter Overrides

To set default overrides for parameters, use the [model provider prefix](https://spice.ai/docs/building-blocks/model-providers#model-provider-prefix) followed by the parameter name. For more details, see the [Parameter Overrides documentation](https://spiceai.org/docs/features/large-language-models/parameter_overrides).

```yaml
models:
  - name: pirate-haikus
    from: openai:gpt-4o
    params:
      openai_temperature: 0.1
      openai_response_format: { 'type': 'json_object' }
```

### Example: Configuring a System Prompt

To configure an additional system prompt, use the `system_prompt` parameter. For more details, see the [Parameter Overrides documentation](https://spiceai.org/docs/features/large-language-models/parameter_overrides).

```yaml
models:
  - name: pirate-haikus
    from: openai:gpt-4o
    params:
      system_prompt: |
        Write everything in Haiku like a pirate
```

### Example: Serving a Local Model

To serve a model from the local filesystem, specify the `from` path as `file` and provide the local path. For more details, see [Filesystem Model Provider](https://spice.ai/docs/building-blocks/model-providers/filesystem).

```yaml
models:
  - from: file://absolute/path/to/my/model.gguf
    name: local_fs_model
```

### Example: Analyzing GitHub Issues with a Chat Model

This example demonstrates how to pull GitHub issue data from the last 14 days, accelerate the data, create a chat model with memory and tools to access the accelerated data, and use Spice to ask the chat model about the general themes of new issues.

#### Step 1: Pull GitHub Issue Data

First, configure a dataset to pull GitHub issue data from the last 14 days.

```yaml
datasets:
  - from: github:github.com/<owner>/<repo>/issues
    name: github_issues
    params:
      github_token: ${secrets:GITHUB_TOKEN}
    acceleration:
      enabled: true
      refresh_mode: append
      refresh_check_interval: 24h
      refresh_data_window: 14d
```

#### Step 2: Create a Chat Model with Memory and Tools

Next, create a chat model that includes memory and tools to access the accelerated GitHub issue data.

```yaml
datasets:
  - from: memory:store
    name: llm_memory
    access: read_write

models:
  - name: github-issues-analyzer
    from: openai:gpt-4o
    params:
      tools: memory, sql
```

#### Step 3: Query the Chat Model

At this step, the `spicepod.yaml` should look like:

```yaml
datasets:
  - from: github:github.com/<owner>/<repo>/issues
    name: github_issues
    params:
      github_token: ${secrets:GITHUB_TOKEN}
    acceleration:
      enabled: true
      refresh_mode: append
      refresh_check_interval: 24h
      refresh_data_window: 14d

  - from: memory:store
    name: llm_memory
    access: read_write

models:
  - name: github-issues-analyzer
    from: openai:gpt-4o
    params:
      openai_api_key: ${ secrets:SPICE_OPENAI_API_KEY }
      tools: memory, sql
```

Finally, use Spice to ask the chat model about the general themes of new issues in the last 14 days. The following `curl` command demonstrates how to make this request using the OpenAI-compatible API.

```bash
curl -X POST http://localhost:8090/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "github-issues-analyzer",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What are the general themes of new issues in the last 14 days?"}
    ]
  }'
```

Refer to the [Create Chat Completion API documentation](https://spiceai.org/docs/api/HTTP/post-chat-completions) for more details on making chat completion requests.