spiceai/docs | Spice Cloud Platform

docs

spiceai/docs | Spice Cloud Platform

spiceai/docs/README.md

title: 'Filesystem Hosted Models' description: 'Instructions for using models hosted on a filesystem with Spice.' sidebar_label: 'Filesystem' sidebar_position: 5

To use a model hosted on a filesystem, specify the path to the model file or folder in the from field:

Supported formats include GGUF, GGML, and SafeTensor for large language models (LLMs) and ONNX for traditional machine learning (ML) models.

Configuration

`from`

An absolute or relative path to the model file or folder:

`params` (optional)

Param	Description
`tools`	Which tools (opens in a new tab) should be made available to the model. Set to `auto` to use all available tools.
`system_prompt`	An additional system prompt used for all chat completions to this model.
`chat_template`	Customizes the transformation of OpenAI chat messages into a character stream for the model. See Overriding the Chat Template (opens in a new tab).

See Large Language Models (opens in a new tab) for additional configuration options.

`files` (optional)

The files field specifies additional files required by the model, such as tokenizer, configuration, and other files.

Examples

Loading a GGML Model

Example: Loading a SafeTensor Model

Loading LLM from a directory

Note: The folder provided should contain all the expected files (see examples above).

Loading an ONNX Model

Loading a GGUF Model

Overriding the Chat Template

Chat templates convert the OpenAI compatible chat messages (see format (opens in a new tab)) and other components of a request into a stream of characters for the language model. It follows Jinja3 templating syntax (opens in a new tab).

Further details on chat templates can be found here (opens in a new tab).

Templating Variables

messages: List of chat messages, in the OpenAI format (opens in a new tab).
add_generation_prompt: Boolean flag whether to add a generation prompt (opens in a new tab).
tools: List of callable tools, in the OpenAI format (opens in a new tab).

:::warning[Limitations]

The throughput, concurrency & latency of a locally hosted model will vary based on the underlying hardware and model size. Spice supports Apple metal (opens in a new tab) and CUDA (opens in a new tab) for accelerated inference.

:::

models:
  - from: file://models/llms/llama3.2-1b-instruct/
    name: llama3

models:
  - from: file://models/llms/llama3.2-1b-instruct/
    name: llama3

from: file://absolute/path/models/llms/llama3.2-1b-instruct/
from: file:models/llms/llama3.2-1b-instruct/

from: file://absolute/path/models/llms/llama3.2-1b-instruct/
from: file:models/llms/llama3.2-1b-instruct/

- name: local-model
  from: file://models/llms/llama3.2-1b-instruct/model.safetensors
  files:
    - path: //models/llms/llama3.2-1b-instruct/tokenizer.json
    - path: //models/llms/llama3.2-1b-instruct/tokenizer_config.json
    - path: //models/llms/llama3.2-1b-instruct/config.json

- name: local-model
  from: file://models/llms/llama3.2-1b-instruct/model.safetensors
  files:
    - path: //models/llms/llama3.2-1b-instruct/tokenizer.json
    - path: //models/llms/llama3.2-1b-instruct/tokenizer_config.json
    - path: //models/llms/llama3.2-1b-instruct/config.json

models:
  - from: file://absolute/path/to/my/model.ggml
    name: local_ggml_model
    files:
      - path: models/llms/ggml/tokenizer.json
      - path: models/llms/ggml/tokenizer_config.json
      - path: models/llms/ggml/config.json

models:
  - from: file://absolute/path/to/my/model.ggml
    name: local_ggml_model
    files:
      - path: models/llms/ggml/tokenizer.json
      - path: models/llms/ggml/tokenizer_config.json
      - path: models/llms/ggml/config.json

models:
  - name: safety
    from: file:models/llms/llama3.2-1b-instruct/model.safetensors
    files:
      - path: models/llms/llama3.2-1b-instruct/tokenizer.json
      - path: models/llms/llama3.2-1b-instruct/tokenizer_config.json
      - path: models/llms/llama3.2-1b-instruct/config.json

models:
  - name: safety
    from: file:models/llms/llama3.2-1b-instruct/model.safetensors
    files:
      - path: models/llms/llama3.2-1b-instruct/tokenizer.json
      - path: models/llms/llama3.2-1b-instruct/tokenizer_config.json
      - path: models/llms/llama3.2-1b-instruct/config.json

models:
  - name: llama3
    from: file:models/llms/llama3.2-1b-instruct/

models:
  - name: llama3
    from: file:models/llms/llama3.2-1b-instruct/

models:
  - from: file://absolute/path/to/my/model.onnx
    name: local_fs_model

models:
  - from: file://absolute/path/to/my/model.onnx
    name: local_fs_model

models:
  - from: file://absolute/path/to/my/model.gguf
    name: local_gguf_model

models:
  - from: file://absolute/path/to/my/model.gguf
    name: local_gguf_model

models:
  - name: local_model
    from: file:path/to/my/model.gguf
    params:
      chat_template: |
        {% set loop_messages = messages %}
        {% for message in loop_messages %}
          {% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}
          {{ content }}
        {% endfor %}
        {% if add_generation_prompt %}
          {{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
        {% endif %}

models:
  - name: local_model
    from: file:path/to/my/model.gguf
    params:
      chat_template: |
        {% set loop_messages = messages %}
        {% for message in loop_messages %}
          {% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}
          {{ content }}
        {% endfor %}
        {% if add_generation_prompt %}
          {{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
        {% endif %}