Model Providers

Filesystem Hosted Models

Instructions for using models hosted on a filesystem with Spice.

To use a model hosted on a filesystem, specify the path to the model file or folder in the from field:

models:
  - from: file://models/llms/llama3.2-1b-instruct/
    name: llama3

Supported formats include GGUF, GGML, and SafeTensor for large language models (LLMs).

Configuration

from

An absolute or relative path to the model file or folder:

from: file://absolute/path/models/llms/llama3.2-1b-instruct/
from: file:models/llms/llama3.2-1b-instruct/

params (optional)

ParamDescription
toolsWhich tools should be made available to the model. Set to auto to use all available tools.
system_promptAn additional system prompt used for all chat completions to this model.
chat_templateCustomizes the transformation of OpenAI chat messages into a character stream for the model. See Overriding the Chat Template.
trust_pickleAllow loading pickle-based weight files (.bin, .pt, .pth, .ckpt). These formats execute arbitrary code when loaded and are rejected by default. Accepts true or false; defaults to false. Set to true only when the weights come from a fully trusted source.

See Large Language Models for additional configuration options.

files (optional)

The files field specifies additional files required by the model, such as tokenizer, configuration, and other files.

- name: local-model
  from: file://models/llms/llama3.2-1b-instruct/model.safetensors
  files:
    - path: //models/llms/llama3.2-1b-instruct/tokenizer.json
    - path: //models/llms/llama3.2-1b-instruct/tokenizer_config.json
    - path: //models/llms/llama3.2-1b-instruct/config.json

Examples

Loading a GGML Model

models:
  - from: file://absolute/path/to/my/model.ggml
    name: local_ggml_model
    files:
      - path: models/llms/ggml/tokenizer.json
      - path: models/llms/ggml/tokenizer_config.json
      - path: models/llms/ggml/config.json

Example: Loading a SafeTensor Model

models:
  - name: safety
    from: file:models/llms/llama3.2-1b-instruct/model.safetensors
    files:
      - path: models/llms/llama3.2-1b-instruct/tokenizer.json
      - path: models/llms/llama3.2-1b-instruct/tokenizer_config.json
      - path: models/llms/llama3.2-1b-instruct/config.json

Loading LLM from a directory

models:
  - name: llama3
    from: file:models/llms/llama3.2-1b-instruct/

Note: The folder provided should contain all the expected files (see examples above).

Loading a GGUF Model

models:
  - from: file://absolute/path/to/my/model.gguf
    name: local_gguf_model

Overriding the Chat Template

Chat templates convert the OpenAI compatible chat messages (see format) and other components of a request into a stream of characters for the language model. It follows Jinja3 templating syntax.

Further details on chat templates can be found here.

models:
  - name: local_model
    from: file:path/to/my/model.gguf
    params:
      chat_template: |
        {% set loop_messages = messages %}
        {% for message in loop_messages %}
          {% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}
          {{ content }}
        {% endfor %}
        {% if add_generation_prompt %}
          {{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
        {% endif %}

Templating Variables

  • messages: List of chat messages, in the OpenAI format.
  • add_generation_prompt: Boolean flag whether to add a generation prompt.
  • tools: List of callable tools, in the OpenAI format.

Cookbook