title: 'HuggingFace' description: 'Instructions for using machine learning models hosted on HuggingFace with Spice.' sidebar_label: 'HuggingFace' sidebar_position: 4

To use a model hosted on HuggingFace, specify the huggingface.co path in the from field and, when needed, the files to include.

Configuration

from

The from key takes the form of huggingface:model_path. Below shows 2 common example of from key configuration.

• huggingface:username/modelname: Implies the latest version of modelname hosted by username.
• huggingface:huggingface.co/username/modelname:revision: Specifies a particular revision of modelname by username, including the optional domain.

The from key follows the following regex format.

\A(huggingface:)(huggingface\.co\/)?(?<org>[\w\-]+)\/(?<model>[\w\-]+)(:(?<revision>[\w\d\-\.]+))?\z

\A(huggingface:)(huggingface\.co\/)?(?<org>[\w\-]+)\/(?<model>[\w\-]+)(:(?<revision>[\w\d\-\.]+))?\z

The from key consists of five components:

1. Prefix: The value must start with huggingface:.
2. Domain (Optional): Optionally includes huggingface.co/ immediately after the prefix. Currently no other Huggingface compatible services are supported.
3. Organization/User: The HuggingFace organization (org).
4. Model Name: After a /, the model name (model).
5. Revision (Optional): A colon (:) followed by the git-like revision identifier (revision).

name

The model name. This will be used as the model ID within Spice and Spice's endpoints (i.e. http://localhost:8090/v1/models). This can be set to the same value as the model ID in the from field.

params

files

The specific file path for Huggingface model. For example, GGUF model formats require a specific file path, other varieties (e.g. .safetensors) are inferred.

Example

models:
  - from: huggingface:huggingface.co/lmstudio-community/Qwen2.5-Coder-3B-Instruct-GGUF
    name: sloth-gguf
    files:
      - path: Qwen2.5-Coder-3B-Instruct-Q3_K_L.gguf

models:
  - from: huggingface:huggingface.co/lmstudio-community/Qwen2.5-Coder-3B-Instruct-GGUF
    name: sloth-gguf
    files:
      - path: Qwen2.5-Coder-3B-Instruct-Q3_K_L.gguf

Access Tokens

:::warning[huggingface_token on v2.2.0 and v2.2.1]

On this release line the parameter is spelled huggingface_token. hf_token is rejected: the runtime drops it with

WARN runtime_parameters_typed: Ignoring parameter `hf_token`: not supported for model huggingface.

WARN runtime_parameters_typed: Ignoring parameter `hf_token`: not supported for model huggingface.

and loads the model anonymously, so a gated or private repository fails with an HTTP 401 that names no cause. A bare token is rejected too, with a warning that it must be prefixed with huggingface_. This affects from: huggingface: models only — the hf_token documented for embeddings (opens in a new tab) and rerankers is read correctly on this release.

Spelling reverted in v2.3.0: hf_token is the parameter again, and huggingface_token is kept as an alias so a Spicepod written against v2.2.x keeps loading. See spiceai/spiceai#13932 (opens in a new tab).

:::

Access tokens can be provided for Huggingface models in two ways:

1. In the Huggingface token cache (i.e. ~/.cache/huggingface/token). Default.
2. Via model params (opens in a new tab).

models:
  - name: llama_3.2_1B
    from: huggingface:huggingface.co/meta-llama/Llama-3.2-1B
    params:
      huggingface_token: ${ secrets:HF_TOKEN }

models:
  - name: llama_3.2_1B
    from: huggingface:huggingface.co/meta-llama/Llama-3.2-1B
    params:
      huggingface_token: ${ secrets:HF_TOKEN }

Examples

Load a LLM model to generate text

models:
  - from: huggingface:huggingface.co/microsoft/Phi-3.5-mini-instruct
    name: phi

models:
  - from: huggingface:huggingface.co/microsoft/Phi-3.5-mini-instruct
    name: phi

Load a private model

models:
  - name: llama_3.2_1B
    from: huggingface:huggingface.co/meta-llama/Llama-3.2-1B
    params:
      huggingface_token: ${ secrets:HF_TOKEN }

models:
  - name: llama_3.2_1B
    from: huggingface:huggingface.co/meta-llama/Llama-3.2-1B
    params:
      huggingface_token: ${ secrets:HF_TOKEN }

For more details on authentication, see access tokens (opens in a new tab).

:::warning[Limitations]

• The throughput, concurrency & latency of a locally hosted model will vary based on the underlying hardware and model size. Spice supports Apple Metal and CUDA for accelerated inference. See CONTRIBUTING.md (opens in a new tab) for build instructions.

:::

Cookbook

• Use the Llama family of models locally from HuggingFace using Spice. Running Llama3 Locally (opens in a new tab)