Cookbook recipes to get up and running with Spice.ai quickly 🚀
No Spice release includes the llama.cpp GGUF engine this recipe needs yet: v2.4.2 and earlier releases refuse this model's qwen35 GGUF architecture. Until a release includes the engine, follow the build steps under Prerequisites (opens in a new tab).
Qwen3.5 (opens in a new tab) is Alibaba's February 2026 family of hybrid-thinking, multimodal open-weight models under the Apache 2.0 license. This recipe serves Qwen3.5-9B on an Apple silicon Mac with Spice's built-in llama.cpp engine and Metal, grounds it in a public dataset through Spice's SQL tools, and shows how to turn its reasoning on and off per request. An abliterated variant (opens in a new tab) of the same model is in abliterated/ (opens in a new tab).
The 9B model is the member of the family that runs on every MacBook Pro, including 16 GB configurations. The table in Memory (opens in a new tab) names the files for larger Macs, up to Qwen3.5-35B-A3B on a 36 GB MacBook Pro.
| Model | Parameters | Fits a 36 GB MacBook Pro | GGUF repository |
|---|---|---|---|
| Qwen3.5-9B | 9B dense | Yes, from 16 GB | unsloth/Qwen3.5-9B-GGUF (opens in a new tab) |
| Qwen3.5-27B | 27B dense | Yes, from 32 GB | unsloth/Qwen3.5-27B-GGUF (opens in a new tab) |
| Qwen3.5-35B-A3B | 36B MoE, 3B active | Yes, from 36 GB | unsloth/Qwen3.5-35B-A3B-GGUF (opens in a new tab) |
| Qwen3.5-122B-A10B | 122B MoE, 10B active | No: 77 GB at 4-bit; a 128 GB Mac | unsloth/Qwen3.5-122B-A10B-GGUF (opens in a new tab) |
| Qwen3.5-397B-A17B | 397B MoE, 17B active | No: a 256 GB Mac Studio and up | unsloth/Qwen3.5-397B-A17B-GGUF |
The 0.8B, 2B and 4B models also fit, with the same spicepod shape and Unsloth repositories.
spice version must report a runtime with the +models.metal suffix and the llama.cpp GGUF engine. Spice installs the Metal build on Apple silicon by default. No release includes the engine yet, and v2.4.2 and earlier releases refuse this model. Run spice upgrade once a release that includes it ships; until then, use a trunk build (see Using a trunk build (opens in a new tab)).~/.cache/huggingface/hub.huggingface.co and to the public S3 bucket spiceai-demo-datasets. No Hugging Face account is needed.Spice loads the whole GGUF into GPU memory. macOS gives the GPU most of unified memory but not all of it: Metal reports the budget as recommendedMaxWorkingSetSize (107 GiB of 128 GiB on an M3 Max). Leave room for the KV cache (a few hundred MB at the engine's default 8,192-token context) and for macOS itself. Pick the file for the Mac:
| Unified memory | File (files[0].path) | Size | Repository |
|---|---|---|---|
| 16 GB and up | Qwen3.5-9B-UD-Q4_K_XL.gguf (the default in spicepod.yaml) | 6.0 GB | unsloth/Qwen3.5-9B-GGUF |
| 16 GB and up | Qwen3.5-9B-UD-Q6_K_XL.gguf | 8.8 GB | unsloth/Qwen3.5-9B-GGUF |
| 24 GB and up | Qwen3.5-9B-Q8_0.gguf (near-lossless) | 9.5 GB | unsloth/Qwen3.5-9B-GGUF |
| 32 GB and up | Qwen3.5-27B-UD-Q4_K_XL.gguf | 17.6 GB | unsloth/Qwen3.5-27B-GGUF |
| 36 GB and up | Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf (3B active: several times faster) | 22.2 GB | unsloth/Qwen3.5-35B-A3B-GGUF |
| 48 GB and up | Qwen3.5-35B-A3B-UD-Q6_K_XL.gguf | 32.1 GB | unsloth/Qwen3.5-35B-A3B-GGUF |
| 64 GB and up | Qwen3.5-122B-A10B-UD-IQ3_XXS.gguf (122B MoE, 10B active) | 44.7 GB | unsloth/Qwen3.5-122B-A10B-GGUF |
| 96 GB and up | UD-Q3_K_XL/Qwen3.5-122B-A10B-UD-Q3_K_XL-00001-of-00003.gguf and its two sibling shards | 57.0 GB | unsloth/Qwen3.5-122B-A10B-GGUF |
| 128 GB | UD-Q4_K_XL/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf and its two sibling shards | 77.0 GB | unsloth/Qwen3.5-122B-A10B-GGUF |
To switch, change from to the repository and files to the file, listing every shard of a split file. The UD- files are Unsloth's dynamic quantizations, which keep the layers that matter most at higher precision.
spicepod.yaml in this directory defines one dataset and one model:
from names the Hugging Face repository and files the GGUF in it. Spice downloads every .gguf listed, so a model split into shards lists each shard. A repository revision can be pinned with from: huggingface:huggingface.co/unsloth/Qwen3.5-9B-GGUF:<commit>.tools: auto gives the model Spice's built-in tools, including list_datasets, table_schema and sql, so it can answer questions about taxi_trips by writing and running SQL.The first start downloads the model (6.0 GB) into the Hugging Face cache and reads the dataset (399 MB) from S3. Later starts load the model from the cache; the dataset is read from S3 again on every start, since the acceleration is in memory (about a second on a fast connection).
The elided lines report settings the runtime derives for this host, and their values differ from machine to machine. The download rate is the network's: this run was on a slow link.
In a second terminal, from the same directory:
Behind each answer the model called list_datasets, then table_schema, then sql. The runtime records every call in runtime.task_history; spice sql shows them:
A model this size sometimes writes a SQL function Spice's engine does not have, such as DATE(). The tool returns the planner's error, and the model rewrites the query; the answer takes a few seconds longer.
Spice serves the model at http://localhost:8090/v1/chat/completions, so OpenAI SDKs and clients work unchanged with the model name qwen3.5.
Qwen3.5's small models (0.8B to 9B) answer directly unless asked to think; the 27B and 35B-A3B models think by default. The reasoning_effort field controls it per request: none turns thinking off, and low, medium or high turn it on. The thinking trace comes back in reasoning_content, separate from content.
To set a default for every request, add hf_reasoning_effort: <level> to the model's params. Qwen3.5's chat template has one thinking switch, so low, medium and high all mean "think" for this model.
On an M3 Max, Qwen3.5-9B at 4-bit decodes about 43 tokens per second with a 1,600-token prompt. The engine uses an 8,192-token context; the dataset tools fit comfortably, and a longer conversation is cut off with an error that says so.
abliterated/ (opens in a new tab) is the same recipe with huihui-ai/Huihui-Qwen3.5-9B-abliterated (opens in a new tab), a version of Qwen3.5-9B with its refusal behavior removed by abliteration, in the GGUF quantization by mradermacher (opens in a new tab). It serves the model without the dataset tools, because its stock Qwen chat template trips a runtime bug when tools: auto is set (spiceai/spiceai#15578 (opens in a new tab)). Read its README before using it: an abliterated model has none of the base model's safety training.
| Symptom | Cause and fix |
|---|---|
Architecture 'qwen35' in <file> is not supported by this Spice release | The runtime predates the llama.cpp GGUF engine. Upgrade Spice, or run a trunk build (see below). |
Unknown GGUF architecture 'qwen35' | A v2.4.2 or earlier runtime, which serves GGUFs with mistral.rs. The runtime then never reports ready (spiceai/spiceai#15744 (opens in a new tab)). Same fix. |
| The model never finishes loading, or macOS becomes unresponsive | The file does not fit the GPU's share of memory. Choose a smaller file from the Memory (opens in a new tab) table. |
the prompt is N tokens, and the model's context holds 8192 | The conversation outgrew the context. Start a new chat. |
spice chat "<question>" prints nothing for a question that needs the dataset tools | spiceai/spiceai#15579 (opens in a new tab): the one-shot form stalls on tool calls. Use the interactive REPL (spice chat, then type the question), pipe the question on standard input (echo "<question>" | spice chat), or call the API. |
| Slow first start | The 6.0 GB download runs at the network's speed; the log reports progress every 30 seconds. Later starts read the cache. |
Until a release includes the llama.cpp GGUF engine, download the latest trunk build of the runtime and point the CLI at it. The CLI's spice run, spice chat and spice sql work unchanged. The download needs the GitHub CLI (opens in a new tab) signed in to a GitHub account (brew install gh, then gh auth login), because Actions artifacts are only served to authenticated users.
Stop Spice with Ctrl+C. The model stays in the Hugging Face cache for the next start; remove it with:
tools: auto.