Cookbook recipes to get up and running with Spice.ai quickly 🚀
No Spice release includes the llama.cpp GGUF engine this recipe needs yet: v2.4.2 and earlier releases refuse this model's kimi-linear GGUF architecture. Until a release includes the engine, follow the build steps under Prerequisites (opens in a new tab).
Moonshot AI's Kimi models are open weights, but the flagship ones are trillion-parameter mixtures of experts: Kimi K2.6 (1T parameters) is 340 GB at 2-bit, and Kimi K3 (2.8T parameters, June 2026) is 466 GB even in Unsloth's 1-bit UD-Q1_0 build, so neither fits a MacBook Pro (a 512 GB Mac Studio could hold K2.6 at 2-bit, which this recipe does not test); they are served from multi-GPU servers, or tensor-parallel across several machines with Spice Enterprise's distributed_backend: ring. The Kimi model that runs on a laptop is Kimi-Linear-48B-A3B-Instruct, Moonshot's 48B mixture-of-experts model with 3B active parameters, built on Kimi Delta Attention, a hybrid linear-attention design that keeps the KV cache small at long context. This recipe serves it on an Apple silicon Mac with Spice's built-in llama.cpp engine and Metal. An abliterated variant (opens in a new tab) is in abliterated/ (opens in a new tab).
| Model | Parameters | Released | Fits a 36 GB MacBook Pro | GGUF repository |
|---|---|---|---|---|
| Kimi-Linear-48B-A3B-Instruct | 48B MoE, 3B active | October 2025 | Yes (this recipe) | bartowski/moonshotai_Kimi-Linear-48B-A3B-Instruct-GGUF (opens in a new tab) |
| Kimi K2.6 | 1T MoE, 32B active | April 2026 | No: 340 GB at 2-bit | unsloth/Kimi-K2.6-GGUF |
| Kimi K2.7-Code | 1T MoE | June 2026 | No | unsloth/Kimi-K2.7-Code-GGUF |
| Kimi K3 | 2.8T MoE, 896 experts | June 2026 | No: 466 GB at 1-bit, 861 GB at 2-bit | unsloth/Kimi-K3-GGUF (opens in a new tab) |
Moonshot publishes Kimi-Linear under the MIT license. The GGUF quantizations are by bartowski, a maintainer of community llama.cpp quantizations, since neither Moonshot nor Unsloth publishes one.
spice version must report a runtime with the +models.metal suffix and the llama.cpp GGUF engine. No release includes the engine yet, and v2.4.2 and earlier releases refuse this model. Run spice upgrade once a release that includes it ships; until then, use a trunk build (see Using a trunk build (opens in a new tab)).~/.cache/huggingface/hub.huggingface.co. No Hugging Face account is needed.Spice loads the whole GGUF into GPU memory. macOS gives the GPU most of unified memory but not all of it: Metal reports the budget as recommendedMaxWorkingSetSize (107 GiB of 128 GiB on an M3 Max). Leave room for the KV cache, which Kimi Delta Attention keeps small, and for macOS itself.
| Unified memory | File (files[0].path) | Size |
|---|---|---|
| 24 GB | moonshotai_Kimi-Linear-48B-A3B-Instruct-IQ2_M.gguf | 15.8 GB |
| 32 GB | moonshotai_Kimi-Linear-48B-A3B-Instruct-IQ3_XXS.gguf | 19.7 GB |
| 36 GB and up | moonshotai_Kimi-Linear-48B-A3B-Instruct-IQ3_M.gguf (the default in spicepod.yaml) | 22.7 GB |
| 48 GB and up | moonshotai_Kimi-Linear-48B-A3B-Instruct-Q4_K_M.gguf | 30.1 GB |
| 64 GB and up | moonshotai_Kimi-Linear-48B-A3B-Instruct-Q6_K.gguf | 40.5 GB |
| 96 GB and up | moonshotai_Kimi-Linear-48B-A3B-Instruct-Q8_0/moonshotai_Kimi-Linear-48B-A3B-Instruct-Q8_0-00001-of-00002.gguf and -00002-of-00002.gguf (near-lossless) | 52.2 GB |
| 128 GB | moonshotai_Kimi-Linear-48B-A3B-Instruct-bf16/moonshotai_Kimi-Linear-48B-A3B-Instruct-bf16-00001-of-00003.gguf and its two sibling shards (full precision) | 98.3 GB |
The IQ files are importance-matrix quantizations, which keep more of the model's quality at the same size than the plain Q files. List every shard of a split file in files. Kimi K2.6 stays out of reach: its 2-bit build is 340 GB.
spicepod.yaml in this directory defines one model:
from names the Hugging Face repository and files the GGUF in it. A repository revision can be pinned with from: huggingface:huggingface.co/<repo>:<commit>. Kimi-Linear's tool-call format is not one Spice's parser reads, so this recipe does not enable tools.
The first start downloads the model (22.7 GB); later starts load it from cache in a few seconds.
The elided lines report settings the runtime derives for this host; the log reports download progress every 30 seconds. Wait for the deployed, ready for inferencing line before chatting: /v1/models lists the model while it is still downloading.
In a second terminal, from the same directory:
Spice serves the model at http://localhost:8090/v1/chat/completions, so OpenAI SDKs and clients work unchanged with the model name kimi-linear.
Kimi-Linear is an instruct model without a thinking mode: it answered "Reply with only the integer 1729." in 2 tokens, and reasoning_effort changes nothing. With 3B active parameters it is fast: about 85 tokens per second on an M3 Max, the fastest of the models in these recipes. The engine uses an 8,192-token context; the model itself supports up to 1M tokens, and a conversation that outgrows the context is cut off with an error that says so.
abliterated/ (opens in a new tab) is the same recipe with huihui-ai/Huihui-Kimi-Linear-48B-A3B-Instruct-abliterated (opens in a new tab), a version of Kimi-Linear with its refusal behavior removed by abliteration, in the importance-matrix GGUF quantization by mradermacher. Read its README before using it: an abliterated model has none of the base model's safety training.
| Symptom | Cause and fix |
|---|---|
Architecture 'kimi-linear' in <file> is not supported by this Spice release | The runtime predates the llama.cpp GGUF engine. Upgrade Spice, or run a trunk build (see below). |
Unknown GGUF architecture 'kimi-linear' | A v2.4.2 or earlier runtime, which has no llama.cpp GGUF engine. The runtime then never reports ready (spiceai/spiceai#15744 (opens in a new tab)). Run a trunk build (see below). |
| The model never finishes loading, or macOS becomes unresponsive | The file does not fit the GPU's share of memory. Choose a smaller file from the Memory (opens in a new tab) table. |
the prompt is N tokens, and the model's context holds 8192 | The conversation outgrew the context. Start a new chat. |
Until a release includes the llama.cpp GGUF engine, download the latest trunk build of the runtime and point the CLI at it. The CLI's spice run, spice chat and spice sql work unchanged. The download needs the GitHub CLI (opens in a new tab) signed in to a GitHub account (brew install gh, then gh auth login), because Actions artifacts are only served to authenticated users.
Stop Spice with Ctrl+C. The model stays in the Hugging Face cache for the next start; remove it with: