Cookbook recipes to get up and running with Spice.ai quickly 🚀
No Spice release includes the llama.cpp GGUF engine this recipe needs yet; until one does, follow the build steps in the base recipe (opens in a new tab).
This directory is the Kimi recipe (opens in a new tab) with the model swapped for huihui-ai/Huihui-Kimi-Linear-48B-A3B-Instruct-abliterated (opens in a new tab), a version of Kimi-Linear-48B-A3B-Instruct with its refusal behavior removed by abliteration, in the importance-matrix i1-IQ3_M GGUF (21.5 GB) published by mradermacher (opens in a new tab). Everything else is unchanged.
Abliteration removes the "refusal direction" from a model's weights, so the model answers requests the base model would decline. huihui-ai describes its method as a proof-of-concept implementation of refusal removal (opens in a new tab). The result has none of the base model's safety training. Use it for research, red-teaming and controlled experiments, add your own moderation before exposing it to users, and review its outputs. huihui-ai's model card carries its own usage warnings and disclaims responsibility for use.
Wait for Model [kimi-linear-abliterated] deployed, ready for inferencing, then in a second terminal:
The API serves the model as kimi-linear-abliterated at http://localhost:8090/v1/chat/completions. On an M3 Max with the trunk Metal build it loaded after a 7-minute download (21.5 GB), answered "Reply with only the integer 1729." in 2 tokens with and without reasoning_effort (no thinking mode, like the base model), answered the three-sentence mixture-of-experts question from the base recipe, and decoded at 87 tokens per second.
The mradermacher repository holds i1-IQ2_M (16.1 GB, 24 GB Macs), i1-IQ3_XXS (19.0 GB, 32 GB), i1-IQ3_M (21.5 GB, the default, 36 GB and up), i1-IQ4_XS (26.3 GB) and i1-Q4_K_M (29.7 GB, 48 GB and up), and i1-Q6_K (40.4 GB, 64 GB and up).