title: Troubleshooting Spice sidebar_label: Troubleshooting sidebar_position: 99 description: 'Review and debug runtime tasks, logs, and diagnostic steps in Spice.' pagination_prev: null pagination_next: null tags:
Spice provides a number of methods to support debugging Runtime operations, including capturing verbose logs and reviewing task history in SQL queries or AI completions.
For hands-on examples, see the Spice.ai Cookbook (opens in a new tab) for recipes demonstrating common troubleshooting scenarios.
spice command not found after installationThe Spice binary directory is not in your PATH. Add it:
Verify with spice version.
Check the runtime output for error messages. Common causes:
pg_host, pg_user, pg_pass, etc.) and that secrets are configured correctly. Use --verbose to see secret resolution logs.curl or psql from the same host.Review the task_history table for detailed error messages:
acceleration: enabled: true to the dataset configuration.EXPLAIN before the query to verify it executes against the local accelerator and not the remote source.Results-Cache-Status HTTP header. A MISS on repeated identical queries may indicate a low item_ttl.Model [name] deployed, ready for inferencing. If the model failed to load, review error messages in the logs.tools: auto is set in the model params so the model can discover and query datasets.spice trace ai_chat: Inspect the trace output to see which tools the model called and whether SQL queries succeeded or failed.runtime.query.memory_limit bounds the query execution pool, not the process. Accelerator caches, serialization buffers, embedded engine pools, and allocator retention sit outside it, so a process can be killed while the query pool still reports unused capacity.
process_resident_memory_bytes is far above query_memory_pool_used_bytes, the memory is off-pool and lowering the query limit will not recover it.runtime.query.max_concurrent_queries reduces the peak directly, whereas lowering runtime.query.memory_limit shrinks each query's budget without reducing how many run at once.See Managing Memory Usage (opens in a new tab) for the sizing model and validation guidance.
If Spice fails to bind to its default ports (8090 for HTTP, 50051 for Flight, 9090 for metrics), another process is already using that port. Override the ports:
Running spiced with --verbose produces immediate debug logs for diagnosing issues in real time. Running spiced with --very-verbose captures trace logs, which are useful for examining function outputs and other low-level details. The verbosity flags are also available for spice run, providing consistent behavior during local testing or production operation.
Example --verbose output:
Example --very-verbose output:
For more information, view the tracing documentation (opens in a new tab)
spice trace for task tracingUse the spice trace ai_chat command to inspect processes involved in generating AI chat responses. The spice trace command supports tracing any task type, like spice trace sql_query:
This step is helpful for reviewing any tool usage or tasks invoked during AI completions, providing more information on the steps an AI took to arrive at a result.
For example, running a chat using the taxi_trips dataset:
To inspect the last chat, run spice trace ai_chat, producing an AI chat trace output:
In this example, the trace logs show that the model attempted an SQL query twice - getting the first query incorrect, due to a syntax issue in the SQL the model generated. Before running the second query, it retrieved the table schema - which it used for the second query to successfully retrieve the last taxi trip time.
For more information, view the spice trace documentation (opens in a new tab).
Query the task_history table to review completed tasks handled by the Runtime. Results in this table can include tasks for accelerator refresh, SQL queries, text embedding, AI calls, and more. Reviewing this table can provide information on SQL query issues or other processes that may produce errors.
Example task_history query:
The task_history table also includes start and end times, including execution duration and any error messages during the operation. An example task_history output with a failed SQL query:
For more information, view the task history documentation (opens in a new tab)
Set runtime.task_history.captured_output to truncated to store summarized SQL query information in the captured_output task_history column. Enable captured output by running spiced with --set-runtime task_history.captured_output=truncated or adjusting the Spicepod parameter runtime.task_history.captured_output.
Example Spicepod excerpt:
Example captured output:
Configure Spice to automatically capture SQL query plans (EXPLAIN or EXPLAIN ANALYZE) in the task history. This feature captures query execution information asynchronously, storing results in the captured_output column for later analysis without impacting query performance.
Configure plan capture using the runtime.task_history settings:
captured_plan: Determines the type of plan captured:
none (default): No plans are capturedexplain: Captures logical and physical query plansexplain analyze: Captures plans with actual execution metricsmin_sql_duration: Minimum query duration before plan capture. Only queries exceeding this threshold generate a plan.min_plan_duration: Minimum plan execution duration before storage. Plans that execute faster than this threshold are discarded.Review captured plans for slow queries:
Plans are captured asynchronously after query completion, ensuring query execution proceeds without blocking. For more details, view the task history documentation (opens in a new tab).
Spice supports generating EXPLAIN plans, which can be used to debug SQL that may not be producing the correct result. An explain plan in Spice can provide information about the data sources the SQL will be executed against, including the re-written SQL that will be executed against that data source (if applicable). For example, an explain plan could be used to debug the SQL that is generated and executed against separate federated sources during an SQL query which joins their results (like a PostgreSQL database and a MySQL database).
Generate an explain plan by adding the EXPLAIN keyword before the query:
If you try to open a shell in a Spice container the way you would with most images, it fails:
On your machine:
The error comes from inside the container: kubectl reached the pod successfully, then found no bash there to execute.
This is expected, not a broken container. The sections below explain why, and give three ways to debug depending on what you need to look at.
The published Spice image is built FROM scratch — an empty base image with no Linux userspace at all. It contains only:
spiced binaryspiced links againstThere is no bash, no sh, not even ls, and no package manager — so nothing can be installed into a running container either. spiced also runs as the unprivileged user 65534 (nobody), whose login shell is set to /usr/sbin/nologin.
This is deliberate: an image with no shell and no packages has a far smaller attack surface and far fewer CVEs to patch. The trade-off is that debugging needs the techniques below instead of kubectl exec ... -- bash. For background on how this image is built, see the Docker Sandbox Guide (opens in a new tab).
| What you need to do | Use | Requirements |
|---|---|---|
| Run SQL queries against the running runtime | SQL REPL (opens in a new tab) | None — works out of the box |
| Inspect files, processes, or the network of a Kubernetes pod | Ephemeral container (opens in a new tab) | Kubernetes v1.25+, and permission to create ephemeral containers |
| Get an interactive shell against a local Docker container | busybox shell (opens in a new tab) | Docker, and the container started with an extra volume |
Start with the SQL REPL if the question is about data or queries. Reach for an ephemeral container when the question is about the environment — config files, mounted volumes, DNS, or connectivity.
Debugging a container means moving between machines, and the same command can behave differently — or fail — depending on where it is typed. Three contexts appear below, and every command block states which one it belongs to:
| Context | Where it actually executes | How you get there |
|---|---|---|
| Your machine | Your own workstation or CI runner — wherever kubectl and docker are installed. These commands talk to the cluster or Docker daemon, not inside it. | The default; no extra step |
| Inside the Spice container | The sandbox container running spiced. No shell here, so only spiced itself, or a mounted binary such as busybox, can be run. | kubectl exec / docker exec |
| Inside the debug container | A separate temporary container in the same pod, with its own filesystem and its own tools. Spice's files are not at their usual paths here. | kubectl debug drops you straight in |
Two consequences that catch people out:
kubectl and docker commands are typed on your machine, but the part after -- (or after the container name) executes inside the container. In kubectl exec -it my-pod -- spiced --repl, kubectl runs locally while spiced --repl runs in the container — which is why localhost in that command means the container's own localhost, not your machine's.kubectl debug does not put you inside the Spice container. It puts you in a new debug container beside it. That is why inspecting Spice's files from there needs the /proc/1/root prefix described below.The REPL needs no shell, because spiced is itself the binary being executed.
On your machine — the spiced --repl part after the container name executes inside the Spice container:
Because spiced --repl runs inside the container, it connects to that container's own http://localhost:50051 Flight endpoint — attaching to the runtime already serving there. The interactive SQL prompt that follows is therefore executing queries inside the deployment, not locally.
This is the quickest way to check whether a dataset loaded, inspect a schema, or reproduce a slow query from inside the deployment.
:::note
Running spiced --repl on your machine instead is a different thing entirely: it would try to reach a runtime on your own localhost:50051. To query a remote runtime from your workstation without kubectl exec, use spice sql --endpoint <url> (it accepts http://, https://, grpc://, and grpc+tls://), pointing at an address the runtime is reachable on — for example one published by kubectl port-forward.
:::
An ephemeral container (opens in a new tab) is a temporary extra container that Kubernetes adds to a pod that is already running. Because it brings its own image, it can supply all the tools the Spice image lacks — while the Spice process keeps running untouched. The pod is not restarted and no configuration is changed.
This is the recommended way to debug Spice on Kubernetes.
On your machine:
The Helm chart names the Spice container spiceai. Run the second command rather than assuming, since a custom manifest may name it something else.
On your machine:
This command returns with an interactive prompt, and from here on you are inside the debug container — a different filesystem from both your machine and the Spice container.
What each flag does:
| Flag | Meaning |
|---|---|
-it | Attach an interactive terminal, as with kubectl exec -it. |
--image | The image providing the debugging tools, and therefore the only source of tools available once inside. ubuntu:24.04 gives a familiar shell and apt to install more; any image works. |
--target | The container in the pod to attach to, from step 1. This is what makes the Spice process and its filesystem visible — omit it and you get an isolated container that sees nothing of the runtime. |
--profile=sysadmin | Applies the sysadmin static debugging profile (opens in a new tab), which grants the elevated capabilities needed to inspect another container's files and processes. |
The debug container has its own filesystem — from ubuntu:24.04, not from Spice — so /app here is empty, and Spice's files are not where you might expect. Because --target shares the process namespace, spiced is visible as PID 1, and Linux exposes any process's root filesystem at /proc/<pid>/root. So everything belonging to Spice is reachable under /proc/1/root:
Inside the debug container (reading the Spice container's files):
The /proc/1/root prefix is what makes the difference: ls /app reads the debug container's empty directory, while ls /proc/1/root/app reads the Spice container's real working directory. Dropping the prefix is the most common source of confusion — it does not error, it just shows you the wrong container's filesystem.
Networking is the exception to that rule. All containers in a pod share a single network namespace, so network tools run inside the debug container already see exactly what the runtime sees — no /proc/1/root prefix applies, and none is needed.
Note that the tools available are whatever the --image you chose provides, not what Spice provides. ubuntu:24.04 is a minimal image, so install what you need first — this is safe, because it modifies only the throwaway debug container:
Inside the debug container:
Images purpose-built for network debugging, such as nicolaka/netshoot, bundle these tools and skip the install step.
When finished, type exit to leave the debug container and return to your machine.
:::note
Ephemeral containers require Kubernetes v1.25 or later, and creating one requires permission on the pods/ephemeralcontainers subresource — a Forbidden error means the account lacks it, not that the command is wrong. An ephemeral container cannot be removed from a pod once added; it remains in the pod spec until the pod is replaced. Some hardened clusters also reject --profile=sysadmin; if so, try --profile=general (fewer capabilities, so cross-container file inspection may not work).
:::
For local Docker debugging, a shell can be added from outside the image. busybox (opens in a new tab) ships a single statically compiled binary containing dozens of standard tools — because it depends on no system libraries, it runs inside the sandbox image even though that image has no userspace of its own. Mounting it as a volume supplies a shell without rebuilding the image.
This technique is for Docker. On Kubernetes, use an ephemeral container (opens in a new tab) instead — it needs no volume and no restart.
Note that the volume must be mounted when the container is created, so this requires starting a new container rather than attaching to a running one.
On your machine (the Docker host) — all four commands:
That last command hands you a shell inside the Spice container itself — unlike kubectl debug, there is no separate debug container here, so paths such as /app are already Spice's own and need no /proc/1/root prefix.
Because the busybox tools are not on PATH, every command in that shell must be prefixed with /busy/busybox:
Inside the Spice container: