title: Troubleshooting Spice sidebar_label: Troubleshooting sidebar_position: 99 description: 'Review and debug runtime tasks, logs, and diagnostic steps in Spice.' pagination_prev: null pagination_next: null tags:
Spice provides a number of methods to support debugging Runtime operations, including capturing verbose logs and reviewing task history in SQL queries or AI completions.
For hands-on examples, see the Spice.ai Cookbook for recipes demonstrating common troubleshooting scenarios.
spice command not found after installationThe Spice binary directory is not in your PATH. Add it:
Verify with spice version.
Check the runtime output for error messages. Common causes:
pg_host, pg_user, pg_pass, etc.) and that secrets are configured correctly. Use --verbose to see secret resolution logs.curl or psql from the same host.Review the task_history table for detailed error messages:
acceleration: enabled: true to the dataset configuration.EXPLAIN before the query to verify it executes against the local accelerator and not the remote source.Results-Cache-Status HTTP header. A MISS on repeated identical queries may indicate a low item_ttl.Model [name] deployed, ready for inferencing. If the model failed to load, review error messages in the logs.tools: auto is set in the model params so the model can discover and query datasets.spice trace ai_chat: Inspect the trace output to see which tools the model called and whether SQL queries succeeded or failed.If Spice fails to bind to its default ports (8090 for HTTP, 50051 for Flight, 9090 for metrics), another process is already using that port. Override the ports:
Running spiced with --verbose produces immediate debug logs for diagnosing issues in real time. Running spiced with --very-verbose captures trace logs, which are useful for examining function outputs and other low-level details. The verbosity flags are also available for spice run, providing consistent behavior during local testing or production operation.
Example --verbose output:
Example --very-verbose output:
For more information, view the tracing documentation
spice trace for task tracingUse the spice trace ai_chat command to inspect processes involved in generating AI chat responses. The spice trace command supports tracing any task type, like spice trace sql_query:
This step is helpful for reviewing any tool usage or tasks invoked during AI completions, providing more information on the steps an AI took to arrive at a result.
For example, running a chat using the taxi_trips dataset:
To inspect the last chat, run spice trace ai_chat, producing an AI chat trace output:
In this example, the trace logs show that the model attempted an SQL query twice - getting the first query incorrect, due to a syntax issue in the SQL the model generated. Before running the second query, it retrieved the table schema - which it used for the second query to successfully retrieve the last taxi trip time.
For more information, view the spice trace documentation.
Query the task_history table to review completed tasks handled by the Runtime. Results in this table can include tasks for accelerator refresh, SQL queries, text embedding, AI calls, and more. Reviewing this table can provide information on SQL query issues or other processes that may produce errors.
Example task_history query:
The task_history table also includes start and end times, including execution duration and any error messages during the operation. An example task_history output with a failed SQL query:
For more information, view the task history documentation
Set runtime.task_history.captured_output to truncated to store summarized SQL query information in the captured_output task_history column. Enable captured output by running spiced with --set-runtime task_history.captured_output=truncated or adjusting the Spicepod parameter runtime.task_history.captured_output.
Example Spicepod excerpt:
Example captured output:
Configure Spice to automatically capture SQL query plans (EXPLAIN or EXPLAIN ANALYZE) in the task history. This feature captures query execution information asynchronously, storing results in the captured_output column for later analysis without impacting query performance.
Configure plan capture using the runtime.task_history settings:
captured_plan: Determines the type of plan captured:
none (default): No plans are capturedexplain: Captures logical and physical query plansexplain analyze: Captures plans with actual execution metricsmin_sql_duration: Minimum query duration before plan capture. Only queries exceeding this threshold generate a plan.min_plan_duration: Minimum plan execution duration before storage. Plans that execute faster than this threshold are discarded.Review captured plans for slow queries:
Plans are captured asynchronously after query completion, ensuring query execution proceeds without blocking. For more details, view the task history documentation.
Spice supports generating EXPLAIN plans, which can be used to debug SQL that may not be producing the correct result. An explain plan in Spice can provide information about the data sources the SQL will be executed against, including the re-written SQL that will be executed against that data source (if applicable). For example, an explain plan could be used to debug the SQL that is generated and executed against separate federated sources during an SQL query which joins their results (like a PostgreSQL database and a MySQL database).
Generate an explain plan by adding the EXPLAIN keyword before the query:
The Spice sandbox container is a minimal container that doesn't include standard Linux tools like bash. This can make it difficult to debug issues in the container itself.
It's possible to run the SQL REPL from the container to debug SQL queries:
Or from kubectl:
Attach an ephemeral container to inspect a running Spice pod without restarting it. The helper container shares the runtime container's namespaces, so diagnostics reflect the live workload.
List pods in the relevant namespace: kubectl get pods -n <namespace>
Start the debugger:
--target names the container inside the pod; Helm installs the Spice container as spiceai by default. --profile applies a static debugging profile; the sysadmin preset adds root-level capabilities. With --profile=sysadmin and --target=spiceai, the Spice filesystem mounts at /proc/1/root inside the debugger for direct inspection.
Run the required commands and exit. For example, review the deployed Spicepod definition:
Ephemeral containers require Kubernetes v1.25+.
To debug issues using a shell, mount a volume that has a statically compiled busybox binary, and exec into the container using the busybox sh command. Here is an example in Docker: