Observability

Continuously identify and fix issues by tracking process actions with Spice monitoring and request logs.

The Observability tab provides real-time visibility into your project's usage, request performance, and API activity. Use it to track request volume, identify failures, and debug issues.

The tab groups the project's metrics, request logs, and traces into sections, listed in the left navigation:

Overview — aggregate metrics for the project.
API — API request metrics.
Cache — results cache metrics. Shown when SQL results caching is enabled.
Data — dataset and acceleration metrics.
Resources — CPU and memory usage for the project's running instances.
Request Logs — a record of individual API requests.
Traces — task traces. Shown for projects with a spicepod.

Accessing Observability

Navigate to your Spice project in the portal.
Click the Observability tab in the project navigation bar.

Overview

The Overview section shows aggregate metrics for:

SQL Queries — total count, success/failure rate, and average duration.
AI Completions — LLM inference request metrics.
Vector Searches — embedding-based search request metrics.
Embedding Calculations — embedding generation metrics.
Dataset Refreshes — accelerated dataset refresh success and timing.

Resources

The Resources section charts CPU and memory usage per running instance of the project.

Open the Observability tab for your project.
Select Resources in the left navigation.

CPU is reported as utilization against the project's configured CPU limit. Projects with no CPU limit set — including projects on dedicated clusters — report absolute core usage instead. Memory is reported in bytes.

Use the resource charts to size a project before raising its limits, and to correlate slow queries with memory or CPU pressure.

Usage Metrics Dashboard

Track request volume, data usage, and query time across configurable time ranges.

Open the Observability tab for your project.
Select a time range: 1 hour, 24 hours, 7 days, or 28 days.
Review the dashboard charts for request counts, data transferred, and query latency.

Use the metrics dashboard to identify usage trends, detect spikes in query duration, and plan capacity.

API Request Logs

The request logs provide a detailed record of individual API requests to your project's endpoints, including status codes, durations, and timestamps.

Open the Observability tab for your project.
Select Request Logs in the left navigation.
Select a time range: past hour, 8 hours, 24 hours, or up to 3 days.
Browse the log entries to inspect individual request details including endpoint, status code, and duration.

Use request logs to debug failing queries, identify slow requests, and audit API usage.

Project monitors

Project monitors send notifications when a project signal meets a condition. Charts show trends; request logs show individual API requests.

Open your project and select Monitoring.
Select Create monitor, then choose a monitor type shown for the project.
Set its condition in When. Available controls depend on the monitor type.
Choose one or more destinations in Then.
Name the monitor and select Create monitor.

Monitor types

The picker lists monitors supported by the project. Most let you set a comparison and threshold. Dataset status also lets you choose a dataset and status; instance health has a fixed condition. Template defaults set the window, sustain period, and severity. A window is a lookback period where used; sustain is how long the condition must hold.

Monitor	What it watches	Condition and configuration
Instance health	A project instance reporting a failed state, runtime error, or eviction.	Fixed condition; no threshold to configure. Evictions use a fixed 5-minute observation window; critical severity, with no additional sustain delay.
Instance memory	Memory use as a percentage of the instance's configured memory limit.	Set the comparison and percentage threshold. Default: above 80%, sustained for 5 minutes; warning.
Instance CPU	CPU use as a percentage of the instance's configured CPU limit.	Set the comparison and percentage threshold. Default: above 85% over a 5-minute window, sustained for 5 minutes; critical. Requires a CPU limit.
Dataset status	A selected dataset reporting a status such as Error.	Choose the dataset and a status: Initializing, Ready, Disabled, Error, Refreshing, or Shutting down. Default: Error sustained for 5 minutes; critical.
Acceleration refresh errors	Errors recorded during dataset acceleration refreshes.	For projects with acceleration. Set the comparison and error-count threshold. Default: more than 0 errors in a 5-minute window; critical, with no additional sustain delay.
HTTP 5xx responses	Estimated server-error responses from the project's HTTP API.	Set the comparison and response-count threshold. Default: more than 0 estimated responses in a 5-minute window; critical, with no additional sustain delay.
SQL p99 latency	The 99th-percentile duration of SQL queries.	For projects with SQL queries. Set the comparison and threshold in milliseconds. Default: above 1000 ms over a 5-minute window, sustained for 5 minutes; warning.
SQL query failures	Server-caused SQL query failures.	Requires SQL query telemetry. Set the comparison and failure-count threshold. Default: more than 0 estimated failures in a 5-minute window; critical, with no additional sustain delay.
Flight SQL DoGet failures	Failed Flight SQL requests.	Requires Flight SQL telemetry. Set the comparison and failure-count threshold. Default: more than 0 estimated failures in a 5-minute window; critical, with no additional sustain delay.
LLM failures	Server-caused model request failures.	Requires a configured model. Set the comparison and failure-count threshold. Default: more than 0 estimated failures in a 5-minute window; critical, with no additional sustain delay.

Saved rate-based failure conditions retain their configured units and values.

In Then, choose one or more destinations: Email, HTTP, or Slack. Enter email recipients, an HTTPS URL for HTTP, or select a Slack channel. HTTP also accepts an optional authorization token.

An organization administrator connects Slack through Settings → Integrations. Each monitor saves its own channel; the organization default only preselects a channel for new monitors. See Connect Slack.

After saving, select Send test notification to check delivery to the configured destinations. The test does not evaluate the monitor condition or add an event to its history.