Observability

Continuously identify and fix issues by tracking process actions with Spice monitoring and request logs.

The Observability tab provides real-time visibility into your project's usage, request performance, and API activity. Use it to track request volume, identify failures, and debug issues.

The tab groups the project's metrics, request logs, and traces into sections, listed in the left navigation:

Overview: aggregate metrics for the project.
API: API request metrics.
Cache: results cache metrics. Shown when SQL results caching is enabled.
Data: dataset and acceleration metrics.
Resources: CPU and memory usage for the project's running instances.
Request Logs: a record of individual API requests.
Monitors: project monitors that send notifications. Shown for projects on Spice-managed compute.
Traces: task traces. Shown for projects with a spicepod.

Accessing Observability

Navigate to your Spice project in the portal.
Click the Observability tab in the project navigation bar.

Overview

The Overview section shows aggregate metrics for:

SQL Queries: succeeded and failed query counts, and p95 and p99 query latency.
AI Completions: succeeded and failed completion counts, and p95 and p99 latency. Shown when the project has a model.
Embeddings: succeeded and failed embedding search counts, and p95 and p99 search latency. Shown when the project has embeddings.
Acceleration: the maximum dataset acceleration refresh duration. Shown when the project accelerates a dataset.

Cache

The Cache section charts the SQL results cache:

Results Cache Hits vs Misses: the share of cache lookups that were hits and misses at each step, from 0% to 100%. Hover a step to see each share and its lookup count. A step with no lookups stays empty.
Results Cache Usage: the cache size in bytes, with a line at the configured maximum (runtime.caching.sql_results.max_size, default 128MiB).

Resources

The Resources section charts CPU and memory usage per running instance of the project.

Open the Observability tab for your project.
Select Resources in the left navigation.

CPU is reported as utilization against the project's configured CPU limit. Projects with no CPU limit set (including projects on dedicated clusters) report absolute core usage instead. Memory is reported in bytes.

Use the resource charts to size a project before raising its limits, and to correlate slow queries with memory or CPU pressure.

Time ranges

Each chart section has a time range menu.

Overview and API: Last hour, Last 24 hours, Last 2 days, Last 7 days, or Last 28 days.
Cache, Data, and Resources: Last 1 hour, Last 24 hours, Last 7 days, or Last 28 days.

Use the longer ranges to identify usage trends, detect spikes in query duration, and plan capacity.

API Request Logs

The request logs provide a detailed record of individual API requests to your project's endpoints, including status codes, durations, and timestamps.

Open the Observability tab for your project.
Select Request Logs in the left navigation.
Select a time range.
Browse the log entries to inspect individual request details including endpoint, status code, and duration.

The Request Logs time ranges are 1 Hour, 8 Hours, 24 Hours, 3 Days, and 7 Days. The ranges available depend on the org's plan. Every plan has 1 Hour.

Use request logs to debug failing queries, identify slow requests, and audit API usage.

Project monitors

Project monitors send notifications when a project signal meets a condition. Charts show trends; request logs show individual API requests.

Open the Observability tab for your project.
Select Monitors in the left navigation. The section is shown for projects on Spice-managed compute.
Select Create monitor, then choose a monitor type shown for the project.
Set its condition in When. Available controls depend on the monitor type.
Choose one or more destinations in Then.
Name the monitor and select Create monitor.

Monitor types

The picker lists monitors supported by the project. Most let you set a comparison and threshold. Dataset status also lets you choose a dataset and status; instance health has a fixed condition. Template defaults set the window, sustain period, and severity. A window is a lookback period where used; sustain is how long the condition must hold.

Monitor	What it watches	Condition and configuration
Instance health	A project instance reporting a failed state, runtime error, or eviction.	Fixed condition; no threshold to configure. Evictions use a fixed 5-minute observation window; critical severity, with no additional sustain delay.
Instance memory	Memory use as a percentage of the instance's configured memory limit.	Set the comparison and percentage threshold. Default: above 80%, sustained for 5 minutes; warning.
Instance CPU	CPU use as a percentage of the instance's configured CPU limit.	Set the comparison and percentage threshold. Default: above 85% over a 5-minute window, sustained for 5 minutes; critical. Requires a CPU limit.
Dataset status	A selected dataset reporting a status such as Error.	Choose the dataset and a status: Initializing, Ready, Disabled, Error, Refreshing, or Shutting down. Default: Error sustained for 5 minutes; critical.
Acceleration refresh errors	Errors recorded during dataset acceleration refreshes.	For projects with acceleration. Set the comparison and error-count threshold. Default: more than 0 errors in a 5-minute window; critical, with no additional sustain delay.
HTTP 5xx responses	Estimated server-error responses from the project's HTTP API.	Set the comparison and response-count threshold. Default: more than 0 estimated responses in a 5-minute window; critical, with no additional sustain delay.
SQL p99 latency	The 99th-percentile duration of SQL queries.	For projects with SQL queries. Set the comparison and threshold in milliseconds. Default: above 1000 ms over a 5-minute window, sustained for 5 minutes; warning.
SQL query failures	Server-caused SQL query failures.	Requires SQL query telemetry. Set the comparison and failure-count threshold. Default: more than 0 estimated failures in a 5-minute window; critical, with no additional sustain delay.
Flight SQL DoGet failures	Failed Flight SQL requests.	Requires Flight SQL telemetry. Set the comparison and failure-count threshold. Default: more than 0 estimated failures in a 5-minute window; critical, with no additional sustain delay.
LLM failures	Server-caused model request failures.	Requires a configured model. Set the comparison and failure-count threshold. Default: more than 0 estimated failures in a 5-minute window; critical, with no additional sustain delay.

Saved rate-based failure conditions retain their configured units and values.

In Then, choose one or more destinations: Email, HTTP, or Slack. Enter email recipients, an HTTPS URL for HTTP, or select a Slack channel. HTTP also accepts an optional authorization token.

An org administrator connects Slack through Settings > Integrations. Each monitor saves its own channel; the org default only preselects a channel for new monitors. See Connect Slack.

After saving, select Send test notification to check delivery to the configured destinations. The test does not evaluate the monitor condition or add an event to its history.

Default monitors

A new project on Spice-managed compute starts with eight monitors:

Instance health
Dataset status
HTTP 5xx responses
Flight SQL DoGet failures
SQL query failures
SQL p99 latency
Instance CPU
Instance memory

Instance CPU and Instance memory fire above 85%. The others use the template defaults in the table above.

The default monitors send email to the org's alert email addresses. When the org has none, they send email to the org owner. Edit or delete the default monitors like any other monitor. To create a project through the Management API without them, set default_alerts to false in POST /v1/projects.

Alert email addresses

An org admin keeps a list of up to 20 alert email addresses in Settings > Integrations > Alert email. Members can choose these addresses when they create or edit a monitor, and the default monitors of new projects use them.

Notifications

A monitor sends a notification when it starts firing and another when it resolves. For email and Slack, Spice combines related alerts into one notification. These alerts are for the same org, arrive within 5 seconds of each other, and have the same state and destination. An HTTP destination receives one request for each alert.

An HTTP destination receives a JSON request with Content-Type: application/json. The method is POST unless the destination sets PUT or PATCH. When the destination has a token, the request carries it as Authorization: Bearer <token>. The URL must be a public HTTPS URL, and the request times out after 10 seconds. The body has these fields:

Field	Description
`alert_id`	The monitor's ID.
`alert_name`	The monitor's name.
`state`	`firing` or `resolved`.
`severity`	`warning` or `critical`.
`occurred_at`	When the event occurred, as a UTC timestamp.
`resolved_at`	When the alert resolved, as a UTC timestamp, or `null` while it fires.
`org_name`	The org.
`scope`	`project` or `cluster`.
`project_name`	The project, when `scope` is `project`.
`cluster_name`, `nodegroup_name`, `node_name`	Optional. The cluster resources, when `scope` is `cluster`.
`kind`	The kind of alert.
`summary`	A one-line summary.
`description`, `condition`, `observed`, `details`, `url`	Optional. More context: a description, the condition, the observed value, details, and a link.