# Observability Continuously identify and fix issues by tracking process actions with Spice monitoring and request logs. The **Observability** tab provides real-time visibility into your project's usage, request performance, and API activity. Use it to track request volume, identify failures, and debug issues. The tab groups the project's metrics, request logs, and traces into sections, listed in the left navigation: - **Overview**: aggregate metrics for the project. - **API**: API request metrics. - **Cache**: results cache metrics. Shown when SQL results caching is enabled. - **Data**: dataset and acceleration metrics. - **Resources**: CPU and memory usage for the project's running instances. - **Request Logs**: a record of individual API requests. - **Monitors**: project monitors that send notifications. Shown for projects on Spice-managed compute. - **Traces**: task traces. Shown for projects with a spicepod. ## Accessing Observability 1. Navigate to your Spice project in the [portal](https://spice.ai/). 2. Click the **Observability** tab in the project navigation bar. ## Overview The **Overview** section shows aggregate metrics for: - **SQL Queries**: succeeded and failed query counts, and p95 and p99 query latency. - **AI Completions**: succeeded and failed completion counts, and p95 and p99 latency. Shown when the project has a model. - **Embeddings**: succeeded and failed embedding search counts, and p95 and p99 search latency. Shown when the project has embeddings. - **Acceleration**: the maximum dataset acceleration refresh duration. Shown when the project accelerates a dataset. ## Cache The **Cache** section charts the SQL results cache: - **Results Cache Hits vs Misses**: the share of cache lookups that were hits and misses at each step, from 0% to 100%. Hover a step to see each share and its lookup count. A step with no lookups stays empty. - **Results Cache Usage**: the cache size in bytes, with a line at the configured maximum (`runtime.caching.sql_results.max_size`, default `128MiB`). ## Resources The **Resources** section charts CPU and memory usage per running instance of the project. 1. Open the **Observability** tab for your project. 2. Select **Resources** in the left navigation. CPU is reported as utilization against the project's configured CPU limit. Projects with no CPU limit set (including projects on dedicated clusters) report absolute core usage instead. Memory is reported in bytes. Use the resource charts to size a project before raising its limits, and to correlate slow queries with memory or CPU pressure. ## Time ranges Each chart section has a time range menu. - **Overview** and **API**: **Last hour**, **Last 24 hours**, **Last 2 days**, **Last 7 days**, or **Last 28 days**. - **Cache**, **Data**, and **Resources**: **Last 1 hour**, **Last 24 hours**, **Last 7 days**, or **Last 28 days**. Use the longer ranges to identify usage trends, detect spikes in query duration, and plan capacity. ## API Request Logs The request logs provide a detailed record of individual API requests to your project's endpoints, including status codes, durations, and timestamps. 1. Open the **Observability** tab for your project. 2. Select **Request Logs** in the left navigation. 3. Select a time range. 4. Browse the log entries to inspect individual request details including endpoint, status code, and duration. The Request Logs time ranges are **1 Hour**, **8 Hours**, **24 Hours**, **3 Days**, and **7 Days**. The ranges available depend on the org's plan. Every plan has **1 Hour**. Use request logs to debug failing queries, identify slow requests, and audit API usage. ## Project monitors Project monitors send notifications when a project signal meets a condition. Charts show trends; request logs show individual API requests. 1. Open the **Observability** tab for your project. 2. Select **Monitors** in the left navigation. The section is shown for projects on Spice-managed compute. 3. Select **Create monitor**, then choose a monitor type shown for the project. 4. Set its condition in **When**. Available controls depend on the monitor type. 5. Choose one or more destinations in **Then**. 6. Name the monitor and select **Create monitor**. ### Monitor types The picker lists monitors supported by the project. Most let you set a comparison and threshold. Dataset status also lets you choose a dataset and status; instance health has a fixed condition. Template defaults set the window, sustain period, and severity. A window is a lookback period where used; sustain is how long the condition must hold. | Monitor | What it watches | Condition and configuration | | - | - | - | | **Instance health** | A project instance reporting a failed state, runtime error, or eviction. | Fixed condition; no threshold to configure. Evictions use a fixed 5-minute observation window; critical severity, with no additional sustain delay. | | **Instance memory** | Memory use as a percentage of the instance's configured memory limit. | Set the comparison and percentage threshold. Default: above 80%, sustained for 5 minutes; warning. | | **Instance CPU** | CPU use as a percentage of the instance's configured CPU limit. | Set the comparison and percentage threshold. Default: above 85% over a 5-minute window, sustained for 5 minutes; critical. Requires a CPU limit. | | **Dataset status** | A selected dataset reporting a status such as Error. | Choose the dataset and a status: Initializing, Ready, Disabled, Error, Refreshing, or Shutting down. Default: Error sustained for 5 minutes; critical. | | **Acceleration refresh errors** | Errors recorded during dataset acceleration refreshes. | For projects with acceleration. Set the comparison and error-count threshold. Default: more than 0 errors in a 5-minute window; critical, with no additional sustain delay. | | **HTTP 5xx responses** | Estimated server-error responses from the project's HTTP API. | Set the comparison and response-count threshold. Default: more than 0 estimated responses in a 5-minute window; critical, with no additional sustain delay. | | **SQL p99 latency** | The 99th-percentile duration of SQL queries. | For projects with SQL queries. Set the comparison and threshold in milliseconds. Default: above 1000 ms over a 5-minute window, sustained for 5 minutes; warning. | | **SQL query failures** | Server-caused SQL query failures. | Requires SQL query telemetry. Set the comparison and failure-count threshold. Default: more than 0 estimated failures in a 5-minute window; critical, with no additional sustain delay. | | **Flight SQL DoGet failures** | Failed Flight SQL requests. | Requires Flight SQL telemetry. Set the comparison and failure-count threshold. Default: more than 0 estimated failures in a 5-minute window; critical, with no additional sustain delay. | | **LLM failures** | Server-caused model request failures. | Requires a configured model. Set the comparison and failure-count threshold. Default: more than 0 estimated failures in a 5-minute window; critical, with no additional sustain delay. | Saved rate-based failure conditions retain their configured units and values. In **Then**, choose one or more destinations: **Email**, **HTTP**, or **Slack**. Enter email recipients, an HTTPS URL for HTTP, or select a Slack channel. HTTP also accepts an optional authorization token. An org administrator connects Slack through **Settings** > **Integrations**. Each monitor saves its own channel; the org default only preselects a channel for new monitors. See [Connect Slack](https://spice.ai/docs/cloud/integrations/slack). After saving, select **Send test notification** to check delivery to the configured destinations. The test does not evaluate the monitor condition or add an event to its history. ### Default monitors A new project on Spice-managed compute starts with eight monitors: - **Instance health** - **Dataset status** - **HTTP 5xx responses** - **Flight SQL DoGet failures** - **SQL query failures** - **SQL p99 latency** - **Instance CPU** - **Instance memory** Instance CPU and Instance memory fire above 85%. The others use the template defaults in the table above. The default monitors send email to the org's alert email addresses. When the org has none, they send email to the org owner. Edit or delete the default monitors like any other monitor. To create a project through the Management API without them, set `default_alerts` to `false` in `POST /v1/projects`. ### Alert email addresses An org admin keeps a list of up to 20 alert email addresses in **Settings** > **Integrations** > **Alert email**. Members can choose these addresses when they create or edit a monitor, and the default monitors of new projects use them. ### Notifications A monitor sends a notification when it starts firing and another when it resolves. For email and Slack, Spice combines related alerts into one notification. These alerts are for the same org, arrive within 5 seconds of each other, and have the same state and destination. An HTTP destination receives one request for each alert. An **HTTP** destination receives a JSON request with `Content-Type: application/json`. The method is `POST` unless the destination sets `PUT` or `PATCH`. When the destination has a token, the request carries it as `Authorization: Bearer `. The URL must be a public HTTPS URL, and the request times out after 10 seconds. The body has these fields: | Field | Description | | - | - | | `alert_id` | The monitor's ID. | | `alert_name` | The monitor's name. | | `state` | `firing` or `resolved`. | | `severity` | `warning` or `critical`. | | `occurred_at` | When the event occurred, as a UTC timestamp. | | `resolved_at` | When the alert resolved, as a UTC timestamp, or `null` while it fires. | | `org_name` | The org. | | `scope` | `project` or `cluster`. | | `project_name` | The project, when `scope` is `project`. | | `cluster_name`, `nodegroup_name`, `node_name` | Optional. The cluster resources, when `scope` is `cluster`. | | `kind` | The kind of alert. | | `summary` | A one-line summary. | | `description`, `condition`, `observed`, `details`, `url` | Optional. More context: a description, the condition, the observed value, details, and a link. |