Spice can be monitored with Datadog (opens in a new tab) using the Spice Metrics Endpoint (opens in a new tab) and pre-built dashboards available in the Spice repository (opens in a new tab).
Prerequisite: Datadog Agent version 6.5.0 or later is installed (opens in a new tab).
Configure the Datadog Agent to scrape the Spice metrics endpoint:
openmetrics.d/conf.yaml file in the conf.d/ folder at the root of your Agent’s configuration directory (opens in a new tab):spiceai to confirm Spice telemetry information is successfully collected.:::note The namespace sets the metric prefix
Every scraped metric is prefixed with the configured namespace, so query_duration_ms is stored as spiceai.query_duration_ms. Keep this value consistent with the prefix used by the queries in the Spice dashboard JSON, and with runtime.telemetry.metric_prefix (opens in a new tab) if the same deployment also pushes metrics over OTLP.
:::
Spice exports latency and size metrics — query_duration_ms, flight_request_duration_ms, http_requests_duration_ms, dataset_acceleration_refresh_duration_ms, and others — as Prometheus histograms. Configuring the Agent to submit those buckets as Datadog distributions (opens in a new tab) makes percentiles queryable over the selected time window:
| Option | Default | Why Spice needs it |
|---|---|---|
collect_histogram_buckets | true | Sends the _bucket series that percentiles are computed from. |
histogram_buckets_as_distributions | false | Submits those buckets as a Datadog distribution rather than a set of counters. Required for p50:, p90:, p95:, and p99: queries. |
:::caution Percentile aggregations must be enabled on the metric
Submitting a distribution is not sufficient on its own. Datadog computes p50/p90/p95/p99 for a distribution metric only once percentile aggregations are enabled for that metric on the Metrics Summary page — see Enabling advanced query functionality (opens in a new tab). Until then a p99: query returns no data and the widget renders empty rather than reporting an error.
:::
:::note Why not the _summary metrics
The metrics endpoint also exposes a <metric>_summary family carrying quantile tags, derived from each histogram at scrape time. The underlying histogram is cumulative, so those quantiles describe the entire lifetime of the process rather than the queried time window: they flatten the longer a pod stays up and will not surface a latency regression. Prefer distributions for any percentile that needs to track a time window.
:::
With the Datadog Agent (opens in a new tab) in the cluster, annotate the Spice container (spiceai in the Helm chart) so OpenMetrics scrapes include the service_instance_id tag used by the dashboard instance filter:
collect_histogram_buckets and histogram_buckets_as_distributions serve the same purpose here as in the host configuration — see Histogram Percentiles (Distributions) (opens in a new tab). max_returned_metrics raises the check's default limit of 2000, which Spice can exceed.
The Spice dashboard identifies each running instance by a service_instance_id tag: the instance filter selects on it, and every per-instance panel groups by it. The Agent supplies no instance-level tag of its own — its unified service tags (opens in a new tab) are env, service, and version — so set it explicitly, as in both examples above:
| Deployment | Value |
|---|---|
| Kubernetes | %%kube_pod_name%% |
| Host, VM, or Docker | Hostname, or another stable per-process identifier |
:::caution Panels collapse silently without this tag
Datadog does not error on a missing tag. Without service_instance_id, each panel renders a single N/A series summing every instance — a 12-replica deployment reports 12 times its real dataset count rather than showing nothing.
:::
Panels from the Kubernetes integration (CPU, memory, and PVC utilization) group by pod_name instead, as those metrics never carry service_instance_id.
As an alternative to scraping the Prometheus endpoint with the Datadog Agent, Spice can push metrics directly to Datadog's OTLP Metrics Intake Endpoint (opens in a new tab) over HTTP. This is the recommended approach for agentless deployments (e.g. serverless, ephemeral containers) and for environments where the Datadog API key is managed through Spice's secret stores (opens in a new tab).
Replace us3 with the Datadog site for the target account (us3, us5, eu, ap1, etc.) and store the Datadog API key in a secret:
Metrics begin appearing in the Datadog Metrics Explorer within a minute or two.
Use runtime.telemetry.metric_prefix (opens in a new tab) to prepend a string to every exported metric name. This avoids collisions with metrics from other services in the same Datadog account:
The runtime metric query_duration_ms is then exported as spiceai.query_duration_ms.
:::caution Combining metric_prefix with metric filtering
If you also set runtime.telemetry.otel_exporter.metrics (opens in a new tab) to whitelist specific metrics, the entries must include the prefix. The filter runs after the prefix is applied, so e.g. query_duration_ms will not match when metric_prefix: 'spiceai.' is set — use spiceai.query_duration_ms instead.
:::
Attach custom key/value pairs to every metric using runtime.telemetry.properties (opens in a new tab). Spice sends these as OpenTelemetry resource attributes:
For these resource attributes to surface as tags in Datadog, the Datadog OTLP intake also requires the dd-otel-metric-config header with resource_attributes_as_tags enabled (see Datadog OTLP Metrics Intake Endpoint (opens in a new tab)):
:::note Tags can lag behind metrics Datadog typically ingests OTLP metrics within seconds, but the associated tags (from resource attributes) can take noticeably longer to appear in the UI — sometimes several minutes after the first datapoints. The metrics and tags do eventually converge. :::
:::caution Manage tag cardinality in Datadog
Datadog bills on custom metric cardinality (opens in a new tab), driven by the number of unique tag-value combinations per metric. The custom tags added via runtime.telemetry.properties are typically low-cardinality (environment, region, team), but Spice metrics also carry a number of automatically populated dimensions — for example dataset, protocol, client, client_version, client_system, user_agent, runtime, runtime_version, runtime_system (see Available Metrics (opens in a new tab)) — some of which can grow with the size of the deployment.
Datadog's Metrics without Limits™ (opens in a new tab) decouples ingestion from indexing for exactly this case. With Metrics without Limits™, every tag Spice emits is still ingested, but each metric is configured with one of:
dataset and environment, drop the rest), oruser_agent or client_version).Only the indexed (queryable) tag combinations count toward custom metric billing. Configuration is done per metric in the Metrics Summary page or via the Metrics API, and the in-app UI surfaces an estimated indexed-metric volume before saving and can pre-populate an allowlist from tags actively queried in dashboards, monitors, and notebooks. :::
A complete runtime.telemetry block combining metric prefixing, custom tags, and Datadog OTLP export:
With this configuration, every Spice metric (e.g. spiceai.query_duration_ms, spiceai.query_executions) arrives in Datadog tagged with environment:prod, region:us-west-2, and team:data-platform.
For general OTLP exporter options (push interval, metric filtering, gRPC vs HTTP), see OpenTelemetry Metrics Exporter (opens in a new tab).