Observability

Spice.ai Enterprise exposes a comprehensive set of metrics, structured logs, and OpenTelemetry traces. This page documents how to wire each into a production observability stack and which signals to alert on.

Metrics

The runtime exposes Prometheus metrics on port 9090 at /metrics. The Helm chart and the Kubernetes operator both ship with first-class scrape integration:

• Helm chart: set monitoring.podMonitor.enabled: true to deploy a PodMonitor.
• Operator: set servicemonitor.enabled: true to deploy a ServiceMonitor for the operator itself.

# values.yaml (Spice Helm chart)
monitoring:
  podMonitor:
    enabled: true
    additionalLabels:
      release: prometheus

Key metrics

Metric names are exported verbatim, without a namespace prefix and without unit or counter suffixes. Durations are milliseconds, and the unit is part of the name.

The spiced_cpu_budget_cores source label reports how the entitlement was determined: configured, cgroup_quota, affinity, or fallback.

For the full list, query the running runtime: curl localhost:9090/metrics | grep -E '^# HELP'.

Memory pools

Memory budgets are derived from the running process's own cgroup limit, so a container sees the memory it was allocated rather than the memory of its host.

A dedicated compaction memory pool is reserved only for Cayenne accelerations that can compact into it — file mode with a small-write refresh profile. Other deployments, including refresh_mode: full, keep the whole memory limit available to queries.

Compare query_memory_pool_used_bytes against the memory limit to see query headroom. A rising cayenne_compaction_memory_exhausted_total means compaction cannot reserve the memory its rewrite working set needs; raise the memory limit for the deployment.

When a memory pool refuses a reservation, the query fails with the ResourcesExhausted error code, the HTTP API returns 503 Service Unavailable, and Arrow Flight reports RESOURCE_EXHAUSTED. Both are retriable, so pool exhaustion stays distinguishable from a rejected query, which returns 400. Alert on it through the err_code label:

sum(rate(query_failures{err_code="ResourcesExhausted"}[5m])) > 0

Cache metrics

Each cache the runtime maintains exports its own metric family. The SQL results cache uses the results_ prefix; the search results and embeddings caches use search_results_ and embeddings_ and expose the same metric names.

The reason label on *_cache_evictions separates three causes that call for different responses:

On an accelerated dataset with a periodic refresh, invalidated is normally the dominant cause and would swamp an unlabelled total. Alert on size instead, which is the reason that indicates real cache pressure:

sum(rate(results_cache_evictions{reason="size"}[5m])) > 0

Cayenne segment cache

Cayenne accelerations read through one in-memory segment cache shared by every Cayenne table in the process. Size it with cayenne_segment_cache_mb under runtime.params:

runtime:
  params:
    cayenne_segment_cache_mb: 1024

When unset, the budget is approximately 1/64 of the detected memory entitlement, clamped to between 256 MiB and 2 GiB. The parameter counts binary megabytes, so cayenne_segment_cache_mb: 1024 reserves 1 GiB. A value of 0 disables segment caching. Every series carries a cache label; the shared cache reports as shared.

The accesses and hits series are cumulative counters, so read them with rate() or increase() rather than as instantaneous values. A hit ratio that falls while cayenne_segment_cache_weighted_bytes sits at cayenne_segment_cache_capacity_bytes means the combined working set of every Cayenne table no longer fits; raise runtime.params.cayenne_segment_cache_mb.

Grafana dashboard

Spice.ai publishes a maintained Grafana dashboard with the panels operations teams need most often (query rate / latency / errors, acceleration freshness and row counts, executor registration, certificate expiry).

Import via dashboard ID or copy the JSON from the Spice.ai Grafana dashboard. The dashboard is compatible with Prometheus, Amazon Managed Prometheus, Azure Managed Prometheus, and Google Cloud Managed Service for Prometheus.

Logs

Spice emits structured JSON logs on stdout. The log level is controlled by SPICED_LOG:

spec:
  env:
    - name: SPICED_LOG
      value: INFO

Log routing

Always retain query-error and acceleration-refresh-failure log lines for at least 30 days for incident review.

Distributed tracing

When the runtime is started with --otel-endpoint or the SPICED_OTEL_ENDPOINT environment variable, Spice exports OpenTelemetry traces over OTLP/gRPC. Traces cover query parsing, optimization, and execution; for SpicepodCluster, scheduler-to-executor RPCs are linked into a single trace.

spec:
  env:
    - name: SPICED_OTEL_ENDPOINT
      value: http://otel-collector.observability:4317
    - name: SPICED_OTEL_RESOURCE_ATTRIBUTES
      value: service.name=spiceai,deployment.environment=prod

Pair with an OpenTelemetry Collector configured for the organization's tracing backend (Tempo, Jaeger, Datadog, Honeycomb, X-Ray).

Alerts

The following alerts are the minimum recommended set for production. Tune thresholds against the deployment's observed baseline.

groups:
  - name: spiceai
    interval: 30s
    rules:
      - alert: SpiceAIQueryErrorRateHigh
        expr: |
          sum(rate(query_failures[5m]))
            /
          sum(rate(query_executions[5m])) > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Query error rate above 5% for 10 minutes."

      - alert: SpiceAIAccelerationRefreshFailing
        expr: |
          increase(dataset_acceleration_refresh_errors[15m]) > 3
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Acceleration refresh failing for {{ $labels.dataset }}."

      - alert: SpiceAIQueryLatencyP95
        expr: |
          histogram_quantile(0.95, sum by (le) (
            rate(query_duration_ms_bucket[5m])
          )) > 2000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "p95 query latency above 2s."

      - alert: SpiceAIExecutorMissing
        expr: |
          scheduler_active_executors_count < 2
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Scheduler {{ $labels.node_id }} has fewer than 2 executors registered."

      - alert: SpiceAIClusterCAExpiringSoon
        expr: |
          spiceai_operator_cluster_ca_expiry_timestamp_seconds - time() < 7 * 24 * 3600
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.cluster }} CA certificate expires in less than 7 days."

The latency threshold is in milliseconds, matching the query_duration_ms histogram.

Crashloop protection is not exposed as a metric. The operator records it on the resource as a pause reason and emits Kubernetes events \u2014 see Crashloop Protection. Alert on container restarts using the kube_pod_container_status_restarts_total series from kube-state-metrics.

Health endpoints

The Spice Helm chart and the operator configure liveness and readiness probes against these endpoints by default. Tune the probe parameters on heavy initial-load workloads via probes:

spec:
  probes:
    readiness:
      initialDelaySeconds: 30
      periodSeconds: 10
      failureThreshold: 6
    liveness:
      initialDelaySeconds: 60
      periodSeconds: 30
      failureThreshold: 5

Gating readiness on executor availability (scheduler role)

In distributed (SpicepodCluster) deployments a scheduler can finish loading its own datasets and accelerations before enough executors have connected to actually serve queries. To keep a scheduler out of rotation until the cluster has capacity, GET /v1/ready accepts two optional executor gates. They apply only to the scheduler role; supplying a non-zero value on a non-scheduler node returns 400.

When both gates are supplied, both must pass for the probe to return 200 OK; otherwise the endpoint returns 503 (gate not yet satisfied) or 400 (invalid parameter, e.g. a percentage outside 0–100, or a non-zero gate requested outside scheduler role). Datasets and accelerations must still have completed their initial load regardless of executor gating.

Pass the gates as query parameters on the readiness probe path:

readinessProbe:
  httpGet:
    path: /v1/ready?min_ready_executors=3&min_ready_executors_percent=80
    port: 8090