The metrics endpoint exposes per Spice project instance runtime metrics in Prometheus format. Use this endpoint to scrape metrics and send them to Datadog, Grafana, Prometheus, and OpenTelemetry compatible systems for dashboards and alerting.
Note
Runtime Endpoint: The metrics endpoint is served by your Spice runtime instance (e.g. https://<project-cname>.spiceai.io), not the Management API (https://api.spice.ai). Authenticate with your project API key , not a Personal Access Token.
Get Metrics
GEThttps://<project-cname>.spiceai.io/v1/metrics
Returns runtime metrics for the Spice project instance in Prometheus exposition format.
Header Type Description X-API-Keystring Your project API key
Response
200: OK 401: Unauthorized
Returns metrics in Prometheus text exposition format.
# HELP http_requests Number of HTTP requests.
# TYPE http_requests counter
http_requests_total{method="GET",path="/v1/sql"} 1024
# HELP query_duration_ms The total amount of time spent planning and executing queries in milliseconds.
# TYPE query_duration_ms histogram
query_duration_ms_bucket{le="10"} 50
query_duration_ms_bucket{le="100"} 200
...
{ "error" : "Unauthorized" } Example
curl https://us-west-2-prod-aws-data.spiceai.io/v1/metrics \ -H "X-API-Key: <API_KEY>" Monitoring Integrations
The metrics output is compatible with standard monitoring systems. Pre-built dashboards and setup guides are available:
Grafana & Prometheus - Scrape metrics with Prometheus, visualize in GrafanaDatadog - Scrape metrics with the Datadog Agent's OpenMetrics integrationOpenTelemetry - Use the OTEL Prometheus receiver
Warning
The metrics endpoint provides Spice runtime metrics only. Kubernetes pod-level metrics (CPU, memory, etc.) are not currently included.
Available Metrics
All metrics include relevant labels (dimensions) for filtering and aggregation.
Acceleration
Metric Type Description accelerated_ready_state_federated_fallbackcount Number of times the federated table was queried due to the accelerated table loading the initial data. accelerated_zero_results_federated_fallbackcount Number of times the federated table was queried due to the accelerated table returning zero results. dataset_acceleration_ingestion_lag_msgauge Lag between the current wall-clock time and the maximum time_column value after the refresh operation, in milliseconds. Disabled by default. dataset_acceleration_last_refresh_time_msgauge Unix timestamp in milliseconds when the last refresh completed. Disabled by default. dataset_acceleration_max_timestamp_after_refresh_msgauge Maximum value of the dataset's time_column after the refresh operation, in milliseconds. Disabled by default. dataset_acceleration_max_timestamp_before_refresh_msgauge Maximum value of the dataset's time_column before the refresh operation, in milliseconds. Disabled by default. dataset_acceleration_refresh_data_fetches_skippedcount Number of refresh data fetches skipped due to unchanged file metadata. dataset_acceleration_refresh_duration_mshistogram Duration in milliseconds to load a full or appended refresh data. dataset_acceleration_refresh_errorscount Number of errors refreshing the dataset. dataset_acceleration_refresh_lag_msgauge Difference between the maximum time_column value after and before the refresh operation, in milliseconds. dataset_acceleration_refresh_worker_panicscount Number of times a refresh worker panicked while refreshing a dataset. dataset_acceleration_snapshot_bootstrap_bytesgauge Number of bytes downloaded when bootstrapping the acceleration from a snapshot. dataset_acceleration_snapshot_bootstrap_checksumgauge Checksum of the snapshot downloaded during bootstrap (emitted with checksum attribute). dataset_acceleration_snapshot_bootstrap_duration_mscount Time in milliseconds taken to download the snapshot used to bootstrap acceleration. dataset_acceleration_snapshot_failure_countcount Number of failures encountered while writing snapshots. dataset_acceleration_snapshot_write_bytesgauge Number of bytes written for the most recent snapshot. dataset_acceleration_snapshot_write_checksumgauge Checksum of the most recent snapshot write (emitted with checksum attribute). dataset_acceleration_snapshot_write_duration_mshistogram Time in milliseconds taken to write the latest snapshot to object storage. dataset_acceleration_snapshot_write_timestampgauge Unix timestamp (seconds) when the most recent snapshot write completed.
Datasets
Metric Type Description dataset_active_countgauge Number of currently loaded datasets. dataset_load_errorscount Number of errors loading the dataset. dataset_load_stategauge Status of the dataset. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. dataset_unavailable_time_msgauge Time dataset went offline in milliseconds.
Catalogs
Metric Type Description catalog_load_errorscount Number of errors loading the catalog provider. catalog_load_stategauge Status of the catalog provider. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown.
Queries
Metric Type Description query_active_counthistogram Number of concurrent top-level queries actively being processed. Includes the protocol dimension (http, flight, flightsql, internal). query_duration_mshistogram The total amount of time spent planning and executing queries in milliseconds. query_execution_duration_mshistogram The total amount of time spent only executing queries (0 for cached queries). query_executionscount Number of query executions. query_failurescount Number of query failures. query_processed_bytescount Number of bytes processed by the runtime. query_produced_spillscount Number of spills produced by the query. query_returned_bytescount Number of bytes returned to query clients. query_returned_rowshistogram Number of rows returned to query clients. query_spilled_bytescount Number of spilled bytes produced by the query. query_spilled_rowscount Number of spilled rows produced by the query.
HTTP & Flight
Metric Type Description http_requestscount Number of HTTP requests. http_requests_duration_mshistogram Measures the duration of HTTP requests in milliseconds. flight_requestscount Total number of Flight requests. flight_request_duration_mshistogram Measures the duration of Flight requests in milliseconds. flight_do_exchange_data_updates_sentcount Number of data updates sent via DoExchange.
AI / LLM
Metric Type Description ai_inferences_with_spice_countcount AI Inferences with Spice count. llm_requestscount Number of LLM requests. llm_failurescount Number of LLM failures. llm_internal_request_duration_mshistogram The duration of running an LLM request internally. llm_load_stategauge Status of the LLM model. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown.
Models
Metric Type Description model_active_countgauge Number of currently loaded models. model_load_duration_mshistogram Duration in milliseconds to load the model. model_load_errorscount Number of errors loading the model. model_load_stategauge Status of the model. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown.
Embeddings
Metric Type Description embeddings_active_countgauge Number of currently loaded embeddings. embeddings_requestscount Number of embedding requests. embeddings_failurescount Number of embedding failures. embeddings_internal_request_duration_mshistogram The duration of running an embedding(s) internally. embeddings_load_errorscount Number of errors loading the embedding. embeddings_load_stategauge Status of the embedding. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown.
Embeddings Cache
Metric Type Description embeddings_cache_hitscount Cache hit count. embeddings_cache_missescount Cache miss count. embeddings_cache_requestscount Number of requests to get a key from the cache. embeddings_cache_evictionscount Number of cache evictions. embeddings_cache_hit_ratiogauge Cache hit ratio (hits / total requests). embeddings_cache_items_countgauge Number of items currently in the cache. embeddings_cache_size_bytesgauge Size of the cache in bytes. embeddings_cache_max_size_bytesgauge Maximum allowed size of the cache in bytes. embeddings_cache_stale_swr_countcount Number of stale-while-revalidate background refreshes skipped due to existing in-flight revalidation. embeddings_cache_swr_background_query_countcount Number of background queries triggered for stale-while-revalidate cache refreshes.
Results Cache
Metric Type Description results_cache_hitscount Cache hit count. results_cache_missescount Cache miss count. results_cache_requestscount Number of requests to get a key from the cache. results_cache_evictionscount Number of cache evictions. results_cache_hit_ratiogauge Cache hit ratio (hits / total requests). results_cache_items_countgauge Number of items currently in the cache. results_cache_size_bytesgauge Size of the cache in bytes. results_cache_max_size_bytesgauge Maximum allowed size of the cache in bytes. results_cache_stale_swr_countcount Number of stale-while-revalidate background refreshes skipped due to existing in-flight revalidation. results_cache_swr_background_query_countcount Number of background queries triggered for stale-while-revalidate cache refreshes.
Search Results Cache
Metric Type Description search_results_cache_hitscount Search cache hit count. search_results_cache_missescount Cache miss count. search_results_cache_requestscount Number of requests to get a key from the search cache. search_results_cache_evictionscount Number of cache evictions. search_results_cache_hit_ratiogauge Cache hit ratio (hits / total requests). search_results_cache_items_countgauge Number of items currently in the search cache. search_results_cache_size_bytesgauge Size of the search cache in bytes. search_results_cache_max_size_bytesgauge Maximum allowed size of the search cache in bytes. search_results_cache_stale_swr_countcount Number of stale-while-revalidate background refreshes skipped due to existing in-flight revalidation. search_results_cache_swr_background_query_countcount Number of background queries triggered for stale-while-revalidate cache refreshes.
Metric Type Description tool_active_countgauge Number of currently loaded LLM tools. tool_load_errorscount Number of errors loading the LLM tool. tool_load_stategauge Status of the LLM tools. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. view_load_errorscount Number of errors loading the view. view_load_stategauge Status of the views. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown.
Runtime
Metric Type Description component_metric_registered_countgauge Number of currently registered component metrics. runtime_http_server_startedcount Indicates the runtime HTTP server has started. runtime_flight_server_startedcount Indicates the runtime Flight server has started. secrets_store_load_duration_mshistogram Duration in milliseconds to load the secret stores. worker_active_countgauge Number of currently loaded workers. workers_load_duration_mshistogram Duration in milliseconds to load the worker.
Note
In addition to these core metrics, individual components can expose their own metrics. For example, the MySQL data connector exposes connection pool metrics. See the Spice OSS Observability docs for more details.
See also: