Spice can be monitored using the Spice Prometheus-compatible Metrics Endpoint (opens in a new tab). Spice also supports distributed tracing by integrating with Zipkin (opens in a new tab) and compatible tracing systems.
Monitoring clients configuration:
The metrics endpoint uses port 9090 by default. The metrics endpoint configuration is logged at startup.
Pass the --metrics parameter to bind to a specific port. For example, to bind to port 9091:
or when using Docker:
Configuration of the metrics endpoint can be verified using a HTTP GET request, for example:
| Metric | Description |
|---|---|
accelerated_ready_state_federated_fallback(count) | Number of times the federated table was queried due to the accelerated table loading the initial data. |
catalog_load_errors(count) | Number of errors loading the catalog provider. |
catalog_load_state(gauge) | Status of the catalog provider. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
dataset_acceleration_last_refresh_time_ms(gauge) | Unix timestamp in seconds when the last refresh completed. |
dataset_acceleration_refresh_duration_ms(histogram) | Duration in milliseconds to load a full or appended refresh data. |
dataset_acceleration_refresh_errors(count) | Number of errors refreshing the dataset. |
dataset_active_count(gauge) | Number of currently loaded datasets. |
dataset_load_state(gauge) | Status of the dataset. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
dataset_unavailable_time_ms(gauge) | Time dataset went offline in milliseconds. |
embeddings_active_count(gauge) | Number of currently loaded embeddings. |
embeddings_load_errors(count) | Number of errors loading the embedding. |
embeddings_load_state(gauge) | Status of the embedding. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
flight_request_duration_ms(histogram) | Measures the duration of Flight requests in milliseconds. |
flight_requests(count) | Total number of Flight requests. |
http_requests_duration_ms(histogram) | Measures the duration of HTTP requests in milliseconds. |
http_requests(count) | Number of HTTP requests. |
llm_load_state(gauge) | Status of the LLM model. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
model_active_count(gauge) | Number of currently loaded models. |
model_load_duration_ms(histogram) | Duration in milliseconds to load the model. |
model_load_errors(count) | Number of errors loading the model. |
model_load_state(gauge) | Status of the model. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
query_duration_ms(histogram) | The total amount of time spent planning and executing queries in milliseconds. |
query_execution_duration_ms(histogram) | The total amount of time spent only executing queries (0 for cached queries). |
query_executions(count) | Number of query executions. |
query_failures(count) | Number of query failures. |
query_processed_bytes(count) | Number of bytes processed by the runtime. |
query_returned_bytes(count) | Number of bytes returned to query clients. |
embeddings_cache_max_size_bytes(gauge) | Maximum allowed size of the cache in bytes. |
embeddings_cache_requests(count) | Number of requests to get a key from the cache. |
embeddings_cache_hits(count) | Cache hit count. |
embeddings_cache_items_count(gauge) | Number of items currently in the cache. |
embeddings_cache_size_bytes(gauge) | Size of the cache in bytes. |
results_cache_max_size_bytes(gauge) | Maximum allowed size of the cache in bytes. |
results_cache_requests(count) | Number of requests to get a key from the cache. |
results_cache_hits(count) | Cache hit count. |
results_cache_items_count(gauge) | Number of items currently in the cache. |
results_cache_size_bytes(gauge) | Size of the cache in bytes. |
search_results_cache_max_size_bytes(gauge) | Maximum allowed size of the search cache in bytes. |
search_results_cache_requests(count) | Number of requests to get a key from the search cache. |
search_results_cache_hits(count) | Search cache hit count. |
search_results_cache_items_count(gauge) | Number of items currently in the search cache. |
search_results_cache_size_bytes(gauge) | Size of the search cache in bytes. |
runtime_flight_server_started(count) | Indicates the runtime Flight server has started. |
runtime_http_server_started(count) | Indicates the runtime HTTP server has started. |
secrets_store_load_duration_ms(histogram) | Duration in milliseconds to load the secret stores. |
tool_active_count(gauge) | Number of currently loaded LLM tools. |
tool_load_errors(count) | Number of errors loading the LLM tool. |
tool_load_state(gauge) | Status of the LLM tools. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
view_load_errors(count) | Number of errors loading the view. |
view_load_state(gauge) | Status of the views. 0=Initializing, 1=Ready, 2=Disabled, 3=Error, 4=Refreshing, 5=ShuttingDown. |
:::note Component Metrics
In addition to these core metrics, individual components can expose their own metrics. For example, the MySQL data connector exposes connection pool metrics (opens in a new tab). See Component Metrics (opens in a new tab) for more information.
:::
