---

title: 'Runtime' sidebar_label: 'Runtime' description: 'Runtime YAML reference'

The runtime section specifies configuration settings for the Spice runtime.

Reload Behavior

Editing a Spicepod on disk reloads it into the running process, but most of runtime.* is consumed once when spiced starts and cannot be rebuilt in place. A reload installs the new value in the app while the process keeps running the old one; spiced logs a warning naming each start-time-only section that changed and telling you to restart. Restart spiced to apply them.

Applied when spiced starts — a reload logs a warning and the previous value stays in effect:

runtime.auth · runtime.caching · runtime.cors · runtime.cpu · runtime.dataset_load_parallelism · runtime.mcp · runtime.metrics · runtime.output_level · runtime.query · runtime.ready_state · runtime.scheduler · runtime.task_history · runtime.telemetry · runtime.tls · runtime.tracing

Applied when spiced starts, except for components the same reload recreates — a connector rebuilt by the reload reads the new value, while the process-wide use of it does not change until a restart:

runtime.flight · runtime.params · runtime.source_rate_control

Applied on reload — no restart needed:

Setting	Why it applies
runtime.shutdown_timeout	Read from the current app when the runtime shuts down.
runtime.functions	Reconciled by the function diff on each reload.
runtime.caching.sql_results.cache_key_type	Resolved per request from the live app.
runtime.query.timeout	Resolved per request from the live app.
runtime.telemetry.user_agent_collection	Resolved per request from the live app.

The three per-request settings sit inside otherwise start-time-only sections. Changing one of them alone takes effect on the next request and is not reported as requiring a restart; changing any other key in the same section is.

:::note runtime.tls certificate and CA files are separately hot-reloadable without a restart — see Certificate Hot-Reload (opens in a new tab). Inline PEM material is loaded once at startup. :::

runtime.auth

runtime.auth.api-key

Spice supports adding optional authentication to its API endpoints via configurable API keys. Learn more (opens in a new tab).

API key authentication supports the following configuration parameters:

Parameter name	Optional	Default	Description
enabled	Yes	true	Defaults to true. Whether API key authentication is enabled
keys	No	-	A list of API keys used to authenticate requests.

runtime.dataset_load_parallelism

This setting specifies the maximum number of datasets that can be loaded in parallel during startup. By default, the number of parallel datasets is unlimited.

runtime.caching

This setting specifies cache settings for supported Runtime components:

• sql_results: Specifies cache settings for results from SQL queries.
• search_results: Specifies cache settings for results from searches.
• embeddings: Specifies cache settings for embeddings requests.

Runtime caches support common configuration parameters:

Parameter name	Optional	Default	Description
enabled	Yes	true	Defaults to true.
max_size	Yes	128MiB	Maximum cache size. Defaults to 128MiB.
eviction_policy	Yes	lru	Cache replacement policy when the cache reaches max_size. Defaults to lru. Supports lru (Least Recently Used) and tiny_lfu (Tiny Least Frequently Used, higher hit rate for skewed access patterns).
item_ttl	Yes	1s	Cache entry expiration duration (Time to Live). Defaults to 1 second.
hashing_algorithm

runtime.caching.search_results

The search results cache section specifies runtime search cache configuration. Learn more (opens in a new tab).

The search results cache supports the common cache configuration parameters.

runtime.caching.embeddings

The embeddings cache section specifies runtime embeddings requests cache configuration. Learn more (opens in a new tab).

The embeddings cache supports the common cache configuration parameters.

runtime.caching.sql_results

The SQL results cache section specifies runtime SQL query cache configuration. Learn more (opens in a new tab).

In addition to the common cache configuration parameters, sql_results also supports the following parameters:

Parameter name	Optional	Default	Description
cache_key_type	Yes	plan	Determines how cache keys are generated. Defaults to plan. plan uses the query's logical plan, while sql uses the raw SQL query string.
encoding	Yes	none	Compression algorithm for cached results. Defaults to none. Supports none or zstd.
stale_while_revalidate_ttl	Yes	0s	Duration to serve stale cache entries while revalidating in the background. When set to a non-zero value, expired cache entries continue to be served while a background refresh occurs. Defaults to 0s (disabled).

:::info

runtime.results_cache has been deprecated and will be removed in a future release. If runtime.results_cache is specifed in the spicepod it will override the runtime.caching.sql_results settings if it is not defined.

:::

Choosing a cache_key_type

• plan (Default): Uses the query's logical plan as the cache key. Matches semantically equivalent queries but requires query parsing.
• sql: Uses the raw SQL string as the cache key. Provides faster lookups but requires exact string matches. Queries with dynamic functions, such as NOW(), may produce unexpected results. Use sql only when results are predictable.

Use sql for the lowest latency with identical queries that do not include dynamic functions. Use plan for greater flexibility.

Choosing a hashing_algorithm

• xxh3 (Default): Uses the XXH3 (opens in a new tab) algorithm for hashing the cache keys. XXH3 is a fast, non-cryptographic hash algorithm that provides high performance and good distribution. It is suitable for scenarios where speed is critical and cryptographic security is not required.
• siphash: Uses the SipHash1-3 algorithm for hashing the cache keys, the default hashing algorithm of Rust (opens in a new tab). This hashing algorithm is a secure algorithm that implements verified protections against "hash flooding" (opens in a new tab) denial of service (DoS) attacks. Reasonably performant, and provides a high level of security.
• ahash: Uses the AHash (opens in a new tab) algorithm for hashing the cache keys. The AHash algorithm is a high quality (opens in a new tab) hashing algorithm, and has claimed resistance against hashing DoS attacks. AHash has higher performance than SipHash1-3, especially when used with cache_key_type: plan.
• blake3: Uses the BLAKE3 (opens in a new tab) cryptographic hash function. BLAKE3 is a fast, parallelizable hash function that provides cryptographic security while maintaining high performance. It is suitable for scenarios requiring both speed and cryptographic guarantees.

Use xxh3 (the default) for its superior speed in most scenarios. Use ahash, xxh64 or xxh128 for reduced collision probability when caching a large number of queries. Use blake3 when cryptographic security is required. Use siphash when protection against hash flooding attacks is a priority.

runtime.params

Optional. Global key-value parameters for the runtime.

HTTP Rate Control

HTTP-based connectors (HTTP/HTTPS, GraphQL, Databricks) support the following rate control defaults. The GitHub connector is not part of this family — it has its own limiter, configured with runtime.source_rate_control.github_concurrent_connections_limit (opens in a new tab):

Parameter Name	Description
http_max_concurrent_requests	Default maximum concurrent HTTP requests per upstream origin. Can be overridden per-dataset with max_concurrent_requests.
http_requests_per_second_limit	Default maximum HTTP requests per second per upstream origin. Can be overridden per-dataset with requests_per_second_limit.
http_requests_per_minute_limit	Default maximum HTTP requests per minute per upstream origin. Can be overridden per-dataset with requests_per_minute_limit.
http_rate_control_jitter_min	Default minimum random delay before HTTP requests when rate control is active. Defaults to 5ms when a rate limit is configured. Can be overridden per-dataset.
http_rate_control_jitter_max	Default maximum random delay before HTTP requests when rate control is active. Defaults to 10ms when a rate limit is configured. Can be overridden per-dataset.

Spatial SQL Functions (opt-in)

PostGIS-style spatial ST_* SQL functions (via geodatafusion (opens in a new tab)) can be optionally registered with the SQL engine.

Parameter Name	Description
geo	Set to enabled to register ST_* spatial functions. Requires a spiced binary built with the geo Cargo feature (cargo build -p spiced --features geo). Unset by default.

Both gates must be satisfied: the binary must be built with --features geo and runtime.params.geo: enabled must be set in the Spicepod. Standard distributions of spiced do not include the geo feature, so spatial functions remain unregistered unless you produce a custom build.

Spice Cayenne (engine-global)

Engine-global tuning for the Spice Cayenne (opens in a new tab) data accelerator. These apply to every Cayenne-accelerated dataset in the instance and are not valid under a dataset's acceleration.params (per-dataset Cayenne parameters are documented on the Cayenne accelerator page (opens in a new tab)).

Parameter Name	Description
cayenne_footer_cache_mb	Size of the engine-wide in-memory Vortex footer cache in megabytes, shared across all Cayenne datasets. Optional; when unset, DataFusion's default file-metadata-cache limit of 50 MB applies (there is no fixed 128 MB default).
cayenne_segment_cache_mb	Total size in megabytes of the in-memory Vortex segment cache, which holds decompressed data segments. One cache serves every Cayenne table in the process, so adding a table divides this budget rather than reserving another cache. Takes a whole number of megabytes; 0 disables segment caching. When unset, derived as ~1/64 of the process's memory entitlement, clamped to 256 MB–2 GB.
cayenne_filter_propagation	Enables Cayenne's filter-propagation optimizer rules. Accepts enabled or disabled; defaults to disabled.
cayenne_optimizer_rules	Selects which Cayenne optimizer rules run. Accepts auto (default), all, none / disabled, or a comma-separated list of rule names.
cayenne_compaction_memory_fraction

runtime.source_rate_control

Optional. Configures how Spice limits outbound requests to upstream data sources, and optionally enables cluster-wide coordination through persisted state in object storage.

Without state_location, rate limits are local to each Spice instance. When state_location is set, Spice instances coordinate through object storage so that a configured limit is shared across the cluster. For example, requests_per_second_limit: 20 means approximately 20 RPS total across all replicas, not 20 RPS per replica.

Parameter Name	Optional	Default	Description
state_location	Yes	-	Root URI for globally persisted rate-control state (e.g. s3://bucket/path/). Enables cluster-wide rate control when set. Without this, limits are local to each Spice instance.
params	Yes	-	Object-store authentication parameters for state_location. Supports the same keys as other object-store configurations (e.g. s3_region, s3_key, s3_secret for S3; account, access_key for Azure). Supports ${ secrets:NAME } references.
refresh_interval	Yes	30s	How often each instance refreshes and persists per-source rate-control state. Longer intervals reduce object-store writes but adapt more slowly to demand changes.
github_concurrent_connections_limit	Yes	4

HTTP/API rate limits are configured through runtime.params (opens in a new tab) (cluster defaults) and per-dataset overrides. Precedence is:

When state_location is set, the configured RPS/RPM quota is converted into a token budget per lease window and distributed across replicas using a demand-weighted leased token-bucket model.

runtime.functions

Controls whether functions (opens in a new tab) declared in the top-level functions: section (and tools: entries with as_sql: true) are registered with the SQL engine. Defaults to disabled.

Parameter	Optional	Default	Description
enabled	Yes	false	When true, the runtime registers functions: entries and exposes them via SQL and /v1/functions.

When disabled, the functions: block is parsed but not registered, list_udfs() returns no user-source rows, and GET /v1/functions returns an empty array.

See the Functions Spicepod reference (opens in a new tab) for the function declaration schema.

runtime.shutdown_timeout

Controls how long Spice waits for connections to be gracefully drained and for components to shut down cleanly during runtime termination. Defaults to 30 seconds.

runtime.tls

The TLS section specifies the configuration for enabling Transport Layer Security (TLS) for all endpoints exposed by the runtime. Learn more about enabling TLS (opens in a new tab).

In addition to configuring TLS via the manifest, TLS can also be configured via spiced command line arguments using the --tls-enabled true flag along with --tls-certificate/--tls-certificate-file and --tls-key/--tls-key-file.

Certificate Hot-Reload

Spice can hot-reload TLS certificates and client CA files for runtime endpoints. Update the certificate, key, or CA file on disk, then send SIGHUP to the Spice process to reload without restart. Only file-based certificates/keys/CA are hot-reloaded (not inline PEM). Existing connections are not interrupted; only new connections use the updated files. If reload fails, the previous certificate remains active and a warning is logged.

Steps:

1. Replace the certificate/key/CA file on disk.
2. Send SIGHUP to the Spice process (e.g., kill -SIGHUP <pid>).
3. Check logs for reload confirmation or errors.

runtime.tls.enabled

Enables or disables TLS for the runtime endpoints.

runtime.tls.certificate

The TLS certificate to use for securing the runtime endpoints. The certificate can also come from secrets (opens in a new tab).

runtime.tls.certificate_file

The path to the TLS PEM-encoded certificate file. Only one of certificate or certificate_file must be used.

runtime.tls.key

The TLS key to use for securing the runtime endpoints. The key can also come from secrets (opens in a new tab).

runtime.tls.key_file

The path to the TLS PEM-encoded key file. Only one of key or key_file must be used.

runtime.tls.client_auth_mode

:::info Enterprise Feature mTLS (client certificate authentication) is included in the Enterprise distribution of Spice.ai. Learn more (opens in a new tab). :::

Controls whether the runtime requires, requests, or ignores client certificates on its public endpoints (HTTP, Flight, Metrics). Defaults to none.

Mode	Behavior
none (default)	Standard one-way TLS. No client certificate is requested.
request	The server sends a CertificateRequest but accepts connections without a certificate. Presented certificates are verified against the configured CA. Useful for migration or audit-only deployments.
required	A valid client certificate is required. The Flight (gRPC) listener rejects connections without a certificate at the TLS handshake. The HTTP listener admits no-cert connections so /health and /v1/ready remain accessible for Kubernetes probes, but all other HTTP endpoints return 401 without a verified client certificate. The metrics listener has no client-auth gate.

Requires client_auth_ca_file or client_auth_ca to be set when mode is request or required.

runtime.tls.client_auth_ca_file

Path to a PEM-encoded CA bundle used to verify client certificates. The file is watched for changes and reloaded atomically alongside the server certificate and key.

runtime.tls.client_auth_ca

Inline PEM (or ${ secrets:... }) form of the client CA bundle. Mutually exclusive with client_auth_ca_file. Inline material is loaded once at startup and is not hot-reloaded.

runtime.task_history

The task history section specifies runtime task history configuration. For more details, see the Task History documentation (opens in a new tab).

Parameter name	Optional	Description
enabled	Yes	Defaults to true.
captured_output	Yes	Specifies the level of output captured by the task history table. Defaults to none.
captured_context	Yes	Controls how much of the input and captured_output payload is stored for AI and search tasks. Options: truncated (default), redacted, or full. Values are case-sensitive.
captured_plan	Yes	Controls SQL query plan capture. Options: none (default), explain, or explain analyze. Query plans are captured asynchronously after query completion.
min_sql_duration

runtime.cors

The CORS section specifies the configuration for enabling Cross-Origin Resource Sharing (CORS) for the HTTP endpoint. By default, CORS is disabled.

Default configuration:

runtime.cors.enabled

Enables or disables CORS for the HTTP endpoint. Defaults to false.

runtime.cors.allowed_origins

A list of allowed origins for CORS requests. Defaults to ["*"], which permits all origins.

Example:

This configuration permits requests only from the https://example.com origin.

runtime.cpu

The CPU section states how many CPUs the runtime should behave as though it has. That single entitlement sizes every CPU-derived pool coherently — the tokio runtimes' worker threads, DataFusion's query fan-out (runtime.query.target_partitions) and query admission bound (runtime.query.max_concurrent_queries), the Cayenne encode, compaction, upload and file-scan concurrency defaults, the Cayenne SQLite metastore pool, the embedding inference pool, DuckDB's per-instance threads, and a cluster executor's concurrent-task advertisement.

runtime.cpu.cores

Accepts a Kubernetes CPU quantity — a whole number of cores (4), a fraction (3.5), or millicores (3500m) — or one of two named values:

Value	Meaning
auto	The default. Detect the entitlement. On a pod that declares a CPU request and no CPU limit, that is twice the request — min(max(2 cores, request × 2), available CPUs). The multiple is deliberate: it exceeds the request so the pod can still burst above its scheduling floor.
all	Every CPU this process may use, regardless of any CPU request. A CPU limit, if set, still applies.
A quantity	Use exactly this, whatever the pod declares.

Millicores must be integral, so 3500m is valid and 3.5m is not. A value of 0, a negative value, or an unparseable one fails startup with an actionable error rather than being clamped silently — 0 is not a spelling of all, so a typo cannot resolve to full-machine sizing.

Three configuration surfaces set the same value. Precedence, highest first:

Surface	Form
Command-line flag	--cpu-cores 4
Environment variable	SPICE_CPU_CORES=4
Spicepod	runtime.cpu.cores: 4

A surface set to auto still takes precedence over the surfaces below it; it simply resolves to detection.

all is the exception: it states that a surface imposes no ceiling of its own, so it defers to a quantity named on a lower-precedence surface. A platform that sets SPICE_CPU_CORES=all on every deployment therefore does not silence an operator who wrote runtime.cpu.cores: 4 in their spicepod. It does not defer to auto, which is an instruction ("detect it") rather than the absence of one.

Applied at startup only. The thread pools it sizes cannot be resized afterwards, so changing runtime.cpu and reloading the spicepod logs a warning that a restart is required rather than taking effect. It is one of several start-time-only sections — see Reload Behavior (opens in a new tab).

Detection

With nothing configured (auto), the entitlement is detected. First match wins:

1. A cgroup CPU quota — cgroup v2 cpu.max, cgroup v1 cpu.cfs_quota_us; Kubernetes resources.limits.cpu. Read along the whole cgroup path, taking the smallest quota found at any level, and capped by the CPU affinity mask.
2. The pod's declared CPU request, as min(max(2 cores, request × 2), available CPUs) — see Sizing from a CPU request (opens in a new tab).
3. The process's CPU affinity mask (sched_getaffinity on Linux, the logical CPU count elsewhere) — the CPUs the process may run on, which a cpuset or taskset may narrow below the host's core count.
4. One core, when nothing can be determined.

A CPU limit outranks a request: bursting past a quota does not produce CPU, it produces CFS throttling. runtime.cpu.cores: all suppresses rung 2 only, so it resolves exactly as it would on a pod that declared no request — a CPU limit if one is set, otherwise every available CPU.

A process that declares no CPU request skips rung 2 entirely and is sized for every CPU it can see. That covers every bare-metal deployment, docker run without CPU flags, and every benchmark.

Sizing from a CPU request

A pod that sets resources.requests.cpu without resources.limits.cpu has no cgroup quota. Sizing for the whole node would build thread pools and query fan-out for a machine the pod does not own, so the entitlement is derived from the request instead — as a bounded multiple of it, currently 2×.

The multiple is what makes this safe to automate. A request is a scheduling floor rather than a ceiling, so sizing at the request would remove the bursting that is the reason the limit was omitted. The result floors at 2 cores, so a requests.cpu: 100m pod still has enough parallelism to overlap a scan with something, and yields to a genuinely smaller host.

The request must be declared — a cgroup CPU share is never an input. Every cgroup carries a share whether or not a request was expressed (a plain docker run reports cpu.weight: 100), and the conversion back to a request varies by container runtime, so a share is read for reporting only and never interpreted as a number.

Declaring it is the deployment surface's job, through SPICE_CPU_REQUEST_MILLICORES:

The Spice Helm chart (opens in a new tab) and the Spice Kubernetes Operator both emit this automatically whenever the pod sets a CPU request, so neither needs configuring. A hand-written pod spec must include it, or the pod falls through to rung 3 and sizes for the machine — the runtime warns at startup when it detects that case.

Two details in that block are load-bearing. The divisor: 1m is what makes the value millicores, which is what the variable's name states; without it a requests.cpu of 4 arrives as 4 and reads as four millicores. And the block must be emitted only when a CPU request is actually set: with no request declared, resourceFieldRef reports the node's allocatable CPU, which is exactly the over-sizing this exists to prevent.

See Resource Allocation (opens in a new tab) for the Kubernetes guidance.

Observability

At startup the runtime logs the effective entitlement, the rung of the ladder or the setting it came from, the readings it sits between, and the defaults derived from it:

CPU budget: 8 cores (source: the declared CPU request (x2); host reports 64, declared CPU request 4 cores, cgroup share unset, cgroup limit unset) → 8 main worker threads, 7 per dedicated runtime pool, 8 target partitions
CPU budget derived sizing: main_runtime_worker_threads=8, dedicated_runtime_worker_threads=7, target_partitions=8, max_concurrent_queries=32, ...

These two lines are the primary diagnostic: between them they name what the runtime sized for, which rung produced it, and every quantity derived from it. A pod sized differently than expected is answered here rather than by inference.

The derived line reports defaults, not necessarily the values in force: several are overridable by their own setting (runtime.query.target_partitions, runtime.query.max_concurrent_queries, DuckDB's threads, a model's parallelism), and the line is logged before that configuration is resolved. Each overridable consumer separately logs the value it used and where that value came from.

The following warnings identify CPU configuration issues and remedies:

Warning	Fires when
CPU request present but not passed through	Running under Kubernetes with a cgroup share but no SPICE_CPU_REQUEST_MILLICORES — the deployment surface is not emitting the block above, so sizing fell through to the machine.
Declared request implausibly small	A declared request below 10 millicores, which is what a resourceFieldRef missing its divisor: 1m produces for a request of one to nine cores.
Configured cores above the real ceiling	The explicit CPU entitlement exceeds the cgroup or affinity limit. The configured value remains unchanged; the warning identifies the limit so the entitlement or limit can be adjusted.
CPU share changed after startup	The cgroup share moved from its value at startup — the pod was resized in place. The entitlement cannot change without a restart, so this reports the drift rather than acting on it.

The spiced_cpu_budget_cores, spiced_cpu_budget_millicores, spiced_cpu_limit_millicores, and spiced_cpu_request_millicores gauges report the same figures — see Observability (opens in a new tab). The source label on spiced_cpu_budget_cores is the authority on which rung won, which is what makes a fleet greppable for pods that resolved somewhere unexpected. tokio_runtime_workers is the cross-check on the thread pools the entitlement sized.

runtime.query.memory_limit

The memory_limit parameter sets a memory usage cap for the Spice runtime query engine. This limit applies only to the query engine and should be used in addition to other memory configuration options, such as duckdb_memory_limit. When the limit is reached, DataFusion spills intermediate data to disk using the directory configured in runtime.query.temp_directory.

If not specified, defaults to 90% of the memory the process may use — its cgroup memory limit when one binds, otherwise total system memory. The limit is read from the process's own cgroup path (cgroup v2 memory.max, cgroup v1 memory.limit_in_bytes), taking the smallest limit found at any level of that path, so a container limit, a systemd unit's MemoryMax=, a capped parent slice, and a Kubernetes pod cgroup all size the default. When a cgroup limit binds, the runtime logs the figure it sized from at startup. When Cayenne acceleration is active, the default is reduced to 70% to reserve headroom for Cayenne's dedicated compaction memory pool and its in-memory CDC tier. When DuckDB accelerators are configured, that default is reduced further so the query pool and the DuckDB instance ceilings together fit within available memory — see Coordinated memory budget (opens in a new tab). An explicitly configured memory_limit is always honored verbatim and is never reduced.

Specify the value as a size, for example 4GiB or 1024MiB.

For detailed memory information, see Memory (opens in a new tab).

runtime.query.max_concurrent_queries

The max_concurrent_queries parameter bounds how many query-executing plans may run concurrently. Excess queries wait (admission control) rather than oversubscribing the shared query runtime and memory pool, which can otherwise cause queries to starve each other under load — for example, analytical queries running alongside CDC ingestion and compaction.

Behavior:

• Applies to ordinary queries, DDL/DML, and EXECUTE. Lightweight session-state statements (PREPARE, DEALLOCATE, SET) are not gated.
• A permit is held for the plan's full execution and result-streaming lifetime. A results-cache hit is never gated.
• If not set, the bound is sized from the CPU entitlement at four concurrent plans per core — 16 on a 4-core budget. Above one per core so a query blocked on I/O does not idle its core, and low enough that the query memory pool is shared between a countable number of plans. See runtime.cpu (opens in a new tab).
• max_concurrent_queries: 0 opts out and leaves concurrency unbounded. Every other configured value is a limit, applied verbatim.

:::warning Behavior change

Prior to this release the default was unbounded, and an explicit 0 was clamped to a minimum of 1 (a single concurrent query). Both have changed: an unset value is now bounded by the CPU entitlement, and 0 now means unbounded rather than maximally throttled. A deployment that set 0 to disable admission control gets the behavior it intended; one that set 0 expecting a throttle gets the opposite.

:::

runtime.query.timeout

The timeout parameter sets a maximum wall-clock duration a query may run before it is automatically cancelled, expressed as a human-readable duration (for example 30s or 5m). The clock covers the query's full lifetime: planning, admission-control waits (max_concurrent_queries), execution, and streaming results to the client.

Behavior:

• Applies to queries issued through the runtime's query APIs (HTTP, Flight, and Flight SQL). Internal runtime queries — acceleration refreshes and health checks — are exempt.
• Enforcement is cooperative (best-effort): the query is cancelled at its next cancellation checkpoint, so actual runtime can slightly exceed the configured value.
• On expiry, the query fails with a timeout error. If the timeout is observed before the response starts, the client receives an HTTP 504 / gRPC DEADLINE_EXCEEDED. If results are already streaming, the status can no longer change, so the in-progress stream is terminated with the error — data streamed before expiry will have been delivered, but the stream never ends silently as if complete.
• If not set, queries run with no timeout (the default behavior). The value must be a positive duration greater than 0.

runtime.query.spill_compression

The spill_compression parameter configures compression for spill files generated during large query execution in the Spice runtime.

Supported values:

• zstd (default): Enables high compression ratios for spill files, reducing disk usage but with moderate (de)compression speed.
• lz4_frame: Provides faster (de)compression, resulting in larger spill files and potentially higher disk usage.
• uncompressed: Disables compression. Spill files will be the largest, but with no (de)compression overhead.

This setting controls the trade-off between disk space usage and query performance for large-scale analytics workloads.

runtime.query.temp_directory

The path to a temporary directory that Spice uses for query and acceleration operations that spill to disk. It is used by DataFusion query execution (spill for sorts, aggregations, and sort-merge joins that exceed runtime.query.memory_limit), by Spice Cayenne's dedicated compaction runtime, by every DuckDB accelerator instance (passed through as DuckDB's own temp_directory), and by cluster-mode executors as their local working directory. When unset, DataFusion spills to the operating system's temporary directory ($TMPDIR, otherwise /tmp) and DuckDB to a .tmp directory beside its database file; when Cayenne acceleration is active and the setting is unset, the runtime logs a reminder at startup.

Set it to a directory on local NVMe or SSD with ample free space — not the root volume, a network file system, or a RAM-backed mount, whose files count against the process's memory. Each spilled batch is a synchronous write the query waits on, so the directory's per-I/O latency lands directly on query time. DataFusion caps total spill at 100 GB per runtime environment; the cap is not configurable — there is no runtime.query setting for it, and SET datafusion.runtime.max_temp_directory_size is rejected because the query APIs do not accept SET statements. For more details, see Spill-to-Disk and the Temporary Directory (opens in a new tab) and Storage (opens in a new tab) in the Performance Tuning guide, the Managing Memory Usage documentation (opens in a new tab), and the DuckDB Data Accelerator documentation (opens in a new tab).

runtime.output_level

Controls verbosity in addition to the existing CLI and environment variable support. (opens in a new tab). Supported values are info, verbose, and very_verbose. The value is applied in the following priority: CLI, environment variables, then YAML configuration.

runtime.telemetry

The telemetry section configures runtime telemetry collection and export. Learn more (opens in a new tab).

runtime.telemetry.enabled

Enables or disables runtime telemetry collection. Defaults to true.

runtime.telemetry.metric_prefix {#runtimetelemetrymetric_prefix}

Optional string prepended to every exported metric name. Useful for namespacing Spice metrics in shared backends (e.g. Datadog, Grafana Cloud, New Relic) so they do not collide with metrics from other services. Defaults to no prefix.

The prefix applies to all metric readers — the Prometheus scrape endpoint (--metrics), the cluster on-demand OTLP reader, and the otel_exporter push exporter — because OpenTelemetry views are configured at the meter-provider level rather than per reader.

With this configuration, the runtime metric query_duration_ms is exported as spiceai.query_duration_ms.

The prefix is validated at spicepod load against the OpenTelemetry instrument name syntax (opens in a new tab) so {prefix}{instrument} stays a valid OTLP/Prometheus metric name. A non-empty prefix must:

• start with an ASCII letter (A–Z or a–z);
• contain only ASCII letters, digits, _, ., -, or /; and
• be at most 128 characters (leaving ≥127 characters of headroom under the OpenTelemetry 255-character instrument-name limit for the base metric name).

An invalid prefix fails fast with an actionable error rather than producing malformed metric names. An empty or unset metric_prefix applies no prefix.

runtime.telemetry.properties {#runtimetelemetryproperties}

Map of custom key/value attributes attached to telemetry metrics emitted by spiced. Applied as OpenTelemetry resource attributes on the runtime's MeterProvider, so they appear as dimensions/tags on every metric exported via the Prometheus scrape endpoint, the cluster on-demand OTLP reader, and the otel_exporter push exporter. Defaults to empty.

The standard OpenTelemetry environment variables (OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES) are still honored and act as defaults; explicit properties entries take precedence on key conflicts.

For backends that map OTLP resource attributes to tags through additional configuration (e.g. Datadog), see the Datadog OTLP guide (opens in a new tab).

runtime.telemetry.otel_exporter

Configures an OpenTelemetry (opens in a new tab) metrics exporter to push metrics to an OpenTelemetry collector. The exporter automatically infers the protocol (gRPC or HTTP) based on the endpoint configuration.

Parameter name	Optional	Default	Description
enabled	Yes	true	Whether the OpenTelemetry exporter is enabled.
endpoint	No	-	The OpenTelemetry collector endpoint. Protocol is inferred from the format (see examples below).
push_interval	Yes	60s	How frequently metrics are pushed to the collector. Specify as a duration (opens in a new tab).
metrics	Yes	[]	List of metric names to export. When empty (default), all metrics are exported.
headers	Yes	{}	Map of headers to send with each export request. For HTTP these are sent as HTTP headers; for gRPC they are sent as metadata entries (keys must be lowercase ASCII). Values support the ${secrets:...} replacement syntax (opens in a new tab) for loading credentials from a secret store.

Protocol inference:

• gRPC (default): Use a bare host:port endpoint without a scheme (e.g., localhost:4317). gRPC uses port 4317 by default.
• HTTP: Include the http:// or https:// scheme and the /v1/metrics path (e.g., http://localhost:4318/v1/metrics). HTTP uses port 4318 by default.

Examples:

gRPC configuration:

HTTP configuration:

With metric filtering (export only specific metrics):

:::caution Filtering happens after metric_prefix is applied The whitelist is matched against the final metric name, after runtime.telemetry.metric_prefix (opens in a new tab) has been prepended. If you set metric_prefix: 'spiceai.', the entries under metrics: must include the prefix (e.g. spiceai.query_duration_ms), otherwise nothing will match and no metrics will be exported. :::

Authenticated exporters:

For collectors that require authentication, set the headers map. Load credentials from a secret store (opens in a new tab) via ${secrets:...} rather than committing them to source.

Datadog (OTLP/HTTP) — replace us3 with your Datadog site:

Grafana Cloud (OTLP/HTTP) — use the base64 instanceID:accessPolicyToken from the Grafana Cloud OpenTelemetry connection page:

gRPC collector with auth metadata (keys must be lowercase ASCII):

runtime.metrics

Specifies metrics that are disabled by default.

Following metrics are disabled by default:

• dataset_acceleration_max_timestamp_before_refresh_ms
• dataset_acceleration_max_timestamp_after_refresh_ms
• dataset_acceleration_refresh_lag_ms
• dataset_acceleration_ingestion_lag_ms

For details about these metrics, see Observability (opens in a new tab).

runtime.flight

Configures Arrow Flight protocol settings for the runtime.

Parameter name	Optional	Default	Description
max_message_size	Yes	-	Maximum size of a single Arrow Flight message.
do_put_rate_limit_enabled	Yes	true	Whether rate limiting is applied to DoPut Arrow Flight operations.

runtime.mcp

Configures settings for the Spice MCP server endpoint (/v1/mcp).

runtime.mcp.allowed_hosts

Controls which Host header values are accepted on the /v1/mcp endpoint. This prevents DNS rebinding (opens in a new tab) attacks against the MCP server.

Behavior	Configuration
Default (not set)	Only localhost, 127.0.0.1, and ::1 are permitted. Requests with any other Host value receive 403 Forbidden.
Explicit list	Replaces the defaults entirely. Only the listed hosts are accepted.
Wildcard (["*"])	Disables host checking — all Host header values are accepted.

To disable host checking entirely:

Each entry can be a bare hostname (example.com), a host-port pair (example.com:8090), or a full origin URL (https://example.com).

runtime.ready_state

Controls when the runtime readiness probe (/v1/ready) reports the runtime as ready. This is particularly useful for Kubernetes readiness probes.

Value	Description
on_load (default)	The runtime reports ready after all components (datasets, models, etc.) have loaded successfully.
on_registration	The runtime reports ready as soon as all components have been registered, before they finish loading.

runtime.scheduler

Configures the cluster scheduler when running Spice in cluster mode (opens in a new tab). This section is relevant only when using --role scheduler.

Parameter name	Optional	Default	Description
state_location	No	-	Root URI for shared cluster state storage (e.g. s3://bucket/path/).
params	Yes	-	Object store parameters (e.g. s3_region).
partition_assignment_interval	Yes	30s	How often the scheduler runs partition assignment cycles.
max_partition_assignments_per_interval	Yes	100	Maximum number of partition assignments per interval.
max_partitions_per_executor	Yes	1000	Maximum number of partitions assigned to a single executor.
partition_discovery_timeout	Yes	60s	How long the scheduler waits for executor discovery before timing out.