title: 'Datasets' sidebar_label: 'Datasets' description: 'Datasets YAML reference' tags:

• reference
• datasets
• spicepod

A Spicepod can contain one or more datasets referenced by relative path or defined inline.

Inline example:

spicepod.yaml

datasets:
  - from: spice.ai/spiceai/quickstart/datasets/taxi_trips
    name: taxi_trips
    acceleration:
      enabled: true
      mode: memory # / file
      engine: arrow # / cayenne / duckdb / sqlite / postgres / turso
      refresh_check_interval: 1h
      refresh_mode: full / append # / changes / caching / snapshot

datasets:
  - from: spice.ai/spiceai/quickstart/datasets/taxi_trips
    name: taxi_trips
    acceleration:
      enabled: true
      mode: memory # / file
      engine: arrow # / cayenne / duckdb / sqlite / postgres / turso
      refresh_check_interval: 1h
      refresh_mode: full / append # / changes / caching / snapshot

spicepod.yaml

datasets:
  - from: databricks:spiceai.datasets.specific_table
    name: uniswap_eth_usd
    params:
      environment: prod
    acceleration:
      enabled: true
      mode: memory # / file
      engine: arrow # / cayenne / duckdb / sqlite / postgres / turso
      refresh_check_interval: 1h
      refresh_mode: full / append # / changes / caching / snapshot

datasets:
  - from: databricks:spiceai.datasets.specific_table
    name: uniswap_eth_usd
    params:
      environment: prod
    acceleration:
      enabled: true
      mode: memory # / file
      engine: arrow # / cayenne / duckdb / sqlite / postgres / turso
      refresh_check_interval: 1h
      refresh_mode: full / append # / changes / caching / snapshot

Relative path example:

spicepod.yaml

datasets:
  - ref: datasets/taxi_trips

datasets:
  - ref: datasets/taxi_trips

datasets/taxi_trips/dataset.yaml

from: spice.ai/spiceai/quickstart/datasets/taxi_trips
name: taxi_trips
acceleration:
  enabled: true
  refresh_check_interval: 1h

from: spice.ai/spiceai/quickstart/datasets/taxi_trips
name: taxi_trips
acceleration:
  enabled: true
  refresh_check_interval: 1h

from

The from field is a string that represents the Uniform Resource Identifier (URI) for the dataset. This URI is composed of two parts: a prefix indicating the Data Connector to use to connect to the dataset, a delimiter, and the path to the dataset within the source.

The syntax for the from field is as follows:

from: <data_connector>:<path>
# OR
from: <data_connector>/<path>
# OR
from: <data_connector>://<path>

from: <data_connector>:<path>
# OR
from: <data_connector>/<path>
# OR
from: <data_connector>://<path>

Where:

• <data_connector>: The Data Connector to use to connect to the dataset

  Currently supported data connectors:

  • spiceai (opens in a new tab)
  • dremio (opens in a new tab)
  • spark (opens in a new tab)
  • databricks (opens in a new tab)
  • s3 (opens in a new tab)
  • postgres (opens in a new tab)
  • mysql (opens in a new tab)
  • flightsql (opens in a new tab)
  • snowflake (opens in a new tab)
  • ftp, sftp (opens in a new tab)
  • http, https (opens in a new tab)
  • clickhouse (opens in a new tab)
  • graphql (opens in a new tab)
  • cosmosdb (opens in a new tab)

  If the Data Connector is not explicitly specified, it defaults to spiceai.

• <delimiter>: The delimiter between the Data Connector and the path. Currently supported delimiters are :, /, and ://. Some connectors place additional restrictions on the allowed delimiters to better conform to the expected syntax of the underlying data source, i.e. s3:// is the only supported delimiter for the s3 connector.

• <path>: The path to the dataset within the source.

:::info[Identifier Case Sensitivity] Unquoted identifiers in the <path> are normalized to lowercase. To reference a table or schema with mixed-case or uppercase characters, wrap each case-sensitive part in double quotes:

datasets:
  # Case is preserved for "ActionExecutions"
  - from: postgres:my_schema."ActionExecutions"
    name: action_executions

  # Quote each part individually as needed
  - from: databricks:my_catalog."MySchema"."MyTable"
    name: my_table

datasets:
  # Case is preserved for "ActionExecutions"
  - from: postgres:my_schema."ActionExecutions"
    name: action_executions

  # Quote each part individually as needed
  - from: databricks:my_catalog."MySchema"."MyTable"
    name: my_table

This applies to all federated database connectors where <path> references a table identifier. Connectors that interpret <path> as a file path (e.g. s3, delta_lake, ftp) do not apply identifier normalization. See Identifier Case Sensitivity and Quoting (opens in a new tab) for details. :::

ref

An alternative to adding the dataset definition inline in the spicepod.yaml file. ref can be use to point to a directory with a dataset defined in a dataset.yaml file. For example, a dataset configured in a dataset.yaml in the "datasets/sample" directory can be referenced with the following:

dataset.yaml

from: spice.ai/spiceai/quickstart/datasets/taxi_trips
name: taxi_trips
acceleration:
  enabled: true
  refresh_check_interval: 1h

from: spice.ai/spiceai/quickstart/datasets/taxi_trips
name: taxi_trips
acceleration:
  enabled: true
  refresh_check_interval: 1h

ref used in spicepod.yaml

version: v1
kind: Spicepod
name: duckdb
datasets:
  - ref: datasets/sample

version: v1
kind: Spicepod
name: duckdb
datasets:
  - ref: datasets/sample

name

The name of the dataset. Used to reference the dataset in the pod manifest, as well as in external data sources. The name cannot be a reserved keyword (opens in a new tab).

Spice follows PostgreSQL SQL syntax conventions, which normalize unquoted identifiers to lowercase. A dataset named LINEITEM is accessible in queries as lineitem.

To preserve uppercase or mixed-case names, wrap the name in double quotes. In YAML, this requires an extra layer of quoting:

datasets:
  - from: snowflake:SNOWFLAKE_SAMPLE_DATA.TPCH_SF100.LINEITEM
    name: '"LINEITEM"'
    params:
      snowflake_account: JYFGIWYEFBW
      snowflake_warehouse: snowflake_wh
      snowflake_password: ${secrets:SNOWFLAKE_PASSWORD}
      snowflake_username: ${secrets:SNOWFLAKE_USERNAME}

datasets:
  - from: snowflake:SNOWFLAKE_SAMPLE_DATA.TPCH_SF100.LINEITEM
    name: '"LINEITEM"'
    params:
      snowflake_account: JYFGIWYEFBW
      snowflake_warehouse: snowflake_wh
      snowflake_password: ${secrets:SNOWFLAKE_PASSWORD}
      snowflake_username: ${secrets:SNOWFLAKE_USERNAME}

-- Query using the preserved uppercase name
SELECT * FROM "LINEITEM";

-- Query using the preserved uppercase name
SELECT * FROM "LINEITEM";

Without the double quotes, the same dataset would be queryable only as lineitem.

See Identifier Case Sensitivity and Quoting (opens in a new tab) for full details on quoting in both the from and name fields.

description

The description of the dataset. Used as part of the Semantic Data Model (opens in a new tab).

access

Optional. Specifies the access level for the dataset. Supported values are:

• read (default): Read-only access.
• read_write: Enables both read and write operations. Only supported for write-capable connectors (opens in a new tab).

To enable write operations, configure your dataset with read_write access:

datasets:
  - from: glue:my_catalog.my_schema.my_table
    name: my_table
    access: read_write
    params:
      # ... connector-specific parameters

datasets:
  - from: glue:my_catalog.my_schema.my_table
    name: my_table
    access: read_write
    params:
      # ... connector-specific parameters

time_column

Optional. The name of the column that represents the temporal (time) ordering of the dataset.

Required to enable a retention policy on the dataset.

time_format

Optional. The format of the time_column. The following values are supported:

• timestamp - Default. Timestamp without a timezone. E.g. 2016-06-22 19:10:25 with data type timestamp.
• timestamptz - Timestamp with a timezone. E.g. 2016-06-22 19:10:25-07 with data type timestamptz.
• unix_seconds - Unix timestamp in seconds. E.g. 1718756687.
• unix_millis - Unix timestamp in milliseconds. E.g. 1718756687000.
• unix_nanos - Unix timestamp in nanoseconds. E.g. 1718756687000000000. Use this for OpenTelemetry's time_unix_nano columns — see OpenTelemetry Data Ingestion (opens in a new tab).
• ISO8601 - ISO 8601 (opens in a new tab) format.
• date - Date in YYYY-MM-DD format. E.g. 2024-01-01.

For an accelerated dataset, Spice checks time_format against the data type of the time_column when the dataset loads. When time_format is omitted, timestamp is assumed. A time_format that does not match the column's data type fails the dataset with an error that names the column's data type and the configured time_format.

:::warning[Limitations]

• String columns are parsed as ISO 8601. An accelerated dataset with a string time_column requires time_format: iso8601, because the default timestamp format does not match a string column.

:::

time_partition_column

(Optional) Specify the column that represents the physical partitioning of the dataset when using append-based acceleration. When the defined time_column is a fine-grained timestamp and the dataset is physically partitioned by a coarser granularity (for example, by date), setting time_partition_column to the partition column (e.g. date_col) improves partition pruning, excludes irrelevant partitions during refreshes, and optimizes scan efficiency.

time_partition_format

(Optional) Define the format of the time_partition_column. For instance, if the physical partitions follow a date format (YYYY-MM-DD), set this value to date. The same format options as time_format are supported for time_partition_column.

Schema Inference and Evolution

Spice infers the dataset schema from the data source at startup. The inferred schema defines the column names, data types, and nullability used for the lifetime of that runtime process. By default, schema changes at the source are not applied at runtime — data refreshes will fail if the source schema drifts, and you must restart the runtime to re-infer the schema.

Accelerated datasets can opt into automatic, in-place schema evolution with the on_schema_change (opens in a new tab) policy, which adopts lossless, widening-compatible source changes without a restart — and can additionally drop and recreate the accelerated table on incompatible changes (drop_and_recreate) when refresh_mode: full is set.

For connector-specific inference parameters, runtime schema change behavior, and recommendations, see Schema Inference (opens in a new tab).

on_schema_change

Optional. Controls how the runtime reacts when the source schema changes after the dataset is registered. Applies to accelerated datasets only — federated (non-accelerated) queries always reflect the live source schema, so the policy is inert for them (a non-default value logs a warning and otherwise has no effect).

The following values are supported:

• block - Default. Schema changes are not applied automatically. The dataset stays healthy and continues serving queries using the registered schema; this preserves the historical behavior.
• fail - Set the dataset to an error status with an actionable message when the projected source schema diverges from the registered schema. Self-heals if the source reverts.
• append_new_columns - Adopt new nullable source columns in place; type changes and relaxed/tightened nullability are treated as block (the dataset keeps serving on the old schema) and a warning is logged.
• sync_all_columns - Adopt the full set of lossless, widening changes in place: new nullable columns, widened column types (for example Int32→Int64, an increase in decimal precision, or Utf8→LargeUtf8), and relaxed nullability. Non-widening changes (column removals, narrowing type changes) remain block-equivalent and are warned.
• drop_and_recreate - Everything sync_all_columns does (adopt lossless widening changes in place), and additionally recreate the accelerated table for changes that cannot be applied in place — column removals, narrowing or otherwise incompatible type changes, and new non-nullable columns. Recreation is destructive: the accelerated data is dropped and rebuilt from the source, so it is only performed with refresh_mode: full (a full refresh re-fetches every row). With refresh_mode: append or changes, an incompatible change is rejected and the existing table is preserved (recreating would drop rows that cannot be re-fetched). Supported on the duckdb, sqlite, turso, and cayenne engines.

datasets:
  - from: postgres:public.events
    name: events
    on_schema_change: drop_and_recreate
    acceleration:
      engine: duckdb
      mode: file
      refresh_mode: full

datasets:
  - from: postgres:public.events
    name: events
    on_schema_change: drop_and_recreate
    acceleration:
      engine: duckdb
      mode: file
      refresh_mode: full

:::note

In-place evolution (no restart) is supported for the duckdb, sqlite, turso, and Spice Cayenne (cayenne) acceleration engines, including for PostgreSQL CDC (refresh_mode: changes). Other engines (for example arrow and the PostgreSQL accelerator) log a clear unsupported message and degrade safely, applying additive changes on restart. Constraint and primary-key columns cannot be widened in place. For destructive schema changes (column removals or narrowing), set on_schema_change: drop_and_recreate with refresh_mode: full to drop and recreate the accelerated table from the source, or use mode: file_update (opens in a new tab), which recreates the acceleration file on any change.

:::

unsupported_type_action

Optional. Specifies the action to take when a data type that is not supported by the data connector is encountered.

The following values are supported:

• error - Default. Return an error when an unsupported data type is encountered.
• warn - Log a warning and ignore the column containing the unsupported data type.
• ignore - Log nothing and ignore the column containing the unsupported data type.
• string - Attempt to convert the unsupported data type to a string. Currently only supports converting the PostgreSQL JSONB type.

:::warning[Limitations]

Not all connectors support specifying an unsupported_type_action. When specified on a connector that does not support the option, the connector will fail to register. The following connectors support unsupported_type_action:

• DuckDB (opens in a new tab)
• PostgreSQL (opens in a new tab)

:::

schema_inference

:::warning[Removed]

The schema_inference dataset field was removed. Schema inference is now always on — Spice attempts the deepest inference each source permits and no longer requires an opt-in. A Spicepod that still specifies schema_inference fails to load with an unknown-field error; remove the field.

For connectors that expose catalog metadata — PostgreSQL (opens in a new tab), MySQL (opens in a new tab), and MongoDB (opens in a new tab) — inference additionally detects the source's primary key (and, where the source exposes them, secondary indexes and sort/clustering columns) and applies them to any acceleration settings left unset. It degrades gracefully — with info-level logs — when the connection role lacks catalog read access. Other connectors infer the base column schema (names, types, nullability) only.

For the previous opt-in standard / extended behavior, see the v2.1.x documentation (opens in a new tab).

:::

ready_state

Supports one of three values (defaults to on_load):

• on_registration: Mark the dataset as ready immediately, and queries on this table will fall back to the underlying source directly until the initial acceleration is complete. When combined with fully declared columns[].type (opens in a new tab) entries, enables deferred dataset initialization (opens in a new tab) — the source connector is not created until the first query.
• on_load: (default) Mark the dataset as ready only after the initial acceleration. Queries against the dataset will return an error before the load has been completed.
• on_schema_resolved: Mark the dataset as ready once the federated source's schema has been resolved (which also verifies access to the source), without waiting for the initial data refresh. Queries fall back to the federated source until the initial load completes; subsequent refresh failures are still reported via dataset status and metrics.

datasets:
  - from: s3://my_bucket/my_dataset/
    name: my_dataset
    ready_state: on_registration # or on_load
    params: ...
    acceleration:
      enabled: true

datasets:
  - from: s3://my_bucket/my_dataset/
    name: my_dataset
    ready_state: on_registration # or on_load
    params: ...
    acceleration:
      enabled: true

check_availability

Spice can monitor whether the source backing a non-accelerated dataset is still reachable, marking the dataset Error while it is not. Availability monitoring is opt-in: it runs only for datasets that set check_availability_interval (opens in a new tab). Note that probing may trigger the startup of compute resources (for example, Databricks or Snowflake), potentially incurring additional costs.

• auto: Monitor the dataset when it is not accelerated and check_availability_interval is set. This is the default value. Accelerated datasets are never monitored.
• disabled: Never monitor the dataset, even with check_availability_interval set.

datasets:
  - from: databricks:catalog.schema.table
    name: my_dataset
    check_availability: disabled
    params: ...

datasets:
  - from: databricks:catalog.schema.table
    name: my_dataset
    check_availability: disabled
    params: ...

check_availability_interval

Optional. How often the runtime probes the source backing this dataset to confirm it is still reachable. Accepts a duration string (for example 60s, 5m). There is no default — leave it unset and the dataset is not monitored at all.

datasets:
  - from: postgres:public.orders
    name: orders
    check_availability_interval: 60s
    params: ...

datasets:
  - from: postgres:public.orders
    name: orders
    check_availability_interval: 60s
    params: ...

Only non-accelerated datasets with check_availability: auto are monitored. Setting the interval on an accelerated dataset logs a warning and has no effect, because an accelerated dataset keeps serving from its accelerator even when the source is unavailable. An unparseable duration fails dataset load rather than silently disabling the check.

A recent successful query counts as proof that the source is reachable: when a dataset was queried successfully more recently than its own interval, that query stands in for the probe, so actively-queried datasets are not probed redundantly. This uses task history (opens in a new tab), and looks back at most one hour regardless of the configured interval; datasets with a longer interval are probed directly.

Availability monitoring is not supported for Iceberg datasets and is skipped for them (see spiceai/spiceai#6994 (opens in a new tab)).

The monitoring works by executing a query that selects one row and all columns from the dataset. i.e.:

SELECT
  "p_partkey",
  "p_name",
  "p_mfgr",
  "p_brand",
  "p_type",
  "p_size",
  "p_container",
  "p_retailprice",
  "p_comment"
FROM
  spiceai_sandbox.tpch.part
LIMIT 1;

SELECT
  "p_partkey",
  "p_name",
  "p_mfgr",
  "p_brand",
  "p_type",
  "p_size",
  "p_container",
  "p_retailprice",
  "p_comment"
FROM
  spiceai_sandbox.tpch.part
LIMIT 1;

If the monitoring query fails, a warning is emitted in the logs, an error is propagated to the task_history table, the dataset_unavailable_time_ms metric is recorded for the failing dataset, and the dataset's status becomes Error — visible via GET /v1/datasets?status=true. A later successful probe restores the dataset to Ready.

The status transition applies only to a dataset that is otherwise Ready (or already in Error from a previous failed probe); transient lifecycle states such as Initializing and Refreshing are left to the load and refresh paths that own them, and a recovery only restores Ready for a dataset the availability monitor itself put into Error.

Deferred dataset initialization

Datasets can defer connector creation and schema inference until the first query by combining ready_state: on_registration with fully declared columns[].type entries. When every column has an explicit type, the runtime registers a placeholder table with the declared Arrow schema at startup — SQL planning and federation analysis work against this schema without contacting the source. On first query, the placeholder is swapped for the real provider.

A dataset is eligible for deferred initialization when:

• It is read-only.
• ready_state: on_registration is set.
• It has no embedding or full-text-search columns.
• Every column has an explicit columns[].type (opens in a new tab).

datasets:
  - from: https://api.example.com/data.json
    name: my_data
    ready_state: on_registration
    columns:
      - name: id
        type: bigint
      - name: name
        type: text
      - name: created_at
        type: timestamptz

datasets:
  - from: https://api.example.com/data.json
    name: my_data
    ready_state: on_registration
    columns:
      - name: id
        type: bigint
      - name: name
        type: text
      - name: created_at
        type: timestamptz

When deferred initialization is active, the runtime:

• Registers the dataset immediately with the declared schema — queries can reference the table in planning before the source is contacted.
• On the first query that references the dataset, initializes the connector and loads the real data transparently.
• Coordinates concurrent triggers so the dataset is only initialized once.
• Supports acceleration — after deferred initialization, the acceleration table, refresh loop, and health monitor are set up normally.

:::warning[Breaking change] load: on_demand has been removed. Replace it with ready_state: on_registration combined with explicit columns[].type declarations. :::

acceleration

Optional. Accelerate queries to the dataset by caching data locally.

acceleration.enabled

Enable or disable acceleration, defaults to true.

acceleration.engine

The acceleration engine to use, defaults to arrow. The following engines are supported:

• arrow - Accelerated in-memory backed by Apache Arrow DataTables.
• cayenne (opens in a new tab) - Accelerated by Spice Cayenne (Vortex) engine (Stable, v1.9.0-rc.1+).
• duckdb (opens in a new tab) - Accelerated by an embedded DuckDB database.
• postgres (opens in a new tab) - Accelerated by a Postgres database.
• sqlite (opens in a new tab) - Accelerated by an embedded SQLite database.
• turso (opens in a new tab) - Accelerated by an embedded Turso (libSQL) database (Beta).

acceleration.mode

Optional. The mode of acceleration. The following values are supported:

• memory - Store acceleration data in-memory. Supported for Spice Cayenne (cayenne), where the acceleration is ephemeral and reloads from its source on restart.
• file - Store acceleration data in a file. Reuses any existing file on startup. Supported for Spice Cayenne (cayenne), duckdb, sqlite, and turso acceleration engines.
• file_create - Always create a new acceleration file on startup, removing any existing file. When snapshots (opens in a new tab) are enabled, the existing file is snapshotted before deletion. Supported for Spice Cayenne (cayenne), duckdb, sqlite, and turso acceleration engines.
• file_update - Open an existing acceleration file if it exists, then check schema compatibility on refresh. If the source schema change is additive (new columns only), the existing file is kept. If the schema change is incompatible (columns removed, renamed, or type changed), the file is snapshotted (if snapshots (opens in a new tab) are enabled) and recreated from scratch. Supported for Spice Cayenne (cayenne), duckdb, sqlite, and turso acceleration engines.

acceleration.storage_profile

Optional. The storage profile for file-backed acceleration. The runtime uses this hint to tune connection-pool sizing, checkpoint thresholds, and file-size defaults for the underlying medium. Only applies to file-mode accelerators (duckdb, sqlite, turso, and Spice Cayenne); memory-mode accelerators ignore this setting.

Supported values:

• auto (default) – Detect the storage profile from the resolved acceleration file path. On Linux, detection reads /proc/self/mountinfo and inspects block-device metadata to recognize Amazon EBS, Azure Managed Disks, Amazon EC2 NVMe instance storage, tmpfs/ramfs, and generic NVMe/SSD. On other platforms, detection returns unknown and the engine defaults apply.
• local_ssd (aliases: ssd, nvme) – Treat the acceleration file location as local SSD/NVMe (for example, EC2 instance store or Azure temporary/NVMe local storage). Uses the engine defaults for connection pool size and checkpoint thresholds.
• ebs (aliases: azure_disk, managed_disk, network_disk) – Treat the acceleration file location as network-attached block storage (for example, Amazon EBS or Azure Managed Disks). Reduces connection-pool size and raises DuckDB's checkpoint threshold so per-IO latency is amortized across larger flushes. Spice Cayenne uses smaller per-file targets to reduce write amplification.
• tmpfs (aliases: ram, ramdisk, ramfs, memory) – Treat the acceleration file location as RAM-backed storage. Raises DuckDB's checkpoint threshold so steady-state workloads don't pay checkpoint cost on small amounts of dirty data; Spice Cayenne uses larger per-file targets to improve scan throughput.

Example:

datasets:
  - from: s3://bucket/data/
    name: analytics
    acceleration:
      engine: duckdb
      mode: file
      storage_profile: ebs
      params:
        duckdb_file: /mnt/ebs/analytics.db

datasets:
  - from: s3://bucket/data/
    name: analytics
    acceleration:
      engine: duckdb
      mode: file
      storage_profile: ebs
      params:
        duckdb_file: /mnt/ebs/analytics.db

acceleration.snapshots

Optional. Controls how this dataset participates in managed acceleration snapshots. Requires the Spicepod to configure the top-level snapshots block (opens in a new tab), the acceleration engine to be duckdb, sqlite, cayenne, or turso, and mode: file with a dataset-specific file path (for example acceleration.params.duckdb_file: /nvme/my_dataset.db).

Supported values:

• enabled – Download the newest snapshot on startup when the acceleration file is missing and write a fresh snapshot after each refresh.
• bootstrap_only – Download snapshots on startup but never write new ones.
• create_only – Write snapshots after refreshes but never download them on startup.
• disabled (default) – Do not use snapshots for this dataset.

Snapshots are written beneath the configured snapshot location using Hive-style partitioning (month=YYYY-MM/day=YYYY-MM-DD/dataset=<dataset>). For more background, see Acceleration snapshots (opens in a new tab).

acceleration.snapshots_trigger

Optional. Controls when Spice creates new snapshots. The available triggers depend on the dataset's refresh mode.

For batch-based datasets (refresh_mode: full, or refresh_mode: append with time_column):

• refresh_complete (default) – Create a snapshot after each data refresh completes.
• time_interval – Create snapshots at a fixed time interval specified by snapshots_trigger_threshold.

For caching datasets (refresh_mode: caching):

• time_interval (default) – The only supported trigger. Create snapshots at a fixed time interval, defaulting to 10m if snapshots_trigger_threshold is not specified. refresh_complete and stream_batches are not supported in caching mode and cause a configuration error.

For stream-based datasets (refresh_mode: changes, or refresh_mode: append without time_column):

• time_interval (default) – Create snapshots at a fixed time interval. Defaults to 10m if snapshots_trigger_threshold is not specified.
• stream_batches – Create a snapshot after a specified number of batches are processed.

See Acceleration snapshots (opens in a new tab) for more details.

acceleration.snapshots_trigger_threshold

Optional. The threshold value for snapshot creation, interpreted based on the configured snapshots_trigger:

• When snapshots_trigger: time_interval – A duration specifying how often to create snapshots (e.g., 10m, 1h). Defaults to 10m for stream-based datasets.
• When snapshots_trigger: stream_batches – An integer specifying the number of batch updates after which to create a snapshot.

Not applicable when snapshots_trigger: refresh_complete.

acceleration.snapshots_compaction

Optional. Enable database compaction before uploading snapshots. Only supported for the duckdb acceleration engine. Defaults to disabled.

When enabled, Spice uses DuckDB's internal compaction mechanism (COPY DATABASE) to optimize the database file before uploading, reducing snapshot size and improving bootstrap performance.

Supported values:

• enabled – Compact the database before creating each snapshot.
• disabled (default) – Upload snapshots without compaction.

acceleration.snapshots_creation_policy

Optional. Controls when new snapshots are created after data refreshes. Defaults to on_change.

Supported values:

• on_change (default) – Only create a new snapshot when the data has changed since the last snapshot.
• always – Create a new snapshot after every refresh, regardless of whether the data has changed.

acceleration.snapshots_reset_expiry_on_load

Optional. Controls whether the snapshot expiry timer is reset when loading from a snapshot during bootstrap. Defaults to disabled.

Supported values:

• disabled (default) – Snapshot expiry is not reset on load; the original expiry time is preserved.
• enabled – Reset the snapshot expiry when it is loaded during bootstrap.

acceleration.refresh_mode

Optional. How to refresh the dataset. The following values are supported:

• full - Refresh the entire dataset.
• append - Append new data to the dataset. When time_column is specified, new records are fetched from the latest timestamp in the accelerated data at the acceleration.refresh_check_interval.
• changes - Apply change data capture (CDC) events to incrementally update the dataset.
• caching - Cache data based on request metadata (HTTP requests). Uses row-level replacement based on cache keys. See Caching Mode (opens in a new tab) for details.
• snapshot - Reload exclusively from the snapshot store (opens in a new tab). The federated source is never queried; the runtime polls for newer snapshots at refresh_check_interval (default: 60s). Requires acceleration.snapshots: enabled or bootstrap_only and a snapshot-capable file-based engine (DuckDB, SQLite, Cayenne, or Turso). Writes (INSERT INTO) are rejected. See Snapshot Refresh Mode (opens in a new tab).

acceleration.write_mode

Optional. Controls how writes to a read_write accelerated dataset propagate between the local accelerator and the federated source. Only applies when the dataset has access: read_write and the source connector supports writes.

Supported values:

• write_through (default) – Writes are sent to the federated source synchronously. The client receives an ACK only after the source commits the change, providing ACID guarantees. The local accelerator is updated through the configured refresh path (for example, the WAL stream when refresh_mode: changes).
• write_back – Writes are applied to the local accelerator first (fast ACK), then forwarded asynchronously to the federated source. Choose this for write throughput when eventual consistency at the source is acceptable.

acceleration.refresh_check_interval

Optional. How often data should be refreshed. For append datasets without a specific time_column, this config is not used. If not defined, the accelerator will not refresh after it initially loads data. Cannot be specified in conjunction with a refresh_cron.

See Duration (opens in a new tab)

acceleration.refresh_cron

Optional. Specifies a cron schedule which controls how often data is refreshed. For append datasets without a specific time_column, this config is not used. If not defined, the accelerator will not refresh after it initially loads data.

See the cron schedule reference (opens in a new tab).

acceleration.params.caching_ttl

Optional. The time-to-live (TTL) for cached data before it is considered stale. Only applicable when refresh_mode: caching. Defaults to 30s.

When cached data exceeds this age (measured from the fetched_at timestamp), it becomes stale. If caching_stale_while_revalidate_ttl is also configured, stale data is immediately served to queries (no delay) while a background refresh is triggered to update the cache, implementing the Stale-While-Revalidate (SWR) pattern. If caching_stale_while_revalidate_ttl is not set, queries wait for fresh data once the TTL expires.

Example:

datasets:
  - from: https://api.tvmaze.com
    name: tv_shows
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: duckdb
      mode: file # Persist cache to disk
      params:
        caching_ttl: 15s # Cache data is fresh for 15 seconds
      refresh_check_interval: 30s # Periodic background refresh

datasets:
  - from: https://api.tvmaze.com
    name: tv_shows
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: duckdb
      mode: file # Persist cache to disk
      params:
        caching_ttl: 15s # Cache data is fresh for 15 seconds
      refresh_check_interval: 30s # Periodic background refresh

See Caching Mode (opens in a new tab) for detailed TTL configuration and behavior.

See Duration (opens in a new tab)

acceleration.params.caching_stale_while_revalidate_ttl

Optional. The duration after caching_ttl expires during which stale data is served while refreshing in the background. Only applicable when refresh_mode: caching. Defaults to none (stale data is not served).

When caching_ttl expires and data becomes stale, this parameter controls how long stale data continues to be served immediately while a background refresh occurs. After the combined caching_ttl + caching_stale_while_revalidate_ttl period, queries wait for fresh data instead of returning stale results.

If omitted, cached data becomes "rotten" immediately after caching_ttl expires, and queries will wait for fresh data rather than returning stale results.

Example:

datasets:
  - from: https://api.tvmaze.com
    name: tv_shows
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: duckdb
      mode: file
      params:
        caching_ttl: 15s # Cache data is fresh for 15 seconds
        caching_stale_while_revalidate_ttl: 30s # Serve stale data for 30 seconds while refreshing
      refresh_check_interval: 60s

datasets:
  - from: https://api.tvmaze.com
    name: tv_shows
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: duckdb
      mode: file
      params:
        caching_ttl: 15s # Cache data is fresh for 15 seconds
        caching_stale_while_revalidate_ttl: 30s # Serve stale data for 30 seconds while refreshing
      refresh_check_interval: 60s

See Caching Mode (opens in a new tab) for detailed TTL configuration and behavior.

See Duration (opens in a new tab)

acceleration.params.caching_stale_if_error

Optional. Controls whether expired cached data is served when the upstream data source returns an error. Only applicable when refresh_mode: caching. Defaults to disabled.

When set to enabled, queries return expired cached data instead of failing if the upstream source returns an error during a refresh attempt. This provides fault tolerance for APIs with intermittent availability or rate limits.

Valid values:

• enabled - Serve expired cached data when upstream errors occur
• disabled (default) - Propagate upstream errors to queries

Example:

datasets:
  - from: https://api.tvmaze.com
    name: tv_shows
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: duckdb
      mode: file
      params:
        caching_ttl: 15s
        caching_stale_while_revalidate_ttl: 30s
        caching_stale_if_error: enabled # Serve stale data on upstream errors
      refresh_check_interval: 60s

datasets:
  - from: https://api.tvmaze.com
    name: tv_shows
    acceleration:
      enabled: true
      refresh_mode: caching
      engine: duckdb
      mode: file
      params:
        caching_ttl: 15s
        caching_stale_while_revalidate_ttl: 30s
        caching_stale_if_error: enabled # Serve stale data on upstream errors
      refresh_check_interval: 60s

See Caching Mode (opens in a new tab) for detailed behavior.

acceleration.refresh_sql

Optional. Filters the data fetched from the source to be stored in the accelerator engine. Supported for full and append refresh mode datasets.

Must be of the form SELECT * FROM {name} WHERE {refresh_filter}. {name} is the dataset name declared above, {refresh_filter} is any SQL expression that can be used to filter the data, i.e. WHERE city = 'Seattle' to reduce the working set of data that is accelerated within Spice from the data source.

:::warning[Limitations]

• The refresh SQL only supports filtering data from the current dataset - joining across other datasets is not supported.
• Queries for data that have been filtered out will not fall back to querying against the federated table.

:::

acceleration.refresh_data_window

Optional. A duration to filter dataset refresh source queries to recent data (duration into past from now). Requires time_column and time_format to also be configured. Supported for full and append refresh mode datasets.

For example, refresh_data_window: 24h will include only records with a timestamp within the last 24 hours.

See Duration (opens in a new tab)

acceleration.refresh_append_overlap

Optional. A duration to specify how far back to include records based on the most recent timestamp found in the accelerated data. Requires time_column to also be configured. Only supported for append refresh mode datasets.

This setting can help mitigate missing data issues caused by late arriving data.

Example: If the latest timestamp in the accelerated data table is 2020-01-01T02:00:00Z, setting refresh_append_overlap: 1h will include records starting from 2020-01-01T01:00:00Z.

See Duration (opens in a new tab)

Supported by the Spice Cayenne (cayenne) acceleration engine from v2.2.1. In v2.2.0 a file-mode Cayenne dataset that sets this fails to load. See Cayenne limitations (opens in a new tab).

acceleration.refresh_retry_enabled

Optional. Specifies whether an accelerated dataset should retry data refresh in the event of transient errors. The default setting is true.

Retries follow a Fibonacci backoff strategy (opens in a new tab). To disable refresh retries, set refresh_retry_enabled: false.

acceleration.refresh_retry_max_attempts

Optional. Defines the maximum number of retry attempts when refresh retries are enabled. The default is undefined, with no upper limit on attempts.

acceleration.refresh_on_startup

Optional. Controls the refresh behavior of an accelerated dataset across restarts. Defaults to auto.

Supported Values

• auto (Default) – Maintains refresh state across restarts:
  • With refresh_check_interval: Schedules next refresh based on last successful refresh time, triggering immediately if interval has already elapsed
  • Without refresh_check_interval: No refresh (on-demand only)
• always – Forces a dataset refresh on every startup, regardless of the existing acceleration state.

Setting refresh_on_startup: always ensures that accelerated data is always refreshed to match the source when the service restarts. This is useful in development environments or when data consistency is critical after deployment.

acceleration.params

Optional. Parameters to pass to the acceleration engine. The parameters are specific to the acceleration engine used.

acceleration.retention_check_enabled

Optional. Enable or disable retention policy check, defaults to false.

acceleration.retention_period

Optional. The retention period for the dataset. Combine with time_column and time_format to determine if the data should be retained or not.

retention_period or retention_sql must be specified when acceleration.retention_check_enabled is true. When both retention_period and retention_sql are configured, both retention policies will be applied during each retention check.

See Duration (opens in a new tab)

acceleration.retention_sql

Optional. Custom SQL statement to define data retention logic. Takes the form of a DELETE FROM <table> WHERE <predicates> statement.

This parameter is useful for scenarios like soft-deleting rows in append-only datasets or removing data based on complex business logic that goes beyond simple time-based retention.

retention_period or retention_sql must be specified when acceleration.retention_check_enabled is true. When both retention_period and retention_sql are configured, both retention policies will be applied during each retention check.

acceleration.retention_check_interval

Optional. How often the retention policy should be checked.

Required when acceleration.retention_check_enabled is true.

See Duration (opens in a new tab)

acceleration.refresh_jitter_enabled

Optional. Enable or disable refresh jitter, defaults to false. The refresh jitter adds/substracts a randomized time period from the refresh_check_interval.

acceleration.refresh_jitter_max

Optional. The maximum amount of jitter to add to the refresh interval. The jitter is a random value between 0 and refresh_jitter_max. Defaults to 10% of refresh_check_interval.

metrics

Optional. Enable component-specific metrics for the dataset. Each component can expose its own set of metrics that can be enabled selectively to monitor specific aspects of its operation.

Most component metrics are disabled by default and can be enabled by adding a metrics section to the dataset configuration. Each metric can be enabled individually by specifying its name in the metrics list.

Some metrics are auto-registered: they export without any metrics configuration, and an entry with enabled: false is what turns one off. See Component Metrics (opens in a new tab) and each component's own documentation for which metrics are auto-registered.

Example Configuration

datasets:
  - from: mysql:my_table
    name: my_dataset
    metrics:
      - name: connection_count
        enabled: true
      - name: connections_in_pool
        enabled: true
      - name: active_wait_requests
        enabled: true
    params:
      mysql_host: localhost
      mysql_tcp_port: 3306
      mysql_user: root
      mysql_pass: ${secrets:MYSQL_PASS}

datasets:
  - from: mysql:my_table
    name: my_dataset
    metrics:
      - name: connection_count
        enabled: true
      - name: connections_in_pool
        enabled: true
      - name: active_wait_requests
        enabled: true
    params:
      mysql_host: localhost
      mysql_tcp_port: 3306
      mysql_user: root
      mysql_pass: ${secrets:MYSQL_PASS}

For detailed information about metrics available for specific components, see the component metrics documentation (opens in a new tab).

acceleration.indexes

Optional. Specify which indexes should be applied to the locally accelerated table. Not supported for in-memory Arrow acceleration engine.

The indexes field is a map where the key is the column reference and the value is the index type.

A column reference can be a single column name or a multicolumn key. The column reference must be enclosed in parentheses if it is a multicolumn key.

See Indexes (opens in a new tab)

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      engine: sqlite
      indexes:
        number: enabled # Index the `number` column
        '(hash, timestamp)': unique # Add a unique index with a multicolumn key comprised of the `hash` and `timestamp` columns

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      engine: sqlite
      indexes:
        number: enabled # Index the `number` column
        '(hash, timestamp)': unique # Add a unique index with a multicolumn key comprised of the `hash` and `timestamp` columns

acceleration.primary_key

Optional. Specify the primary key constraint on the locally accelerated table. Not supported for in-memory Arrow acceleration engine.

The primary_key field is a string that represents the column reference that should be used as the primary key. The column reference can be a single column name or a multicolumn key. The column reference must be enclosed in parentheses if it is a multicolumn key.

See Constraints (opens in a new tab)

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      engine: sqlite
      primary_key: hash # Define a primary key on the `hash` column

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      engine: sqlite
      primary_key: hash # Define a primary key on the `hash` column

acceleration.on_conflict

Optional. Specify what should happen when a constraint is violated. Not supported for in-memory Arrow acceleration engine.

The on_conflict field is a map where the key is the column reference and the value is the conflict resolution strategy.

A column reference can be a single column name or a multicolumn key. The column reference must be enclosed in parentheses if it is a multicolumn key.

Only a single on_conflict target can be specified, unless all on_conflict targets are specified with drop.

The possible conflict resolution strategies are:

• upsert - Upsert the incoming data when the primary key constraint is violated.
• upsert_dedup - Same as upsert, but also deduplicates the data if there are duplicate rows that trigger a violation constraint within a single update. See Advanced upsert behavior (opens in a new tab).
• upsert_dedup_by_row_id - Same as upsert, but resolves any violations by arbitrarily choosing the row with the highest row id. See Advanced upsert behavior (opens in a new tab).
• drop - Drop the data when the primary key constraint is violated.

See Constraints (opens in a new tab)

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      engine: sqlite
      primary_key: hash
      indexes:
        '(number, timestamp)': unique
      on_conflict:
        # Upsert the incoming data when the primary key constraint on "hash" is violated,
        # alternatively "drop" can be used instead of "upsert" to drop the data update.
        hash: upsert

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      engine: sqlite
      primary_key: hash
      indexes:
        '(number, timestamp)': unique
      on_conflict:
        # Upsert the incoming data when the primary key constraint on "hash" is violated,
        # alternatively "drop" can be used instead of "upsert" to drop the data update.
        hash: upsert

acceleration.on_zero_results

Optional. Controls the behavior when an accelerated query returns zero results. Defaults to return_empty.

The following values are supported:

• return_empty - Default. Return an empty result set when the accelerated query returns no rows.
• use_source - Fall back to querying the original data source when the accelerated query returns no rows.

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      on_zero_results: use_source

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      on_zero_results: use_source

acceleration.partition_by

Optional. Specifies columns to partition the accelerated data by, enabling partition-level operations and optimized storage. Defaults to no partitioning (empty).

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      partition_by: block_date

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    acceleration:
      enabled: true
      partition_by: block_date

columns

Optional. Define metadata, semantic details and features (e.g. embeddings, or table indexes) for specific columns in the dataset.

datasets:
  - from: file:sales_data.parquet
    name: sales
    columns:
      - name: address_line1
        description: The first line of the address.
        embeddings:
          - from: hf_minilm
            row_id: order_number
            chunking:
              enabled: true
              target_chunk_size: 256
              overlap_size: 32
        full_text_search:
          enabled: true

datasets:
  - from: file:sales_data.parquet
    name: sales
    columns:
      - name: address_line1
        description: The first line of the address.
        embeddings:
          - from: hf_minilm
            row_id: order_number
            chunking:
              enabled: true
              target_chunk_size: 256
              overlap_size: 32
        full_text_search:
          enabled: true

columns[*].name

The name of the column in the table schema.

columns[*].type

Optional. Declares the expected data type for the column, overriding the type inferred from the data source. Accepts three families of type expressions:

• Arrow display forms — Int64, Utf8, Float64, Bool, Date32, Timestamp(Microsecond, UTC), List<Int64>, Decimal128(p, s), Map<Utf8, Int64>, etc.
• SQL / Postgres forms — BIGINT, INTEGER, TEXT, VARCHAR(n), BOOLEAN, DOUBLE PRECISION, NUMERIC(p,s), DATE, TIMESTAMP, TIMESTAMP WITH TIME ZONE, BYTEA, etc.
• Postgres aliases — int2, int4, int8, float4, float8, serial, bigserial, timestamptz, uuid, and the T[] array suffix (e.g. int4[]).

datasets:
  - from: postgres:public.events
    name: events
    columns:
      - name: event_id
        type: bigint
        nullable: false
      - name: payload
        type: Map<Utf8, Utf8>
      - name: tags
        type: text[]
      - name: amount
        type: numeric(18,4)

datasets:
  - from: postgres:public.events
    name: events
    columns:
      - name: event_id
        type: bigint
        nullable: false
      - name: payload
        type: Map<Utf8, Utf8>
      - name: tags
        type: text[]
      - name: amount
        type: numeric(18,4)

columns[*].nullable

Optional. Declares whether the column allows null values. When omitted, the column defaults to nullable. Set to false to enforce a non-null constraint.

columns[*].description

Optional. A description of the column's contents and purpose. Used as part of the Semantic Data Model (opens in a new tab).

columns[*].embeddings

Optional. Create vector embeddings for this column.

columns[*].embeddings[*].from

The embedding model to use, specify the component name.

columns[*].embeddings[*].row_id

Optional. For datasets without a primary key, used to explicitly specify column(s) that uniquely identify a row.

Specifying a row_id enables unique identifier lookups for datasets from external systems that may not have a primary key.

columns[*].embeddings[*].chunking {#columns-embeddings-chunking}

Optional. The configuration to enable and define the chunking strategy for the embedding column.

columns:
  - name: description
    embeddings:
      - from: hf_minilm
        chunking:
          enabled: true
          target_chunk_size: 512
          overlap_size: 128
          trim_whitespace: false

columns:
  - name: description
    embeddings:
      - from: hf_minilm
        chunking:
          enabled: true
          target_chunk_size: 512
          overlap_size: 128
          trim_whitespace: false

See embeddings[*].chunking (opens in a new tab) for details.

columns[*].embeddings[*].vector_size

Optional. Specifies the size (number of dimensions) of the embedding vector for use in federated queries to databases that do not support arrays with fixed lengths.

columns:
  - name: review_body
    embeddings:
      - from: embed-static-retrieval
        vector_size: 1024

columns:
  - name: review_body
    embeddings:
      - from: embed-static-retrieval
        vector_size: 1024

columns[*].embeddings[*].aggregation

Optional. For multi-vector columns (List<Utf8> source), the strategy used to combine per-element similarities into a single per-row score during vector search. Only meaningful when the underlying column is list-typed.

See Multi-Vector Search (opens in a new tab) for details.

columns:
  - name: tags
    embeddings:
      - from: local_embedding_model
        aggregation: max

columns:
  - name: tags
    embeddings:
      - from: local_embedding_model
        aggregation: max

columns[*].embeddings[*].max_elements_per_row

Optional. For multi-vector columns, the maximum number of list elements embedded per row. Defaults to 32; hard-capped at 1024. Excess elements are dropped with a warning log.

columns:
  - name: tags
    embeddings:
      - from: local_embedding_model
        max_elements_per_row: 128

columns:
  - name: tags
    embeddings:
      - from: local_embedding_model
        max_elements_per_row: 128

columns[*].full_text_search {#columns-search-full-text}

columns[*].full_text_search.enabled

Optional. Enable or disable full text search support for specific column in the dataset. Default false.

columns[*].full_text_search.row_id

Optional. For datasets without a primary key, used to explicitly specify column(s) that uniquely identify a row.

Specifying a row_id enables unique identifier lookups for datasets from external systems that may not have a primary key.

columns[*].metadata

Optional. Specific metadata associated to the column.

columns[*].metadata.vectors

Optional. If provided, a vector engine (see below (opens in a new tab)) should store this column for a particular use, determined by the value, which is one of:

• non-filterable: Store the column in the vector engine.
• filterable: Store the column in the vector engine, and ensure the engine can filter on the column (if possible in the engine).

For the vector engine, only applicable if vectors.enabled is both defined and true.

The distinction between the two values is the vector engine's. When the dataset also has a full-text search index (opens in a new tab), either value carries the column into that index, where it is returned by searches and filterable (opens in a new tab) — whether or not a vector engine is configured. A column whose type the full-text index cannot represent (a date or timestamp) is skipped there, with a warning logged at startup.

embeddings

Optional. Create vector embeddings for specific columns of the dataset.

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    embeddings:
      - column: extra_data
        use: hf_minilm

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    embeddings:
      - column: extra_data
        use: hf_minilm

embeddings[*].column

The column name to create an embedding for.

embeddings[*].use

The embedding model to use, specific the component name embeddings[*].name.

embeddings[*].column_pk

Optional. For datasets without a primary key, explicitly specify column(s) that uniquely identify a row.

embeddings[*].chunking

Optional. The configuration to enable and define the chunking strategy for the embedding column.

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    embeddings:
      - column: extra_data
        use: hf_minilm
        chunking:
          enabled: true
          target_chunk_size: 512
          overlap_size: 128
          trim_whitespace: false

datasets:
  - from: spice.ai/eth.recent_blocks
    name: eth.recent_blocks
    embeddings:
      - column: extra_data
        use: hf_minilm
        chunking:
          enabled: true
          target_chunk_size: 512
          overlap_size: 128
          trim_whitespace: false

embeddings[*].chunking.enabled

Optional. Enable or disable chunking for the embedding column. Defaults to false.

embeddings[*].chunking.target_chunk_size

The desired size of each chunk, in tokens.

If the desired chunk size is larger than the maximum size of the embedding model, the maximum size will be used.

embeddings[*].chunking.overlap_size

Optional. The number of tokens to overlap between chunks. Defaults to 0.

embeddings[*].chunking.trim_whitespace

Optional. If enabled, the content of each chunk will be trimmed to remove leading and trailing whitespace. Defaults to false.

metadata {#metadata}

Optional. Additional key-value metadata for the dataset.

The metadata field serves two purposes:

1. Semantic metadata — Arbitrary key-value pairs used as part of the Semantic Data Model (opens in a new tab).

   datasets:
     - from: spice.ai/eth.recent_blocks
       name: eth.recent_blocks
       metadata:
         instructions: The last 128 blocks.
   

   datasets:
     - from: spice.ai/eth.recent_blocks
       name: eth.recent_blocks
       metadata:
         instructions: The last 128 blocks.
   

2. File metadata columns — For file-based connectors (opens in a new tab) (S3, ABFS, File, FTP, SFTP, SMB, NFS, HTTP/HTTPS), the following reserved keys enable virtual columns that expose per-file object store metadata in query results:

   Key	Value	Column Type	Description
   _location	enabled	Utf8	Full URI of the source file
   _last_modified	enabled	Timestamp(µs, "UTC")	When the file was last modified
   _size	enabled	UInt64	File size in bytes

   datasets:
     - from: s3://bucket/data/
       name: my_data
       params:
         file_format: parquet
       metadata:
         _location: enabled
         _last_modified: enabled
         _size: enabled
   

   datasets:
     - from: s3://bucket/data/
       name: my_data
       params:
         file_format: parquet
       metadata:
         _location: enabled
         _last_modified: enabled
         _size: enabled
   

   If a data file already contains a column with the same name as a metadata column, the metadata column is not added.

full_text_search {#dataset-full-text-search}

Optional. Dataset-level full-text search engine configuration. When absent, the built-in Tantivy in-process engine is used (controlled by column-level columns[*].full_text_search (opens in a new tab) settings).

full_text_search.enabled

Enable or disable the dataset-level FTS engine, defaults to true.

full_text_search.engine

The full-text search engine to use. Currently only elasticsearch is supported. When absent, the built-in Tantivy engine is used.

full_text_search.params

Optional. Engine-specific connection and tuning parameters. See Full-Text Search — Elasticsearch (opens in a new tab) for available parameters.

datasets:
  - from: file:./articles.parquet
    name: articles
    acceleration:
      enabled: true
    full_text_search:
      engine: elasticsearch
      params:
        elasticsearch_endpoint: http://localhost:9200
        elasticsearch_index: articles-fts
    columns:
      - name: body
        full_text_search:
          enabled: true
          row_id:
            - id

datasets:
  - from: file:./articles.parquet
    name: articles
    acceleration:
      enabled: true
    full_text_search:
      engine: elasticsearch
      params:
        elasticsearch_endpoint: http://localhost:9200
        elasticsearch_index: articles-fts
    columns:
      - name: body
        full_text_search:
          enabled: true
          row_id:
            - id

vectors

vectors.enabled

Enable or disable vector storage, defaults to true.

vectors.engine

The vector engine to use. The following engines are supported for datasets:

• s3_vectors (opens in a new tab) - Vectors are created and indexed into Amazon S3 Vectors (opens in a new tab).
• elasticsearch (opens in a new tab) - Vectors are created and indexed into an Elasticsearch (opens in a new tab) cluster.

Dataset and view support may differ. If a view uses vectors, refer to the views reference for the engines supported there.

vectors.params

Optional. Parameters to pass to the vector engine. The parameters are specific to the vector engine used.