Change Data Capture (CDC) captures insert, update, and delete events from a database's transaction log and delivers them to consumers with low latency. This technique enables Spice to keep locally accelerated (opens in a new tab) datasets synchronized with the source data in near real-time. CDC is efficient because it transfers only changed rows instead of re-fetching the entire dataset.
Using locally accelerated datasets configured with CDC enables Spice to provide high-performance accelerated queries and efficient real-time updates.
Consider a fraud detection application that needs to determine whether a pending transaction is likely fraudulent. The application queries a Spice-accelerated, real-time updated table of recent transactions to check if a pending transaction resembles known fraudulent ones. With CDC, the table is kept up-to-date, so the application can quickly identify potential fraud.
When configuring datasets to be accelerated with CDC, ensure that the data connector (opens in a new tab) supports CDC and can return a stream of row-level changes. See the Supported Data Connectors (opens in a new tab) section for more information.
The startup time for CDC-accelerated datasets may be longer than for non-CDC-accelerated datasets due to the initial synchronization.
:::tip
It is recommended to use CDC-accelerated datasets with persistent data accelerator configurations (i.e., file mode for DuckDB (opens in a new tab)/SQLite (opens in a new tab) or PostgreSQL (opens in a new tab)). This ensures that when Spice restarts, it can resume from the last known state of the dataset instead of re-fetching the entire dataset.
:::
Spice applies CDC events through a single apply loop that coalesces a contiguous run of buffered change events ("envelopes") into one accelerator write. The coalescing behavior is controlled by the following runtime.params, set once under the top-level runtime.params and applied to every CDC-accelerated dataset in the instance. Each parameter also accepts a SPICE_-prefixed environment variable; the runtime.params value takes precedence, falling back to the environment variable, then the default. Cayenne-accelerated datasets can additionally override any of these values per-dataset — see Per-dataset overrides (opens in a new tab) below.
| Parameter | Description | Default |
|---|---|---|
cdc_prefetch_buffer | Number of source change events buffered ahead of the apply loop. Range 1–16384. | 128 |
cdc_max_coalesced_envelopes | Maximum number of change events combined into a single accelerator write. Range 1–16384. | 256 |
cdc_max_coalesced_bytes | Maximum in-memory Arrow size (bytes) of a single coalesced write. Range 1–1073741824 (1 GiB). | 134217728 (128 MiB) |
cdc_max_coalesce_age_ms | Apply-loop linger window in milliseconds. When > 0, the loop keeps accumulating change events into one write until the envelope cap, the byte budget, or this window elapses — whichever comes first. The window is measured from the start of the previous apply. 0 disables lingering, so each buffered event is applied as soon as it arrives. | 0 (no linger) |
cdc_commit_timeout_ms | Maximum time to wait for the previous source-side commit before surfacing ingestion as stalled. Range 1–3600000 (1 hour). | 30000 (30s) |
Out-of-range or unparseable values are rejected with a warning and fall back to the default.
For Cayenne (opens in a new tab)-accelerated datasets, any of the five parameters above can also be set per-dataset under the dataset's acceleration.params to override the instance-wide value for that dataset only. A per-dataset value layers on top of the resolved global configuration (runtime.params → environment variable → default): a dataset overrides only the parameters it sets and inherits the global value for the rest. Out-of-range or unparseable per-dataset values are rejected with a warning and keep the global value.
Enabling CDC by setting refresh_mode: changes in the acceleration settings requires support from the data connector to provide a stream of row-level changes.
Spice currently supports streaming ingestion via:
wal_level=logical + pgoutput) and streams INSERT/UPDATE/DELETE events into the accelerator. No Kafka, no Debezium, no external services.INSERT/UPDATE/DELETE events to the accelerator.refresh_mode: append for real-time, append-only acceleration (no separate CDC connector required).See an example of configuring a dataset to use CDC with Debezium by following the recipe at Streaming changes in real-time with Debezium CDC (opens in a new tab).