Real-time change data capture for accelerated datasets

How does change data capture work in Spice?

Spice change data capture reads row-level changes from a source's transaction log or change stream and applies them to a local accelerated dataset. With a native CDC connector (PostgreSQL, MySQL, DynamoDB, or MongoDB), each dataset runs in three stages:

1. Snapshot: when no persisted state exists, Spice loads the table's existing rows into the acceleration engine. This happens on first start, and on every start of an in-memory acceleration.
2. Stream: Spice reads new inserts, updates, and deletes from the source, such as the PostgreSQL write-ahead log, the MySQL binlog, DynamoDB Streams, or MongoDB change streams. Only changed rows cross the network.
3. Apply: an apply loop buffers change events and coalesces them into batched writes to the accelerated dataset. With the default cdc_max_coalesce_age_ms of 0, it does not wait to fill a batch. A higher value trades a short delay for larger writes.

With Debezium sources, Debezium captures the changes instead. Push ingest posts events to Spice, which applies them. With Debezium over Kafka, Spice consumes the Kafka topic, then applies the events. A native CDC dataset sets refresh_mode: changes on a connector dataset in the Spicepod. A push-ingest dataset uses from: cdc:<name> with declared columns, and receives events on the runtime's CDC endpoint. Applications query the dataset through the same SQL endpoint as any other table, and Spice serves reads from the local copy:

datasets:
  - from: postgres:public.orders
    name: orders
    params:
      pg_host: db.example.com
      pg_db: production
      pg_user: ${secrets:PG_USER}
      pg_pass: ${secrets:PG_PASS}
      pg_sslmode: verify-full
    acceleration:
      enabled: true
      engine: duckdb
      mode: file
      refresh_mode: changes
      primary_key: order_id
      on_conflict:
        order_id: upsert

For the general pattern, including log-based, trigger-based, and polling CDC, see the change data capture guide.

Which databases does Spice CDC read from?

Spice captures changes natively from four databases and accepts Debezium change events for the rest:

Debezium push ingest sends JSON or Avro change events directly to the Spice runtime, so sources without a native Spice path also skip the message bus. Plain Kafka topics use refresh_mode: append instead. By default it adds new messages as rows. With a primary key and on_conflict upsert, a message replaces the row with the same key, but deletes are not applied.

How does CDC compare with interval refresh?

The refresh mode sets how an accelerated dataset follows its source. CDC is the only mode that applies updates and deletes without rereading the table:

A full refresh of a large table rescans the source on every interval and adds read load to the production database. CDC processes each change event as it arrives and sends only the changed rows. The DynamoDB Streams case study shows this pattern propagating configuration changes to thousands of nodes in under a second.

What does CDC require from the source?

CDC trades setup on the source for freshness. Plan for these requirements before enabling it:

• Read access to changes: for PostgreSQL and MySQL, the Spice user needs replication privileges. For DynamoDB and MongoDB, it needs permission to read the table's stream or change stream. With Debezium push ingest, Debezium reads the source and posts events to Spice.
• Row identity: updates and deletes are applied by primary key. Set acceleration.primary_key and an on_conflict upsert for that key on every engine except Arrow, including DuckDB, SQLite, PostgreSQL, and Spice Cayenne. For a PostgreSQL table without a primary key, also set a replica identity that sends those columns, such as REPLICA IDENTITY FULL.
• Replication slots: by default, each PostgreSQL dataset on each Spice instance gets its own replication slot. Datasets that set the same pg_replication_slot share one slot, and each Spice instance still needs its own. Size max_replication_slots for the slots actually configured. An unused slot keeps WAL on the source until it is dropped.
• Initial load time: the first snapshot makes startup slower than for a non-CDC dataset. After the snapshot, a PostgreSQL dataset reports Ready when replication lag falls below pg_replication_ready_lag (2 seconds by default). This is a readiness threshold, not a latency guarantee.
• Encrypted transport: set pg_sslmode to require, verify-ca, or verify-full so the replication stream uses TLS.

CDC pairs with SQL federation and acceleration: tables that change often get CDC, and the rest stay federated or refresh on an interval in the same query. The same mechanism keeps the analytics replica and the operational data lakehouse current. To run CDC on a managed runtime, compare Spice Cloud plans and pricing.