Stream every INSERT, UPDATE, and DELETE from an Amazon DynamoDB table directly into a Spice-accelerated dataset using DynamoDB Streams.
This is the recommended way to keep a Spice accelerator (DuckDB, SQLite, PostgreSQL, Cayenne) continuously in sync with a DynamoDB source — no Kafka, no Debezium, no Lambda required.
┌──────────────────┐ DynamoDB Streams ┌───────────────────┐ ChangeBatch ┌───────────────┐
│ DynamoDB │──────────────────────▶│ Spice runtime │──────────────────▶│ Accelerator │
│ table │ shard iterators │ (dynamodb │ (INSERT/ │ DuckDB / │
│ + stream │ │ connector) │ UPDATE / │ SQLite / │
│ │ │ │ DELETE) │ Postgres / │
└──────────────────┘ └───────────────────┘ │ Cayenne │
└───────────────┘
On first start the connector:
Scan of the source table so initial state is captured.INSERT / MODIFY / REMOVE event as a row-level change to the accelerator.The dataset reports Ready once stream lag drops below ready_lag (default 2s).
On subsequent restarts, file-backed accelerators resume from the persisted shard checkpoint instead of re-scanning the source table.
Streams must be enabled with view type NEW_AND_OLD_IMAGES so Spice receives both the old and new image of every modified item.
The credentials Spice uses need both table and stream actions:
refresh_mode: changesUsing DynamoDB Streams requires acceleration with refresh_mode: changes. Supported engines are duckdb, sqlite, postgres, and cayenne.
scan_interval — Polling frequency for new records. Lower values give lower latency at the cost of more GetRecords API calls.ready_lag — Maximum stream lag before the dataset is reported Ready for queries. Defaults to 2s.lag_exceeds_shard_retention_behavior — What to do when stream lag exceeds the DynamoDB shard retention window (24h). One of error (default), ready_before_load, or ready_after_load. See the connector parameter reference for the full description.snapshots_trigger: stream_batches and snapshots_trigger_threshold let you trigger acceleration snapshots based on stream-batch counts rather than wall time. See Acceleration Snapshots.The connector exposes the following component metrics for monitoring streaming health:
| Metric | Type | Description |
|---|---|---|
shards_active | Gauge | Current number of active shards in the stream |
records_consumed_total | Counter | Total number of records consumed from the stream |
lag_ms | Gauge | Current lag in milliseconds between stream watermark and now |
errors_transient_total | Counter | Total number of transient errors encountered while polling from the stream |
reinitializations_on_lag_exceeds_shard_retention_total | Counter | Total rebootstrap operations triggered due to expired shards |
These metrics are opt-in. See the DynamoDB Streams Metrics section of the connector docs for an example metrics: block and a sample Grafana dashboard.
lag_exceeds_shard_retention_behavior (default: error).refresh_sql is not supported with DynamoDB Streams.refresh_mode: changes — refresh-mode reference.