title: 'DynamoDB Data Connector Deployment Guide' sidebar_label: 'Deployment Guide' description: 'Operating guide for the DynamoDB data connector in production: IAM, streams, checkpointing, lag behavior, and observability.' sidebar_position: 10 pagination_prev: null pagination_next: null tags:
Production operating guide for the DynamoDB data connector covering IAM, DynamoDB Streams CDC, checkpointing, and lag handling.
DynamoDB authentication uses the standard AWS credential chain. Configure via the same parameters as the S3 connector (opens in a new tab):
| Parameter | Description |
|---|---|
dynamodb_aws_region | AWS region of the DynamoDB table. |
dynamodb_aws_access_key_id | Explicit access key (optional; falls back to the credential chain when unset). |
dynamodb_aws_secret_access_key | Explicit secret key (optional). |
dynamodb_aws_session_token | Session token for temporary credentials (optional). |
For production on EKS/ECS, leave access-key parameters unset and rely on instance-profile, IRSA, or ECS task-role credentials. Grant the role dynamodb:Scan, dynamodb:Query, and dynamodb:DescribeTable on the table; for streams, additionally grant dynamodb:DescribeStream, dynamodb:GetShardIterator, and dynamodb:GetRecords.
Secrets should be sourced from a secret store (opens in a new tab) when not using IAM role auth.
The DynamoDB connector supports CDC via DynamoDB Streams with an accelerated dataset as the sink. Stream state is persisted as a checkpoint alongside the accelerator, allowing resumption after a restart.
| Parameter | Default | Description |
|---|---|---|
scan_interval | 0s | Interval between polls for new records in a DynamoDB stream. |
dynamodb_replication_ready_lag | 2s | Once replication lag falls below this threshold, the dataset is reported as Ready. (Previously ready_lag, still accepted as a deprecated alias.) |
dynamodb_replication_initial_snapshot | auto | When refresh_mode: changes first loads existing items: auto scans only when no resumable checkpoint exists, disabled streams changes only, always scans on every start. |
dynamodb_replication_invalid_checkpoint_behavior | error | Behavior when the persisted checkpoint can no longer be honored (past ~24h shard retention): error or restart. (Previously lag_exceeds_shard_retention_behavior, still accepted as a deprecated alias.) |
DynamoDB Streams retain records for 24 hours. If Spice is offline longer than the retention window, the checkpoint becomes stale and the next stream open returns ShardNotFound. Behavior is controlled by dynamodb_replication_invalid_checkpoint_behavior:
error (default): Mark the dataset Error. Requires operator intervention to re-bootstrap.restart: Drop the stale checkpoint and re-bootstrap the accelerated dataset from a fresh scan.A checkpoint older than 18 hours is treated as near-expired and triggers the same recovery path even if the shard has not yet been dropped by DynamoDB.
The deprecated lag_exceeds_shard_retention_behavior alias is still accepted: ready_after_load maps to restart, and ready_before_load (which briefly served stale data before reloading) has been removed and also maps to restart.
The DynamoDB connector registers the following metrics:
| Metric | Type | Description |
|---|---|---|
shards_active | Gauge | Number of active DynamoDB Streams shards being consumed. |
records_consumed_total | Counter | Total stream records consumed. |
lag_ms | Gauge | Current lag behind the stream head in milliseconds (approximate, summed across shards). |
errors_transient_total | Counter | Transient stream read errors (retried automatically). |
reinitializations_on_lag_exceeds_shard_retention_total | Counter | Number of times the stream was reinitialized due to lag exceeding shard retention. |
Metrics are exposed with the dataset_dynamodb_ prefix. Monitor lag_ms together with errors_transient_total — a climbing lag with rising transient errors indicates the connector is falling behind retention.
See Component Metrics (opens in a new tab) for enabling and exporting metrics.
Stream polling and bootstrap operations emit spans that participate in task history (opens in a new tab) under the enclosing acceleration_refresh and changes-stream tasks.
attribute_exists) are not supported in DML operations.dynamodb_aws_region to match the region of the source table; cross-region access requires resource policies and is not recommended.StreamSpecification disabled: CDC mode is unavailable; fall back to full-table refresh.| Symptom | Likely cause | Resolution |
|---|---|---|
Dataset stuck in Error after restart with stream enabled | Checkpoint older than 18h or exceeded 24h retention. | Set dynamodb_replication_invalid_checkpoint_behavior: restart to auto-recover, or trigger a manual refresh. |
ProvisionedThroughputExceededException | RCU exhausted during initial scan. | Switch to on-demand billing, raise RCU for the refresh window, or slow the refresh via acceleration settings. |
TrimmedDataAccessException | Records trimmed from the stream before they could be processed. | Same recovery path as ShardNotFound — re-bootstrap. Reduce bootstrap duration via parallel segments if supported. |
AccessDeniedException on DescribeStream | IAM role lacks stream permissions. | Add dynamodb:DescribeStream, GetShardIterator, GetRecords to the role. |
ResourceNotFoundException on stream start | Stream not enabled on the table. | Enable streams on the DynamoDB table (NEW_AND_OLD_IMAGES recommended). |