Acceleration Snapshots

Acceleration snapshots persist accelerated dataset state to an object store so that a Spice runtime can bootstrap a new replica or executor without re-fetching from the federated source. Snapshots dramatically reduce cold-start time, support fast scale-out, and provide a simple disaster-recovery path for accelerated state.

How it works

1. The runtime writes the accelerated dataset to an object store (S3-compatible, Azure Blob, GCS, or any object store with conditional writes).
2. On startup, an accelerated dataset that opts into snapshots reads the most recent snapshot from the configured location and hydrates its local accelerator before serving queries.
3. While running, snapshots are produced according to the dataset's configured trigger and creation policy. Older snapshots can be compacted to bound storage cost.
4. In a SpicepodCluster, each executor reads the snapshot for the partitions it owns and writes new snapshots after refreshes complete on the partitions it is responsible for.

Configuration

Snapshots are configured in two places:

1. Top-level snapshots: declares the snapshot store location, credentials, and global behavior.
2. Per-dataset acceleration.snapshots*: opts the dataset into snapshots and chooses behavior, trigger, and compaction.

# Top-level snapshot store configuration
snapshots:
  enabled: true                         # default: true
  location: "s3://my-bucket/spice/snapshots/"
  bootstrap_on_failure_behavior: warn   # warn | retry | fallback
  params:
    region: us-east-1
    s3_auth: iam_role                   # default for snapshots; override with `key`
    # s3_key: ${secrets:AWS_ACCESS_KEY_ID}
    # s3_secret: ${secrets:AWS_SECRET_ACCESS_KEY}

datasets:
  - from: s3://lake/sales/
    name: sales
    acceleration:
      enabled: true
      engine: cayenne
      partition_by:
        - region: "region"
      # Per-dataset opt-in
      snapshots: enabled                  # disabled | enabled | bootstrap_only | create_only
      snapshots_trigger: refresh_complete # refresh_complete | time_interval | stream_batches
      snapshots_trigger_threshold: "10m"  # required when trigger is time_interval / stream_batches
      snapshots_compaction: enabled       # disabled (default) | enabled
      snapshots_creation_policy: on_change # on_change (default) | always
      snapshots_reset_expiry_on_load: disabled # disabled (default) | enabled

Top-level snapshots

Per-dataset acceleration snapshot fields

Behavior modes

A common topology in SpicepodCluster is one create_only (or enabled) executor per partition writing snapshots, with newly added executors using bootstrap_only to come online quickly.

Triggers

snapshots_creation_policy: on_change (default) suppresses snapshots when there is no change to commit; pair it with time_interval or stream_batches to bound snapshot frequency.

Failure handling on bootstrap

bootstrap_on_failure_behavior controls what happens if the runtime cannot read the most recent snapshot:

• warn: Log a warning and continue with an empty acceleration. The next refresh will repopulate it from the federated source. Suitable when the source is fast and snapshots are an optimization.
• retry: Retry loading the newest snapshot indefinitely. Suitable for replicas that should not serve stale-empty data.
• fallback: Try progressively older snapshots until one loads successfully. Suitable for resilience against a corrupt or partially written latest snapshot.

Distributed clusters

In a SpicepodCluster:

• The snapshot location should point to the same object store used for cluster shared state (or a dedicated bucket) so all schedulers and executors share a single view.
• Each executor writes snapshots only for the partitions it owns; reads target the same partition layout (Hive-style key1=v1/key2=v2/...).
• New executors joining the cluster receive their partition assignments via AllocateInitialPartitions (see Distributed Query) and then bootstrap each owned partition from the snapshot store, avoiding a federated re-scan.
• Setting snapshots: bootstrap_only on executors prevents accidental dual-writers when more than one executor temporarily believes it owns a partition during a transition; the scheduler-confirmed owner uses enabled or create_only.

Observability

Acceleration snapshots emit Prometheus / OTLP metrics on every node:

Production checklist

• Snapshot bucket has versioning enabled and a lifecycle policy that retains a documented number of snapshot generations.
• bootstrap_on_failure_behavior: fallback is used on critical datasets that cannot tolerate empty starts.
• Object-store credentials use IRSA / workload identity, not long-lived access keys.
• An alert is configured on dataset_acceleration_snapshot_failure_count greater than zero over a rolling 15-minute window.
• An alert is configured on stale dataset_acceleration_snapshot_write_timestamp for datasets that should snapshot regularly.
• In SpicepodCluster, the snapshot bucket is in the same region as the runtime nodes to minimise bootstrap latency.

See also

• Distributed Query: Acceleration snapshots (Cayenne)
• SpicepodCluster CRD reference
• Storage
• Observability