The append refresh mode incrementally adds new rows to the acceleration on each refresh. It is designed for append-only or immutable datasets such as time-series, event, and log data.
Use append when:
full mode on each interval.append mode requires a time_column (opens in a new tab) that identifies new rows by comparing the local maximum value to the source. Data is incrementally refreshed where time_column in the source is greater than max(time_column) in the acceleration — see Day-Granular Time Columns (opens in a new tab) for the one case where that comparison is inclusive instead.
To account for clock skew or late-arriving rows, configure an overlap window with acceleration.refresh_append_overlap (opens in a new tab). Rows within the overlap are re-read on each refresh. A row that arrives later than the overlap, with a time_column value older than the window, is not loaded by any later append refresh; a full refresh loads it.
Iceberg sources (opens in a new tab) do not depend on the overlap for late rows: their append refresh reads what each new snapshot added. The exception is a keyed acceleration on an engine other than Cayenne whose primary_key does not include the time_column, which reads by time (opens in a new tab).
An append refresh of an Iceberg (opens in a new tab) dataset, or of an Iceberg table read through the Glue (opens in a new tab) connector, follows the table's snapshots instead of a time window. Spice saves the snapshot it last loaded. The next refresh reads the data files that the snapshots committed since then added, whatever their time_column values, so rows that a writer commits out of time order are loaded. A refresh of a table with no new snapshot scans no data.
An acceleration on an engine other than Cayenne, with a primary_key that does not include the time_column, resolves a repeated key by arrival order under its on_conflict (opens in a new tab) setting, not by time_column. Its append refresh of an Iceberg source does not follow snapshots. It reads rows newer than the latest stored time_column value, less refresh_append_overlap. A late row with a time_column value older than that window is not loaded, so it cannot replace the newer row already stored. A late row inside the window is loaded, and with on_conflict: upsert it can replace a newer stored row with the same key. Cayenne keeps the newest row for each key by time_column, so it follows snapshots and loads every new row.
Such a dataset with no refresh_append_overlap logs this warning when it loads:
The first refresh, which has no saved snapshot, reads by time as described in Configuration (opens in a new tab). Each kind of snapshot since the saved one is handled as follows:
| Snapshot operation | What the refresh reads |
|---|---|
append | The data files the snapshot added, as of that snapshot, so a compaction committed before the refresh does not hide them. |
replace (compaction) | Nothing. The rows are already loaded. |
delete, or any snapshot adding no data file | Nothing. An append refresh never removes a row; see Iceberg delete files (opens in a new tab). |
overwrite | On Spice Cayenne (opens in a new tab) with a primary_key that does not include the time_column, the files the snapshot added, keeping the newest version of each key. Otherwise this refresh reads by time. |
Some table histories cannot be followed. For these, that one refresh reads by time, as an append refresh of any other source does, and later refreshes follow snapshots again from the snapshot it loaded:
overwrite snapshot rewrote rows, and the acceleration does not keep the newest version of each key as described above.A refresh that falls back for one of these reasons logs a warning naming the reason, and repeats it only when the reason changes. Rows that arrived later than refresh_append_overlap before such a refresh are not loaded by it; run a full refresh to reload them. The dataset_acceleration_refresh_fallbacks (opens in a new tab) counter counts each fallback by reason: table_replaced, snapshot_not_ancestor, table_has_no_snapshot, or rows_rewritten_without_primary_key.
A file-mode acceleration keeps the saved snapshot with its data, so following continues across restarts. Other accelerations keep it in memory, and the first refresh after a restart reads by time. The saved snapshot is discarded when a schema change recreates the accelerated table or the acceleration is restored from a snapshot (opens in a new tab), so the next refresh reads by time. A full refresh saves the snapshot it read, and later append refreshes continue from there.
A dataset that computes columns over the Iceberg table, such as embedding columns (opens in a new tab), and a one-off refresh with its own refresh_sql, read by time. These reads, like the first refresh, are not fallbacks: they log no fallback warning and are not counted in dataset_acceleration_refresh_fallbacks.
A date-typed time_column (Arrow Date32 or Date64, configured as time_format: date (opens in a new tab)) carries no time of day, so every row of a given day shares one value. A strictly-greater comparison against the accelerated maximum would exclude every row that arrives later for the day already loaded, on that refresh and on every one after it.
For a day-granular time column the comparison is therefore inclusive, against the start of the high-water mark's day, and Spice's exact-row de-duplication drops the already-loaded rows that come back so nothing is appended twice. A time_partition_column of the same type is floored to that same day boundary, so the partition predicate does not exclude the day the new rows are in.
The trade-off is that each refresh re-reads the boundary day from the source rather than nothing — the minimum needed to see that day's new rows at all. Sub-day time columns are unaffected and keep the strict comparison; if a single day is large, bound the re-read by using a sub-day time_column.
refresh_append_overlap still subtracts its duration from the high-water mark, but on a day-granular column the resulting window is quantized to the day boundary, so an overlap shorter than a day re-reads the whole boundary day.
time_partition_columnDatasets partitioned by a less-granular time column (day, month, year) can specify time_partition_column (opens in a new tab) in addition to time_column for efficient partition pruning at the source.
For object-store sources, set time_column or time_partition_column to the special value last_modified to append only newly created or updated files. Spice uses file metadata to determine which files are new, dramatically reducing scan time for large datasets.
If last_modified exists as a column in the data, the column value takes precedence over file metadata.
This is supported for connectors that accept the file format parameter (opens in a new tab), such as s3://, abfs://, and file://.
An append-mode acceleration that defines a time_column and meets the conditions in Serving an Existing Acceleration While the Source Is Unavailable (opens in a new tab) is served as soon as its existing acceleration is available, whether it was reopened from disk or restored by snapshot bootstrap (opens in a new tab), including while the source is unavailable. The first append refresh then catches up from the newest time_column value in the acceleration. In v2.4.0-rc.3 and earlier, such a dataset with snapshots enabled waited to report ready until its first append refresh completed.
Pair refresh_mode: append with a primary_key to handle source rows that are occasionally updated. On Spice Cayenne (opens in a new tab), the default engine on Linux and macOS (Windows defaults to Arrow), a primary_key alone keeps one row per key: with a time_column that is not part of the primary_key, the newest version by time_column, including when one poll holds several versions of a key. Set refresh_append_overlap (opens in a new tab) to how late rows can arrive so that a late update is re-read:
Other accelerators do not order a key's versions by time_column; see Handling conflicts (opens in a new tab). The End-to-End Incremental Ingestion Example (opens in a new tab) combines this with retention.
Iceberg v2 position and equality delete files are applied on federated scans and on refresh_mode: full (opens in a new tab). They are not applied by append: once a row is in the acceleration, a later delete file does not retract it. Prefer soft deletes plus a filtering view (opens in a new tab) until Iceberg snapshot-diff into changes (opens in a new tab) exists. See Delete files on federated reads (opens in a new tab).
An append poll of an Iceberg source that reads by time skips files and Parquet row groups from partition values and column statistics. How much it skips depends on whether the table is partitioned or sorted by the time column, or compacted so that old and new rows share a file. See Append refresh and table layout (opens in a new tab).