The append refresh mode incrementally adds new rows to the acceleration on each refresh. It is designed for append-only or immutable datasets such as time-series, event, and log data.
Use append when:
full mode on each interval.append mode requires a time_column (opens in a new tab) that identifies new rows by comparing the local maximum value to the source. Data is incrementally refreshed where time_column in the source is greater than max(time_column) in the acceleration — see Day-Granular Time Columns (opens in a new tab) for the one case where that comparison is inclusive instead.
To account for clock skew or late-arriving rows, configure an overlap window with acceleration.refresh_append_overlap (opens in a new tab). Rows within the overlap are re-read on each refresh.
A date-typed time_column (Arrow Date32, configured as time_format: date (opens in a new tab)) carries no time of day, so every row of a given day shares one value. A strictly-greater comparison against the accelerated maximum would exclude every row that arrives later for the day already loaded, on that refresh and on every one after it.
For a day-granular time column the comparison is therefore inclusive, against the start of the high-water mark's day, and Spice's exact-row de-duplication drops the already-loaded rows that come back so nothing is appended twice. A time_partition_column of the same type is floored to that same day boundary, so the partition predicate does not exclude the day the new rows are in.
The trade-off is that each refresh re-reads the boundary day from the source rather than nothing — the minimum needed to see that day's new rows at all. Sub-day time columns are unaffected and keep the strict comparison; if a single day is large, bound the re-read by using a sub-day time_column.
refresh_append_overlap still subtracts its duration from the high-water mark, but on a day-granular column the resulting window is quantized to the day boundary, so an overlap shorter than a day re-reads the whole boundary day.
time_partition_columnDatasets partitioned by a less-granular time column (day, month, year) can specify time_partition_column (opens in a new tab) in addition to time_column for efficient partition pruning at the source.
For object-store sources, set time_column or time_partition_column to the special value last_modified to append only newly created or updated files. Spice uses file metadata to determine which files are new, dramatically reducing scan time for large datasets.
If last_modified exists as a column in the data, the column value takes precedence over file metadata.
This is supported for connectors that accept the file format parameter (opens in a new tab), such as s3://, abfs://, and file://.
Append-mode accelerations that define a time_column wait to report ready until the first append refresh completes after snapshot bootstrap (opens in a new tab). This keeps the dataset out of rotation until the freshest data is available while still benefiting from snapshot-assisted startup.
Pair refresh_mode: append with a primary_key and on_conflict: upsert to handle source rows that are occasionally updated. See End-to-End Incremental Ingestion Example (opens in a new tab).