1Modern applications have two fundamentally different data access patterns, and no single deployment model serves both well. Large analytical queries — scanning terabytes of Iceberg data, joining Delta Lake tables, running cross-dataset aggregations — need distributed execution across many nodes. Hot operational queries — serving the working set a microservice actually uses, answering user-facing requests in under 5 milliseconds, feeding fresh context to an AI agent — need data materialized right next to the application, with no network hop.
2
3Those workloads share a third requirement: **isolation**. Giving an application — especially an AI agent that writes its own queries — direct credentials to production Postgres, a data lake, or a warehouse means a bad plan, a runaway loop, or a prompt injection can exhaust connection pools, scan petabytes, or touch rows it shouldn't. The retrieval layer needs to be a sandbox, not a passthrough.
4
5<img
6 width='740'
7 alt='cluster-sidecar'
8 src='/img/deployment/cluster-sidecar.png'
9/>
10
11The [cluster-sidecar architecture](https://spice.ai/blog/cluster-sidecar-architecture) addresses all three. Application sidecars handle the hot path with a scoped, locally accelerated working set, while a centralized Spice [cluster](./cluster) (or the [Spice Cloud Platform](./hosted)) provides distributed compute for heavy queries, data ingestion, acceleration, and refresh. When a sidecar needs to reach beyond its materialized working set — a historical query, a cross-dataset join, a broad search — it transparently delegates to the cluster over Arrow Flight, which executes the query and returns results. The sidecar can then cache those results for future use.
12
13From the application's perspective, everything is `localhost`. From an infrastructure perspective, the system delivers the throughput of a distributed query engine, the latency of an embedded database, and a hard isolation boundary between the application and origin data systems — without ETL between them, sync jobs, or consistency gaps.
14
15Think of it as a [CDN for your data](../../use-cases/data/database-cdn): the cluster is the origin server, the sidecars are the edge nodes, and Spice handles the caching, invalidation, and routing. Origin databases, data lakes, and CDC streams never see the application fleet.
16
17```mermaid
18flowchart LR
19 subgraph Node1["Node / Pod"]
20 direction LR
21 A1["App"] <-->|"loopback"| SC1["Spice Sidecar (cache)"]
22 end
23
24 subgraph Node2["Node / Pod"]
25 direction LR
26 A2["App"] <-->|"loopback"| SC2["Spice Sidecar (cache)"]
27 end
28
29 SC1 & SC2 -->|"Arrow Flight (gRPC)"| Cluster
30
31 subgraph Cluster["Spice Cluster or Spice Cloud"]
32 direction LR
33 S1["Spice Node 1"]
34 S2["Spice Node 2"]
35 S3["Spice Node 3"]
36 end
37
38 Cluster --> Sources["Data Sources (S3, PostgreSQL, Databricks, ...)"]
39```
40
41Each sidecar is configured declaratively via a `spicepod.yaml` — the datasets, views, acceleration engines, search indices, and AI models it manages. Sidecars start in seconds, typically run on a few hundred megabytes of memory, and scale horizontally with application pods: scale a deployment from 5 to 50 replicas and 50 sidecars come up automatically, each materializing only the working set its spicepod declares. Sidecars never talk to each other; they only talk to the cluster.
42
43For the engineering walkthrough of this pattern, including a request-path example and FAQ, see [Localhost Latency at Scale: The Spice Cluster-Sidecar Architecture](https://spice.ai/blog/cluster-sidecar-architecture).
44
45## How it works
46
47### The sidecar as a sandbox
48
49The most important property of the sidecar is isolation, not just latency. The sidecar is the only data-plane surface the application touches:
50
51- **Scoped working set.** A sidecar's `spicepod.yaml` declares exactly which datasets, views, and search indices the application may query. Anything not declared is physically absent from the catalog — not filtered by a policy, not hidden by a row-level rule. Compare this with row-level security, where the underlying data is still present and a single policy misconfiguration can expose it. Even a perfectly crafted prompt injection cannot query a table that is not in the catalog.
52- **No origin credentials in the application.** The application connects to its sidecar with a local token. The sidecar connects to the cluster over Arrow Flight. Only the cluster holds credentials for Postgres, Snowflake, Databricks, S3, Kafka, and the rest. Compromising an application pod cannot leak origin credentials, because the pod never had them.
53- **Narrow network surface.** The application's only outbound data dependency is the loopback interface. Network policy can pin the sidecar's egress to the cluster endpoint only.
54- **Per-application data views.** Sidecars can be specialized per application class or per tenant. A customer-service agent, a fraud-review agent, and an internal dashboard can each run with different spicepods pointing at different slices of the same cluster. This is physical isolation, not policy-based filtering on a shared database. See [Multi-Tenant AI Agents](/docs/use-cases/ai/multi-tenant-agents).
55- **Bounded resource use.** A rogue query plan or runaway loop exhausts the sidecar's local memory and CPU budget, not the cluster's and not the origin database's.
56- **Local inference.** Sidecars can serve [LLM inference and tool calls](../../features/large-language-models) on loopback so sensitive prompts and retrieved context stay in the pod. Heavier models can still be routed to the cluster.
57- **One audit point.** Every query, search, and inference call flows through the sidecar, so there is one place to log, rate-limit, and enforce policy.
58
59Data only moves along one path: **application → sidecar → (optionally) cluster → origin**. It never skips a tier.
60
61### Three latency tiers
62
63A request is served from the first tier that can answer it:
64
651. **Sidecar results cache** — repeat queries return from the in-memory [results cache](../../features/caching) in microseconds (`Results-Cache-Status: HIT`).
662. **Sidecar working set** — novel queries that fit the locally materialized dataset execute on-node in single-digit milliseconds (Arrow, DuckDB, or SQLite).
673. **Cluster delegation** — queries that exceed the working set are forwarded to the cluster. [Apache Ballista](../../features/distributed-query) distributes execution; [Spice Cayenne](../../components/data-accelerators/cayenne) accelerates large scans. The sidecar caches the result so subsequent reads drop back to tier 1.
68
69The cluster has its own results cache, so one sidecar's miss can become a cluster cache hit for every subsequent sidecar. Typical application-visible latency: p50 at tier 1, p95 at tier 2, tail at tier 3.
70
71Useful cache knobs in this topology:
72
73- `cache_key_type: plan` (default) shares a cache entry across semantically equivalent SQL, which matters for ORM-generated queries. `sql` is a faster, string-exact lookup.
74- `stale_while_revalidate_ttl` lets the sidecar serve a stale cached result immediately while a background refresh runs.
75- Clients can send `Cache-Control` directives per request (`no-cache`, `min-fresh`, `max-stale`, `only-if-cached`).
76- `encoding: zstd` typically cuts results-cache memory by 50–90%.
77
78### The cluster ingests once
79
80Refreshing from upstream sources, running Cayenne acceleration on large Iceberg tables, and keeping CDC streams connected are resource-intensive. Doing any of that *N* times for *N* sidecars is wasteful and often infeasible — source systems have connection limits, and per-pod CDC multiplies cloud cost with fleet size.
81
82The cluster ingests each dataset once and produces one authoritative materialization. Source load is bounded by cluster size, not fleet size. A new pod's sidecar pulls its working set from the cluster (or from an [acceleration snapshot](../../features/data-acceleration/snapshots)) rather than re-scanning the source, so new nodes are operational in seconds.
83
84Sidecars stay lightweight because they do **not** own ingest:
85
86- They start in seconds — important when application pods autoscale with traffic.
87- They typically run on a few hundred megabytes of memory.
88- Scaling from 5 to 50 replicas does **not** add 50 new connections to the source database.
89- Each sidecar materializes only the working set its spicepod declares, not a full copy of the warehouse. Full accelerated datasets stay on the cluster.
90
91### Query delegation over Arrow Flight
92
93Sidecar-to-cluster communication is [Arrow Flight](https://arrow.apache.org/docs/format/Flight.html) over gRPC. Results flow as Arrow record batches directly into the sidecar's query engine with no JSON or row-based serialization detour.
94
95The sidecar decides locally whether a query can be served from its working set. If not, it forwards and streams results back. The application sees one endpoint and one query. It never knows whether execution happened locally or whether Ballista fanned the query out across cluster executors.
96
97Configure delegation with the [Spice.ai Data Connector](../../components/data-connectors/spiceai): point a sidecar dataset at a Spice Cloud app (`spice.ai/<org>/<app>/datasets/<name>`) or at a self-hosted cluster (`spice.ai:https://cluster.example:50051`). Combine with local acceleration to materialize the hot working set; leave acceleration off to always delegate.
98
99Kubernetes is the most common place to run this topology, but it is not required. The architecture needs only a network path from each sidecar to the cluster. Sidecars run wherever the application runs — Kubernetes, a VPC, on-prem, or at the edge.
100
101### Acceleration snapshots
102
103The cluster-sidecar split maps onto [acceleration snapshots](../../features/data-acceleration/snapshots) as a single-writer / many-reader topology:
104
105- The **cluster** is the single writer (`snapshots: create_only`). After each refresh it uploads a snapshot to object storage. It never downloads snapshots on startup; it always refreshes from the source.
106- Each **sidecar** is a reader (`snapshots: bootstrap_only`). On startup — or when ephemeral NVMe is recycled — the sidecar downloads the most recent snapshot and is immediately ready. It never writes snapshots back.
107
108This avoids snapshot conflicts, keeps the cluster as the authoritative refresh point, and gives every sidecar a warm start from the same materialization. For CDC-backed datasets with large initial state, the difference between a snapshot bootstrap and a full re-sync can be seconds versus minutes.
109
110Use DuckDB or SQLite in `mode: file` on the sidecar when snapshots are in play — snapshots persist and restore the acceleration file itself. Heavyweight [Cayenne](../../components/data-accelerators/cayenne) acceleration stays on the cluster.
111
112For production Spicepod and Helm reference configurations, see [Read/Write Separation](/docs/deployment/read-write-separation).
113
114### Cache coherency is a refresh policy, not a protocol
115
116Sidecars pull from the cluster on a configurable interval using append or full refresh. They do **not** participate in a distributed invalidation protocol. A pull-based model with explicit refresh intervals makes staleness bounded, predictable, and debuggable.
117
118For workloads that need sub-second freshness, the cluster consumes [CDC streams](../../features/cdc) (Postgres logical replication, DynamoDB Streams, Debezium, Kafka) once, and the sidecars pull the resulting accelerated dataset on a short interval. That gives near-real-time propagation without a fleet-wide invalidation bus.
119
120If the cluster is temporarily unavailable — a rolling upgrade, a network blip, a zone event — sidecars keep serving cached data and their accelerated working sets. Refreshes pause and resume when connectivity returns. The blast radius of a cluster incident is slightly staler data, not application downtime.
121
122## Example Spicepods
123
124The cluster materializes and accelerates datasets, consumes CDC, and exposes a results cache. It is the only tier with origin credentials.
125
126```yaml
127version: v1
128kind: Spicepod
129name: platform-cluster
130
131runtime:
132 caching:
133 sql_results:
134 enabled: true
135 max_size: 4GiB
136 item_ttl: 1m
137 stale_while_revalidate_ttl: 30s
138 encoding: zstd
139
140snapshots:
141 enabled: true
142 location: s3://my-bucket/spice-snapshots/
143 params:
144 s3_auth: iam_role
145
146datasets:
147 - from: postgres:public.orders
148 name: orders
149 acceleration:
150 enabled: true
151 engine: cayenne
152 mode: file
153 refresh_mode: changes
154 snapshots: create_only
155 primary_key: id
156 on_conflict:
157 id: upsert
158
159 - from: s3://lakehouse/events/
160 name: events
161 params:
162 file_format: parquet
163 acceleration:
164 enabled: true
165 engine: cayenne
166 mode: file
167 refresh_mode: append
168 refresh_check_interval: 15m
169 snapshots: create_only
170```
171
172The sidecar is much smaller. It pulls from the cluster, keeps a working-set engine local, and caches results. It holds no origin credentials.
173
174```yaml
175version: v1
176kind: Spicepod
177name: app-sidecar
178
179runtime:
180 caching:
181 sql_results:
182 enabled: true
183 max_size: 256MiB
184 item_ttl: 30s
185 stale_while_revalidate_ttl: 30s
186
187snapshots:
188 enabled: true
189 location: s3://my-bucket/spice-snapshots/
190 bootstrap_on_failure_behavior: warn
191 params:
192 s3_auth: iam_role
193
194datasets:
195 - from: spice.ai/<your-org>/<your-app>/datasets/orders
196 name: orders
197 acceleration:
198 enabled: true
199 engine: duckdb
200 mode: file
201 refresh_mode: append
202 refresh_check_interval: 10s
203 snapshots: bootstrap_only
204
205 - from: spice.ai/<your-org>/<your-app>/datasets/events
206 name: events
207 # No local acceleration — delegate to the cluster on demand.
208 # Queries that match recent hot events still hit the results cache.
209```
210
211For a self-hosted cluster, replace the `from:` URIs with `spice.ai:https://cluster.example:50051` and set `name:` to the upstream table name. See the [Spice.ai Data Connector](../../components/data-connectors/spiceai) for Cloud and self-hosted URI formats.
212
213## A request path
214
2151. The application (or agent) queries its sidecar on `localhost:8090`: `SELECT ... FROM orders WHERE tenant_id = $1 ORDER BY created_at DESC LIMIT 20`.
2162. The sidecar checks its results cache. Hit → return in microseconds.
2173. On a miss, the sidecar plans against its local catalog. `orders` is materialized locally (refreshed from the cluster every 10 seconds). It executes against DuckDB, returns in single-digit milliseconds, and populates the results cache. Origin Postgres sees no traffic.
2184. A follow-up hybrid search over a multi-gigabyte `tickets` index is not in the sidecar catalog as a local acceleration. The sidecar opens an Arrow Flight stream to the cluster. Ballista distributes the search; Cayenne segment statistics prune most files.
2195. Results stream back as Arrow record batches. The sidecar returns them to the application and caches them. A later LLM call can be served by a small local model on loopback, with the large-model call delegated to the cluster.
2206. The cluster independently caches its own results. The next replica that asks the same question gets it from the cluster cache.
221
222The application sees one endpoint, one wire format, and one latency distribution.
223
224## Benefits
225
226- **Kubernetes-native** — designed to run on Kubernetes, leveraging pod-level sidecars with cluster-level orchestration. The same topology also works outside Kubernetes wherever the application has a network path to the cluster.
227- Sub-millisecond reads via sidecar caching on loopback, with centralized data management in the cluster.
228- Structural isolation — sidecars expose a scoped catalog, hold no origin credentials, and bound resource use per pod.
229- Transparent query delegation — sidecars automatically route queries beyond their cached working set to the cluster.
230- Sidecars remain lightweight — only a working set and a results cache, no ingest or heavy acceleration overhead.
231- Cluster (or Spice Cloud) handles complex operations: data ingestion, [Spice Cayenne](../../components/data-accelerators/cayenne) acceleration, [distributed query](../../features/distributed-query), hybrid search, and refresh from sources.
232- Works with both self-managed Spice clusters and the managed [Spice Cloud Platform](./hosted) as the centralized backend. The Spice Cloud cluster-sidecar model is the most common production topology.
233- Sidecars can run anywhere — in your VPC, on-premises, at the edge, or in any Kubernetes cluster — while connecting securely to the managed cluster.
234- Horizontal scalability — add sidecars without increasing load on data sources.
235- Resilience — sidecars serve cached data even if the cluster is temporarily unavailable.
236- Secure by default — mTLS encryption across all sidecar-to-cluster communication, with data encrypted at rest and in transit.
237
238## Considerations
239
240- More complex deployment structure requiring both sidecar and cluster infrastructure. [Spice Cloud](./hosted) reduces this burden by managing the cluster.
241- Cache coherency — sidecars must be configured with appropriate refresh intervals or TTLs to balance freshness with performance. There is no distributed invalidation bus.
242- Requires a Spice cluster deployment or [Spice Cloud Platform](./hosted) subscription ([Spice.ai Enterprise](https://spice.ai/enterprise) for self-managed clustering with SSO, RBAC, and audit logs).
243- Network connectivity between sidecars and the cluster must be reliable for cache refreshes and query delegation.
244
245## Use This Approach When
246
247- Applications or AI agents require sub-millisecond reads, unified retrieval (SQL, full-text, vector, hybrid), and inference on `localhost`, without giving the application direct database credentials.
248- You need a clear isolation boundary between application code and origin data systems — scoped catalogs, no origin credentials in the pod, and one audit point per replica.
249- Multiple application instances need fast access to the same datasets without each independently querying data sources.
250- Reducing load on upstream data sources is a priority — the cluster ingests once, sidecars cache locally.
251- The system benefits from separating the caching tier (sidecars) from the data processing tier (cluster).
252- Workloads span both real-time operational queries and large-scale analytical queries on the same data (for example, an operational data lakehouse on S3/Iceberg).
253- You already run Kubernetes sidecars for other concerns (service mesh, logging, config).
254
255## Not Ideal When
256
257- The application is simple with a single instance and no isolation requirement — the overhead of both sidecar and cluster infrastructure isn't justified. Consider [Sidecar](./sidecar) or [Microservice](./microservice).
258- All queries are batch or analytical with relaxed latency requirements — a [Microservice](./microservice) deployment is simpler and sufficient.
259- Network connectivity between sidecars and the cluster is unreliable — query delegation and cache refreshes will fail, leading to stale data. Consider standalone [Sidecar](./sidecar) deployments with direct source access.
260
261## Example Use Case
262
263A multi-tenant SaaS platform runs an AI support agent. Each tenant's agent pods include a Spice sidecar that materializes that tenant's working set (recent tickets, active customer records, the last 7 days of events, the tenant's private knowledge-base embeddings) into a local DuckDB + vector index. The spicepod exposes exactly those datasets — and no others — to the agent. The sidecar holds no credentials for Postgres, S3, or Snowflake.
264
265Agent turns hit the sidecar on `localhost` and return in single-digit milliseconds; repeat retrievals return from the results cache in microseconds. Behind the sidecars, a Spice cluster (often Spice Cloud) ingests from PostgreSQL via CDC, S3 Iceberg tables, and Databricks. Cayenne acceleration and refresh schedules run on the cluster. When an agent asks a broader question — "summarize this tenant's churn signal across the last 12 months" — the sidecar recognizes the query exceeds its working set and delegates over Arrow Flight. The cluster executes it, the sidecar caches the result, and subsequent turns for that tenant return in microseconds.
266
267The origin Postgres sees exactly one consumer (the cluster). If a replica is compromised, the attacker gets a loopback endpoint scoped to that tenant's working set, not database credentials and not a query interface to the whole warehouse.
268
269The same pattern — single ingestion path, per-pod sandboxing, tiered latency, one isolation boundary — works for any application fleet, not just agents: microservices serving real-time dashboards, services powering search, or internal tools querying operational data.
270
271## See also
272
273- [Localhost Latency at Scale: The Spice Cluster-Sidecar Architecture](https://spice.ai/blog/cluster-sidecar-architecture) — engineering walkthrough, request-path example, and FAQ.
274- [Read/Write Separation](/docs/deployment/read-write-separation) — production guide for splitting ingest from reads using shared acceleration snapshots, including Spicepod and Helm reference configurations.
275- [Spice.ai Data Connector](../../components/data-connectors/spiceai) — how sidecars federate to a cluster or to Spice Cloud over Arrow Flight.
276- [Caching](../../features/caching) — results cache, stale-while-revalidate, and `Cache-Control`.
277- [Acceleration Snapshots](../../features/data-acceleration/snapshots) — warm-start file-mode accelerations from object storage.
278- [Distributed Query](../../features/distributed-query) — Apache Ballista schedulers and executors on the cluster.
279- [Spice Cayenne](../../components/data-accelerators/cayenne) — cluster-side acceleration for datasets beyond 1 TB.
280- [Multi-Tenant AI Agents](/docs/use-cases/ai/multi-tenant-agents) — per-tenant isolation patterns that pair with this architecture.
281
1Modern applications have two fundamentally different data access patterns, and no single deployment model serves both well. Large analytical queries — scanning terabytes of Iceberg data, joining Delta Lake tables, running cross-dataset aggregations — need distributed execution across many nodes. Hot operational queries — serving the working set a microservice actually uses, answering user-facing requests in under 5 milliseconds, feeding fresh context to an AI agent — need data materialized right next to the application, with no network hop.
2
3Those workloads share a third requirement: **isolation**. Giving an application — especially an AI agent that writes its own queries — direct credentials to production Postgres, a data lake, or a warehouse means a bad plan, a runaway loop, or a prompt injection can exhaust connection pools, scan petabytes, or touch rows it shouldn't. The retrieval layer needs to be a sandbox, not a passthrough.
4
5<img
6 width='740'
7 alt='cluster-sidecar'
8 src='/img/deployment/cluster-sidecar.png'
9/>
10
11The [cluster-sidecar architecture](https://spice.ai/blog/cluster-sidecar-architecture) addresses all three. Application sidecars handle the hot path with a scoped, locally accelerated working set, while a centralized Spice [cluster](./cluster) (or the [Spice Cloud Platform](./hosted)) provides distributed compute for heavy queries, data ingestion, acceleration, and refresh. When a sidecar needs to reach beyond its materialized working set — a historical query, a cross-dataset join, a broad search — it transparently delegates to the cluster over Arrow Flight, which executes the query and returns results. The sidecar can then cache those results for future use.
12
13From the application's perspective, everything is `localhost`. From an infrastructure perspective, the system delivers the throughput of a distributed query engine, the latency of an embedded database, and a hard isolation boundary between the application and origin data systems — without ETL between them, sync jobs, or consistency gaps.
14
15Think of it as a [CDN for your data](../../use-cases/data/database-cdn): the cluster is the origin server, the sidecars are the edge nodes, and Spice handles the caching, invalidation, and routing. Origin databases, data lakes, and CDC streams never see the application fleet.
16
17```mermaid
18flowchart LR
19 subgraph Node1["Node / Pod"]
20 direction LR
21 A1["App"] <-->|"loopback"| SC1["Spice Sidecar (cache)"]
22 end
23
24 subgraph Node2["Node / Pod"]
25 direction LR
26 A2["App"] <-->|"loopback"| SC2["Spice Sidecar (cache)"]
27 end
28
29 SC1 & SC2 -->|"Arrow Flight (gRPC)"| Cluster
30
31 subgraph Cluster["Spice Cluster or Spice Cloud"]
32 direction LR
33 S1["Spice Node 1"]
34 S2["Spice Node 2"]
35 S3["Spice Node 3"]
36 end
37
38 Cluster --> Sources["Data Sources (S3, PostgreSQL, Databricks, ...)"]
39```
40
41Each sidecar is configured declaratively via a `spicepod.yaml` — the datasets, views, acceleration engines, search indices, and AI models it manages. Sidecars start in seconds, typically run on a few hundred megabytes of memory, and scale horizontally with application pods: scale a deployment from 5 to 50 replicas and 50 sidecars come up automatically, each materializing only the working set its spicepod declares. Sidecars never talk to each other; they only talk to the cluster.
42
43For the engineering walkthrough of this pattern, including a request-path example and FAQ, see [Localhost Latency at Scale: The Spice Cluster-Sidecar Architecture](https://spice.ai/blog/cluster-sidecar-architecture).
44
45## How it works
46
47### The sidecar as a sandbox
48
49The most important property of the sidecar is isolation, not just latency. The sidecar is the only data-plane surface the application touches:
50
51- **Scoped working set.** A sidecar's `spicepod.yaml` declares exactly which datasets, views, and search indices the application may query. Anything not declared is physically absent from the catalog — not filtered by a policy, not hidden by a row-level rule. Compare this with row-level security, where the underlying data is still present and a single policy misconfiguration can expose it. Even a perfectly crafted prompt injection cannot query a table that is not in the catalog.
52- **No origin credentials in the application.** The application connects to its sidecar with a local token. The sidecar connects to the cluster over Arrow Flight. Only the cluster holds credentials for Postgres, Snowflake, Databricks, S3, Kafka, and the rest. Compromising an application pod cannot leak origin credentials, because the pod never had them.
53- **Narrow network surface.** The application's only outbound data dependency is the loopback interface. Network policy can pin the sidecar's egress to the cluster endpoint only.
54- **Per-application data views.** Sidecars can be specialized per application class or per tenant. A customer-service agent, a fraud-review agent, and an internal dashboard can each run with different spicepods pointing at different slices of the same cluster. This is physical isolation, not policy-based filtering on a shared database. See [Multi-Tenant AI Agents](/docs/use-cases/ai/multi-tenant-agents).
55- **Bounded resource use.** A rogue query plan or runaway loop exhausts the sidecar's local memory and CPU budget, not the cluster's and not the origin database's.
56- **Local inference.** Sidecars can serve [LLM inference and tool calls](../../features/large-language-models) on loopback so sensitive prompts and retrieved context stay in the pod. Heavier models can still be routed to the cluster.
57- **One audit point.** Every query, search, and inference call flows through the sidecar, so there is one place to log, rate-limit, and enforce policy.
58
59Data only moves along one path: **application → sidecar → (optionally) cluster → origin**. It never skips a tier.
60
61### Three latency tiers
62
63A request is served from the first tier that can answer it:
64
651. **Sidecar results cache** — repeat queries return from the in-memory [results cache](../../features/caching) in microseconds (`Results-Cache-Status: HIT`).
662. **Sidecar working set** — novel queries that fit the locally materialized dataset execute on-node in single-digit milliseconds (Arrow, DuckDB, or SQLite).
673. **Cluster delegation** — queries that exceed the working set are forwarded to the cluster. [Apache Ballista](../../features/distributed-query) distributes execution; [Spice Cayenne](../../components/data-accelerators/cayenne) accelerates large scans. The sidecar caches the result so subsequent reads drop back to tier 1.
68
69The cluster has its own results cache, so one sidecar's miss can become a cluster cache hit for every subsequent sidecar. Typical application-visible latency: p50 at tier 1, p95 at tier 2, tail at tier 3.
70
71Useful cache knobs in this topology:
72
73- `cache_key_type: plan` (default) shares a cache entry across semantically equivalent SQL, which matters for ORM-generated queries. `sql` is a faster, string-exact lookup.
74- `stale_while_revalidate_ttl` lets the sidecar serve a stale cached result immediately while a background refresh runs.
75- Clients can send `Cache-Control` directives per request (`no-cache`, `min-fresh`, `max-stale`, `only-if-cached`).
76- `encoding: zstd` typically cuts results-cache memory by 50–90%.
77
78### The cluster ingests once
79
80Refreshing from upstream sources, running Cayenne acceleration on large Iceberg tables, and keeping CDC streams connected are resource-intensive. Doing any of that *N* times for *N* sidecars is wasteful and often infeasible — source systems have connection limits, and per-pod CDC multiplies cloud cost with fleet size.
81
82The cluster ingests each dataset once and produces one authoritative materialization. Source load is bounded by cluster size, not fleet size. A new pod's sidecar pulls its working set from the cluster (or from an [acceleration snapshot](../../features/data-acceleration/snapshots)) rather than re-scanning the source, so new nodes are operational in seconds.
83
84Sidecars stay lightweight because they do **not** own ingest:
85
86- They start in seconds — important when application pods autoscale with traffic.
87- They typically run on a few hundred megabytes of memory.
88- Scaling from 5 to 50 replicas does **not** add 50 new connections to the source database.
89- Each sidecar materializes only the working set its spicepod declares, not a full copy of the warehouse. Full accelerated datasets stay on the cluster.
90
91### Query delegation over Arrow Flight
92
93Sidecar-to-cluster communication is [Arrow Flight](https://arrow.apache.org/docs/format/Flight.html) over gRPC. Results flow as Arrow record batches directly into the sidecar's query engine with no JSON or row-based serialization detour.
94
95The sidecar decides locally whether a query can be served from its working set. If not, it forwards and streams results back. The application sees one endpoint and one query. It never knows whether execution happened locally or whether Ballista fanned the query out across cluster executors.
96
97Configure delegation with the [Spice.ai Data Connector](../../components/data-connectors/spiceai): point a sidecar dataset at a Spice Cloud app (`spice.ai/<org>/<app>/datasets/<name>`) or at a self-hosted cluster (`spice.ai:https://cluster.example:50051`). Combine with local acceleration to materialize the hot working set; leave acceleration off to always delegate.
98
99Kubernetes is the most common place to run this topology, but it is not required. The architecture needs only a network path from each sidecar to the cluster. Sidecars run wherever the application runs — Kubernetes, a VPC, on-prem, or at the edge.
100
101### Acceleration snapshots
102
103The cluster-sidecar split maps onto [acceleration snapshots](../../features/data-acceleration/snapshots) as a single-writer / many-reader topology:
104
105- The **cluster** is the single writer (`snapshots: create_only`). After each refresh it uploads a snapshot to object storage. It never downloads snapshots on startup; it always refreshes from the source.
106- Each **sidecar** is a reader (`snapshots: bootstrap_only`). On startup — or when ephemeral NVMe is recycled — the sidecar downloads the most recent snapshot and is immediately ready. It never writes snapshots back.
107
108This avoids snapshot conflicts, keeps the cluster as the authoritative refresh point, and gives every sidecar a warm start from the same materialization. For CDC-backed datasets with large initial state, the difference between a snapshot bootstrap and a full re-sync can be seconds versus minutes.
109
110Use DuckDB or SQLite in `mode: file` on the sidecar when snapshots are in play — snapshots persist and restore the acceleration file itself. Heavyweight [Cayenne](../../components/data-accelerators/cayenne) acceleration stays on the cluster.
111
112For production Spicepod and Helm reference configurations, see [Read/Write Separation](/docs/deployment/read-write-separation).
113
114### Cache coherency is a refresh policy, not a protocol
115
116Sidecars pull from the cluster on a configurable interval using append or full refresh. They do **not** participate in a distributed invalidation protocol. A pull-based model with explicit refresh intervals makes staleness bounded, predictable, and debuggable.
117
118For workloads that need sub-second freshness, the cluster consumes [CDC streams](../../features/cdc) (Postgres logical replication, DynamoDB Streams, Debezium, Kafka) once, and the sidecars pull the resulting accelerated dataset on a short interval. That gives near-real-time propagation without a fleet-wide invalidation bus.
119
120If the cluster is temporarily unavailable — a rolling upgrade, a network blip, a zone event — sidecars keep serving cached data and their accelerated working sets. Refreshes pause and resume when connectivity returns. The blast radius of a cluster incident is slightly staler data, not application downtime.
121
122## Example Spicepods
123
124The cluster materializes and accelerates datasets, consumes CDC, and exposes a results cache. It is the only tier with origin credentials.
125
126```yaml
127version: v1
128kind: Spicepod
129name: platform-cluster
130
131runtime:
132 caching:
133 sql_results:
134 enabled: true
135 max_size: 4GiB
136 item_ttl: 1m
137 stale_while_revalidate_ttl: 30s
138 encoding: zstd
139
140snapshots:
141 enabled: true
142 location: s3://my-bucket/spice-snapshots/
143 params:
144 s3_auth: iam_role
145
146datasets:
147 - from: postgres:public.orders
148 name: orders
149 acceleration:
150 enabled: true
151 engine: cayenne
152 mode: file
153 refresh_mode: changes
154 snapshots: create_only
155 primary_key: id
156 on_conflict:
157 id: upsert
158
159 - from: s3://lakehouse/events/
160 name: events
161 params:
162 file_format: parquet
163 acceleration:
164 enabled: true
165 engine: cayenne
166 mode: file
167 refresh_mode: append
168 refresh_check_interval: 15m
169 snapshots: create_only
170```
171
172The sidecar is much smaller. It pulls from the cluster, keeps a working-set engine local, and caches results. It holds no origin credentials.
173
174```yaml
175version: v1
176kind: Spicepod
177name: app-sidecar
178
179runtime:
180 caching:
181 sql_results:
182 enabled: true
183 max_size: 256MiB
184 item_ttl: 30s
185 stale_while_revalidate_ttl: 30s
186
187snapshots:
188 enabled: true
189 location: s3://my-bucket/spice-snapshots/
190 bootstrap_on_failure_behavior: warn
191 params:
192 s3_auth: iam_role
193
194datasets:
195 - from: spice.ai/<your-org>/<your-app>/datasets/orders
196 name: orders
197 acceleration:
198 enabled: true
199 engine: duckdb
200 mode: file
201 refresh_mode: append
202 refresh_check_interval: 10s
203 snapshots: bootstrap_only
204
205 - from: spice.ai/<your-org>/<your-app>/datasets/events
206 name: events
207 # No local acceleration — delegate to the cluster on demand.
208 # Queries that match recent hot events still hit the results cache.
209```
210
211For a self-hosted cluster, replace the `from:` URIs with `spice.ai:https://cluster.example:50051` and set `name:` to the upstream table name. See the [Spice.ai Data Connector](../../components/data-connectors/spiceai) for Cloud and self-hosted URI formats.
212
213## A request path
214
2151. The application (or agent) queries its sidecar on `localhost:8090`: `SELECT ... FROM orders WHERE tenant_id = $1 ORDER BY created_at DESC LIMIT 20`.
2162. The sidecar checks its results cache. Hit → return in microseconds.
2173. On a miss, the sidecar plans against its local catalog. `orders` is materialized locally (refreshed from the cluster every 10 seconds). It executes against DuckDB, returns in single-digit milliseconds, and populates the results cache. Origin Postgres sees no traffic.
2184. A follow-up hybrid search over a multi-gigabyte `tickets` index is not in the sidecar catalog as a local acceleration. The sidecar opens an Arrow Flight stream to the cluster. Ballista distributes the search; Cayenne segment statistics prune most files.
2195. Results stream back as Arrow record batches. The sidecar returns them to the application and caches them. A later LLM call can be served by a small local model on loopback, with the large-model call delegated to the cluster.
2206. The cluster independently caches its own results. The next replica that asks the same question gets it from the cluster cache.
221
222The application sees one endpoint, one wire format, and one latency distribution.
223
224## Benefits
225
226- **Kubernetes-native** — designed to run on Kubernetes, leveraging pod-level sidecars with cluster-level orchestration. The same topology also works outside Kubernetes wherever the application has a network path to the cluster.
227- Sub-millisecond reads via sidecar caching on loopback, with centralized data management in the cluster.
228- Structural isolation — sidecars expose a scoped catalog, hold no origin credentials, and bound resource use per pod.
229- Transparent query delegation — sidecars automatically route queries beyond their cached working set to the cluster.
230- Sidecars remain lightweight — only a working set and a results cache, no ingest or heavy acceleration overhead.
231- Cluster (or Spice Cloud) handles complex operations: data ingestion, [Spice Cayenne](../../components/data-accelerators/cayenne) acceleration, [distributed query](../../features/distributed-query), hybrid search, and refresh from sources.
232- Works with both self-managed Spice clusters and the managed [Spice Cloud Platform](./hosted) as the centralized backend. The Spice Cloud cluster-sidecar model is the most common production topology.
233- Sidecars can run anywhere — in your VPC, on-premises, at the edge, or in any Kubernetes cluster — while connecting securely to the managed cluster.
234- Horizontal scalability — add sidecars without increasing load on data sources.
235- Resilience — sidecars serve cached data even if the cluster is temporarily unavailable.
236- Secure by default — mTLS encryption across all sidecar-to-cluster communication, with data encrypted at rest and in transit.
237
238## Considerations
239
240- More complex deployment structure requiring both sidecar and cluster infrastructure. [Spice Cloud](./hosted) reduces this burden by managing the cluster.
241- Cache coherency — sidecars must be configured with appropriate refresh intervals or TTLs to balance freshness with performance. There is no distributed invalidation bus.
242- Requires a Spice cluster deployment or [Spice Cloud Platform](./hosted) subscription ([Spice.ai Enterprise](https://spice.ai/enterprise) for self-managed clustering with SSO, RBAC, and audit logs).
243- Network connectivity between sidecars and the cluster must be reliable for cache refreshes and query delegation.
244
245## Use This Approach When
246
247- Applications or AI agents require sub-millisecond reads, unified retrieval (SQL, full-text, vector, hybrid), and inference on `localhost`, without giving the application direct database credentials.
248- You need a clear isolation boundary between application code and origin data systems — scoped catalogs, no origin credentials in the pod, and one audit point per replica.
249- Multiple application instances need fast access to the same datasets without each independently querying data sources.
250- Reducing load on upstream data sources is a priority — the cluster ingests once, sidecars cache locally.
251- The system benefits from separating the caching tier (sidecars) from the data processing tier (cluster).
252- Workloads span both real-time operational queries and large-scale analytical queries on the same data (for example, an operational data lakehouse on S3/Iceberg).
253- You already run Kubernetes sidecars for other concerns (service mesh, logging, config).
254
255## Not Ideal When
256
257- The application is simple with a single instance and no isolation requirement — the overhead of both sidecar and cluster infrastructure isn't justified. Consider [Sidecar](./sidecar) or [Microservice](./microservice).
258- All queries are batch or analytical with relaxed latency requirements — a [Microservice](./microservice) deployment is simpler and sufficient.
259- Network connectivity between sidecars and the cluster is unreliable — query delegation and cache refreshes will fail, leading to stale data. Consider standalone [Sidecar](./sidecar) deployments with direct source access.
260
261## Example Use Case
262
263A multi-tenant SaaS platform runs an AI support agent. Each tenant's agent pods include a Spice sidecar that materializes that tenant's working set (recent tickets, active customer records, the last 7 days of events, the tenant's private knowledge-base embeddings) into a local DuckDB + vector index. The spicepod exposes exactly those datasets — and no others — to the agent. The sidecar holds no credentials for Postgres, S3, or Snowflake.
264
265Agent turns hit the sidecar on `localhost` and return in single-digit milliseconds; repeat retrievals return from the results cache in microseconds. Behind the sidecars, a Spice cluster (often Spice Cloud) ingests from PostgreSQL via CDC, S3 Iceberg tables, and Databricks. Cayenne acceleration and refresh schedules run on the cluster. When an agent asks a broader question — "summarize this tenant's churn signal across the last 12 months" — the sidecar recognizes the query exceeds its working set and delegates over Arrow Flight. The cluster executes it, the sidecar caches the result, and subsequent turns for that tenant return in microseconds.
266
267The origin Postgres sees exactly one consumer (the cluster). If a replica is compromised, the attacker gets a loopback endpoint scoped to that tenant's working set, not database credentials and not a query interface to the whole warehouse.
268
269The same pattern — single ingestion path, per-pod sandboxing, tiered latency, one isolation boundary — works for any application fleet, not just agents: microservices serving real-time dashboards, services powering search, or internal tools querying operational data.
270
271## See also
272
273- [Localhost Latency at Scale: The Spice Cluster-Sidecar Architecture](https://spice.ai/blog/cluster-sidecar-architecture) — engineering walkthrough, request-path example, and FAQ.
274- [Read/Write Separation](/docs/deployment/read-write-separation) — production guide for splitting ingest from reads using shared acceleration snapshots, including Spicepod and Helm reference configurations.
275- [Spice.ai Data Connector](../../components/data-connectors/spiceai) — how sidecars federate to a cluster or to Spice Cloud over Arrow Flight.
276- [Caching](../../features/caching) — results cache, stale-while-revalidate, and `Cache-Control`.
277- [Acceleration Snapshots](../../features/data-acceleration/snapshots) — warm-start file-mode accelerations from object storage.
278- [Distributed Query](../../features/distributed-query) — Apache Ballista schedulers and executors on the cluster.
279- [Spice Cayenne](../../components/data-accelerators/cayenne) — cluster-side acceleration for datasets beyond 1 TB.
280- [Multi-Tenant AI Agents](/docs/use-cases/ai/multi-tenant-agents) — per-tenant isolation patterns that pair with this architecture.
281