title: 'Kubernetes - Local NVMe Storage' sidebar_label: 'Local NVMe Storage' sidebar_position: 4 description: 'Step-by-step guide to giving Spice acceleration files and query spill a local NVMe volume on Kubernetes — EKS, GKE, AKS, and self-hosted clusters.' tags:
Spice accelerations and query spill run at the per-I/O latency of the disk beneath them, and the fastest disk a Kubernetes node has is its local NVMe. This guide walks through giving a Spice deployment a local NVMe volume: which nodes to use, how to mount the disk, how to turn it into a PersistentVolume, and how to point the Helm chart and the Spicepod at it. When you are done, /data inside the Spice container is a directory on the node's NVMe, and both acceleration files and spill live there.
Why this matters — and what to do when a node's NVMe is not available — is covered in Storage (opens in a new tab) in the Performance Tuning guide. This page is the how-to.
:::tip[Quick path]
| Where Spice runs | Fastest route |
|---|---|
| GKE | Create the node pool with --local-nvme-ssd-block, apply Google's two provisioner manifests, then deploy Spice with stateful.storageClass: nvme-ssd-block (Step 2 (opens in a new tab), Step 4 (opens in a new tab)). |
| EKS with Karpenter | Set instanceStorePolicy: RAID0 on the EC2NodeClass and use an emptyDir on NVMe-backed ephemeral storage (opens in a new tab). |
| EKS managed or self-managed nodes | Mount the disks with nodeadm (localStorage.strategy: Mount), install the static provisioner, deploy with stateful.storageClass: local-storage (Step 2 (opens in a new tab), Step 3 (opens in a new tab), Step 4 (opens in a new tab)). |
| AKS | Use an NVMe VM size, mount the disks, install the static provisioner (Step 2 (opens in a new tab), Step 3 (opens in a new tab), Step 4 (opens in a new tab)). |
| Self-hosted / bare metal | Mount the NVMe under /mnt/disks, install the static provisioner, deploy with stateful.storageClass: local-storage (Step 2 (opens in a new tab), Step 3 (opens in a new tab), Step 4 (opens in a new tab)). |
| A single node, Kind, or a lab | Create one hand-made local PersistentVolume (opens in a new tab) and deploy with stateful.storageClass: local-storage. |
:::
Four facts explain everything that follows:
local, pinned to that node with a node affinity. Nothing can move it; a pod that uses it runs on that node.volumeBindingMode: WaitForFirstConsumer, so the scheduler first picks a node for the pod and then binds the claim to a free local volume on that node, instead of binding first and pinning the pod to wherever the volume happens to be.stateful.enabled: true renders a StatefulSet with one PersistentVolumeClaim per replica. The claim outlives pod restarts and rollouts, so the acceleration on the node survives them.Something has to turn a disk into a PersistentVolume. The Local Volume Static Provisioner (opens in a new tab) is the standard tool: a DaemonSet that watches a directory on each node (/mnt/disks by convention) and publishes every filesystem mounted beneath it as one PersistentVolume. GKE and Azure ship their own variants of the same idea.
Local NVMe is a property of the instance type, not something a cluster adds later:
| Platform | Instance types with local NVMe |
|---|---|
| AWS | Storage-optimized i4i, i7ie, i8g, and the d-suffixed general-purpose, compute, and memory families (m6id, m7gd, c7gd, r7gd, …). The instance store (opens in a new tab) volumes arrive unformatted at launch. |
| Azure | Storage-optimized Lsv3 / Lasv3 (opens in a new tab) (Standard_L8s_v3 and larger) and the v6 families with ephemeral NVMe data disks. Prefer a size with two or more NVMe disks, or a managed OS disk, so the OS does not claim the only NVMe — see AKS NVMe best practices (opens in a new tab). |
| GCP | Machine types with Local SSD (opens in a new tab): the -lssd variants (c3-standard-*-lssd, n2-standard-*-lssd) and the z3 series. Local SSD needs n1-standard-1 or larger; the default e2-medium does not support it. |
| Self-hosted | Any node with an NVMe device (lsblk shows it as nvme0n1, nvme1n1, …). |
Give those nodes a label so Spice — and only Spice — schedules onto them. The examples below use local-nvme=true; add a taint as well if the node pool should be reserved:
The provisioner in Step 3 publishes mounted filesystems, so each node needs its NVMe formatted (ext4 or xfs) and mounted under a discovery directory, one mount point per volume — /mnt/disks/nvme0, /mnt/disks/nvme1, and so on. Mount with noatime. How to arrange that depends on the platform.
Karpenter. Set the instance store policy on the EC2NodeClass:
Karpenter formats the instance store volumes as a RAID 0 array, mounts it at /mnt/k8s-disks/0, and uses it as the filesystem for the kubelet and containerd, so the node's allocatable ephemeral-storage becomes the size of the instance store — see spec.instanceStorePolicy (opens in a new tab). With the disks serving as ephemeral storage, the simplest way to use them is an emptyDir — skip to Alternative A (opens in a new tab). To publish PersistentVolumes instead, bind-mount a subdirectory of /mnt/k8s-disks/0 into /mnt/disks from user data and continue with Step 3.
Managed and self-managed node groups (Amazon Linux 2023). nodeadm mounts the instance store for you. In the node's NodeConfig user data:
Mount formats each instance store volume and mounts it under /mnt/k8s-disks/; use that path as the provisioner's hostDir in Step 3. The RAID0 strategy instead combines the disks into one filesystem for the kubelet and containerd, which is the emptyDir route of Alternative A. See the nodeadm API reference (opens in a new tab).
Bottlerocket. Bottlerocket's bootstrap commands apiclient ephemeral-storage init and apiclient ephemeral-storage bind --dirs /var/lib/containerd /var/lib/kubelet /var/log/pods put the kubelet and containerd directories on the instance store, which is again the emptyDir route; Karpenter's RAID0 policy does the same on Bottlerocket nodes. See Bottlerocket bootstrap commands (opens in a new tab).
Create the node pool with Local SSD attached as raw block devices:
First- and second-generation machine types take a disk count (--local-nvme-ssd-block count=N); third- and fourth-generation types carry a fixed number of Local SSDs, so the flag stands alone. GKE labels the nodes cloud.google.com/gke-local-nvme-ssd=true.
Then apply the two manifests from Google's raw block Local SSD guide (opens in a new tab): gke-daemonset-raid-disks.yaml assembles the Local SSDs into a RAID 0 array, formats it, and mounts it; gke-nvme-ssd-block-raid.yaml installs the static provisioner and a StorageClass named nvme-ssd-block with WaitForFirstConsumer binding. That covers Step 3, so continue with Step 4 (opens in a new tab) using stateful.storageClass: nvme-ssd-block.
:::warning GKE node upgrades and auto-repair delete the underlying VM, and Local SSD data with it. Treat every upgrade as a node replacement — see Plan for the data being ephemeral (opens in a new tab). :::
Add a node pool on an NVMe VM size:
AKS does not format the ephemeral NVMe disks. Two ways to make them usable:
/mnt/disks — Google's gke-daemonset-raid-disks.yaml (opens in a new tab) is a good template (it finds the NVMe devices, builds a RAID 0 array, runs mkfs.ext4, and mounts the result). Then continue with Step 3 (opens in a new tab).local-csi (provisioner localdisk.csi.acstor.io, WaitForFirstConsumer), striped across the VM's NVMe disks. It hands out generic ephemeral volumes by default; a PersistentVolumeClaim needs the annotation localdisk.csi.acstor.io/accept-ephemeral-storage: "true" on its claim template. The Spice Helm chart's claim template cannot carry that annotation, so use local-csi with a StatefulSet you write yourself or with the Spice.ai Enterprise Operator's SpicepodSet (opens in a new tab), or mount it through the chart's volumes as a generic ephemeral volume (pod-lifetime, like Alternative A).On each NVMe node, once per disk:
Use lsblk -o NAME,SIZE,MODEL,MOUNTPOINT to identify the device and make sure it is not the root disk. To split one large NVMe into several PersistentVolumes, mount it once elsewhere and bind-mount subdirectories into /mnt/disks — the provisioner accepts bind mounts, though volumes sharing a disk do not get capacity isolation from each other.
Skip this step on GKE (Google's manifest includes it) and for the emptyDir route.
Create the StorageClass. kubernetes.io/no-provisioner means "volumes are created out of band"; WaitForFirstConsumer is required for local volumes:
Install the provisioner with values that name the class and the discovery directory from Step 2:
On EKS nodes mounted by nodeadm, set hostDir: /mnt/k8s-disks.
Check that the volumes appeared — one Available PersistentVolume per mounted disk, each pinned to its node:
If nothing shows up, check the provisioner's DaemonSet logs on an NVMe node: the usual causes are a disk that is not mounted under hostDir, or a nodeSelector that does not match the node's labels.
With reclaimPolicy: Delete, deleting a claim makes the provisioner wipe the volume's contents and publish the PersistentVolume again for the next claim.
Enable the StatefulSet in the Helm chart, name the StorageClass, pin the pods to the NVMe nodes, and put every acceleration path plus the spill directory under the mount:
Notes on the values:
nodeSelector; a replica with nowhere to bind stays Pending.stateful.size must fit the volume. A local PersistentVolume's capacity is the size of the mounted filesystem; a claim larger than any free volume never binds.runtime.query.temp_directory at startup when its parent exists (/data/tmp under the /data mount needs no init container); keep it on the NVMe with the data rather than leaving it at the default, which lands on the node's root disk. Size the volume for the data plus 2–4× the largest spillable input — see Spill-to-Disk (opens in a new tab).65534 with fsGroup: 65534, and local PersistentVolumes honor fsGroup, so the mount is writable without further setup. A hostPath volume does not apply fsGroup; if you use one, chown 65534:65534 the directory on the node.SpicepodSet (opens in a new tab) manages the per-replica StatefulSets and claims for you; give it the same StorageClass.Spice detects the storage tier itself. For a Cayenne dataset, the cayenne_data_storage_class and cayenne_metastore_storage_class gauges report 0 for local SSD (1 is network-attached, 2 tmpfs, 3 unknown), and the Cayenne auto-tuned config startup log line names the detected classes. If a table reports 3 on a node you know has NVMe — a container runtime that hides the block-device metadata, for example — set storage_profile: local_ssd on the acceleration; see acceleration.storage_profile (opens in a new tab) and Storage tier detection (opens in a new tab).
A local volume lives exactly as long as its node. Decide up front how a replacement pod gets its data back:
Refresh from the source — the default. Nothing to configure; the new pod re-materializes every dataset. Fine when a refresh is fast enough for your recovery objective.
Bootstrap from snapshots (Spice.ai Enterprise) — the pattern this storage was designed for. Each refresh uploads the acceleration file to object storage, and a pod that starts with an empty /data downloads the newest snapshot instead of refreshing:
See Acceleration Snapshots (opens in a new tab), and Read/Write Separation (opens in a new tab) for the pattern where one ingesting cluster writes snapshots that many read replicas on local NVMe bootstrap from.
Cayenne on S3 Express One Zone — keeps Cayenne data files off the node entirely, at single-digit-millisecond latency; see Storage (opens in a new tab).
Two operational details of local volumes on node replacement:
Pending. The StatefulSet's claim still references a PersistentVolume whose node no longer exists, so the replacement pod cannot schedule. Delete the claim (and the stale PersistentVolume) and the StatefulSet recreates the pod with a fresh claim on a live node. The provisioner project's node cleanup controller (opens in a new tab) automates exactly this.reclaimPolicy: Delete, the provisioner then wipes and republishes them).emptyDir on NVMe-backed ephemeral storageWhen the node's own ephemeral storage is on the NVMe — Karpenter's instanceStorePolicy: RAID0, nodeadm's RAID0 strategy, Bottlerocket's ephemeral-storage bootstrap, or a GKE node pool created with --ephemeral-storage-local-ssd — a plain emptyDir already lands on NVMe, with no PersistentVolumes to manage. The trade-off is that the volume lives only as long as the pod: every restart or rollout starts from empty, so this route suits read replicas that bootstrap from snapshots, or datasets that refresh quickly.
Use the chart's volumes and volumeMounts passthroughs with stateful.enabled left at false:
Never use emptyDir with medium: Memory here: its files count against the container's memory limit, which turns a spill into an OOM kill.
For a single node, a Kind cluster, or a lab, skip the provisioner and declare the volume yourself. The node affinity is mandatory:
Create the local-storage StorageClass from Step 3, apply this PersistentVolume, and deploy with the Step 4 values. With Retain, the volume is not reused after its claim is deleted until you clear the directory and remove the claimRef from the PersistentVolume.
| Symptom | Likely cause | Resolution |
|---|---|---|
Pod Pending with didn't find available persistent volumes to bind | No free local volume on a node that matches the nodeSelector, or the claim is larger than every free volume | kubectl get pv — check capacity, status, and node; verify the disk is mounted under hostDir and the node carries the label. |
Pod Pending after a node was replaced | The claim is bound to a volume on the vanished node. | Delete the claim and stale volume; consider the node cleanup controller. |
permission denied writing to /data | hostPath volume, or a directory owned by root on a volume that does not apply fsGroup. | Use a local PersistentVolume, or chown 65534:65534 the directory on the node. |
PersistentVolume stuck in Released | Cleanup is running, or the disk is gone (node replaced). | Wait for the provisioner to republish it; if the node is gone, delete the PersistentVolume object. |
cayenne_data_storage_class reports 3 (unknown) on an NVMe node | The container cannot read the block-device metadata under /sys. | Set storage_profile: local_ssd on the acceleration. |
| Volume fills during large queries | Spill (/data/tmp) sharing the volume with the data, sized without spill headroom. | Size the volume for data plus 2–4× the largest spillable input; a spill that runs out of space fails the query with ResourcesExhausted. |
| Data gone after a cluster upgrade | Expected: upgrades replace nodes and their NVMe. | Enable snapshots, or accept a refresh; on GKE, schedule upgrades deliberately. |