title: 'Data Ingestion' sidebar_label: 'Data Ingestion' description: 'Learn how to ingest data in Spice.' sidebar_position: 6 pagination_prev: null pagination_next: null tags:
Data can be ingested by the Spice runtime into a Data Connector using the following methods:
Spice supports writing data to compatible data connectors using standard SQL INSERT INTO syntax.
Data connectors that support write operations are tagged as write:
To enable write operations, configure your dataset or catalog with read_write access:
For more details on the INSERT statement syntax, see the SQL INSERT documentation.
By default, the runtime exposes an OpenTelemetry (OTEL) endpoint at grpc://127.0.0.1:50052 for the OTEL data ingestion.
OTEL metrics will be inserted into datasets with matching names (metric name = dataset name) and optionally replicated to the dataset source.
Spice.ai OSS includes built-in data ingestion support, allowing the collection of the latest data from edge nodes for use in subsequent queries. This feature eliminates the need for additional ETL pipelines and enhances the speed of the feedback loop.
For example, consider CPU usage anomaly detection. When CPU metrics are sent to the Spice OpenTelemetry endpoint, the loaded machine learning model can use the most recent observations for inferencing and provide recommendations to the edge node. This process occurs quickly on the edge itself, within milliseconds, and without generating network traffic.
Additionally, Spice will periodically replicate the data to the data connector for further use.
Data Quality: Use Spice SQL capabilities to transform and cleanse ingested edge data, ensuring high-quality inputs.
Data Security: Evaluate data sensitivity and secure network connections between the edge and data connector when replicating data for further use. Implement encryption, access controls, and secure protocols.
Start Spice with the following dataset:
Start telegraf with the following config:
SMART data will be available in the smart_attribute_raw_value dataset in Spice.ai OSS and replicated to the coolorg.smart.drive_stats dataset in Spice.ai Cloud.
:::warning[Current Limitations]
:::
