Delta Lake data connector enables SQL queries from Delta Lake tables.
datasets : - from : delta_lake:s3://my_bucket/path/to/s3/delta/table/ name : my_delta_lake_table params : delta_lake_aws_access_key_id : ${secrets:aws_access_key_id} delta_lake_aws_secret_access_key : ${secrets:aws_secret_access_key} Configuration
fromThe from field for the Delta Lake connector takes the form of delta_lake:path where path is any supported path, either local or to a cloud storage location. See the examples section below.
nameThe dataset name. This will be used as the table name within Spice.
Example:
datasets : - from : delta_lake:s3://my_bucket/path/to/s3/delta/table/ name : cool_dataset params : ... SELECT COUNT ( * ) FROM cool_dataset; +----------+ | count(* ) | +----------+ | 6001215 | +----------+ The dataset name cannot be a reserved keyword .
paramsUse the secret replacement syntax to reference a secret, e.g. ${secrets:aws_access_key_id}.
Delta Lake object store parameters
AWS S3
Azure Blob
Note
Note
One of the following auth values must be provided for Azure Blob:
delta_lake_azure_storage_account_key,
delta_lake_azure_storage_client_id, delta_lake_azure_storage_client_secret, and delta_lake_azure_storage_tenant_id, or
delta_lake_azure_storage_sas_key.
Google Storage (GCS)
Examples Delta Lake + Local - from : delta_lake:/path/to/local/delta/table # A local filesystem path to a Delta Lake table name : my_delta_lake_table Delta Lake + S3 - from : delta_lake:s3://my_bucket/path/to/s3/delta/table/ # A reference to a table in S3 name : my_delta_lake_table params : delta_lake_aws_region : us-west-2 # Optional delta_lake_aws_access_key_id : ${secrets:aws_access_key_id} delta_lake_aws_secret_access_key : ${secrets:aws_secret_access_key} delta_lake_aws_endpoint : s3.us-west-2.amazonaws.com # Optional Delta Lake + MinIO - from : delta_lake:s3://my_bucket/path/to/s3/delta/table/ # A reference to a table in MinIO name : my_delta_lake_table params : delta_lake_aws_region : us-east-1 # Best practice for MinIO delta_lake_aws_access_key_id : ${secrets:aws_access_key_id} delta_lake_aws_secret_access_key : ${secrets:aws_secret_access_key} delta_lake_aws_endpoint : http://localhost:9000 # MinIO Endpoint delta_lake_aws_allow_http : true Delta Lake + Azure Blob - from : delta_lake:abfss://my_container@my_account.dfs.core.windows.net/path/to/azure/delta/table/ # A reference to a table in Azure Blob name : my_delta_lake_table params : # Account Name + Key delta_lake_azure_storage_account_name : my_account delta_lake_azure_storage_account_key : ${secrets:my_key} # OR Service Principal + Secret delta_lake_azure_storage_client_id : my_client_id delta_lake_azure_storage_client_secret : ${secrets:my_secret} delta_lake_azure_storage_tenant_id : ${secrets:my_tenant_id} # OR SAS Key delta_lake_azure_storage_sas_key : my_sas_key Delta Lake + Google Storage params : delta_lake_google_service_account : /path/to/service-account.json Types The table below shows the Delta Lake data types supported, along with the type mapping to Apache Arrow types in Spice.
Column Mapping Delta Lake tables that use column mapping with columnMapping.mode = name or columnMapping.mode = id are supported. Tables in either mode store data in Parquet files under opaque physical column names (e.g., col-b70f7585-…) that differ from the logical names exposed by the table schema. Spice resolves logical-to-physical names at read time, including for partitions, predicate pushdown, and nested Struct, List, and Map field renames.
Limitations
Delta Lake connector does not support reading Delta tables with the V2Checkpoint feature enabled. To use the Delta Lake connector with such tables, drop the V2Checkpoint feature by executing the following command:
ALTER TABLE <table-name> DROP FEATURE v2Checkpoint [TRUNCATE HISTORY]; For more details on dropping Delta table features, refer to the official documentation: Drop Delta table features
Secrets Spice integrates with multiple secret stores to help manage sensitive data securely. For detailed information on supported secret stores, refer to the secret stores documentation . Additionally, learn how to use referenced secrets in component parameters by visiting the using referenced secrets guide .
Cookbook