# Delta Lake Data Connector

Query Delta Lake tables with SQL from a local path or from S3, MinIO, Azure Blob Storage, or Google Cloud Storage.

Delta Lake data connector enables SQL queries from [Delta Lake](https://delta.io/) tables.

```yaml
datasets:
  - from: delta_lake:s3://my_bucket/path/to/s3/delta/table/
    name: my_delta_lake_table
    params:
      delta_lake_aws_access_key_id: ${secrets:aws_access_key_id}
      delta_lake_aws_secret_access_key: ${secrets:aws_secret_access_key}
```

## Configuration

### `from`

The `from` field for the Delta Lake connector takes the form of `delta_lake:path` where `path` is any supported path, either local or to a cloud storage location. See the [examples](https://spice.ai/docs/building-blocks/data-connectors/delta-lake#examples) section below.

### `name`

The dataset name. This will be used as the table name within Spice.

Example:

```yaml
datasets:
  - from: delta_lake:s3://my_bucket/path/to/s3/delta/table/
    name: cool_dataset
    params: ...
```

```sql
SELECT COUNT(*) FROM cool_dataset;
```

```shell
+----------+
| count(*) |
+----------+
| 6001215  |
+----------+
```

The dataset name cannot be a [reserved keyword](https://spiceai.org/docs/reference/spicepod/keywords).

### `params`

Use the [secret replacement syntax](https://spiceai.org/docs/components/secret-stores) to reference a secret, e.g. `${secrets:aws_access_key_id}`.

| Parameter Name | Description |
| - | - |
| `client_timeout` | Optional. Specifies timeout for object store operations. No default; uses the underlying HTTP client timeout when not set. E.g. `client_timeout: 60s` |

## Delta Lake object store parameters

### AWS S3

| Parameter Name | Description |
| - | - |
| `delta_lake_aws_region` | Optional. The AWS region for the S3 object store. E.g. `us-west-2`. |
| `delta_lake_aws_access_key_id` | The access key ID for the S3 object store. |
| `delta_lake_aws_secret_access_key` | The secret access key for the S3 object store. |
| `delta_lake_aws_session_token` | Optional. The AWS session token for S3 object store. |
| `delta_lake_aws_endpoint` | Optional. The endpoint for the S3 object store. E.g. `s3.us-west-2.amazonaws.com`. |
| `delta_lake_aws_allow_http` | Optional. Enables insecure HTTP connections to `delta_lake_aws_endpoint`. Defaults to `false`. |

### Azure Blob

> **Note**
>
> **Note**
>
> **One** of the following auth values must be provided for Azure Blob:
>
> - `delta_lake_azure_storage_account_key`,
>
> - `delta_lake_azure_storage_client_id`, `delta_lake_azure_storage_client_secret`, and `delta_lake_azure_storage_tenant_id`, or
>
> - `delta_lake_azure_storage_sas_key`.
>
> | Parameter Name | Description |
> | - | - |
> | `delta_lake_azure_storage_account_name` | The Azure Storage account name. |
> | `delta_lake_azure_storage_account_key` | The Azure Storage master key for accessing the storage account. |
> | `delta_lake_azure_storage_client_id` | The service principal client id for accessing the storage account. |
> | `delta_lake_azure_storage_client_secret` | The service principal client secret for accessing the storage account. |
> | `delta_lake_azure_storage_tenant_id` | The service principal tenant id for accessing the storage account. |
> | `delta_lake_azure_storage_sas_key` | The shared access signature key for accessing the storage account. |
> | `delta_lake_azure_storage_endpoint` | Optional. The endpoint for the Azure Blob storage account. |
>
> ### Google Storage (GCS)
>
> | Parameter Name | Description |
> | - | - |
> | `delta_lake_google_service_account` | Filesystem path to the Google service account JSON key file. |
>
> ## Examples
>
> ### Delta Lake + Local
>
> ```yaml
> - from: delta_lake:/path/to/local/delta/table # A local filesystem path to a Delta Lake table
>   name: my_delta_lake_table
> ```
>
> ### Delta Lake + S3
>
> ```yaml
> - from: delta_lake:s3://my_bucket/path/to/s3/delta/table/ # A reference to a table in S3
>   name: my_delta_lake_table
>   params:
>     delta_lake_aws_region: us-west-2 # Optional
>     delta_lake_aws_access_key_id: ${secrets:aws_access_key_id}
>     delta_lake_aws_secret_access_key: ${secrets:aws_secret_access_key}
>     delta_lake_aws_endpoint: s3.us-west-2.amazonaws.com # Optional
> ```
>
> ### Delta Lake + MinIO
>
> ```yaml
> - from: delta_lake:s3://my_bucket/path/to/s3/delta/table/ # A reference to a table in MinIO
>   name: my_delta_lake_table
>   params:
>     delta_lake_aws_region: us-east-1 # Best practice for MinIO
>     delta_lake_aws_access_key_id: ${secrets:aws_access_key_id}
>     delta_lake_aws_secret_access_key: ${secrets:aws_secret_access_key}
>     delta_lake_aws_endpoint: http://localhost:9000 # MinIO Endpoint
>     delta_lake_aws_allow_http: true
> ```
>
> ### Delta Lake + Azure Blob
>
> ```yaml
> - from: delta_lake:abfss://my_container@my_account.dfs.core.windows.net/path/to/azure/delta/table/ # A reference to a table in Azure Blob
>   name: my_delta_lake_table
>   params:
>     # Account Name + Key
>     delta_lake_azure_storage_account_name: my_account
>     delta_lake_azure_storage_account_key: ${secrets:my_key}
>
>     # OR Service Principal + Secret
>     delta_lake_azure_storage_client_id: my_client_id
>     delta_lake_azure_storage_client_secret: ${secrets:my_secret}
>     delta_lake_azure_storage_tenant_id: ${secrets:my_tenant_id}
>
>     # OR SAS Key
>     delta_lake_azure_storage_sas_key: my_sas_key
> ```
>
> ### Delta Lake + Google Storage
>
> ```yaml
> params:
>   delta_lake_google_service_account: /path/to/service-account.json
> ```
>
> ## Types
>
> The table below shows the Delta Lake data types supported, along with the type mapping to Apache Arrow types in Spice.
>
> | Delta Lake Type | Arrow Type |
> | - | - |
> | `String` | `Utf8` |
> | `Long` | `Int64` |
> | `Integer` | `Int32` |
> | `Short` | `Int16` |
> | `Byte` | `Int8` |
> | `Float` | `Float32` |
> | `Double` | `Float64` |
> | `Boolean` | `Boolean` |
> | `Binary` | `Binary` |
> | `Date` | `Date32` |
> | `Timestamp` | `Timestamp(Microsecond, Some("UTC"))` |
> | `TimestampNtz` | `Timestamp(Microsecond, None)` |
> | `Decimal` | `Decimal128` |
> | `Array` | `List` |
> | `Struct` | `Struct` |
> | `Variant` | `Struct` |
> | `Map` | `Map` |
>
> ## Column Mapping
>
> Delta Lake tables that use [column mapping](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-mapping) with `columnMapping.mode = name` or `columnMapping.mode = id` are supported. Tables in either mode store data in Parquet files under opaque physical column names (e.g., `col-b70f7585-…`) that differ from the logical names exposed by the table schema. Spice resolves logical-to-physical names at read time, including for partitions, predicate pushdown, and nested `Struct`, `List`, and `Map` field renames.
>
> ## Limitations
>
> - Delta Lake connector does not support reading Delta tables with the `V2Checkpoint` feature enabled. To use the Delta Lake connector with such tables, drop the `V2Checkpoint` feature by executing the following command:
>
>   ```sql
>   ALTER TABLE <table-name> DROP FEATURE v2Checkpoint [TRUNCATE HISTORY];
>   ```
>
>   For more details on dropping Delta table features, refer to the official documentation: [Drop Delta table features](https://docs.delta.io/latest/delta-drop-feature.html)
>
> ## Secrets
>
> Spice integrates with multiple secret stores to help manage sensitive data securely. For detailed information on supported secret stores, refer to the [secret stores documentation](https://spiceai.org/docs/components/secret-stores). Additionally, learn how to use referenced secrets in component parameters by visiting the [using referenced secrets guide](https://spiceai.org/docs/components/secret-stores#using-secrets).
>
> ## Cookbook
>
> - A cookbook recipe to configure Delta Lake as a data connector in Spice. [Delta Lake Data Connector](https://github.com/spiceai/cookbook/tree/trunk/delta-lake#readme)