title: 'Databricks Data Connector' sidebar_label: 'Databricks Data Connector' description: 'Databricks Data Connector Documentation' pagination_prev: null tags:
Databricks as a connector for federated SQL query against Databricks using Spark Connect (opens in a new tab), directly from Delta Lake (opens in a new tab) tables, or using the SQL Statement Execution API (opens in a new tab).
fromThe from field for the Databricks connector takes the form databricks:catalog.schema.table where catalog.schema.table is the fully-qualified path to the table to read from.
:::info
Unquoted identifiers are normalized to lowercase. To reference a table, schema, or catalog with mixed-case characters, wrap each case-sensitive part in double quotes: databricks:my_catalog."MySchema"."MyTable". See Identifier Case Sensitivity (opens in a new tab).
:::
nameThe dataset name. This will be used as the table name within Spice.
Example:
The dataset name cannot be a reserved keyword (opens in a new tab).
paramsUse the secret replacement syntax (opens in a new tab) to reference a secret, e.g. ${secrets:my_token}.
| Parameter Name | Description |
|---|---|
mode | The execution mode for querying against Databricks. The default is spark_connect. Possible values: spark_connect: Use Spark Connect to query against Databricks. Requires a Spark cluster to be available.delta_lake: Query directly from Delta Tables. Requires the object store credentials to be provided.sql_warehouse: Use the SQL Statement Execution API to query against a Databricks SQL Warehouse. |
databricks_endpoint | The endpoint of the Databricks instance. Required for all modes. |
databricks_sql_warehouse_id | The ID of the SQL Warehouse in Databricks to use for the query. Only valid when mode is sql_warehouse. |
databricks_cluster_id | The ID of the compute cluster in Databricks to use for the query. Only valid when mode is spark_connect. |
databricks_credential_vending | When set to enabled (requires mode to be delta_lake), short-lived storage credentials for each table are fetched from the Unity Catalog credential vending API instead of using static object store credentials. Possible values: enabled, disabled. Defaults to disabled. |
databricks_use_ssl | If true, use a TLS connection to connect to the Databricks Spark Connect endpoint. Only valid when mode is spark_connect. Default is true. |
client_timeout | Optional. Specifies timeout for HTTP operations. In delta_lake mode, applies to object store operations. In sql_warehouse mode, applies per-HTTP-call (statement submit, status poll, chunk fetch) — not total query duration. Default: 30s. E.g. client_timeout: 2m |
connect_timeout | Optional. Timeout for establishing TCP/TLS connections. Applies in sql_warehouse mode. Default: 10s. E.g. connect_timeout: 15s |
databricks_token | The Databricks API token to authenticate with the Unity Catalog API. Can't be used with databricks_client_id and databricks_client_secret. |
databricks_client_id | The Databricks OAuth client ID. Used with databricks_client_secret for service-principal (M2M) auth, or alone for interactive User-to-Machine (U2M) auth. Can't be used with databricks_token. |
databricks_client_secret | The Databricks Service Principal Client Secret. Required for M2M auth; omit for U2M auth. Can't be used with databricks_token. |
databricks_auth_mode | Optional. Pins the authentication flow instead of inferring it from the credentials that are set. One of auto, token, m2m, u2m (case-insensitive). Defaults to auto. See Pinning the authentication mode (opens in a new tab). |
The following parameters apply only when mode is sql_warehouse and control connection resilience and concurrency:
| Parameter Name | Description |
|---|---|
connect_timeout | Optional. Timeout for establishing TCP/TLS connections to the Databricks API. Default: 10s. E.g. connect_timeout: 15s |
client_timeout | Optional. Per-HTTP-call timeout (statement submit, status poll, chunk fetch) — not total query duration. Overall query time is bounded by statement_max_retries × backoff. Default: 30s. E.g. client_timeout: 2m |
max_concurrent_requests | Optional. Maximum number of concurrent HTTP requests to the SQL Warehouse API. Default: 8. |
http_max_retries | Optional. Maximum number of HTTP-level retries for transient failures (429, 5xx). Default: 3. |
backoff_method | Optional. Backoff strategy for transient HTTP retries. Options: fibonacci, exponential. Default: fibonacci. |
statement_max_retries | Optional. Maximum number of poll retries when waiting for async statement completion. Default: 14. |
disable_on_permanent_error | Optional. When true, non-retryable errors (401, 403, 404) permanently disable the connector. Default: true. |
The Databricks connector supports per-dataset rate control parameters when mode is spark_connect or sql_warehouse. These override runtime.params (opens in a new tab) HTTP rate control defaults. When runtime.source_rate_control.state_location (opens in a new tab) is configured, rate limits are coordinated across the cluster.
| Parameter Name | Description |
|---|---|
requests_per_second_limit | Optional. Maximum HTTP requests per second to the Databricks endpoint. Overrides runtime.params.http_requests_per_second_limit. |
requests_per_minute_limit | Optional. Maximum HTTP requests per minute to the Databricks endpoint. Overrides runtime.params.http_requests_per_minute_limit. |
rate_control_jitter_min | Optional. Minimum random delay before HTTP requests when rate control is active. Defaults to 5ms when a rate limit is configured. Accepts durations like 5ms. |
rate_control_jitter_max | Optional. Maximum random delay before HTTP requests when rate control is active. Defaults to 10ms when a rate limit is configured. Accepts durations like 10ms. |
To learn more about how to set up personal access tokens, see Databricks PAT docs (opens in a new tab).
Spice supports the Machine-to-Machine (M2M) OAuth flow with service principal credentials by utilizing the databricks_client_id and databricks_client_secret parameters. The runtime will automatically refresh the token.
Ensure that you grant your service principal the "Data Reader" privilege preset for the catalog and "Can Attach" cluster permissions when using Spark Connect mode.
To Learn more about how to set up the service principal, see Databricks M2M OAuth docs (opens in a new tab).
Spice supports the User-to-Machine (U2M) OAuth flow for interactive sign-in against Databricks. To use U2M auth, supply only databricks_client_id (without databricks_token or databricks_client_secret), or set databricks_auth_mode: u2m to pin the flow — see Pinning the authentication mode (opens in a new tab).
When U2M auth is configured, the connector defers initialization until first use. On the first query the runtime opens a browser to complete the Databricks OAuth sign-in, then caches and refreshes the resulting token for subsequent requests.
To learn more about how to set up U2M OAuth, see the Databricks U2M OAuth docs (opens in a new tab).
:::note
U2M auth is supported with mode: delta_lake and mode: sql_warehouse. It is not supported with mode: spark_connect — use a personal access token or service principal credentials when querying through Spark Connect.
:::
By default (databricks_auth_mode: auto) the connector infers the flow from whichever credentials resolve: a databricks_token alone selects personal access token, databricks_client_id alone selects U2M, and databricks_client_id together with databricks_client_secret selects M2M.
Credentials do not only come from the Spicepod. databricks_token, databricks_client_secret, and the other secret parameters are auto-loaded from the secret stores (opens in a new tab) and the environment (e.g. DATABRICKS_CLIENT_SECRET, SPICE_DATABRICKS_CLIENT_SECRET) when the Spicepod omits them — so an ambient client secret on the host is enough to switch a U2M dataset to machine-to-machine, which then fails with a service principal 401.
Set databricks_auth_mode to pin the flow. A pinned mode ignores the credentials the flow does not use:
| Value | Flow | Requires |
|---|---|---|
auto (default) | Inferred from the credentials set | — |
token | Personal access token | databricks_token |
m2m | Service principal (M2M) OAuth | databricks_client_id, databricks_client_secret |
u2m | User-to-machine OAuth | databricks_client_id |
Values are matched case-insensitively, so M2M and U2M are also accepted. A required parameter that is missing for the pinned mode fails at load with an error naming it; an unrecognized value is rejected.
Configure the connection to the object store when using mode: delta_lake. Use the secret replacement syntax (opens in a new tab) to reference a secret, e.g. ${secrets:aws_access_key_id}.
| Parameter Name | Description |
|---|---|
databricks_aws_region | Optional. The AWS region for the S3 object store. E.g. us-west-2. |
databricks_aws_access_key_id | The access key ID for the S3 object store. |
databricks_aws_secret_access_key | The secret access key for the S3 object store. |
databricks_aws_session_token | Optional. The AWS session token for the S3 object store. Required with temporary (STS) credentials. |
databricks_aws_endpoint | Optional. The endpoint for the S3 object store. E.g. s3.us-west-2.amazonaws.com. |
databricks_aws_allow_http | Optional. Enables insecure HTTP connections to databricks_aws_endpoint. Defaults to false. |
:::info Note One of the following auth values must be provided for Azure Blob:
databricks_azure_storage_account_key,databricks_azure_storage_client_id and databricks_azure_storage_client_secret, ordatabricks_azure_storage_sas_key.
:::| Parameter Name | Description |
|---|---|
databricks_azure_storage_account_name | The Azure Storage account name. |
databricks_azure_storage_account_key | The Azure Storage key for accessing the storage account. |
databricks_azure_storage_client_id | The Service Principal client ID for accessing the storage account. |
databricks_azure_storage_client_secret | The Service Principal client secret for accessing the storage account. |
databricks_azure_storage_sas_key | The shared access signature key for accessing the storage account. |
databricks_azure_storage_endpoint | Optional. The endpoint for the Azure Blob storage account. |
| Parameter Name | Description |
|---|---|
databricks_google_service_account | Filesystem path to the Google service account JSON key file. |
The table below shows the Databricks (mode: delta_lake) data types supported, along with the type mapping to Apache Arrow types in Spice.
| Databricks SQL Type | Arrow Type |
|---|---|
STRING | Utf8 |
BIGINT | Int64 |
INT | Int32 |
SMALLINT | Int16 |
TINYINT | Int8 |
FLOAT | Float32 |
DOUBLE | Float64 |
BOOLEAN | Boolean |
BINARY | Binary |
DATE | Date32 |
TIMESTAMP | Timestamp(Microsecond, Some("UTC")) |
TIMESTAMP_NTZ | Timestamp(Microsecond, None) |
DECIMAL | Decimal128 |
ARRAY | List |
STRUCT | Struct |
MAP | Map |
Spice integrates with multiple secret stores to help manage sensitive data securely. For detailed information on supported secret stores, refer to the secret stores documentation (opens in a new tab). Additionally, learn how to use referenced secrets in component parameters by visiting the using referenced secrets guide (opens in a new tab).
Databricks connector (mode: delta_lake) does not support reading Delta tables with the V2Checkpoint feature enabled. To use the Databricks connector (mode: delta_lake) with such tables, drop the V2Checkpoint feature by executing the following command:
For more details on dropping Delta table features, refer to the official documentation: Drop Delta table features (opens in a new tab)
When using mode: spark_connect, correlated scalar subqueries can only be used in filters, aggregations, projections, and UPDATE/MERGE/DELETE commands. Spark Docs (opens in a new tab)
:::warning[Memory Considerations]
When using the Databricks (mode: delta_lake) Data connector without acceleration, data is loaded into memory during query execution. Ensure sufficient memory is available, including overhead for queries and the runtime, especially with concurrent queries.
Memory limitations can be mitigated by storing acceleration data on disk, which is supported by duckdb (opens in a new tab) and sqlite (opens in a new tab) accelerators by specifying mode: file.
mode: spark_connect) does not yet support streaming query results from Spark.:::