# Apache Spark Connector

Run federated SQL queries against an Apache Spark cluster through Spark Connect, and keep the connection URI in a secret store.

Apache Spark as a connector for federated SQL query against a Spark Cluster using [Spark Connect](https://spark.apache.org/docs/latest/spark-connect-overview.html)

```yaml
datasets:
  - from: spark:spiceai.datasets.my_awesome_table
    name: my_table
    params:
      spark_remote: sc://localhost:15002
```

> **Note**
>
> Unquoted identifiers are normalized to lowercase. To reference a table with mixed-case characters, wrap each case-sensitive part in double quotes: `spark:my_catalog."MySchema"."MyTable"`. See [Identifier Case Sensitivity](https://spice.ai/docs/building-blocks/data-connectors#identifier-case-sensitivity-and-quoting).

## Configuration

- `spark_remote`: Required. A [spark remote](https://spark.apache.org/docs/latest/spark-connect-overview.html#set-sparkremote-environment-variable) connection URI. Refer to [spark connect client connection string](https://github.com/apache/spark/blob/master/sql/connect/docs/client-connection-string.md) for parameters in URI.

The dataset name cannot be a [reserved keyword](https://spiceai.org/docs/reference/spicepod/keywords).

### Auth Examples

Spark clusters configured to accept authenticated requests should not set `spark_remote` as an inline dataset param, as it will contain sensitive data. For this case, use the [secret replacement syntax](https://spiceai.org/docs/components/secret-stores) to load the secret from a secret store, e.g. `${secrets:my_spark_remote}`.

Check [Secrets Stores](https://spiceai.org/docs/components/secret-stores) for more details.

**Env**

```bash
SPICE_SPARK_REMOTE="<spark-remote>" \
spice run
# Or using the CLI to configure the secrets into an `.env` file
spice login spark --spark_remote "<spark-remote>"
```

`.env`

```bash
SPICE_SPARK_REMOTE="<spark-remote>"
```

`spicepod.yaml`

```yaml
version: v1
kind: Spicepod
name: spice-app

secrets:
  - from: env
    name: env

datasets:
  - from: spark:spiceai.datasets.my_awesome_table
    name: my_table
    params:
      spark_remote: ${env:SPICE_SPARK_REMOTE}
```

Learn more about [Env Secret Store](https://spiceai.org/docs/components/secret-stores/env).

**Kubernetes**

```bash
kubectl create secret generic spark \
  --from-literal=spark_remote='<spark-remote>'
```

`spicepod.yaml`

```yaml
version: v1
kind: Spicepod
name: spice-app

secrets:
  - from: kubernetes:spark
    name: spark

datasets:
  - from: spark:spiceai.datasets.my_awesome_table
    name: my_table
    params:
      spark_remote: ${spark:spark_remote}
```

Learn more about [Kubernetes Secret Store](https://spiceai.org/docs/components/secret-stores/kubernetes).

**Keyring**

Add new keychain entry (macOS) with the spark remote:

```bash
security add-generic-password -l "Spark Remote" \
-a spiced -s spice_spark_remote \
-w "<spark-remote>"
```

`spicepod.yaml`

```yaml
version: v1
kind: Spicepod
name: spice-app

secrets:
  - from: keyring
    name: keyring

datasets:
  - from: spark:spiceai.datasets.my_awesome_table
    name: my_table
    params:
      spark_remote: ${keyring:spice_spark_remote}
```

Learn more about [Keyring Secret Store](https://spiceai.org/docs/components/secret-stores/keyring).

## Limitations

- Correlated scalar subqueries are only supported in filters, aggregations, projections, and UPDATE/MERGE/DELETE commands. [Spark Docs](https://spark.apache.org/docs/latest/sql-error-conditions.html#unsupported-subquery-expression-category-unsupported-correlated-scalar-subquery)

- The Spark connector does not yet support streaming query results from Spark.

- Spice-specific functions, such as the [JSON functions](https://spiceai.org/docs/reference/sql/json) and user-defined functions, are evaluated in Spice, not sent to Spark. For a query such as `WHERE json_get_str(doc, 'id') = 'a'`, Spark still reads the table, and Spice applies the filter. See [Federation and Pushdown](https://spiceai.org/docs/reference/sql/json#federation-and-pushdown).

## Cookbook

- A cookbook recipe to configure Spark as a data connector in Spice. [Apache Spark Data Connector](https://github.com/spiceai/cookbook/tree/trunk/spark#readme)