# Elasticsearch Data Connector Deployment Guide

Operating guide for the Elasticsearch data connector in production: authentication, TLS, resilience, and operational tuning.

Production operating guide for the Elasticsearch data connector covering authentication, TLS, resilience, capacity planning, and search routing.

## Authentication & Secrets

The connector uses HTTP Basic authentication. Credentials must be sourced from a [secret store](https://spiceai.org/docs/components/secret-stores) in production.

| Parameter | Description |
| - | - |
| `elasticsearch_endpoint` | Cluster URL. Required. Use `https://...` to enable TLS. |
| `elasticsearch_user` | Username for HTTP Basic authentication. Use `${secrets:...}`. |
| `elasticsearch_pass` | Password for HTTP Basic authentication. Use `${secrets:...}`. |

Scope the user to the minimum required permissions:

- **Read-only access** to the indexes the connector will query (`read` privilege).
- **`monitor`** cluster privilege if you intend to inspect mappings programmatically.

For Elastic Cloud and self-managed deployments protected by API keys, generate a dedicated user (or service account) for Spice rather than reusing administrative credentials.

### TLS

Use `https://` endpoints in production. TLS is enabled automatically when the endpoint scheme is HTTPS. Self-signed certificates require a trusted CA bundle in the container or host OS trust store.

The connector does not currently expose certificate-pinning or custom CA-bundle parameters: rely on the system trust store, or front the cluster with a TLS-terminating proxy you trust.

## Resilience Controls

### Retries

The Elasticsearch client library includes a retry mechanism with exponential backoff for transient errors (HTTP 429 and 5xx). However, retries are currently only active on the **write path** used by the [Elasticsearch Vector Engine](https://spiceai.org/docs/components/vectors/elasticsearch) (`bulk_index` operations). The data connector's read operations (`_search`, `_mapping`) do **not** retry transient errors: failures are surfaced immediately.

Retry tuning is exposed only on the [Elasticsearch Vector Engine](https://spiceai.org/docs/components/vectors/elasticsearch) (`elasticsearch_max_retries`, `elasticsearch_retry_initial_backoff`).

### Timeouts

| Setting | Default | Behavior |
| - | - | - |
| Connect timeout | `10s` | Maximum time to establish a TCP/TLS connection to the cluster. |
| Request timeout | `30s` | Maximum time for each individual HTTP request. |

Long-running search responses (very large `LIMIT`, deep pagination, or expensive aggregations) may exceed the default request timeout. Either narrow the query, accelerate the dataset, or use the [vector engine](https://spiceai.org/docs/components/vectors/elasticsearch) `client_timeout` parameter when running the workload through the embedding-write path.

## Capacity & Sizing

- **Throughput**: Bounded by the Elasticsearch cluster's request handling and (for kNN) HNSW search cost. Plan refresh intervals and concurrent query load to stay within the cluster's tested capacity.
- **Result size**: For queries with `LIMIT N` where N ≤ 10,000, the connector issues a single `_search` request. For larger result sets or queries without `LIMIT`, the connector automatically paginates using Point-In-Time (PIT) + `search_after`, fetching all matching documents in 10,000-hit batches (bounded by the Elasticsearch `index.max_result_window` setting per batch).
- **Mapping fetches**: At dataset registration the connector fetches the index mapping once via `GET /<index>/_mapping`. Mapping changes after registration are not picked up until the runtime restarts.

## Search Routing

When an index has a `dense_vector` field, Spice's search UDTFs compile to native Elasticsearch queries:

- `vector_search(...)` → kNN query against the `dense_vector` field. By default the candidate pool (`num_candidates`) is twice the requested `k`.
- `text_search(...)` → BM25 `match` query on the specified text field.
- `rrf(...)` → both queries issued in parallel and fused using Reciprocal Rank Fusion. RRF tuning (per-query `rank_weight`, recency decay, smoothing `k`) is evaluated by Spice rather than Elasticsearch.

For more, see [Search Functionality](https://spiceai.org/docs/features/search) and the [SQL search reference](https://spiceai.org/docs/reference/sql/search).

## Pushdown Behavior

| Predicate | Pushdown to ES Query DSL |
| - | - |
| `WHERE` filters (any field) | None: the connector pushes down no `WHERE` predicates; all are evaluated locally by DataFusion after fetch (the scan issues `match_all`). |
| `LIMIT N` | Translated to `size: N`. |
| `ORDER BY` | Evaluated locally unless paired with a search UDTF. |
| `vector_search` / `text_search` / `rrf` | Native: issued as kNN / BM25 query bodies. |

For workloads dominated by selective filters, accelerate the dataset (`acceleration.enabled: true`) into DuckDB / SQLite / Cayenne so DataFusion can apply filters at acceleration time rather than fetching unfiltered hits.

## Schema Stability

The connector derives an Arrow schema from `GET /<index>/_mapping` at registration time. Once registered, the schema is locked for the lifetime of the runtime process: adding fields or changing types in Elasticsearch does **not** re-trigger schema inference. Restart the runtime to pick up mapping changes.

For schema-evolution-friendly workloads, prefer accelerating the dataset and refreshing on a schedule against a stable subset of fields.

## Metrics

The Elasticsearch connector does not register connector-specific instruments in the current release. Monitor via:

- Spice query execution metrics (`query_duration_ms`, `query_returned_rows`, `query_failures`) from `runtime.metrics`.
- Elasticsearch's own `/_nodes/stats` endpoint and Kibana dashboards for cluster-side request latency, CPU, JVM heap, and shard health.

See [Component Metrics](https://spiceai.org/docs/features/observability/component_metrics) for general configuration.

## Task History

Elasticsearch requests participate in [task history](https://spiceai.org/docs/reference/task_history) through the HTTP client's span. Each `_search` and `_mapping` call is a child of the enclosing `sql_query` or `acceleration_refresh` task.

## Known Limitations

- **Read-only**: The connector is read-only. Writes (indexing documents, updating mappings) are not supported. Use the [Elasticsearch Vector Engine](https://spiceai.org/docs/components/vectors/elasticsearch) when Spice should manage an index.
- **Schema is frozen at registration**: Mapping changes after startup are not picked up. Restart the runtime to refresh the schema.
- **`date` and `date_nanos` are strings**: Elasticsearch accepts heterogeneous date formats. The connector preserves them as `Utf8`: cast to `TIMESTAMP` in SQL when comparison is needed.
- **`nested` and `object` are JSON strings**: Nested objects are exposed as `Utf8` JSON, not structured Arrow types.
- **`dense_vector` without `dims`**: Falls back to `Utf8` and is not usable as a vector column. Declare `dims` in the index mapping.
- **No filter pushdown**: The connector pushes down no SQL `WHERE` predicates: every filter is evaluated locally by DataFusion after fetch (the scan issues `match_all`; only `LIMIT` is pushed down, as the query `size`). For selective filters, accelerate the dataset.
- **Tested against Elasticsearch 8.17**: Other major versions (7.x, 9.x) may work but are not part of the integration test matrix.

## Troubleshooting

| Symptom | Likely cause | Resolution |
| - | - | - |
| `401 Unauthorized` on dataset registration | Wrong/expired credentials or insufficient privileges. | Verify `elasticsearch_user`/`elasticsearch_pass`; confirm the user has `read` on the target index. |
| `Elasticsearch index 'X' not found in mapping response` | The index does not exist or the user lacks read access. | Create the index, or grant `view_index_metadata` privilege. |
| `dense_vector` column missing from query results | The mapping omits `dims` for that field. | Add `dims` to the index mapping; reconfirm with `GET /<index>/_mapping`. |
| `vector_search` / `text_search` returns nothing | Wrong vector field name, or the index has no documents. | Verify the field is a populated `dense_vector` / `text` field; check via `GET /<index>/_count`. |
| Schema drift after deploying mapping changes | Schema is frozen at registration time. | Restart the runtime to re-infer the schema. |
| Refresh exceeds `request_timeout` | Large response or slow cluster. | Narrow the query, accelerate the dataset, or front Elasticsearch with a cache. |
| TLS handshake fails with self-signed certificate | The certificate's CA is not in the runtime's trust store. | Install the CA bundle in the container/host trust store; do not disable TLS verification in production. |