title: 'GCS Data Connector' sidebar_label: 'GCS Data Connector' description: 'GCS (Google Cloud Storage) Data Connector Documentation' tags:
The GCS Data Connector enables federated SQL queries on files stored in Google Cloud Storage. Both gcs:// and gs:// URI schemes are accepted.
When a folder path is provided, all the contained files will be loaded.
File formats are specified using the file_format parameter, as described in File Formats.
fromDefines the GCS URI to a folder or object. Both schemes are supported and equivalent:
from: gs://<bucket>/<path>from: gcs://<bucket>/<path>Example: from: gs://my-bucket/path/to/file.parquet
nameDefines the dataset name, which is used as the table name within Spice.
Example:
The dataset name cannot be a reserved keyword.
params| Parameter name | Description |
|---|---|
file_format | Specifies the data format. Required if it cannot be inferred from the object URI. Options: parquet, csv, json. Refer to File Formats for details. |
allow_http | Allow insecure HTTP connections. Defaults to false. |
client_timeout | Optional. Timeout for GCS client operations. |
hive_partitioning_enabled | Enable partitioning using hive-style partitioning from the folder structure. Defaults to false. |
schema_source_path | Specifies the URL used to infer the dataset schema. Defaults to the most recently modified file. |
The following authentication methods are mutually exclusive — only one can be set at a time. The runtime will fail to start if more than one is specified.
gcs_service_account_pathgcs_service_account_keygcs_application_default_credentialsgcs_skip_signatureIf none of these are set, the connector accesses the bucket without explicit credentials. For public buckets, set gcs_skip_signature: true to skip request signing.
| Parameter name | Description |
|---|---|
gcs_service_account_path | Path to a GCS service account JSON key file. |
gcs_service_account_key | GCS service account JSON key as a string. |
gcs_application_default_credentials | Set to true to use Google Application Default Credentials. If GOOGLE_APPLICATION_CREDENTIALS is set, that path is used. Defaults to false. |
gcs_skip_signature | Set to true to skip signing requests. Use for public buckets. |
| Parameter name | Description |
|---|---|
gcs_max_retries | Maximum number of retries. Defaults to 3. |
gcs_retry_timeout | Total timeout for retries (e.g., 5s, 1m). |
gcs_backoff_initial_duration | Initial retry delay (e.g., 5s). |
gcs_backoff_max_duration | Maximum retry delay (e.g., 1m). |
gcs_backoff_base | Exponential backoff base (e.g., 0.1). |
GCS connector supports four mutually-exclusive authentication modes, as detailed in the authentication parameters.
Configure a service account by setting gcs_service_account_path to the file path of a downloaded service account JSON key:
To create the key file, follow the Google Cloud documentation for service account keys and grant the service account roles/storage.objectViewer (or higher) on the bucket via the Cloud Storage IAM settings.
When mounting a key file is not practical (e.g., when keying off a secret store), pass the JSON contents directly via gcs_service_account_key:
The value should be the full JSON key as a single string, ideally provided through a supported secret store.
To use Application Default Credentials — for example, when running inside Google Cloud with attached service accounts (GKE Workload Identity, Compute Engine metadata, etc.) or when using gcloud auth application-default login locally — set gcs_application_default_credentials: true:
If the GOOGLE_APPLICATION_CREDENTIALS environment variable is set to a service account JSON key path, that file is used. Otherwise, the ADC chain searches the well-known locations described in the Google Cloud documentation.
For unauthenticated access to a public bucket, set gcs_skip_signature: true:
Specify the file format using the file_format parameter. More details in File Formats.
Spice integrates with multiple secret stores to help manage sensitive data securely. For detailed information on supported secret stores, refer to the secret stores documentation.
gcs_service_account_path and gcs_service_account_key are marked as secrets and can be supplied through any supported secret store using the ${secrets:KEY} replacement syntax.