Data Connectors

Glue Data Connector

Glue Data Connector Documentation

The Glue Data Connector enables federated SQL querying on tables in an AWS Glue Data Catalog.

datasets:
  - from: glue:tpch.lineitem
    name: lineitem
    params:
      glue_region: us-east-1
      glue_key: ${env:SPICE_AWS_KEY}
      glue_secret: ${env:SPICE_AWS_SECRET}

Configuration

from

Specify a table using the format, glue:<database>.<table> by replacing <database> with the name of the Glue database and <table>with the name of the table inside of the <database>.

name

The dataset name. This will be used as the table name within Spice.

Example:

SELECT COUNT(*) FROM lineitem;
+----------+
| count(*) |
+----------+
| 6001215  |
+----------+

params

The following parameters are supported for configuring the connection to the Glue Data Catalog:

Parameter NameDefinition
glue_regionThe AWS region for the Glue Data Catalog. E.g. us-west-2.
glue_keyAccess key (e.g. AWS_ACCESS_KEY_ID for AWS)
glue_secretSecret key (e.g. AWS_SECRET_ACCESS_KEY for AWS)
glue_session_tokenSession token (e.g. AWS_SESSION_TOKEN for AWS) for temporary credentials

Authentication

The minimum IAM policy for Glue access is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetTables"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Limitations