Spice currently supports CSV, JSON, and Parquet data file-formats for data connectors that can read files from a file system or cloud object storage (i.e. s3:// (opens in a new tab), abfs:// (opens in a new tab), file:// (opens in a new tab), etc.). Support for Iceberg and other file-formats are on the roadmap.
The parameters supported for specific file-formats are detailed on this page.
Spice automatically supports reading any Parquet file, regardless of the compression codec or data encoding used.
Compression codecs:
UNCOMPRESSED (opens in a new tab)SNAPPY (opens in a new tab)GZIP (opens in a new tab)LZO (opens in a new tab)BROTLI (opens in a new tab)LZ4 (opens in a new tab) (deprecated in favor of LZ4_RAW)LZ4_RAW (opens in a new tab)ZSTDData encodings:
PLAIN (opens in a new tab)PLAIN_DICTIONARY / RLE_DICTIONARY (opens in a new tab)RLE (opens in a new tab)BIT_PACKED (opens in a new tab) (deprecated in favor of RLE)DELTA_BINARY_PACKED (opens in a new tab)DELTA_LENGTH_BYTE_ARRAY (opens in a new tab)csv_has_header: Optional. Indicate if the CSV file has header row. Defaults to truecsv_quote: Optional. A one-character string used to quote fields containing special characters. Defaults to "csv_escape: Optional. A one-character string used to represent special characters or to include characters that would normally be interpreted as delimiters or new line characters within a field value. Defaults to nullcsv_schema_infer_max_records: Optional. A number used to set the limit in terms of records to scan to infer the schema. Defaults to 1000csv_delimiter: Optional. A one-character string used to separate individual fields. Defaults to ,json_format: Optional. Specifies the JSON format to parse. Valid values are array, ndjson, and jsonl. Defaults to jsonlDELTA_BYTE_ARRAY