Skip to main content
Version: v0.12

Iceberg Query Engine Configurations

Apache Iceberg stores analytic data but does not execute queries. Enable an Iceberg query engine alongside iceberg to create and query Iceberg tables.

Full Query Enginesโ€‹

Full query engines are integrated with the DataSQRL server and can execute generated API queries against Iceberg tables.

DuckDB (duckdb)โ€‹

DuckDB is a vectorized database query engine that excels at analytical queries and can read Iceberg tables efficiently.

Configuration Optionsโ€‹

KeyTypeDefaultDescription
urlstring"jdbc:duckdb:"Full JDBC URL for the database connection
memory-limitstring-Sets DuckDB's memory_limit, for example "8GB"
use-disk-cachebooleanfalseLoads the cache_httpfs extension (requires DUCKDB_EXTENSIONS_DIR, see below)
use-version-guessingbooleanfalseSets unsafe_enable_version_guessing flag to be able to read uncommitted data (requires DUCKDB_EXTENSIONS_DIR, see below)
scan-cte-cardinality-divisorinteger1Controls when a query that scans the same Iceberg table multiple times reads it once into a shared CTE. The scan is shared when the number of scans exceeds max((log2(row count) - 20) / divisor, 1); a larger divisor shares scans more eagerly. Requires a known row count (estimated or set via the row_count hint). Must be a positive integer

Example Configurationโ€‹

{
"engines": {
"duckdb": {
"url": "jdbc:duckdb:",
"memory-limit": "4GB",
"use-disk-cache": true,
"use-version-guessing": true
}
}
}
note

The DataSQRL server only loads DuckDB extensions (iceberg, httpfs, and cache_httpfs) and applies use-version-guessing when the DUCKDB_EXTENSIONS_DIR environment variable points to a directory with the preinstalled extensions. The official datasqrl/cmd and datasqrl/sqrl-server Docker images set it. When running the server elsewhere without it, the server logs a warning and silently skips these settings.

Usage Notesโ€‹

  • Ideal for local development and testing of analytical workloads
  • Excellent performance on analytical queries with vectorized execution
  • Can read Iceberg tables directly without additional infrastructure
  • Supports both in-memory and persistent database modes
  • Perfect for prototyping before deploying to cloud query engines like Snowflake
  • Lightweight alternative to larger analytical databases

Shallow Query Enginesโ€‹

Shallow query engines generate engine-specific Iceberg table definitions and query SQL, but are not integrated with the DataSQRL server. They cannot execute generated API queries. When an API server is enabled, pair a shallow query engine with the full DuckDB query engine.

Snowflake (snowflake)โ€‹

Snowflake is a cloud-based analytic database query engine that can read Iceberg tables and provide enterprise-scale analytical capabilities.

Configuration Optionsโ€‹

KeyTypeDefaultDescription
catalog-namestring-Glue catalog name for metadata
external-volumestring-Snowflake external volume name
urlstring-Full JDBC URL including auth params

Example Configurationโ€‹

{
"engines": {
"snowflake": {
"catalog-name": "my-glue-catalog",
"external-volume": "my-external-volume",
"url": "jdbc:snowflake://account.snowflakecomputing.com/?user=username&password=password&warehouse=warehouse&db=database&schema=schema"
}
}
}

Usage Notesโ€‹

  • Requires all three configuration parameters
  • Designed for large-scale analytical workloads in the cloud
  • Integrates with AWS Glue for metadata management
  • Uses external volumes for accessing Iceberg data
  • Authentication parameters should be included in the JDBC URL
  • For local development, consider using DuckDB as a substitute
  • Provides enterprise features like data sharing and governance

Spark SQL (sparksql)โ€‹

Spark SQL generates Spark SQL definitions for Iceberg tables. Enable it by adding "sparksql" to enabled-engines.

Configuration Optionsโ€‹

KeyTypeDefaultDescription
view-catalogstring-Catalog that contains generated views
view-databasestringdefaultDatabase that contains generated views when a catalog is set

When configured, generated view names are qualified as catalog.database.view. Setting only view-database qualifies views as database.view in the current catalog. Setting view-catalog without view-database uses Spark SQL's default database.

Redshift (redshift)โ€‹

Redshift generates Amazon Redshift SQL definitions for Iceberg tables. Enable it by adding "redshift" to enabled-engines.

Configuration Optionsโ€‹

KeyTypeDefaultDescription
view-databasestring-Database that contains generated views
view-schemastringpublicSchema that contains generated views when a database is set

When configured, generated view names are qualified as database.schema.view. Setting only view-schema qualifies views as schema.view in the current database. Setting view-database without view-schema uses Redshift's public schema.

Trino (trino)โ€‹

Trino generates Trino SQL definitions for Iceberg tables. Enable it by adding "trino" to enabled-engines.

Configuration Optionsโ€‹

KeyTypeDefaultDescription
view-catalogstring-Catalog that contains generated views
view-schemastringpublicSchema that contains generated views when a catalog is set

When configured, generated view names are qualified as catalog.schema.view. Setting only view-schema qualifies views as schema.view in the current catalog. Setting view-catalog without view-schema uses Trino's public schema.