DataSQRL Configuration (package.json
file)
DataSQRL projects are configured with one or more JSON files.
Unless a file is passed explicitly to datasqrl compile -c β¦
, the compiler looks for a package.json
in the working directory; if none is found the built-in default (shown here) is applied.
Multiple files can be provided; they are merged in order β later files override earlier ones, objects are deep-merged, and array values are replaced wholesale.
Top-Level Keysβ
Key | Type | Default | Purpose |
---|---|---|---|
version | number | 1 | Configuration schema version β must be 1 . |
enabled-engines | string[] | ["vertx","postgres","kafka","flink"] | Ordered list of engines that form the runtime pipeline. |
engines | object | β | Per-engine configuration (see below). |
compiler | object | see defaults | Controls compilation, logging, and generated artefacts. |
dependencies | object | {} | Aliases for packages that can be IMPORT -ed from SQRL. |
discovery | object | {} | Rules for automatic table discovery when importing data files. |
script | object | β | Points to the main SQRL script and GraphQL schema. |
package | object | β | Optional metadata (name, description, etc.) for publishing. |
values | object | {} | Arbitrary runtime values (e.g. create-topics ). |
test-runner | object | {"delay-sec":30} | Integration-test execution settings. |
1. Engines (engines
)β
Each sub-key below engines
must match one of the IDs in enabled-engines
.
{
"engines": {
"<engine-id>": {
"type": "<engine-id>", // optional; inferred from key if omitted
"config": { β¦ }, // engine-specific knobs (Flink SQL options, etc.)
"connectors": { β¦ } // templates for table sources & sinks
}
}
}
Flink (flink
)β
Key | Type | Default | Notes |
---|---|---|---|
mode | "streaming" | "batch" | "streaming" | Execution mode used during plan compilation and job submission. |
config | object | {} | Copied verbatim into the generated Flink SQL job (e.g. "table.exec.source.idle-timeout": "5 s" ). |
connectors | object | see default list | Connector templates (JDBC, Kafka, files, Icebergβ¦). Field values support variable interpolation (below). |
Built-in connector templates
postgres
,postgres_log-source
,postgres_log-sink
,
kafka
,kafka-keyed
,kafka-upsert
,
iceberg
,localfile
,
Kafka (kafka
)β
The default configuration only declares the engine; topic definitions are injected at plan time.
Additional keys (e.g. bootstrap.servers
) may be added under config
.
Vert.x (vertx
)β
A GraphQL server that routes queries to the backing database/log engines.
No mandatory keys; connection pools are generated from the overall plan.
Postgres (postgres
)β
No mandatory keys. Physical DDL (tables, indexes, views) is produced automatically.
Iceberg (iceberg
)β
Used as a table-format engine together with a query engine such as Flink or Snowflake.
Snowflake (snowflake
)β
Key | Type | Default | Description |
---|---|---|---|
schema-type | "aws-glue" | β | External catalog implementation. |
catalog-name | string | β | Glue catalog. |
external-volume | string | β | Snowflake external volume name. |
url | string | β | Full JDBC URL including auth params. |
Compiler (compiler
)β
{
"compiler": {
"logger": "print", // "print" | any configured log engine | "none"
"extendedScalarTypes": true, // expose extended scalar types in generated GraphQL
"snapshotPath": "snapshots", // snapshots output directory written during test runs
"compilePlan": true, // compile physical plans where supported
"explain": { // artifacts in build/pipeline_*.*
"visual": true,
"text": true,
"sql": false,
"logical": true,
"physical": false,
"sorted": true // deterministic ordering (mostly for tests)
}
}
}
3. Dependencies (dependencies
)β
"dependencies": {
"myalias": {
"name": "folder-name",
"version": "1", // version identifier
"variant": "test" // identifier for the source variant
}
}
If only name
is given the key acts as a local folder alias.
Discovery (discovery
)β
Key | Type | Default | Purpose |
---|---|---|---|
pattern | string (regex) | null | Filters which external tables are automatically exposed in IMPORT β¦ statements. Example: "^public\\..*" |
Script (script
)β
Key | Type | Description |
---|---|---|
main | string | Path to the main .sqrl file. |
graphql | string | Optional GraphQL schema file (defaults to schema.graphqls ). |
Package Metadata (package
)β
Key | Required | Description |
---|---|---|
name | yes | Reverse-DNS style identifier (org.project.module ). |
description | no | Short summary. |
license | no | SPDX license id or free-text. |
homepage | no | Web site. |
documentation | no | Docs link. |
topics | no | String array of tags/keywords. |
Test-Runner (test-runner
)β
Key | Type | Default | Meaning |
---|---|---|---|
delay-sec | number | 30 | Wait between data-load and snapshot. Set -1 to disable. |
required-checkpoints | number | 0 | Minimum completed Flink checkpoints before assertions run (requires delay-sec = -1 ). |
mutation-delay | number | 0 | Pause (s) between mutation queries. |
Values (values
)β
Arbitrary key/value pairs that are exposed to the runtime launcher.
The default launcher recognises:
Key | Type | Description |
---|---|---|
create-topics | string[] | Kafka topics to create before tests start. |
Template & Environment Variablesβ
- Environment variables use
${VAR_NAME}
β resolved by the DataSQRL launcher at runtime.
Example:${JDBC_PASSWORD}
. - SQRL variables use
${sqrl:<identifier>}
and are filled automatically by the compiler, mostly inside connector templates.
Common identifiers includetable-name
,original-table-name
,filename
,format
, andkafka-key
.
Unresolved ${sqrl:*}
placeholders raise a validation error.
Default Configurationβ
The built-in fallback (excerpt - full version here):
{
"version": 1,
"enabled-engines": ["vertx","postgres","kafka","flink"],
"engines": {
"flink": {
"connectors": {
"postgres": { "connector": "jdbc-sqrl", β¦ },
"kafka": { "connector": "kafka", β¦ },
"kafka-keyed": { β¦ },
"kafka-upsert": { β¦ },
"localfile": { β¦ },
"iceberg": { β¦ },
"print": { "connector": "print" }
}
},
"snowflake": {
"schema-type": "aws-glue",
"catalog-name": "${SNOWFLAKE_CATALOG_NAME}",
"external-volume": "${SNOWFLAKE_EXTERNAL_VOLUME}",
"url": "jdbc:snowflake://${SNOWFLAKE_ID}.snowflakecomputing.com/?β¦"
}
},
"test-runner": {
"delay-sec": 30,
"mutation-delay": 1,
"headers": {
"Authorization": "Bearer token123",
"Content-Type": "application/json"
}
}
}