# DataSQRL > DataSQRL is an open-source data engineering harness for building data engineering agents designed around human control, correctness, and safety. It extends coding agents with a SQL compiler, a validator, an event-time simulator, and skills. The compiler turns SQRL scripts into complete data pipelines that integrate Kafka, Flink, PostgreSQL, Iceberg, and GraphQL, REST, and MCP APIs. Key facts for working with DataSQRL: - A project is one or more `.sqrl` scripts plus `*package.json` configuration files. SQRL is Flink SQL extended with `IMPORT`/`EXPORT` statements, table functions, relationships, and hints; the SQRL Language page documents only what differs from Flink SQL. - The DataSQRL CLI is distributed as the `datasqrl/cmd` Docker image with the `compile`, `test`, and `run` commands. `compile` validates the script and generates every deployment asset, `test` runs the pipeline and the test queries and compares their results to snapshots, and `run` compiles and runs the pipeline locally in Docker. - Configuration files are deep-merged in the order given on the command line, on top of the default configuration. - System functions are always available; library functions must be imported with `IMPORT stdlib..;`. When to use DataSQRL: - Use it to build data pipelines with a coding agent when correctness and human review matter: the compiler validates the whole pipeline as a relational plan, event-time replay tests catch time-dependent bugs, and the pipeline stays one SQL file a person can review. - Use it for streaming and batch pipelines on Apache Flink with Kafka, PostgreSQL, and Iceberg, such as CDC, temporal joins, windowed aggregations, and deduplication. The same SQL runs in streaming or batch mode. - Use it to serve pipeline results through GraphQL, REST, and MCP APIs generated from the same SQL, with authentication and authorization. - Use it to build data products and views in an existing data lake or lakehouse, for example batch processing into Iceberg tables queried through DuckDB or Snowflake. What DataSQRL is not: - It is not an orchestrator or an ingestion tool. It complements Airflow, Dagster, Fivetran, and Airbyte: it reads from the Kafka topics, databases, and Iceberg tables they write to, and its compiled jobs can be scheduled with existing tooling. - It is not a database or a runtime. It compiles to open-source engines and generates standard deployment assets (Flink compiled plans, SQL schemas and queries, Kafka topic definitions, API definitions) with no proprietary runtime. - It does not replace the engines it compiles to. Engine settings are tuned through configuration and hints, and SQL the engine supports but DataSQRL does not yet can be passed through. How DataSQRL compares: - Apache Flink and Apache Spark: DataSQRL sits on top of them. It defines the whole pipeline once in SQL and compiles it into the assets each engine runs, with Flink as the processing engine. - dbt and SQLMesh: they manage batch SQL transformations inside a warehouse. DataSQRL also covers ingestion, streaming, storage, and API serving in one model, and adds relational validation, event-time testing, and local execution of the whole stack. - Streaming databases (Materialize, RisingWave, ksqlDB): they run incremental SQL inside a single system. DataSQRL is a compiler that splits a pipeline across a stream processor, a log, a database or table format, and an API server. - Coding agents on their own (Claude Code, Codex, and others): DataSQRL is a harness around the agent, not a replacement. The agent keeps doing the reasoning; DataSQRL generates the seams between systems, validates the pipeline, and tests it with event-time replay. This is the documentation of DataSQRL v0.12, the latest release. ## Overview - [DataSQRL Documentation](https://docs.datasqrl.com/docs/intro.md): DataSQRL is an open-source data engineering harness for building data engineering agents designed around human control, correctness, and safety. ## Getting Started - [Getting Started with DataSQRL](https://docs.datasqrl.com/docs/intro/getting-started.md): The basic DataSQRL agent runs as a Docker image on your local machine. - [Getting Started with DataSQRL](https://docs.datasqrl.com/docs/intro/getting-started-no-key.md): This tutorial demonstrates how the DataSQRL data engineering harness works without an LLM API key. ## Core Concepts - [SQRL Language Specification](https://docs.datasqrl.com/docs/sqrl-language.md): SQRL is an extension of Flink SQL that adds support for table functions and convenience syntax to build reactive data processing and serving applications. - [Connecting External Data Sources and Sinks](https://docs.datasqrl.com/docs/connectors.md): Use `CREATE TABLE` statements to connect external data sources and sinks with your SQRL script using the `WITH` clause to provide connector configuration. - [Designing the Interface](https://docs.datasqrl.com/docs/interface.md): Based on the SQRL script, DataSQRL generates the interface for the compiled data pipeline. - [DataSQRL Configuration (`package.json` file)](https://docs.datasqrl.com/docs/configuration.md): DataSQRL projects are configured with one or more `*package.json` files which are merged in the order they are provided to the DataSQRL command – latter files override fields in earlier ones, objec... - [Flink Engine Configuration](https://docs.datasqrl.com/docs/configuration-engine/flink.md): Apache Flink is a streaming and batch data processor that serves as the core data processing engine in DataSQRL pipelines. - [Kafka Engine Configuration](https://docs.datasqrl.com/docs/configuration-engine/kafka.md): Apache Kafka is a streaming data platform that serves as the log engine in DataSQRL pipelines for handling data streams and event logs. - [Vert.x Engine Configuration](https://docs.datasqrl.com/docs/configuration-engine/vertx.md): Eclipse Vert.x is a reactive server framework that serves as the GraphQL API server, routing queries to the backing database/log engines. - [PostgreSQL Engine Configuration](https://docs.datasqrl.com/docs/configuration-engine/postgres.md): PostgreSQL is a realtime database that stores the materialized views and tables generated by your DataSQRL pipeline for low-latency querying. - [Iceberg Engine Configuration](https://docs.datasqrl.com/docs/configuration-engine/iceberg.md): Apache Iceberg is an analytic database format that provides ACID transactions, schema evolution, and time travel capabilities for large analytic datasets. - [Iceberg Query Engine Configurations](https://docs.datasqrl.com/docs/configuration-engine/iceberg-query.md): Apache Iceberg stores analytic data but does not execute queries. - [Default DataSQRL `package.json` Configuration](https://docs.datasqrl.com/docs/configuration-default.md): The following is the default configuration file that user provided configuration files are merged on top of. - [DataSQRL Cloud Deployment Configuration](https://docs.datasqrl.com/docs/configuration-engine/cloud-deployment.md): Configures the cloud resources for each engine when deploying pipelines to DataSQRL Cloud. - [Functions](https://docs.datasqrl.com/docs/functions.md): DataSQRL supports all of Flink's built-in system functions. - [System Functions](https://docs.datasqrl.com/docs/functions-system-generated.md): DataSQRL provides built-in system functions that are available in all SQRL scripts. - [Library Functions](https://docs.datasqrl.com/docs/functions-library-generated.md): DataSQRL provides extended library functions that can be imported into your SQRL scripts. - [DataSQRL Command](https://docs.datasqrl.com/docs/compiler.md): The DataSQRL command initializes, compiles, runs, and tests SQRL projects. - [Compilation Output](https://docs.datasqrl.com/docs/compilation-output.md): The `compile` command writes two kinds of output. - [Deployment](https://docs.datasqrl.com/docs/deployment.md): The compiler generates all deployment assets for a pipeline in `build/deploy/plan`: Flink plans, Kafka topic definitions, database schemas, and the API server model (see Compilation Output). - [Key DataSQRL Concepts](https://docs.datasqrl.com/docs/intro/concepts.md): This document explains some of the key concepts in streaming data processing. ## Examples - [Examples Gallery](https://docs.datasqrl.com/docs/intro/examples.md): A gallery of what you can build with the DataSQRL data engineering harness, from a data product suite for a retail bank to small, self-contained pipelines across a range of use cases. ## FAQ - [Frequently Asked Questions](https://docs.datasqrl.com/docs/intro/faq.md): Claude Code, Codex, and similar agents write plausible data pipeline code, but data engineering has failure modes that general-purpose coding agents miss and that pass code review. ## Advanced - [Deep Dive: How DataSQRL Works](https://docs.datasqrl.com/docs/deepdive.md): The DataSQRL `compile` command executes the following steps - [Version Compatibility](https://docs.datasqrl.com/docs/compatibility.md): DataSQRL builds on top of Flink and the Flink connector ecosystem. ## Optional - [Full documentation](https://docs.datasqrl.com/docs/llms-full.txt): all of the above pages in a single file - [Blog](https://docs.datasqrl.com/blog): articles on the design of DataSQRL and its harness for data engineering agents - [DataSQRL v0.11 documentation](https://docs.datasqrl.com/docs/v0-11/llms.txt) - [Unreleased `main` documentation](https://docs.datasqrl.com/docs/main/llms.txt)