Skip to main content

Agentic Data Engineering for Operational Data

Real-time pipelines, data APIs, and data products on Kafka, Flink, Postgres, and Iceberg. Built by your coding agent, verified by a compiler.

For teams building on Kafka, Flink, Postgres, and Iceberg who need data APIs and real-time pipelines. Not sure it fits?

DataSQRL harness: your skills and policies, a coding agent, and the DataSQRL compiler, validator, and simulator packaged as one DataSQRL agent

Agents write data code faster than humans can verify it

Writing code is no longer the bottleneck for data teams. Trusting and managing it is.
General-purpose coding agents solve data engineering tasks with dozens of files in multiple languages. The results look plausible but contain subtle bugs and misalignments that are hard to spot. In operational systems that code feeds live applications, so errors reach customers before anyone reviews them.

A coding agent on its own produces dozens of files; with the DataSQRL harness it produces one readable SQL file that the compiler turns into every deployment asset

Automate operational data engineering without giving up control

Human Control

One SQL file your team can read.

Not thousands of lines of Python, dbt, YAML, and glue. The whole pipeline, from ingest to API, is expressed in SQL you can understand, run, and approve.

Correctness

A compiler instead of guesswork.

Deterministic generation removes errors at the seams between systems. Relational validation and event-time replay tests catch the bugs that code review misses.

Safety

Guardrails you can inspect and extend.

Problems surface at compile time. Every compile writes out lineage, schemas, and the full data flow, so you can enforce your organization's policies automatically.

banking.sqrl
IMPORT banking_data.*;

-- Latest version of each account from the CDC stream
Accounts := DISTINCT AccountsCDC ON account_id ORDER BY update_time DESC;

-- Enrich transactions with the account as of transaction time
SpendingTransactions :=
SELECT t.*, h.name AS creditor_name, h.type AS creditor_type
FROM Transactions t
JOIN Accounts FOR SYSTEM_TIME AS OF t.tx_time a
ON t.credit_account_id = a.account_id
JOIN AccountHolders FOR SYSTEM_TIME AS OF t.tx_time h
ON a.holder_id = h.holder_id;

/** Spending transactions for the authenticated account
within [from_time, to_time). Exposed via GraphQL, REST, and MCP. */
SpendingTransactionsByTime(
account_id STRING NOT NULL METADATA FROM 'auth.accountId',
from_time TIMESTAMP NOT NULL,
to_time TIMESTAMP NOT NULL
) :=
SELECT * FROM SpendingTransactions
WHERE debit_account_id = :account_id
AND :from_time <= tx_time AND :to_time > tx_time
ORDER BY tx_time DESC;

Human Control: Review the Logic, Not the Plumbing

The agent's output is declarative SQL covering ingestion, transformation, storage, and a secure MCP/REST/GraphQL endpoint. Easy to understand, review, and argue about.

Aggregations, units, filters, and joins sit on one screen. One command runs the whole pipeline locally, API included, so you can inspect real results and experiment quickly.

DataSQRL compiles one logical model into assets for every engine

Correctness: Deterministic Where It Matters

Language models are probabilistic. Mapping types, keys, schemas, and connectors across Flink, Kafka, Postgres, Iceberg, and the API layer requires strict rule-following that's better handled by a compiler.

Every deployment asset is generated from one logical model, so the systems cannot disagree.

tests
-- Snapshot test, replayed at original event timestamps
/*+ test */
SpendingByHolderTest :=
SELECT creditor_name, COUNT(*) AS tx_count, SUM(amount) AS total
FROM SpendingTransactions
GROUP BY creditor_name ORDER BY creditor_name;

-- Assert that every transaction was enriched with its creditor
/*+ test(no_rows) */
NoUnenrichedTransactions :=
SELECT * FROM SpendingTransactions WHERE creditor_name IS NULL;

Correctness: Test the Pipeline in Motion

Static test fixtures miss the bugs that only appear over time. The simulator runs the real deployment artifacts and replays events at their original timestamps.

Late and out-of-order data, races between streams, idle sources, updates, and deletes become deterministic tests.

Late events, duplicates, and races between streams do not exist in a batch model, and they are where operational pipelines fail.

Why standard integration tests fall short →
build/pipeline_explain.txt
=== CustomerTransaction
Type: stream
Stage: flink
Inputs: _CardAssignment, _Merchant, sources.Transaction
Annotations:
- stream-root: Transaction
Primary Key: transactionId, time
Timestamp : time
Schema:
- transactionId: BIGINT NOT NULL
- cardNo: VARCHAR NOT NULL
- time: TIMESTAMP_LTZ(3) *ROWTIME* NOT NULL
- amount: DOUBLE NOT NULL
- merchantName: VARCHAR NOT NULL
- customerId: BIGINT NOT NULL

Safety: Every Pipeline Is Fully Inspectable

Each compile writes out the complete data flow: table types, inferred keys, timestamps, schemas, engine assignments, and every deployment asset.

Use these artifacts for lineage, impact analysis, and audit. Add custom rules that enforce your policies on PII, naming, retention, or residency for every pipeline the agent builds.

DataSQRL data flow DAG with full lineage

Safety: Guardrails by Construction

The compiler rejects invalid engine assignments, capability mismatches, and inconsistent data flows before anything is deployed.

API endpoints use parameterized queries, and authorization is bound to JWT claims in the SQL itself. Agent-built endpoints get these protections from the compiler, not from a prompt.

Your Harness, Your Organization

Every data organization has its own conventions, domain vocabulary, compliance requirements, and target infrastructure. DataSQRL is a harness you use to build an agent that honors those.

SkillsHow your team gathers requirements, plans, implements, tests, and deploys, plus domain and catalog knowledge
Validators & policiesCustom rules for governance, security, and data quality
Functions & connectorsYour UDFs, sources, sinks, and formats
Engines & deploymentFlink, Kafka, Postgres, Iceberg on Docker, Kubernetes, or cloud
Coding agentClaude Code, Codex, OpenCode, Pi, etc: your choice
# The agent's inner loop, driven by the harness
# 1. Validate and explain the data flow
docker run --rm -v $PWD:/workspace \
datasqrl/cmd compile package.json
# 2. Event-time replay tests
docker run --rm -v $PWD:/workspace \
datasqrl/cmd test test-package.json
# 3. Deployment assets for K8s or cloud
ls build/deploy/plan

What a DataSQRL Agent Builds

Data APIs

GraphQL, REST, and MCP endpoints generated from SQL, with authentication and authorization built in.

Real-Time Pipelines

CDC, temporal joins, windowed aggregations, and deduplication on Flink and Kafka, with low-latency results for applications and AI agents.

Data Products

Curated, documented, and tested pipelines and data APIs, each built for one business need.

Batch ETL into Iceberg

The same SQL in batch mode, landing enriched data in Iceberg with views for Snowflake, Databricks, Athena, and DuckDB.

DataSQRL compiles to open-source technologies

Runs on Infrastructure You Own

DataSQRL compiles to Flink, Kafka, Postgres, and Iceberg and deploys to Docker, Kubernetes, or managed cloud services. There is no proprietary runtime, and the harness is open source.

If Kafka is already in your stack, DataSQRL fits on top of it.

All you need is Docker and an API key from your LLM provider. Run the DataSQRL agent in your project folder:

docker run -e ANTHROPIC_API_KEY -it --rm \
--detach-keys="ctrl-],ctrl-]" -e TERM -e COLORTERM \
-v "$PWD":/workspace -w /workspace datasqrl/core-agent

Using OpenAI, Bedrock, Azure, or Vertex AI? Swap in that provider's environment variables, as shown in the getting started guide.

Getting Started

The DataSQRL core agent wraps the Pi coding agent with the DataSQRL harness and skills. Once it is running, tell it what you need in plain English:

Build a pipeline that ingests our order data from Kafka in real time and serves hourly revenue per product through an API.

You get SQL scripts with tests that you can read, run, and verify.

Want verifiers, judges, and implementation loops? Run the advanced agent from Claude Code, Codex, Cursor, or GitHub Copilot with the DataSQRL plugin.

Getting Started GuidePlugin Docs