Introduction​

"Why do I need that? Isn't Claude Code good enough?"

We get that question a lot. We are building an open-source data engineering harness, the tooling and guardrails that a coding agent uses to build data pipelines. Claude Code, Codex, and OpenCode already write plausible data pipeline code. So what is the harness for?

Parts 1, 2, and 3 were about generating the pipeline, validating it, and testing it. Part 4 is about the check that cannot be automated away: a person understanding what the pipeline actually means.

When an agent builds a conventional pipeline, the logic ends up spread across dozens or hundreds of files in two or three languages. Ingestion scripts, transformation jobs, table definitions, glue, API resolvers. Nobody holds that in their head. So the bugs where every line is correct and the meaning is wrong slip through, and the human review that was supposed to be the last line of defense collapses under fatigue.

1. Assumptions that never meet​

When ingestion, transformation, and serving live in separate files written at different times, each stage quietly assumes a different grain, unit, or shape. Nothing forces those assumptions to meet.

Example: the transform emits one row per order line, expecting something downstream to aggregate. The serving query assumes one row per order and sums nothing. The API reports inflated totals, and the two assumptions never appear on the same screen.

2. The same business rule, implemented twice, differently​

Scattered pipelines duplicate logic. A filter, a currency conversion, a status definition. The copies drift, and two parts of the system end up disagreeing about what a number means.

Example: "active customer" means status equals active in the ingestion filter and status is not closed in an API query. Two endpoints return different customer counts, and no file shows both definitions together.

3. Units, currency, and time zone drifting between stages​

The most expensive bugs are not broken code. They are correct code about the wrong meaning. Cents treated as dollars. UTC treated as local. One currency treated as another. The mismatch lives across a file boundary where the unit is never written down.

Example: the stream layer stores an amount in cents and a serving view applies a tax rate as if it were dollars, producing charges off by a factor of 100. Each file is internally consistent. The error lives only in the gap between them.

4. Hidden coupling that breaks on edit​

In a sprawling codebase, files depend on each other through implicit contracts. A column name, an ordering, a nullability. An agent changing one file cannot see what else relies on it.

Example: the agent renames a column in a transform script to tidy things up. A resolver in a different directory that referenced the old name starts returning null. Nothing connects the two at edit time.

5. A filter applied on one path and forgotten on another​

When the same source feeds several downstream paths through different files, a filter or a deduplication applied on one path and missing on another lets inconsistent data through. The omission is invisible because the paths are never read together.

Example: an "exclude test accounts" filter is applied in the reporting path but missing from the path feeding a fraud model. The model quietly scores synthetic accounts, and no single file shows that one path is filtered and the other is not.

6. Reviewer fatigue​

Human review is the real bottleneck once agents write pipelines quickly. Asking a person to work through thousands of lines across many files and languages produces fatigue and missed bugs, which is the opposite of validation.

Example: a reviewer signs off on a 2,000-line, fifteen-file change after skimming, missing a one-line grain error in file nine. Nobody sustains attention across that surface.

7. Handoffs that lose the design​

When the logic is distributed across many files and languages, no single person holds the whole picture. Handoffs lose context, and the next engineer or agent changes things blind to the original design.

Example: the engineer who built the pipeline leaves. Their replacement cannot work out why a particular deduplication exists, removes it, and reintroduces the duplicate-records bug it was silently preventing.

8. Requirements you cannot actually verify​

The point of review is confirming the pipeline does what was asked. You cannot confirm that against logic you cannot read as a whole, so requirements get marked done on faith.

Example: a compliance rule requires closed accounts to be excluded from a report. Verifying it means tracing the rule across ingestion, transformation, and serving files. The reviewer assumes it is handled and ships a violation.

Summary​

None of these is a bug a validator can flag, because every individual line is correct. The error is in the meaning: how stages align, what a number represents, an assumption two files quietly disagree about. The only thing that catches a misalignment of meaning is a person who understands the pipeline, and understanding is impossible when the logic is scattered across hundreds of files in several languages.

One readable file collapses that surface. The entire data flow, from ingest to transform to store to serve, is expressed as one declarative SQL logic in the language most data engineers already read. That makes the whole pipeline comprehensible in one sitting, which turns review from a fatiguing rubber stamp back into real scrutiny, and frees the engineer to spend that attention on the requirements and the meaning that agents and validators cannot judge for themselves.

As pipeline creation speeds up, human understanding becomes the bottleneck. Keeping the logic in one readable place is how you keep that bottleneck open.

It is open source, feel free to try it out yourself: DataSQRL on GitHub.

Introduction​

"Why do I need that? Isn't Claude Code good enough?"

We get that question a lot. We are building an open-source data engineering harness, the tooling and guardrails that a coding agent uses to build data pipelines. Claude Code, Codex, and OpenCode already write plausible data pipeline code. So what is the harness for?

Parts 1 and 2 were about generating the pipeline correctly and validating it. Part 3 is about the accomplice in every failure so far: the test that passed.

An agent writes a test, runs it against a fixed snapshot of data, sees green, and ships. But a pipeline's hardest bugs only exist in motion which standard tests don't catch, leaving you to troubleshoot in production.

1. Testing on a static snapshot with nothing updating mid-run​

The default agent test loads a fixed dataset, runs the pipeline once, and checks the output. Every bug that requires data to change during execution is invisible. This is the largest blind spot, because it makes time-dependent correctness untestable by construction.

Example: the time-dependent join from Part 2, where a transaction gets enriched with a later version of an account, cannot reproduce on a static snapshot. Nothing updates while the test runs, so the naive join and the correct join produce identical output. The test proves the wrong code right.

2. Tests that depend on the wall clock​

Agents write tests against the system time, so the result depends on how fast the machine ran and when the test happened to execute. It passes locally, flakes in CI, and certifies nothing.

Example: a windowed aggregation tested against the current timestamp produces different bucket boundaries on every run. The snapshot never matches twice, so the agent "stabilizes" it by loosening the assertion until it no longer tests anything.

3. No late or out-of-order data in the test set​

Real streams deliver records late and out of order. An agent's test data is conveniently sorted and on time, so lateness handling and retraction logic never get exercised.

Example: an event that arrives after its window closed should either update the result or be dropped, depending on the policy. The test never includes one, so a pipeline that quietly discards late data passes while losing records in production.

4. Races between interleaved streams​

When two streams feed a join, correctness depends on how they interleave. An agent testing each stream in isolation never sees the race.

Example: a transaction arrives in the same instant an account flips from "active" to "frozen," and which value the enrichment picks decides whether a fraud check fires. The agent's test loads every account before any transaction, so the race never happens and the bug ships.

5. Idle sources and stalled time​

A source that goes quiet stalls the progress of event time and can freeze a join indefinitely. An agent's test keeps every source busy, so the stall never appears.

Example: in production, one low-volume source stops emitting overnight. Time stops advancing and the joined output silently halts. An always-busy test set cannot produce that condition.

6. Updates and deletes never tested​

Updates and deletes propagate through a streaming pipeline as retractions of earlier results. An agent that only tests inserts never checks that a later update or delete correctly replaces what came before.

Example: a record is inserted, then deleted upstream. A pipeline tested only on inserts keeps emitting the deleted entity, and the API serves a row that no longer exists.

7. Production bugs that cannot be turned into a test​

When something breaks in production, the fix starts with reproducing it. Without timestamp-accurate replay, a time-dependent failure cannot be reconstructed, so the fix is a guess.

Example: a customer reports numbers that drifted last Tuesday. The agent cannot recreate Tuesday's exact sequence of events, patches speculatively, and never confirms the fix worked.

8. The rare scenarios only time control can construct​

The scenarios most likely to cause an outage are a leap in event time, a burst, a duplicate replay, a backfill colliding with live data. None of them can be built out of static fixtures, so they go untested until they happen.

Example: a backfill of historical data is replayed alongside the live stream, and the pipeline double-counts because it was never tested against overlapping time ranges.

Summary​

The thread through all eight is time. The bugs that matter in a data pipeline are time-dependent, and an agent testing on static, on-time, single-stream snapshots is testing the one regime where time cannot hurt it.

A simulator changes what the test runs against. It executes the real generated deployment assets in a container and replays events at their original timestamps, so time-consistent semantics hold and the same input always produces the same output. This allows testing those things standard integration tests cannot: mid-run updates, late and out-of-order data, interleavings and races, idle sources, retraction sequences, and faithful replay of a production incident. Tests run deterministically, on a laptop, in a tight loop with the agent before anything is proposed for deployment.

A race condition that would take weeks to surface in production becomes a test case you write on purpose.

The event-time simulator is part of the open source data engineering harness, so you can replay your own scenarios and break the pipeline yourself: DataSQRL on GitHub.

Introduction​

"Why do I need that? Isn't Claude Code good enough?"

We get that question a lot. We are building an open-source data engineering harness, the tooling and guardrails that a coding agent uses to build data pipelines. Claude Code, Codex, and OpenCode already write plausible data pipeline code. So what is the harness for?

Part 1 was about the seams between systems, and how a transpiler generates them deterministically. Part 2 is about a harder problem.

Some bugs have no correctness condition in the query text at all. The condition lives in the relationship between a query, the data it reads, and how that data changes over time. An agent reads SQL/code as text, and at the text level these bugs are invisible until the production deployment fails.

1. No indexes for the paths the API actually queries​

An agent generating a database schema writes the table definitions and stops. It never reasons about which columns the API filters and sorts on, so the serving database has no index matching its access paths. The pipeline is fast on a thousand test rows and collapses into full table scans at production volume.

Example: An endpoint that looks up a customer by email scans every customer row on every call. Nothing about the query is wrong. Tail latency goes from milliseconds to seconds as the table grows.

2. The wrong primary key, or none at all​

Moving a changing table into the database needs two things. A primary key that matches the upstream key, and a write mode that replaces rather than appends. Agents omit the key, pick the wrong columns, or append what is logically a changelog. You get duplicate rows that accumulate forever, or the last writer wins at the wrong grain.

Example: An accounts table built from a feed of database changes is written with no primary key. Every account update inserts a new row instead of replacing the previous one, and the API starts returning three conflicting records for the same account.

3. Treating a changelog as an event stream, or the reverse​

The most consequential property of a source is what kind of table it is. An append-only stream of events, a changelog where later rows replace earlier ones, and a lookup table are three different things, and which operations are valid depends entirely on which one you have. An agent that cannot see the distinction ingests a stream of updates as if each were a new event, or collapses an event stream that was never meant to be deduplicated.

Example: The agent reads an account update feed as a plain append stream. Every status change becomes another "account," and the count of active accounts inflates by the number of edits each account has ever received.

4. Joins whose answer depends on when they run​

Join a stream to a table that changes over time with a plain join, and the result depends on when the join executed rather than on the state of the world when the event happened. It also changes retroactively whenever the other table updates. The query reads correctly. The time semantics are wrong.

Example: Transactions are joined to accounts with a regular join, so each transaction is enriched with whatever account classification exists at processing time. A Tuesday transaction gets labeled with Thursday's account type, and enrichments already written quietly change after the fact.

5. Broken time attributes and stalled watermarks​

Stream processing needs every source to carry an event timestamp and a watermark, the signal that tells the engine how far event time has advanced. That time attribute then has to survive joins and aggregations. Agents leave the watermark off a source, fall back to processing time, or break the attribute partway through. Temporal joins stall, windows never fire, or results stop being reproducible.

Example: One of three joined sources is declared with an incorrect watermark. The join waits forever for a time that never advances and the pipeline emits nothing. There is no error, just an empty result that reads as "no matching data."

6. NOT NULL contracts a nullable source cannot honor​

NOT NULL is a contract that has to hold across every system. An agent declares a column non-null downstream of a source or a join that can legitimately produce null values. One unmatched row then fails an insert or aborts a batch.

Example: An enriched column is declared NOT NULL in the database, but the outer join that produces it leaves the value null for unmatched transactions. The first unmatched row throws a constraint violation and halts the writer.

7. Ordering that does not actually order anything​

Deduplication and "latest version" logic depend on an ordering column that really does increase over time. Stable results depend on an order being defined at all. An agent that deduplicates on a column which can move in either direction, or that relies on an implicit order, produces results that change between runs and keeps the wrong version of a row.

Example: The agent deduplicates products by ordering on price, meaning "the latest one." Price is not monotonic, so the row that survives is whichever one carries the highest price rather than the most recent update. The current product record is simply wrong.

8. State that grows forever​

Streaming aggregations and joins hold state. An agent that groups on an ever-growing key, omits a retention bound, or aggregates at the wrong grain produces a pipeline whose memory grows without limit until it falls over. That happens long after the demo passed.

Example: The agent keys a running aggregation by transaction id. State grows by one entry per transaction forever, and the job degrades and then crashes weeks into production.

Summary​

None of these bugs is visible in the query text. They live in the semantics of the data flow: how data changes over time, what its key is, when its timestamp is valid, which system can run which operator, how much state a step accumulates. An agent reviewing that text approves every one of them. So does a human reviewing it.

A validator doesn't review text. It parses the pipeline into a relational plan and traverses it the way a query optimizer would, and every fix above falls out of that one traversal. It infers keys and pushes them into the write configuration. It tracks timestamps and watermarks through each join and aggregation. It classifies every table as a stream, a changelog, or a lookup, and checks each operator against that. It propagates nullability, verifies that ordering columns actually increase, and reads the query workload to choose indexes. It rejects any plan that puts an operation on a system that cannot run it, or serves data that was never materialized.

When something fails, the validator names the table, the column, and a suggested fix. That matters more than it sounds. In our testing, an agent handed structured feedback fixes the problem far more reliably than one reasoning backward from an opaque runtime error.

The validator is open source, and you can add your own rules: DataSQRL on GitHub.

Introduction​

"Why do I need that? Isn't Claude Code good enough?"

We get that question a lot. We are building an open-source data engineering harness, the tooling and guardrails that a coding agent uses to build data pipelines. Claude Code, Codex, and OpenCode already write plausible data pipeline code. So what is the harness for?

This series answers that. It looks at where general-purpose coding harnesses fail on data engineering work, and why those failures survive code review.

Part 1 is about the seams. A pipeline spans multiple data systems, and the same fact (a type, a name, an encoding) has to be restated in each one. Restating facts consistently across systems is strict rule-following. The thing doing the restating is a probabilistic model. It gets most of them right. The ones it gets wrong compile cleanly, pass the demo, and surface in production as overflowed numbers and fields that were never there.

A transpiler removes that entire class of error. It derives every boundary asset from one logical model, so the systems cannot disagree. The agent also writes less code, which means fewer tokens and fewer duplicated schema definitions cluttering its context.

1. The same column, typed three different ways​

A column in a pipeline exists at least three times. Once in the stream processor, once in the database, once in the API schema. Every system has its own type system, and an agent hand-maps between them one file at a time. A 64-bit integer becomes a 32-bit one at the API. A timestamp with a time zone becomes a bare local timestamp. A fixed-precision decimal lands as a floating point number. Nothing errors. The numbers are just wrong at the edges.

Example: an agent maps a transaction amount, stored as a 64-bit count of cents, to a 32-bit API type. Every transaction above roughly $21M wraps to a negative number. Every test built on small sample values passes.

A transpiler infers the type once from the logical model and generates the matching type for each system, so the three representations cannot drift apart.

2. Query dialects that look the same and mean different things​

Every engine speaks its own dialect. They share a vocabulary and disagree in the details: function names, null handling, division, casts, date arithmetic. An agent writes a transformation in one dialect and a serving query in another. Both run, neither complains, but they compute different answers.

Example: an integer division floors in the database and returns a fraction in the stream engine. A "share of total" computed in the stream layer and recomputed in a database view disagree for the same row, and nothing in either system flags it.

A transpiler generates each engine's dialect from one logical plan. A single declared computation is translated consistently instead of agent-written twice.

3. A field added in one place but not everywhere​

Adding one output column is a multi-system edit. The transformation, the connector that moves the data, the database table definition, the API type, and every endpoint that exposes it all have to gain the field together. An agent updates two or three of them and forgets the rest. The column either never reaches the database or arrives and stays invisible to every consumer.

Example: you ask for a merchant category on an enriched transaction stream. The agent updates the transformation and the database table but not the API type. The field does not exist for any consumer, and there is no error to point at.

With a transpiler you add the column once. Every downstream asset is regenerated from that one definition.

4. Connector configuration where two systems meet​

Wherever two systems meet, a connection has to be configured: names, formats, key fields, write mode. This is hand-written glue, and an agent gets it wrong in ways that never fail loudly. A key field that does not match the table's key. An append-only write for something that is logically an update. A missing time attribute.

Example: the agent configures the write from the stream layer into the database but never aligns the update key. Concurrent updates race, and the stored row flickers between values.

A transpiler treats every cut between systems as a generated connection. Both sides of every boundary derive from the same source of truth.

5. An API contract the data model cannot honor​

When the API schema is hand-written separately from the queries behind it, the contract drifts away from the data. A field marked as always present that can return nothing. A list marked non-empty that can be empty. An argument whose type does not match the query parameter. The API now advertises guarantees the pipeline cannot keep, and consumers get errors where they were promised values.

Example: the agent declares a customer's order summary as always non-empty. A customer with zero orders violates the contract and fails the whole query.

A transpiler builds the API schema from the data model. When you supply your own schema, it raises a compile error wherever schema and model disagree, rather than letting the mismatch reach consumers.

6. Internal columns leaking into the public interface​

Tables carry columns the system fills in itself: generated ids, ingest timestamps, computed values. An interface that accepts writes must ask clients for the columns they actually supply and nothing else. An agent hand-writing that interface routinely includes the internal ones, so valid writes are rejected, or omits a real one, so writes arrive incomplete.

Example: the write interface for an event asks for the generated id and the ingest timestamp. Every well-formed client insert is rejected for sending fields the server computes on its own.

A transpiler derives the input contract from the table definition and excludes system-populated columns, so the interface matches what a client is meant to send.

7. The same operation exposed three ways, three different shapes​

Operations are usually published over more than one protocol. An agent hand-coding each one produces views that disagree. An operation present on one protocol and missing on another. A route using the wrong method for its payload. A path whose arguments do not match the query signature. Consumers on one protocol get a broken endpoint, or none at all.

Example: a query that takes a filter is exposed as a request that carries the filter in URL parameters. The filter is silently truncated, and every call returns the unfiltered result set.

A transpiler generates every protocol from one authoritative API model, so names, paths, and signatures stay aligned across all of them.

8. Identifier casing, quoting, and reserved words​

Systems disagree about case sensitivity, quoting, and which words are reserved. An agent carrying a name across them by hand introduces mismatches that break the mapping from API field to stored column. A mixed-case name folded to lowercase by one system no longer matches the code looking for it. A name that is a reserved word parses on one engine and fails on the next.

Example: a column named order compiles in the stream layer and produces invalid table definitions in the database. The agent "fixes" it by quoting the name in one place and leaving it unquoted in another, and the API field resolves to nothing.

A transpiler applies one identifier mapping across every generated asset, so a logical name resolves to the right form on every system.

Summary​

We could keep going down our list of seam-breaking errors we have seen in agentic implementations, but you get the idea: Don't generate probabilistically what can be derived deterministically from a shared ground truth.

That's what the transpiler in the data engineering harness does. It takes one verified logical model as the ground truth and generates every boundary asset from it: the stream transformations, the database schema and queries, the connector configuration at each cut, the API schema, and every protocol endpoint. A type is inferred once and projected into each system. A field added once propagates to every asset. A name maps the same way everywhere. Both ends of a connection come from the same definition.

Because a compiler does that mapping instead of a language model, the entire category of "the systems disagree at the seam" does not need review to catch. It cannot be expressed. The agent decides what the pipeline computes. The transpiler guarantees the boundaries agree.

This has the additional benefit that there is a lot less code for the agent to generate at the seams. That means faster implementation, fewer tokens, less context pollution, and less technical debt.

Want to learn more about what exactly gets generated? The transpiler is part of the open source data engineering harness, so you can read exactly what gets generated: DataSQRL on GitHub.

How do we ensure that AI-generated data systems meet the rigorous non-functional requirements that production data platforms demand? This article presents our framework for integrating AI coding agents into data engineering workflows while maintaining data quality, reliability, governance, and trust.

The Scaling Challenge​

Organizations are targeting 3-5x productivity improvements through AI-assisted development. The velocity is real, but it creates an unsustainable burden on traditional data engineering practices:

• Manual code review can't scale when agents generate dozens of pipeline changes daily
• Ad-hoc data quality checks miss subtle issues that only manifest at scale or over time
• Tribal knowledge about production constraints doesn't transfer to AI agents
• Integration testing becomes a bottleneck when deployment velocity outpaces validation capacity
• Operations and troubleshooting overwhelm teams when they have to manage dozens of pipelines in production

The fundamental problem is coding agents optimize for functional correctness (e.g. does the query return the right results?) while production data systems require a much broader set of guarantees.

Without systematic guardrails, AI-generated data pipelines work in demos but fail in production and overwhelm the data engineering teams that have to fill the gaps.

A Data Quality Governance Framework​

Successfully integrating AI into data engineering requires a governance framework that addresses three dimensions: transparency, validation, and operations.

Transparency: Exposing Data Lineage and Reasoning​

AI agents need to operate within systems that expose their reasoning and the data flows they create. This serves two purposes: enabling human oversight and providing feedback for iterative refinement.

What does transparency look like for data engineering?

• Clear Transformations: Well-defined data transformations that make it obvious what was transformed and why
• Data lineage tracking: Trace every output field back through transformations to source systems
• Computational DAGs: Visualize how data flows from ingestion through processing to serving
• Schema inference: Make explicit the types, keys, and timestamps the agent has assumed
• Execution plans: Show which engines execute which computations

When an agent proposes a data pipeline, we should see not just the SQL code but the complete picture: where data originates, how it transforms, what guarantees apply at each stage, and how it ultimately reaches consumers.

=== SpendingTransactions
ID:     default_catalog.default_database.SpendingTransactions
Type:   stream
Stage:  flink
Inputs: Transactions, Accounts, AccountHolders

Annotations:
 - temporal-join: uses event-time semantics
 - stream-root: Transactions

Primary Key: transactionId, tx_time
Timestamp  : tx_time

Schema:
 - transactionId: BIGINT NOT NULL
 - amount: DECIMAL(10,2) NOT NULL
 - tx_time: TIMESTAMP_LTZ(3) *ROWTIME* NOT NULL
 - creditor_name: VARCHAR NOT NULL

This representation combines inferences from the logical layer (primary keys, timestamps) with physical mappings (execution stage, inputs) to give us complete visibility into agent-generated pipelines.

Validation: Real-Time Quality Assessment​

Every agent action must pass through validation layers that assess correctness before execution. Here's what makes data pipelines different from application code: bugs often produce silently wrong results: queries that execute successfully but return incorrect data.

Effective validation operates at multiple levels:

Logical Validation

• Syntax and schema verification
• Data type inference and consistency checking
• Primary key and timestamp propagation validation
• Table type verification (stream vs. state semantics)

Physical Validation

• Engine capability matching (can Postgres execute this temporal join?)
• Data type mapping consistency across engines
• Connector configuration verification
• Topological constraint satisfaction (data must reach database before API can serve it)

Semantic Validation

• Business rule assertions
• Data quality constraints (nullability, referential integrity, value ranges)
• SLA verification (latency bounds, freshness guarantees)

The validation system needs to provide actionable feedback when checks fail. Rather than cryptic error messages, agents need comprehensive context and suggested fixes:

ERROR: Temporal join requires timestamp column on probe side

Table 'Transactions' is used in a temporal join with 'Accounts'
but lacks a rowtime attribute.

Suggestion: Add a timestamp column with WATERMARK definition:
  tx_time TIMESTAMP_LTZ(3),
  WATERMARK FOR tx_time AS tx_time - INTERVAL '5' SECOND

Operations: Continuous Monitoring and Autonomous Troubleshooting​

Building pipelines is only half the challenge. When you're managing dozens of AI-generated data pipelines in production, operations becomes the bottleneck. Traditional monitoring approaches (dashboards, manual alerts, runbooks) can't scale when pipeline count grows faster than team headcount.

The framework needs to support autonomous operations:

Continuous Monitoring

• Real-time data quality assertions that validate business rules on every record
• SLA tracking that measures end-to-end latency from source event to API availability
• Schema drift detection that catches upstream changes before they break downstream consumers
• Resource utilization monitoring that identifies capacity issues before they cause failures

Autonomous Troubleshooting

• Automatic correlation of symptoms to root causes using lineage information
• Self-healing for common failure modes (connector reconnection, checkpoint recovery, partition rebalancing)
• Intelligent alerting that groups related issues and suppresses noise
• Runbook automation that executes standard remediation steps without human intervention

Observability Integration

• Structured logging that links every record to its source transformation
• Distributed tracing across the complete pipeline (Kafka → Flink → Postgres → API)
• Metrics export to existing observability platforms (Prometheus, Datadog, CloudWatch)

The goal is a team of three data engineers being able to operate dozens pipelines in production. That's only possible when the harness handles routine operations autonomously and escalates only the issues that genuinely require human judgment.

Capturing Data Engineering Expertise​

The effectiveness of AI in data engineering depends on systematically capturing and encoding human expertise. This falls into three categories: domain knowledge, operational patterns, and failure modes.

Domain Knowledge Encoding​

Data engineers carry implicit knowledge about their data domains: which fields contain PII, how upstream systems behave during maintenance windows, what query patterns consumers actually use. This knowledge needs to be made explicit for agents to leverage.

What works:

• Schema annotations: Capture business semantics beyond technical types
• Data contracts: Formalize expectations between producers and consumers
• Quality rules: Encode domain-specific validity constraints
• Access patterns: Document how data gets queried in practice

-- Domain knowledge encoded in table definition
/** Customer spending transactions enriched with merchant details.
    PII: contains customer_id (indirect identifier)
    Freshness SLA: < 5 minutes from source event
    Primary consumer: Fraud detection system (latency-sensitive)
*/
SpendingTransactions := SELECT ...

Operational Pattern Libraries​

Production data pipelines exhibit recurring patterns: CDC deduplication, temporal enrichment joins, windowed aggregations, slowly changing dimensions. We encode them as reusable patterns to reduce the occurrence of subtle bugs when agents try to recreate them.

-- Pattern: CDC deduplication to current state
Accounts := DISTINCT AccountsCDC ON account_id ORDER BY update_time DESC;

-- Pattern: Temporal enrichment join
EnrichedTransactions := SELECT t.*, a.account_type
    FROM Transactions t
    JOIN Accounts FOR SYSTEM_TIME AS OF t.tx_time a
    ON t.account_id = a.account_id;

-- Pattern: Tumbling window aggregation
HourlyMetrics := SELECT
    window_start, COUNT(*) as event_count
    FROM TABLE(TUMBLE(TABLE Events, DESCRIPTOR(event_time), INTERVAL '1' HOUR))
    GROUP BY window_start;

These patterns encode not just the SQL syntax but the semantic intent and operational characteristics. When an agent needs CDC deduplication, it applies the established pattern rather than improvising a potentially incorrect solution.

Failure Mode Documentation​

Every production incident represents encoded knowledge about what can go wrong. Systematically capturing failure modes and their resolutions creates a corpus that agents can learn from:

• Symptoms: How the failure manifested (data delays, incorrect aggregates, schema mismatches)
• Root cause: The underlying issue (late data handling, join key mismatch, type coercion)
• Resolution: How we fixed it
• Prevention: What validation or pattern would have caught this earlier

Over time, this corpus enables agents to anticipate failure modes and proactively avoid them.

The Data Engineering Harness​

Implementing governance, validation, and expertise capture requires purpose-built infrastructure. We call this a data engineering harness: a system that provides the guardrails and feedback loops coding agents need to produce production-grade data systems.

The harness has three integrated components:

Conceptual Framework​

The framework provides a precise vocabulary for reasoning about data transformations:

Logical Layer: Expresses what transformations are needed using SQL extended with stream processing semantics. The declarative nature enables deep introspection. We can analyze query structure, infer schemas, and validate semantics.

Physical Layer: Represents how data gets processed through engine assignment and configuration. A cost-based optimizer maps logical operations to physical engines (Flink, Kafka, Postgres, Iceberg) while respecting capability constraints.

Why does this separation matter? Agents should reason about business logic (logical layer) while the harness handles infrastructure complexity (physical layer). This division produces higher quality results by keeping agent context focused on the problem domain.

Comprehensive Validation​

Validation operates continuously throughout the development lifecycle:

• Compile-time: Schema consistency, type safety, semantic correctness
• Plan-time: Physical feasibility, capability matching, optimization validity
• Test-time: Functional correctness against known inputs and expected outputs
• Deploy-time: Configuration validity, resource availability, dependency satisfaction
• Run-time: Data quality assertions, SLA monitoring, anomaly detection

Each validation stage produces structured feedback that agents consume for iterative refinement. The harness transforms validation failures into actionable guidance.

Real-World Feedback​

Static validation catches many issues but can't substitute for execution feedback. The harness provides two mechanisms for real-world validation:

Simulation: Execute pipelines locally with timestamp-accurate event replay. The simulator runs the complete stack (Flink, Kafka, Postgres) in Docker, enabling agents to test against realistic data volumes and timing scenarios. Crucially, simulation is deterministic: the same inputs always produce the same outputs, enabling reliable regression testing.

Production Telemetry: Monitor deployed pipelines and correlate observations back to source code. When latency increases or data quality degrades, the harness links metrics to specific transformations, enabling autonomous troubleshooting.

Implementation: The DataSQRL Approach​

DataSQRL implements this harness architecture as an open-source framework. Here's how the abstract governance principles translate to concrete tooling.

SQL as the Logical Layer​

DataSQRL uses SQRL as the logical representation, which is SQL extended with stream processing from Flink SQL and interface definitions. Why SQL?

• LLM familiarity: Most models are extensively trained on SQL
• Human readability: Engineers can verify agent output without learning new syntax
• Mathematical foundation: Relational algebra enables rigorous validation
• Declarative introspection: We can analyze and transform queries programmatically

-- Agent-generated pipeline in SQRL
IMPORT banking_data.*;

-- Deduplicate CDC stream to current state
Accounts := DISTINCT AccountsCDC ON account_id ORDER BY update_time DESC;

-- Enrich transactions with temporal join
SpendingTransactions := SELECT
    t.*,
    h.name AS creditor_name
FROM Transactions t
JOIN Accounts FOR SYSTEM_TIME AS OF t.tx_time a
    ON t.credit_account_id = a.account_id
JOIN AccountHolders FOR SYSTEM_TIME AS OF t.tx_time h
    ON a.holder_id = h.holder_id;

-- Define API endpoint
SpendingByAccount(account_id STRING NOT NULL) :=
    SELECT * FROM SpendingTransactions
    WHERE debit_account_id = :account_id
    ORDER BY tx_time DESC;

Deterministic Transpilation​

The mapping from logical to physical layer happens through deterministic transpilation, not agent generation. This eliminates an entire class of subtle bugs:

• Schema mismatches between engines
• Incorrect data type coercions
• Missing index structures
• Inconsistent serialization formats

The transpiler generates deployment artifacts (Flink plans, Kafka topics, Postgres schemas, GraphQL models) that are guaranteed consistent with the logical definition. Agents focus on business logic while the harness handles infrastructure integration.

Neuro-Symbolic Optimization​

Certain data engineering tasks are better handled by dedicated optimizers than LLM reasoning. DataSQRL implements a neuro-symbolic approach: agents handle high-level design while specialized solvers handle constraint satisfaction.

• Query Optimization: Apache Calcite's Volcano optimizer rewrites queries for performance
• Physical Planning: Cost-based optimizer assigns operations to engines while respecting topological constraints
• Index Selection: Lattice-based optimizer selects index structures that support query access patterns

Agents can provide hints to guide optimization (e.g. forcing specific engine assignments or partition keys) but the optimizer ensures constraint satisfaction. This leverages LLM strengths (reasoning under uncertainty, creative problem-solving) while delegating deterministic optimization to purpose-built systems.

Continuous Evaluation​

The harness supports continuous evaluation through automated testing infrastructure:

/*+test */
TransactionEnrichmentTest :=
    SELECT creditor_name, COUNT(*) as tx_count
    FROM SpendingTransactions
    GROUP BY creditor_name
    ORDER BY creditor_name;

Test definitions execute against known inputs with expected outputs captured as snapshots. The simulator replays events with precise timestamps, enabling tests for complex scenarios:

• Late-arriving events and watermark handling
• Out-of-order data processing
• Race conditions in temporal joins
• Schema evolution compatibility

Tests provide immediate feedback to agents and spot regressions in CI/CD infrastructure.

Organizational Implications​

Successfully integrating AI into data engineering requires organizational adaptation beyond tooling. What shifts as AI assumes greater responsibility for pipeline development?

From Implementation to Oversight​

As agents handle routine implementation, data engineers shift focus toward:

• Architecture review: Evaluating agent-proposed designs against organizational patterns
• Pipeline auditing: Reviewing agent-generated pipelines for correctness, efficiency, and compliance
• Domain encoding: Capturing business knowledge in schemas, contracts, and quality rules
• Failure analysis: Investigating production issues and encoding learnings
• Governance evolution: Refining validation rules and operational criteria

This shift parallels the evolution in other engineering disciplines where automation handles routine tasks while humans focus on judgment-intensive decisions.

From Manual Testing to Continuous Validation​

Traditional data pipeline testing, manual verification against sample datasets, can't scale with AI-accelerated development. Organizations need to invest in:

• Comprehensive test suites that encode expected behavior across edge cases
• Automated regression detection that flags behavioral changes between versions
• Production monitoring that validates data quality continuously
• Anomaly detection that identifies novel failure modes

The testing burden shifts from per-deployment verification to continuous infrastructure maintenance.

From Tribal Knowledge to Encoded Expertise​

AI agents can't access knowledge that exists only in engineers' heads. Organizations need to systematically externalize:

• Data domain semantics and business rules
• Operational patterns and anti-patterns
• Historical failure modes and resolutions
• Consumer requirements and SLAs

This documentation effort has value beyond AI enablement: it improves onboarding, reduces key-person dependencies, and creates institutional memory that persists through team changes.

The Path Forward​

AI-assisted data engineering is here. Organizations that successfully integrate AI into their data platforms will achieve significant productivity gains while maintaining the reliability that production systems demand.

AI integration requires infrastructure, not just tools. Coding agents operating without guardrails produce pipelines that work in demos but fail in production. Agents operating within a purpose-built harness with comprehensive validation, real-world feedback, and encoded expertise produce pipelines that meet production requirements.

The data engineering harness represents this infrastructure: a system that provides the governance, validation, and feedback loops necessary for AI-assisted data platform automation.

DataSQRL implements this harness as an open-source framework. You can customize it to encode your domain knowledge, integrate your validation rules, and build an automated data platform tailored to your requirements.

The question is no longer whether AI will transform data engineering, but how we adapt our practices, tooling, and teams to harness its potential while maintaining the trust that data consumers depend on.

Getting Started​

To explore AI-assisted data engineering with DataSQRL:

1. Build a project from scratch to understand harness components
2. Explore example projects demonstrating common patterns
3. Read about the harness architecture for detailed technical background
4. Contribute to the open-source project to shape the future of AI-assisted data engineering

Why a Data Engineering Harness?​

Coding agents are transforming software development. Tools like Claude Code, Copilot, and Codex can generate application code, write tests, and even refactor entire codebases. But data engineering presents unique challenges that general-purpose coding agents struggle to address.

The difference lies in non-functional requirements. When you build a data pipeline, functional correctness (does the query return the right results?) is just the starting point. Production data systems must also deliver:

• Data Quality: Consistent, accurate data with proper handling of late-arriving events, duplicates, and schema evolution
• Scalability: Performance that holds up as data volumes grow from gigabytes to terabytes
• Governance: Lineage tracking, access controls, and audit trails for regulatory compliance
• Reliability: Exactly-once semantics, failure recovery, and graceful degradation under load
• Cost Efficiency: Optimal resource utilization across compute, storage, and network

A coding agent can generate a SQL query that produces correct results on a test dataset. But will that query perform at scale? Does it handle late data correctly? Will it maintain data quality guarantees when upstream schemas change? These are the questions that justify data engineering as its own discipline and that general-purpose coding agents are not equipped to answer consistently.

A data engineering harness provides the guardrails, feedback loops, and domain-specific constraints that coding agents need to produce production-grade data systems. Without a harness, you get code that works in demos but fails in production. With a harness, you get data pipelines that embody decades of hard-won data engineering and domain-specific knowledge.

DataSQRL is that harness. It encodes the conceptual framework of data systems, validates implementations against data engineering best practices, and provides real-world feedback through simulation and production telemetry. The goal is to constrain coding agents into building data systems you'd actually trust to run in production.

The DataSQRL Harness​

For the purposes of automating data platforms, a comprehensive harness captures data schemas, data processing, and data serving to consumers. Specifically, we are building a harness for non-transactional data processing and serving.

The harness provides the frame of reference for implementing safe, reliable data processing systems. It captures the knowledge from Database Systems: The Complete Book combined with 25 years of data engineering experience.

DataSQRL breaks the harness into logical and physical layers.

Logical Layer​

The logical layer expresses what data transformations are needed to produce the desired results.

An obvious choice for the logical layer is Codd's relational model and its most popular implementation SQL.

The relational model is widely adopted, proven, and provides a solid mathematical foundation. Most LLMs are trained on lots of SQL code and related documentation. And it is easy for humans to read. Modern versions of SQL (e.g., the SQL:2023 standard) support semi-structured data (JSON), polymorphic table functions, and complex pattern matching to address the messy reality of data platforms.

While the relational model and SQL are a good starting point, we need two additions to achieve the expressibility that modern data platforms require.

1. Dataflow​

The relational model uses set semantics. That is inconvenient for representing data flows which are important for data pipelines.

Jennifer Widom's Continuous Query Language extends the relational model with data streams and relational operators for moving between streams and sets.

Flink SQL, based on Apache Calcite, is the most widely adopted implementation of this extended relational model. That's why we use Flink SQL as the basis of the logical layer in DataSQRL.

Using a declarative language for the harness has a number of advantages from concise representation to deep introspection, but a practical shortcoming is the fact that some data transformations are easier to express imperatively. Flink SQL overcomes this by supporting user defined functions and custom table operators in programming languages like Java. This gives us a logical layer grounded in relational algebra with flexible extensibility to express complex data transformations imperatively.

DataSQRL builds on Flink SQL and adds 1) concise syntax for common transformations, 2) dbt-style templating, and 3) modular file management and importing. These features help with context management for LLMs by reducing the size of the active context that needs to be maintained during implementation and refinement.

-- Ingest data from connected systems
IMPORT banking_data.AccountHoldersCDC; -- CDC stream from masterdata
IMPORT banking_data.AccountsCDC;       -- CDC stream from database
IMPORT banking_data.Transactions;      -- Kafka topic for transactions

-- Convert the CDC stream of updates to the most recent version
Accounts       := DISTINCT AccountsCDC ON account_id ORDER BY update_time DESC;
AccountHolders := DISTINCT AccountHoldersCDC ON holder_id ORDER BY update_time DESC;

-- Enrich debit transactions with creditor information using time-consistent join
SpendingTransactions :=
SELECT
    t.*,
    h.name AS creditor_name,
    h.type AS creditor_type
FROM Transactions t
         JOIN Accounts FOR SYSTEM_TIME AS OF t.tx_time a
           ON t.credit_account_id = a.account_id
         JOIN AccountHolders FOR SYSTEM_TIME AS OF t.tx_time h
           ON a.holder_id = h.holder_id;

We call this SQL dialect SQRL. You can read the documentation for a complete reference of the SQRL language.

2. Serving​

In addition to data processing, a critical function of data platforms is serving data to consumers as data streams, datasets, or data APIs. Data APIs, in particular, are becoming more important with the rise of operational analytics and MCP (Model Context Protocol) for making data accessible to AI agents.

To support data serving, DataSQRL adds support for endpoint definitions via table functions and explicit relationships.

Table functions are part of the SQL:2016 standard and return entire tables as result sets computed dynamically based on provided parameters. In DataSQRL, table functions can be defined as API entry points.

/** Retrieve spending transactions within the given time-range.
  from_time (inclusive) and to_time (exclusive) must be RFC-3339 compliant date time.
*/
SpendingTransactionsByTime(
  account_id STRING NOT NULL METADATA FROM 'auth.accountId',
  from_time TIMESTAMP NOT NULL,
  to_time TIMESTAMP NOT NULL
) :=
SELECT * FROM SpendingTransactions
WHERE debit_account_id = :account_id
  AND :from_time <= tx_time
  AND :to_time > tx_time
ORDER BY tx_time DESC;

Furthermore, DataSQRL allows for explicit relationship definitions between tables which are important for API-based data access where results need to include related entities like most recent orders or recommendations for movie category. The relational model does not support traversing through an entity-relationship model, which is usually handled by an object-relational mapping layer when exposing an API. To avoid that extra complexity and impedance mismatch in our logical layer, DataSQRL provides first-class support for relationships.

-- Create a relationship between holder and accounts filtered by status
AccountHolders.accounts(status STRING) :=
SELECT * FROM Accounts a
WHERE a.holder_id = this.holder_id
  AND a.status = :status
ORDER BY a.account_type ASC;

With the addition of access functions and relationships, the logical layer maps directly to the entity-relationship model of GraphQL which DataSQRL uses as the logical representation for API-based data retrieval. This gives DataSQRL a highly expressive interface with a simple extension of the logical layer which retains conceptual simplicity of the harness.

The interface documentation provides more details on the serving layer of DataSQRL.

Physical Layer​

The physical layer represents how the data gets processed and served. It's a translation of the logical layer into executable code that runs on actual data systems.

Pipeline Architecture​

With hundreds of database systems and many more data infrastructure choices, it is a daunting challenge to construct a simple and coherent physical layer that is flexible enough to cover the diverse needs of data platforms.

After analyzing a wide range of data platforms, we identified that the vast majority of implementations combine multiple data systems from these categories:

• Database: for storing and querying data, e.g., PostgreSQL, MySQL, SQLServer, Apache Cassandra, Clickhouse, etc.
  • Table Formats and Query Engines: For analytic data, separating compute from storage can save money and support multiple consumers. DataSQRL conceptualizes this as a "disintegrated database" with table formats for storage (e.g., Apache Iceberg, DeltaLake, Apache Hudi) and query engines for access (e.g., Apache Spark, Apache Flink, Snowflake).
• Data Processor: for batch or realtime transformation of data, e.g., Apache Spark, Apache Flink, etc.
• Log/Queue: for reliably capturing data and moving it between data systems, e.g., Apache Kafka, RedPanda, Kinesis, etc.
• Server: for capturing and exposing data through an API
  • Cache: sits between server and database to speed up frequent queries over less-frequently changing data.

We call each data system an engine and the above categories engine types. When looking at data platform implementations at the level of engine types, we see about 15 patterns emerge (the 10 most popular are documented here) that arrange those engines in a directed-acyclic graph (DAG) of data processing.

Hence, we use a computational DAG that models the flow of data from source to interface as the basis of our physical layer. Each node in the DAG represents a logical computation mapped to be executed by an engine. Thus, the physical layer provides an integrated view of the entire data flow.

Transpiler​

While the physical layer gives the AI control over what engine executes which computation, the actual mapping of logical to physical plan is done by a deterministic transpiler built in Apache Calcite. This avoids subtle bugs in data mapping and execution. The results of the transpilation are deployment assets which are executed by each engine. For example, the transpiler generates the database schema and queries for Postgres.

In the transpiler component, we make the following simplifying assumptions:

• The database engines support a version of SQL (e.g., PostgreSQL, T-SQL) or a subset thereof (e.g., Cassandra Query Language)
• The data processor supports a SQL-based dialect (e.g., Spark SQL, Flink SQL)
• The log engine is Apache Kafka compatible (e.g., RedPanda, Azure EventHub)
• The server has a GraphQL execution engine.

This modular architecture allows new engines to be added by conforming to the engine type interface and implementing the transpiler rules in Calcite where needed. At the same time, it abstracts much of the physical plan mapping complexity from the AI, which produces higher quality results and preserves context for higher-level reasoning.

Configuration​

DataSQRL uses a package.json file to configure the engines used to execute a data pipeline. The configuration file defines the overall pipeline topology, the individual engine configurations, and the compiler configuration. One file controls how the physical layer is derived and executed, making it simple for the AI to experiment with and fine-tune the physical layer.

Interface​

For the data serving interface, we use GraphQL schema as the physical representation which bidirectionally maps to the access functions, table schema, and relationships defined in the logical plan by naming convention. GraphQL fields are mapped to SQL or Kafka queries based on their respective definitions in SQRL. This allows the AI to fine-tune the API within the GraphQL schema.

Furthermore, REST and MCP APIs can be explicitly or implicitly defined through GraphQL operations. Implicit definition traverses the GraphQL schema from root query and mutation fields. Explicitly defined operations are provided as separate GraphQL files.

Using GraphQL as the physical representation for the API combines simplicity with flexibility while benefiting from the prevalence of GraphQL in LLM training data.

Validator​

The harness gives AI coding agents a frame of reference to reason about data pipeline and data product implementations. DataSQRL provides validation to support that reasoning and give users tools to ensure the correctness and quality of the generated pipelines and APIs.

Verification & Introspection​

Verification and introspection complement the harness by reinforcing the concepts, rules, and dependencies. DataSQRL provides validation at 3 levels: the logical layer, physical layer, and deployment assets (the code that gets executed by the engines).

Logical​

At the logical level, the DataSQRL compiler verifies syntax, schemas, and data flow semantics. This ensures that the data pipeline is logically coherent and that data integration points (e.g., between the SQL definitions and GraphQL schema) are consistent.

One of the benefits of using relational algebra as the basis for our harness is the ability to run rules and deep traversals over the operators in the relational algebra tree. The DataSQRL compiler uses Apache Calcite's rule and RelNode traversal framework to validate timestamp propagation, infer primary keys and data types, validate table types, and more. This validation component can be extended with custom rules to validate domain-specific semantics and constraints.

The validation component was designed to provide comprehensive context and suggested fixes for validation errors. In our testing, this produces significantly better results compared to the AI coding agent having to look up and reason about encountered errors.

Physical​

On compilation, DataSQRL produces the computational data flow DAG that represents the physical layer. DataSQRL generates a visual representation as shown above for human validation as well as a concise textual representation that is consumed by coding agents as feedback on their proposed solutions and to reinforce the conceptual data flow of the harness.

=== CustomerTransaction
ID:     default_catalog.default_database.CustomerTransaction
Type:   stream
Stage:  flink
Inputs: default_catalog.default_database._CardAssignment, default_catalog.default_database._Merchant, default_catalog.sources.Transaction
Annotations:
 - stream-root: Transaction
Primary Key: transactionId, time
Timestamp  : time
Schema:
 - transactionId: BIGINT NOT NULL
 - cardNo: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
 - time: TIMESTAMP_LTZ(3) *ROWTIME* NOT NULL
 - amount: DOUBLE NOT NULL
 - merchantName: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
 - category: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
 - customerId: BIGINT NOT NULL

This representation of the physical layer combines the inferences from the logical layer with the mapping to execution engines to provide a source-to-interface definition of the data flow.

Validation at the physical level ensures that data type mappings are consistent and that the engine assignments are valid, i.e., that an assigned engine has the capabilities to execute a particular operator. DataSQRL uses a capabilities component that extracts all requirements from an operator (e.g., temporal join, or a particular function execution) and validates that the engine supports the corresponding capabilities.

Deployment Assets​

The executable deployment assets are transpiled from the physical layer. Since the transpilation is deterministic, this yields better results than letting the coding agent generate them, and it keeps the harness concise. However, we generate all deployment assets in a text representation that the coding agent can easily consume as another source of feedback. This is particularly useful during troubleshooting where the deployment assets are the ultimate source of truth of what is being executed and allow the agent to reason "backwards" to the logical layer and how to fix it.

Specifically, we generate:

• Database: The database schema, index structures, and (parameterized) SQL queries for all views and API entrypoints.
• Data Processor: The optimized physical plan and compiled execution plan.
• Log: The topic definitions and filter predicates.
• Server: The mapping from GraphQL fields to database or Kafka queries as well as operation definitions. Also, the GraphQL schema if it is not provided.

Optimization​

While LLMs' reasoning ability under uncertainty is outstanding, we have found LLMs to perform worse and less consistently on deterministic optimization and constraint satisfaction problems. This finding is supported by a rich body of research in neuro-symbolic AI which researches the integration of neural networks (like LLMs) with symbolic computation (e.g., solvers, planners) and has documented how neural networks alone fall short for such tasks.

DataSQRL follows the neuro-symbolic approach and provides 3 types of planners for deterministic sub-tasks in the implementation and maintenance of data pipelines:

Query Optimization​

Query rewriting and optimization is a well-established technique for producing high-performing physical plans from relational algebra. DataSQRL relies on Apache Calcite's Volcano optimizer and HEP rule engine for this purpose.

Physical Planning​

The physical plan DAG is subject to a number of constraints forced by the real-world constraints of physical data movement. For example, when the API needs to serve data on request, that data must first be available in the database. It cannot be served directly from a data processor. These topological constraints combined with the capabilities of individual engines render many AI-proposed solutions invalid.

Hence, we implement a physical planner that uses a cost model with a greedy heuristic to assign logical operators to engines in a way that is consistent. The AI can provide hints to force the assignment of certain operators to specific engines which are added as constraints to the optimizer. This gives the AI control over allocations but shifts the burden of constraint satisfaction to a dedicated solver.

Index Selection​

Efficiently querying data in the database or table format requires index structures (or partition + sort keys) that support the access paths to the data. Otherwise, we execute inefficient table scans.

Index structure selection is another optimization problem that is better handled by a dedicated optimizer. We use an adaptation of Ullman et al's lattice framework for data cube selection since data cube selection and index selection are related problems with different optimization functions.

Real World Feedback​

A harness with complementary verification and introspection provides the foundation of a concise model for data pipelines with feedback on proposed solutions. However, that feedback is limited to the plan and does not account for the complexities of actual execution. Real-world feedback is critical for iterative refinement of production-grade implementations and troubleshooting issues that arise in operation.

DataSQRL provides two sources of real-world feedback: a simulator that's used at implementation time and telemetry collection from production deployments that captures the operational status of the pipeline.

Simulator​

The DataSQRL simulator executes the configured engines with the generated deployment assets within a Docker environment. The simulator can replay events and records at their original timestamp, allowing for deterministic reproducibility of real-world scenarios. This is important for creating realistic test cases as well as reproducing production issues for troubleshooting and regression testing.

By capturing and faithfully replaying records at their original timestamp, the simulator ensures time-consistent semantics of data flows and makes it simple to construct complex test cases for scenarios like race conditions.

Simulation is important in agentic coding workflows because it allows the agent to execute and refine the implementation in a feedback loop that is executed locally and can simulate scenarios that only occur rarely in production.

Read more about invoking the simulator and writing reproducible test cases.

Operations and Telemetry​

The most important source of real-world feedback is observing the deployed data pipeline in a production environment (or a closely approximated pre-prod environment). Observability is critical for assessing the health of the pipeline and troubleshooting any issues that may occur.

Logs and telemetry collection is a well-established practice for DevOps. What DataSQRL adds is the ability to link observed data back to the physical computation DAG so the agent can accurately reason about cause and effect. For data pipelines that execute across multiple engines, many complex errors arise at system boundaries and require reasoning across multiple systems. For example, an issue in the data processing layer may cause excessive writes to the database, degrading overall performance. To automate such troubleshooting, we need to correlate observations back to the physical data flow and logical layer.

DataSQRL currently assumes production operation in Kubernetes or Docker and provides hooks for extracting logs and telemetry data. That data is correlated back to the SQL code and configuration defining the pipeline via the deployment assets, allowing coding agents to reason about effective solutions for troubleshooting production issues autonomously.

Summary​

Data engineering is entering a new era of automation. Coding agents can now write SQL, configure pipelines, and deploy data systems. But without proper guardrails, they produce solutions that fail under the weight of production requirements.

DataSQRL is the data engineering harness that ensures AI coding agents produce high-quality pipelines. DataSQRL encodes decades of data engineering knowledge into a structured framework that coding agents can leverage to build production-grade data systems.

The harness provides three critical capabilities:

1. A Conceptual Framework grounded in relational algebra and stream processing that gives agents a precise vocabulary for reasoning about data transformations, with logical and physical layers that separate what from how.

2. Comprehensive Validation at every level ensures that agent-generated code meets data engineering standards before it reaches production: from syntax and schema validation through physical plan verification to deployment asset generation.

3. Real-World Feedback Loops through simulation and production telemetry that enable agents to iteratively refine implementations based on actual execution behavior, not just static analysis.

For any organization pursuing data platform automation, a data engineering harness is foundational to avoid shipping data pipelines that fail in production. Without it, you're asking general-purpose coding agents to navigate the complex constraints of distributed data systems without a map. With DataSQRL, you're equipping them with the domain expertise and multiple sources of feedback to succeed.

DataSQRL is open-source so you can customize it to build a self-driving data platform tailored to your organization.

Getting Started​

To try out DataSQRL:

1. Build a project from scratch with DataSQRL to see how the components of DataSQRL work
2. Explore the AI generated data products for a fictional bank based on the bank's catalog definition as an example.
3. Read the documentation
4. Check out the open-source project on GitHub

DataSQRL 0.10 has been released and the headline feature is supporting mutations for Iceberg tables. DataSQRL can now manage Apache Iceberg tables as sources and sinks.

Why is that a big deal? Up to this point, DataSQRL could read and write to Apache Iceberg tables, but you had to manage them explicitly. This new release makes it easy to share data through Apache Iceberg between DataSQRL pipelines.

Data Fast and Slow​

Let's back up a bit. Before 0.10 you could create tables in DataSQRL like this:

CREATE TABLE Clickstream (
    user_id STRING,
    event_time TIMESTAMP_LTZ(3) NOT NULL METADATA FROM 'timestamp',
    ad_id STRING,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
);

If you don't specify a connector explicitly, DataSQRL will manage the table for you, create the Kafka topics, and wire everything up. This allows you to expose mutation endpoints in GraphQL that store mutation events in Kafka and process them in Flink, making it easy to build event-driven microservices.

Kafka is great if you need data fast – in milliseconds. But it requires a separate Kafka cluster, which is challenging to maintain and costly. For many data use cases, you don't need data that quickly. Minutes and hours are just fine.

That's where Apache Iceberg shines. It's a table format for sharing data that does not require a separate data system to operate. All you need is local or cloud storage.

With the 0.10 release, DataSQRL supports Apache Iceberg for managed tables. Use Kafka for the fast data and Iceberg for the slow data. This allows you to balance speed with operational simplicity and cost.

To make this possible, DataSQRL 0.10 introduces a breaking change: You have to explicitly annotate where you want the tables you create to be persisted: iceberg or kafka. Use the engine SQL hint:

/*+ engine(iceberg) */
CREATE TABLE Clickstream (
    user_id STRING,
    event_time TIMESTAMP_LTZ(3) NOT NULL METADATA FROM 'timestamp',
    ad_id STRING,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
);

That's it. And make sure to upgrade your existing DataSQRL projects with /*+ engine(kafka) */ on any managed tables as you migrate to 0.10.

There are a lot more goodies in the 0.10 release. Check out the complete changelog for details. Shout out to Ferenc for driving this release and to our newest team members Wellington and Mate for making their first contributions to DataSQRL. Thank you!

What's Next?​

We are working steadily toward the 1.0 release of DataSQRL. Most of the big features are in, and we are primarily focusing on hardening what is there: additional test coverage, handling edge cases, and extending support.

One area of work that's particularly interesting is partition strategy optimization. We are extending support for statistics to make that happen. More on this soon.

As a result, Flink SQL significantly lowers the barrier to entry for building real-time data systems.

However, developing streaming applications differs fundamentally from traditional SQL query processing. One key difference is that streaming jobs often have multiple sinks populated by a single pipeline, sharing large portions of common data processing logic.

While Flink SQL provides mechanisms to express this concisely—using views and statement sets—in practice, this often results in duplicate processing in the generated job graph.

The Core Problem: Why Duplication Happens​

In Flink SQL, each sink maps to its own relational tree. These trees are:

1. Optimized individually.
2. Combined afterward into a single job graph.

By the time they are merged, the query optimization has introduced subtle difference in the shared data processing which render the subgraph merging ineffective. As a result, common processing logic, such as joins, gets duplicated.

Ironically, common SQL optimization techniques like predicate pushdown and projection pruning can make this worse in streaming contexts. While these optimizations are beneficial in traditional query processing (because they avoid computing unused data), they can fragment pipelines in streaming jobs and prevent subgraph sharing.

Example: Clickstream Enrichment with Two Aggregations​

Consider a streaming application that:

1. Ingests ad click events.
2. Enriches them with ad metadata.
3. Produces two separate aggregations.

We perform a temporal join between the clickstream and the ad inventory to enrich each click event with ad metadata.

After enrichment, we compute two aggregations:

• Hourly tumbling window that counts number of clicks per ad category.
• Daily tumbling window that counts clicks per advertiser.

Both results are written to a PostgreSQL database for querying.

Below is a simplified Flink SQL script that implements our clickstream aggregation.

-- Clickstream source
CREATE TABLE Clickstream (
    user_id STRING,
    event_time TIMESTAMP_LTZ(3) NOT NULL METADATA FROM 'timestamp',
    ad_id STRING,
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
    'connector' = 'kafka', ...
);

-- Ad inventory source
CREATE TABLE AdInventory (
     ad_id STRING,
     category STRING,
     advertiser STRING,
     launch_date TIMESTAMP_LTZ(3) NOT NULL METADATA FROM 'timestamp',
     PRIMARY KEY (ad_id) NOT ENFORCED,
     WATERMARK FOR launch_date AS launch_date - INTERVAL '5' SECOND
) WITH (
    'connector' = 'upsert-kafka', ...
);

-- Enrich clickstream with ads
CREATE VIEW EnrichedClicks AS
SELECT
    c.user_id,
    c.event_time,
    c.ad_id,
    a.category,
    a.advertiser
FROM Clickstream AS c
LEFT JOIN AdInventory FOR SYSTEM_TIME AS OF c.event_time AS a
ON c.ad_id = a.ad_id;

BEGIN STATEMENT SET;

-- Hourly tumbling window: clicks per category
INSERT INTO hourly_category_clicks
SELECT
    category,
    window_start,
    COUNT(*) AS click_count
FROM TABLE(
    TUMBLE(
        TABLE EnrichedClicks,
        DESCRIPTOR(event_time),
        INTERVAL '1' HOUR
    )
)
GROUP BY
    category,
    window_start;

-- Daily tumbling window: clicks per advertiser
INSERT INTO daily_advertiser_clicks
SELECT
    advertiser,
    window_start,
    COUNT(*) AS click_count
FROM TABLE(
    TUMBLE(
        TABLE EnrichedClicks,
        DESCRIPTOR(event_time),
        INTERVAL '1' DAY
    )
)
GROUP BY
    advertiser,
    window_start;

END;

Why Does this Cause Duplicate Processing?​

When running this Flink SQL script, the generated job graph duplicates the temporal join.

Why? Because the two aggregations needs slightly different data: The hourly aggregation needs category. The daily aggregation needs advertiser. Thus, predicate pushdown and projection pruning cause the optimizer to generate slightly different pipelines for each sink. As a result, the temporal join is executed twice.

In traditional SQL systems, this behavior is desirable because it avoids processing unnecessary fields when you submit a query. In streaming systems, however, this approach is suboptimal because it considers each sink in isolation and not the overall data processing of the entire streaming job.

For our simple application, it is much more efficient to compute the temporal join once and enrich the clickstream with all the ad metadata we need downstream. That eliminates an operator and redundant data processing. More importantly, it cuts the number of state requests to RocksDB in half which is the primary bottleneck for this job.

Fixing Job-Graph Duplication in Flink SQL​

Apache Flink already provides two important building blocks to eliminate this wasteful duplication:

• Subgraph elimination within the physical RelNode graph to remove duplicate processing.
• Compile plans, which generate a static artifact representing the generated job graph from Flink SQL.

The compile plan is especially useful because it allows validation of the generated job graph at compile time and assigns stable operator IDs. That's important for job evolution by preserving state mappings across job changes or Flink version upgrades.

Selective Rule Control in Calcite​

The issue lies in how the Calcite optimizer applies certain optimization rules, particularly those related to projection pruning and filter pushdown. These are the most likely culprits for causing minor differences in the optimized versions of shared logical plans.

In many real-world streaming scenarios with multiple sinks, these rules prevent effective subgraph elimination because they cause subtle difference in the RelNode graph whereas the subgraph elimination requires strict equality.

The solution is to selectively disable specific Calcite rules so that intermediate views do not get optimized for each RelNode tree and remain identical. Identical RelNode trees are then removed during the subgraph elimination phase of the Flink SQL optimizer.

How Do I use this?​

An easy way to remove job graph duplication is to use the DataSQRL compiler and configure the following in your project's package.json:

  "compiler": {
    "predicate-pushdown-rules": "LIMITED_RULES"
  }

This ensures only a limited set of Calcite rules are applied during the compiled plan optimization. DataSQRL is a data automation framework that compiles Flink SQL to data pipelines and it can compile your Flink SQL to a compiled plan with a single command you execute locally. For more information, check out the getting started tutorial.

As an alternative, you can selectively disable Calcite rules in your own instrumentation framework for Flink. Check out this code snippet to see what rules we are disabling. We highly recommend that you produce a compiled plan for your Flink SQL jobs for introspection and predictability. You can use the open-source Flink SQL Runner to execute compiled plans.

In effect, it allows you to define a streaming-specific optimization ruleset, rather than relying solely on optimizations designed primarily for traditional query workloads.

What's Next?​

While the ruleset tweaks described above address most cases of subgraph duplication in streaming Flink SQL, we still have some more work to do for certain edge cases.

In particular, SQL constructs that introduce correlation variables (e.g. UNNEST) into the Calcite logical plan do not get deduplicated yet because correlation variable have a static counter that makes each variable unique.

We implemented a normalization algorithm for correlation variables and are looking for ways to contribute it directly to the Apache Flink project.

DataSQRL 0.7 marks a major milestone in our journey to automate data pipelines, thanks to significant improvements to the serving layer:

• Support for the Model Context Protocol (MCP) for tooling and resource access
• REST API support
• JWT-based authentication and authorization

These features enable developers to build a wide range of production-ready data interfaces. This release also includes performance and configuration improvements to the serving layer of DataSQRL-generated pipelines.

You can find the full release notes and source code on our GitHub release page. To update your local installation of DataSQRL, simply pull the latest Docker image:

docker pull datasqrl/cmd:0.7.0

The Last Mile: Data Delivery​

Data delivery is the final and most visible stage of any data pipeline. It's how users, applications, and AI agents actually access and consume data. Most enterprise data interactions happen through APIs, making the delivery interface a critical component. At DataSQRL, we've invested heavily in automating the upstream parts of the pipeline: from Flink-powered data processing to Postgres-backed storage. With version 0.7, we turn our focus to the serving layer: introducing support for the Model Context Protocol (MCP) and REST APIs, as well as JWT-based authentication and authorization. These additions ensure seamless integration with most authentication providers and enable secure, token-based data access, with fine-grained authorization logic enforced directly in the SQRL script. This completes our vision of end-to-end pipeline automation, where consumption patterns inform data storage and processing—closing the loop between data production and usage.

Check out the interface documentation for more information.

Major Contributions​

In addition to the flagship features—MCP, REST, and JWT support—which we’ll discuss in more detail in future blog posts, the 0.7.0 release contains a number of additional features and improvements.

Configuration & CLI Improvements​

• Refactored CLI and Flink configuration logic.
• Improved error messages during package config validation.
• Enforced predictable ordering and sorting of config keys.
• Migrated deprecated config key naming (-dir → -folder), now with compile-time warnings.
• Standardized configuration schema and structured logging to /build/logs.

Testing Infrastructure Enhancements​

• Added new sqrl-container-testing module.
• Converted tests to use AssertJ.
• Increased test coverage for SqrlConfig and dependency mapping.
• Fixed test runner error reporting and exit code handling.
• Reworked dependent service startup to trigger post-compilation.

Flink-SQL Runner Integration​

• Integrated flink-sql-runner into DatasqrlRun.
• Temporarily merged sqrl-test module into sqrl-run.
• Simplified Docker image setup (public ghcr.io images).
• Updated submodule paths and version to 0.7.0.

Authentication & API Enhancements​

• Added initial JWT-based authentication support.
• Published documentation for JWT and Swagger-based OpenAPI specs for REST endpoints.
• Added batch GraphQL mutation support with transactional semantics.
• Replaced GraphQLBigInteger with native Long handling.

Kafka & Runtime Improvements​

• Kafka topic names now support templating.
• Added an async OpenAI test use case and resolved snapshot issues.
• Fixed intermittent WebSocket failures in SubscriptionClient.

Project Structure and CI Pipeline​

• Simplified project structure and removed outdated dependency declarations.
• Refactored CI pipeline and added automated GitHub package cleanup workflow.

Updated Dependencies​

Upgraded versions for the following dependencies and patched critical vulnerabilities:

• Apache Flink and Flink connectors (e.g., Postgres CDC)
• Vert.x and related plugins
• Apache Iceberg, DuckDB, PostgreSQL JDBC
• AWS SDK BOM
• JSON Schema Validator, Netty, OpenCSV
• Micrometer, Log4j, Reactor, Immutables, Testcontainers
• Maven plugins: Enforcer, GPG, Build-helper

Flink SQL Runner is here to change that. It's an open-source toolkit that simplifies development, deployment, and operation of Flink SQL applications—locally or in Kubernetes—without manual JAR assembly or scripting custom infrastructure pipelines.

From SQL to Production, Minus the Plumbing​

Imagine you're writing a Flink SQL job that reads from Kafka, enriches the data, and sinks to Iceberg. In theory, it's just SQL. But in practice, production deployment requires:

• Assembling dependencies into a JAR
• Writing YAML to configure connectors
• Injecting secrets for different environments

Flink SQL Runner eliminates those headaches. You get:

• Declarative execution with SQL scripts or compiled plans
• Simple deployments on Kubernetes via Flink Operator
• Environment isolation with variable substitution and UDF packaging

All without leaving the SQL layer.

Key Features​

• SQL and Plan Execution: Run raw SQL scripts or pre-compiled execution plans.
• Kubernetes-Native: Built for the Flink Kubernetes Operator—deploy SQL jobs without writing infrastructure code.
• Composable Toolkit: Use the pieces you need—Docker image, libraries, extensions—to suit your environment.
• Environment Variable Substitution: Inject secrets and environment-specific config into SQL or plan files using ${ENV_VAR} syntax.
• UDF Infrastructure: Load custom JARs and register system functions easily.
• Function Libraries: Drop-in UDFs for advanced math and OpenAI integration.

Flexible and Extensible​

Flink SQL Runner is not a monolith. You can:

• Run it standalone with Docker.
• Deploy it with the Flink Kubernetes Operator.
• Extend it via Maven or Gradle in your own Flink stack:

<dependency>
  <groupId>com.datasqrl.flinkrunner</groupId>
  <artifactId>flink-sql-runner</artifactId>
  <version>0.6.0</version>
</dependency>

Get Started​

The Flink SQL Runner project is open source and available on Github. Check out the README for more information on how to use and deploy the Flink SQL Runner.

Try it out, report issues, or contribute your own UDFs.

FlinkSQL adds the ability to process data incrementally to the classic set-based semantics of SQL. In addition, FlinkSQL supports source and sink connectors making it easy to ingest data from and move data to other systems. That's a powerful combination which covers a lot of data processing use cases.

In fact, it only takes a few extensions to FlinkSQL to build entire data applications. Let's see how that works.

Building Data APIs with FlinkSQL​

/*+ engine(kafka) */
CREATE TABLE UserTokens (
userid BIGINT NOT NULL,
tokens BIGINT NOT NULL,
request_time TIMESTAMP_LTZ(3) NOT NULL METADATA FROM 'timestamp'
);

/*+query_by_all(userid) */
TotalUserTokens := SELECT userid, sum(tokens) as total_tokens,
count(tokens) as total_requests
FROM UserTokens GROUP BY userid;

UserTokensByTime(userid BIGINT NOT NULL, fromTime TIMESTAMP NOT NULL, toTime TIMESTAMP NOT NULL):=
                SELECT * FROM UserTokens WHERE userid = :userid
                AND request_time >= :fromTime AND request_time < :toTime ORDER BY request_time DESC;

UsageAlert := SUBSCRIBE SELECT * FROM UserTokens WHERE tokens > 100000;

This script defines a sequence of tables. We introduce := as syntactic sugar for the verbose CREATE TEMPORARY VIEW syntax.

The UserTokens table does not have a configured connector, which mean we treat it as an API mutation endpoint connected to Flink via a Kafka topic that captures the events. This makes it easy to build APIs that capture user activity, transactions, or other types of events.

Next, we sum up the data collected through the API for each user. This is a standard FlinkSQL aggregation query and we expose the result in our API through the query_by_all hint which defines the arguments for the query endpoint of that table.

We can also explicitly define query endpoints with arguments through SQL table functions. FlinkSQL supports table functions natively. All we had to do is provide the syntax for defining the function signature.

And last, the SUBSCRIBE keyword in front of the query defines a subscription endpoint for requests exceeding a certain token count which get pushed to clients in real-time.

Voila, we just build ourselves a complete GraphQL API with mutation, query, and subscription endpoints. Run the above script with DataSQRL to see the result. Save it as usertokens.sqrl next to a usertokens-package.json file that sets script.main to usertokens.sqrl (see the configuration documentation), then run:

docker run -it --rm -p 8888:8888 -v $PWD:/workspace datasqrl/cmd run usertokens-package.json

Relationships for Complex Data Structures​

And for extra credit, we can define relationships in FlinkSQL to represent the structure of our data explicitly and expose it in the API:

User.totalTokens := SELECT * FROM TotalUserTokens t WHERE this.userid = t.userid LIMIT 1;

The User table in this example is read from an upsert Kafka topic using a standard FlinkSQL CREATE TABLE statement.

Code Modularity and Connector Management​

Many FlinkSQL projects break the codebase into multiple files for better code readability, modularity, or to swap out sources and sinks. That requires extra infrastructure to manage FlinkSQL files and stitch them together.

How about we do that directly in FlinkSQL?

IMPORT source-data.User;

Here, we import the User table from a separate file within the source-data directory, allowing us to separate the data processing logic from the source configurations. It also enables us to use dependency management to swap out sources for local testing vs production.

And we can do the same for sinks:

EXPORT UsageAlert TO mysinks.UsageAlert;

In addition to breaking out the sink configuration from the main script, the EXPORT statement functions as an INSERT INTO statement and creates a STATEMENT SET implicitly. That makes the code easier to read.

Learn More​

FlinkSQL is phenomenal extension of the SQL ecosystem to stream processing. With DataSQRL, we are trying to make it easier to build end-to-end data pipelines and complete data applications with FlinkSQL.

Check out the complete example which also covers testing, customization, and deployment. Or read the documentation to learn more.

You can find the full release notes and source code on our GitHub release page. To get started with the latest compiler, simply pull the latest Docker image:

docker pull datasqrl/cmd:0.6.0

A New Chapter: Flink SQL Integration​

With DataSQRL 0.6, we are embracing the Flink ecosystem more deeply than ever before. This release introduces a complete re-architecture of the DataSQRL compiler to build directly on top of Flink SQL's parser and planner. By aligning our internal model with Flink SQL semantics, we unlock a host of new capabilities and bring DataSQRL users closer to the vibrant Flink ecosystem.

This architectural shift allows DataSQRL to:

• Use Flink SQL syntax as the foundation, enabling more intuitive query definitions and easier onboarding for users familiar with Flink.
• Extend Flink SQL with domain-specific features, such as declarative relationship definitions and functions to define the data interface.
• Transpile FlinkSQL to database dialects for query execution.

Serving-Layer Power: Functions & Relationships​

DataSQRL 0.6 introduces first-class support for defining functions and relationships in your SQRL scripts. These constructs make it easier to model complex application logic in a modular, declarative fashion.

These features are purpose-built for powering LLM-ready APIs, event-driven architectures, and real-time user-facing applications.

Check out the language documentation for details.

Developer Tooling​

DataSQRL 0.6 provides a docker image for compiling, running, and testing SQRL projects. You can now quickly iterate and check the results. Or run automated tests in CI/CD.

Deployment Artifacts​

DataSQRL 0.6 removes deployment profiles and instead generates all deployment artifacts in the build/deploy/plan folder. This makes it easier to integrate with Kubernetes deployment processes (e.g. via Helm) or cloud managed service deployments (e.g. via Terraform).

Breaking Changes & Migration Path​

As this is a major release, DataSQRL 0.6 is not backwards compatible with version 0.5. The syntax and internal representation have been updated to align with Flink SQL and to support the new compiler architecture.

To help you transition, we’ve provided updated examples and migration guidance in the DataSQRL examples repository. We recommend starting with one of the updated use cases to get a feel for the new workflow.

Thanks to the Community​

This release wouldn’t have been possible without the contributions, bug reports, and thoughtful feedback from our growing community. Whether you opened a pull request, filed an issue, or joined a discussion, thank you. Your support drives this project forward.

We’re excited to see what you build with DataSQRL 0.6. If you haven’t joined the community yet, now’s a great time to get involved: star us on GitHub, try out the latest release, and share your thoughts.

Stay tuned for more updates, and happy building.

This article introduces the temporal join, compares it to the traditional inner join, explains when to use it, and why it is a secret superpower.

Table of Contents:

• The Join: A Quick Review
• The Temporal Join: Linking Stream and State
• Temporal Join vs Inner Join
• Why Temporal Joins are Fast and Efficient
• Summary

The Join: A Quick Review​

Let's take a quick detour down memory lane and revisit the good ol' join operation. That trusty sidekick in your SQL utility belt helps you link data from two or more tables based on a related column between them.

Suppose we are operating a factory with a number of machines that roast and package coffee. We place sensors on each machine to monitor the temperature and detect overheating.

We keep track of the sensors and machines in two database tables.

The Sensor table contains the serial number and machine id that the sensor is placed on.

The Machine table contains the name of each machine.

To identify all the sensors on the machine “Iron Roaster” we use the following SQL query which joins the Sensor and Machine tables:

SELECT s.id, s.serialNo FROM Sensor s 
    JOIN Machine m ON s.machineid = m.id 
    WHERE m.name = “Iron Roaster”

Why are joins important? Without it, your data tables are like islands, isolated and lonely. Joins bring them together, creating meaningful relationships between data, and enriching data records with context to see the bigger picture.

By default, databases execute joins as inner joins which means only matching records are included in the join.

So, now that we've refreshed our memory about the classic join, let's dive into the exciting world of temporal joins in stream processing systems like Apache Flink.

The Temporal Join: Linking Stream and State​

Picture this: you're a time traveler. You have the power to access any point in time, past or future, at your will. Now, imagine that your data could do the same. Enter the Temporal Join, the DeLorean of data operations, capable of taking your data on a time-traveling adventure.

A Temporal Join is like a regular join but with a twist. It allows you to join a stream of data (the time traveler) with a versioned table (the timeline) based on the time attribute of the data stream. This means that for each record in the stream, the join will find the most recent record in the versioned table that is less than or equal to the stream record's time.

The versioned table is a normal state table where we keep track of data changes over time. That is, we keep older versions of each record around to allow the stream to match the correct version in time. Like time travel, temporal joins can make your head spin a bit. Let’s look at an example to break it down.

Temporal Join vs Inner Join​

Back to our coffee roasting factory, we collect the temperature readings from each sensor in a data stream.

And we want to know the maximum temperature recorded for each machine.

Easy enough, let’s join the temperature data stream with the Sensors table and aggregate by machine id:

SELECT s.machineid, MAX(r.temperature) AS maxTemp 
FROM SensorReading r INNER JOIN Sensor s 
    ON r.sensorid = s.id GROUP BY s.machineid

But here is a problem: What if we moved a sensor from one machine to another during the day? With an inner join, all of the sensor’s readings would be linked to the machine it was last placed on. So, if sensor 1 records a high temperature of 105 degrees in the morning and we move the sensor to the “Iron Roaster” machine in the afternoon, then we might see the 105 degrees falsely show up as the maximum temperature for the Iron Roaster. See how time played a trick on our join?

And this happens whenever we join a data stream with a state table that changes over time, like our sensors that get moved around the factory. What to do? Let’s call the temporal join to our rescue:

SELECT s.machineid, MAX(r.temperature) AS maxTemp 
FROM SensorReading r JOIN Sensor FOR SYSTEM_TIME AS OF r.`timestamp` s
    ON r.sensorid = s.id GROUP BY s.machineid

Pretty much the same query, just a different join type. Just a heads-up: the syntax for temporal joins in Flink SQL is more complex.

As a temporal join, we are joining each sensor reading with the version of the sensor record at the time of the data stream. In other words, the join not only matches the sensor reading with the sensor record based on the id but also based on the timestamp of the reading to ensure it matches the right version of the sensor record. Pretty neat, right?

Whenever you join a data stream with a state that changes over time, you want to use the temporal join to make sure your data is lined up correctly in time. Temporal joins are a powerful feature of stream processing engines that would be difficult to implement in a database.

Why Temporal Joins are Fast and Efficient​

Not only do temporal joins solve the time-alignment problem when joining data streams with changing state, modern stream processors like Apache Flink are also incredibly efficient at executing temporal joins. A powerful feature with great performance? Sounds too good to be true. Let’s peek behind the stream processing curtain to find out why.

In stream processing, joins are maintained as the underlying data changes over time. That requires the stream engine to hold all the data it needs to update join records when either side of the join changes. This makes inner joins pretty expensive on data streams.

Consider our max-temperature query with the inner join: When we join a temperature reading with the corresponding sensor record, and that record changes, the engine has to update the result join record. To do so, it has to store all the sensor readings to determine which join results are affected by a change in a sensor record. This can lead to a lot of updates and hence a lot of downstream computation. It can also cause system failure when there are a lot of temperature readings in our data stream because the stream engine has to store all of them.

Temporal joins, on the other hand, can be executed much more efficiently. The stream engine only needs to store the versions of the sensor table that are within the time bounds of the sensor reading data stream. And it only has to briefly store (if at all) the sensor reading records to ensure they are joined with the most up-to-date sensor records. Moreover, temporal joins don’t require sending out a massive amount of updated join records when sensors change placement since the join is fixed in time.

Time to Wrap Up This Temporal Journey​

We've reached the end of our time-traveling adventure through the universe of temporal joins. We've seen how they're like the DeLorean of data operations, zipping us back and forth through time to make sure our data matches up just right. We've also compared them to the good ol' inner join.

Temporal joins help us avoid the pitfalls of time-alignment problems when joining data streams with changing state. They're also super efficient, making them a great choice for high-volume, real-time data processing.

And that’s why the temporal join is stream processing's secret superpower.

DataSQRL makes using temporal joins a breeze. With its simplified syntax and smart defaults, it's like having a personal tour guide leading you through the sometimes bewildering landscape of stream processing. Take a look at our Getting Started to see a complete example of temporal joins in action or take a look at our other tutorials for a step-by-step guide to stream processing including temporal joins.

Happy data time-traveling, folks!

You need a queue like Kafka to hold your events, a stream processor like Flink to process data, a database like Postgres to store and query the result data, and an API layer to tie it all together.

And that’s just the price of admission. To get a functioning data layer, you need to make sure that all these components talk to each other and that data flows smoothly between them. Schema synchronization, data model tuning, index selection, query batching … all that fun stuff.

The point is, you need to do a ton of data plumbing if you want to build a data-driven application. All that data plumbing code is time-consuming to develop, hard to maintain, and expensive to operate.

We need to make building with data easier. That’s why we are sending out this call to action to uplevel our database game. Join us in figuring out how to simplify the data layer.

We have an idea to get us started: Meet DataSQRL.

Introducing DataSQRL​

DataSQRL is a build tool that compiles your application’s data layer from a high-level data development language, dubbed SQRL.

Our goal is to create a new abstraction layer above the low-level languages often used in data layers, allowing a compiler to handle the tedious tasks of data plumbing, infrastructure assembly, and configuration management.

Much like how you use high-level languages such as Javascript, Python, or Java instead of Assembly for software development, we believe a similar approach should be used for data.

SQRL is designed to be a developer-friendly version of SQL, maintaining familiar syntax while adding features necessary for building data-driven applications, like support for nested data and data streams.

Check out this simple SQRL script to build a recommendation engine from clickstream data.

IMPORT clickstream.Clickstream; --Import clickstream data from Kafka
IMPORT content.Content;         --Import content from CDC stream

/* Find next page visits within 10 minutes */
_CoVisits := SELECT b.url AS beforeURL, a.url AS afterURL,
                    a.event_time AS `timestamp`
             FROM Clickstream b INNER JOIN Clickstream a ON b.userid=a.userid
                 AND b.event_time < a.event_time AND
                     b.event_time >= a.event_time - INTERVAL 10 MINUTE;
/* Recommend pages that are visited shortly after */
/*+query_by_all(url) */
Recommendation := SELECT beforeURL AS url, afterURL AS recommendation,
                         count(1) AS frequency FROM _CoVisits
                  GROUP BY beforeURL, afterURL
                  ORDER BY url ASC, frequency DESC;

This little SQRL script imports clickstream data, identifies pairs of URLs visited within a 10-minute interval, and compiles these pairs into a set of recommendations, ordered by the frequency of co-visits.

DataSQRL then takes this script and compiles it into an integrated data pipeline, complete with all necessary data plumbing pre-installed. It configures access to the clickstream. It generates an executable for the stream processor that ingests, validates, joins, and aggregates the clickstream data. It creates the data model and writes the aggregated data to the database. It synchronizes timestamps and schemas between all the components. And it compiles a server executable that queries the database and exposes the computed recommendations through a GraphQL API.

The bottom line: These 9 lines of SQRL code can replace hundreds of lines of complex data plumbing code and save hours of infrastructure setup.

We believe that all this low-level data plumbing work should be done by a compiler since it is tedious, time-consuming, and error-prone. Let’s uplevel our data game, so we can focus on what we are trying to build with data and less on the how.

Join Us on this Journey​

We have the ambitious goal of designing a higher level of abstraction for data to enable millions of developers to build data-driven applications.

We just released the first version of DataSQRL, and we recognize that we are at the beginning of a long, long road. We need your help. If you are a data nerd, like building with data, or wish it was easier, please join us on this journey. DataSQRL is an open-source project, and all development activity is transparent.

Here are some ideas for how you can contribute:

• Share your thoughts: Do you have ideas on how we can improve the SQRL language or the DataSQRL compiler? Jump into our community and let us know!
• Test the waters: Do you like playing with new technologies? Try out DataSQRL and let us know if you find any bugs or missing features.
• Spread the word: Think DataSQRL has potential? Share this blog post and star DataSQRL on Github. Your support can help us reach more like-minded individuals.
• Code with us: Do you enjoy contributing to open-source projects? Dive into the code with us and pick up a ticket.

Let’s uplevel our database game. With your help, we can make building with data fun and productive.

More Information​

You probably have a ton of questions now. How do I import my own data? How do I customize the API? How do I deploy SQRL scripts to production? How do I import functions from my favorite programming language?

Those are all great questions. Check out the documentation for answers.