The Whole Pipeline in One File a Human Can Actually Read

Part 4 of 4: How a data engineering harness eliminates the AI coding agent errors that survive review.

Introduction

"Why do I need that? Isn't Claude Code good enough?"

We get that question a lot. We are building an open-source data engineering harness, the tooling and guardrails that a coding agent uses to build data pipelines. Claude Code, Codex, and OpenCode already write plausible data pipeline code. So what is the harness for?

Parts 1, 2, and 3 were about generating the pipeline, validating it, and testing it. Part 4 is about the check that cannot be automated away: a person understanding what the pipeline actually means.

When an agent builds a conventional pipeline, the logic ends up spread across dozens or hundreds of files in two or three languages. Ingestion scripts, transformation jobs, table definitions, glue, API resolvers. Nobody holds that in their head. So the bugs where every line is correct and the meaning is wrong slip through, and the human review that was supposed to be the last line of defense collapses under fatigue.

1. Assumptions that never meet

When ingestion, transformation, and serving live in separate files written at different times, each stage quietly assumes a different grain, unit, or shape. Nothing forces those assumptions to meet.

Example: the transform emits one row per order line, expecting something downstream to aggregate. The serving query assumes one row per order and sums nothing. The API reports inflated totals, and the two assumptions never appear on the same screen.

2. The same business rule, implemented twice, differently

Scattered pipelines duplicate logic. A filter, a currency conversion, a status definition. The copies drift, and two parts of the system end up disagreeing about what a number means.

Example: "active customer" means status equals active in the ingestion filter and status is not closed in an API query. Two endpoints return different customer counts, and no file shows both definitions together.

3. Units, currency, and time zone drifting between stages

The most expensive bugs are not broken code. They are correct code about the wrong meaning. Cents treated as dollars. UTC treated as local. One currency treated as another. The mismatch lives across a file boundary where the unit is never written down.

Example: the stream layer stores an amount in cents and a serving view applies a tax rate as if it were dollars, producing charges off by a factor of 100. Each file is internally consistent. The error lives only in the gap between them.

4. Hidden coupling that breaks on edit

In a sprawling codebase, files depend on each other through implicit contracts. A column name, an ordering, a nullability. An agent changing one file cannot see what else relies on it.

Example: the agent renames a column in a transform script to tidy things up. A resolver in a different directory that referenced the old name starts returning null. Nothing connects the two at edit time.

5. A filter applied on one path and forgotten on another

When the same source feeds several downstream paths through different files, a filter or a deduplication applied on one path and missing on another lets inconsistent data through. The omission is invisible because the paths are never read together.

Example: an "exclude test accounts" filter is applied in the reporting path but missing from the path feeding a fraud model. The model quietly scores synthetic accounts, and no single file shows that one path is filtered and the other is not.

6. Reviewer fatigue

Human review is the real bottleneck once agents write pipelines quickly. Asking a person to work through thousands of lines across many files and languages produces fatigue and missed bugs, which is the opposite of validation.

Example: a reviewer signs off on a 2,000-line, fifteen-file change after skimming, missing a one-line grain error in file nine. Nobody sustains attention across that surface.

7. Handoffs that lose the design

When the logic is distributed across many files and languages, no single person holds the whole picture. Handoffs lose context, and the next engineer or agent changes things blind to the original design.

Example: the engineer who built the pipeline leaves. Their replacement cannot work out why a particular deduplication exists, removes it, and reintroduces the duplicate-records bug it was silently preventing.

8. Requirements you cannot actually verify

The point of review is confirming the pipeline does what was asked. You cannot confirm that against logic you cannot read as a whole, so requirements get marked done on faith.

Example: a compliance rule requires closed accounts to be excluded from a report. Verifying it means tracing the rule across ingestion, transformation, and serving files. The reviewer assumes it is handled and ships a violation.

Summary

None of these is a bug a validator can flag, because every individual line is correct. The error is in the meaning: how stages align, what a number represents, an assumption two files quietly disagree about. The only thing that catches a misalignment of meaning is a person who understands the pipeline, and understanding is impossible when the logic is scattered across hundreds of files in several languages.

One readable file collapses that surface. The entire data flow, from ingest to transform to store to serve, is expressed as one declarative SQL logic in the language most data engineers already read. That makes the whole pipeline comprehensible in one sitting, which turns review from a fatiguing rubber stamp back into real scrutiny, and frees the engineer to spend that attention on the requirements and the meaning that agents and validators cannot judge for themselves.

As pipeline creation speeds up, human understanding becomes the bottleneck. Keeping the logic in one readable place is how you keep that bottleneck open.

It is open source, feel free to try it out yourself: DataSQRL on GitHub.