AI-Augmented DataOps

Policy-Governed DataOps Agents

Over the past several months I've written about keeping modern data platforms intentionally simple.

My argument has been consistent:

• Let the warehouse perform computation.
• Let dbt manage transformations.
• Let Airflow orchestrate.
• Resist adding complexity until complexity has been earned.

I still believe that.

But I have also become convinced that something does belong above the orchestration layer.

Not another scheduler.

Not another transformation engine.

Not another proprietary metadata catalog.

An operational reasoning layer.

We Have Automated Pipelines. We Have Not Automated Operations.

Most mature data platforms today already know when something has gone wrong.

A dbt test fails.

A source freshness check exceeds its threshold.

An Airflow task retries three times and eventually fails.

A row count suddenly drops by 95%.

A schema changes unexpectedly.

Cloud monitoring raises an alert.

The problem is rarely discovering that something failed.

The problem is deciding what happens next.

Today that responsibility almost always falls to a human.

Someone opens Slack.

Someone reads logs.

Someone compares today's run with yesterday's.

Someone remembers that Vendor X occasionally delivers malformed files at month end.

Someone knows that this particular failure can safely be rerun while another requires immediate escalation.

In other words, our platforms are excellent at generating evidence.

Humans remain responsible for judgment.

The Separation That Matters

As large language models become increasingly capable, there is a temptation to hand operational control directly to an "AI agent."

I think that is the wrong abstraction.

Instead, I believe we should deliberately separate four responsibilities:

Policy

What constitutes success?

What constitutes failure?

Which conditions are warnings?

What actions are permitted?

These rules should be explicit, version-controlled, and reviewable.

Evidence

What actually happened?

Collect logs.

Collect dbt artifacts.

Collect Airflow metadata.

Collect data quality metrics.

Collect lineage information.

Evidence should be assembled mechanically and reproducibly.

Judgment

Given the evidence and the governing policy, what is the most likely explanation?

Is this probably a transient infrastructure problem?

A source data issue?

A schema evolution?

An upstream outage?

A deployment regression?

This is where an LLM becomes valuable—not because it is making policy, but because it is exceptionally good at synthesizing disparate evidence into a coherent operational narrative.

Action

Finally, what should happen?

Perhaps nothing more than a notification.

Perhaps automatically rerun a failed model.

Perhaps quarantine an input file.

Perhaps open a ticket with all supporting context already attached.

Perhaps recommend a code change—but never apply it automatically.

Policy determines what actions are permissible.
The agent determines which permissible action best fits the available evidence.

That distinction matters.

Deterministic Systems Should Decide What Is Wrong

One principle keeps surfacing as I think through this architecture.

Deterministic systems should determine whether something is wrong.
Agents should help determine what it means.

That may seem like a subtle distinction, but it has profound implications.

A row count either exceeded its threshold or it did not.

A dbt test either passed or failed.

A freshness SLA was either violated or it wasn't.

These are objective questions.

An LLM should not decide whether a uniqueness test passed.

It should decide whether five seemingly unrelated failures are actually symptoms of the same underlying issue.

Those are fundamentally different kinds of reasoning.

Data Contracts Become Operational Contracts

This perspective also changes how I think about data contracts.

Today we often think of contracts in terms of schema, ownership, and interface compatibility.

I think they should evolve further.

Each dataset should carry an operational contract describing:

• freshness expectations
• quality thresholds
• acceptable volume ranges
• required tests
• notification paths
• escalation rules
• permissible automated remediations

In effect, every important dataset gains its own runbook.

The operational behavior of the platform becomes configuration rather than application code.

The Incident Packet

One implementation idea that I find increasingly compelling is what I think of as an Incident Packet.

Rather than granting an agent unrestricted access to an environment, a deterministic workflow assembles all relevant evidence into a structured package.

That packet might include:

• dbt artifacts
• pipeline logs
• source freshness
• historical execution statistics
• row count anomalies
• schema differences
• upstream dependency status
• permitted remediation actions

Only then is the packet presented to the reasoning engine.

The agent never needs unrestricted filesystem access.

It receives exactly the information required to reason about the incident—and only the actions that policy explicitly allows.

This strikes me as both a safer and a more scalable architecture.

Beyond Data Engineering

Although I've been thinking about this in the context of Airflow and dbt, I don't believe the pattern is specific to data platforms.

The same architecture could govern:

• cloud infrastructure operations
• manufacturing systems
• medical device telemetry
• IoT fleets
• cybersecurity workflows
• enterprise application support

Any environment where evidence is objective, policy is explicit, and judgment benefits from contextual reasoning could adopt the same pattern.

Looking Ahead

I don't believe the future belongs to autonomous agents wandering freely through production systems.

I believe it belongs to policy-governed agents operating inside carefully designed boundaries.

Deterministic systems establish facts.

Policies establish constraints.

Agents provide reasoning.

Humans retain authority over consequential decisions.

If we separate those responsibilities deliberately, we gain something more valuable than automation.

We gain operational systems that are explainable, auditable, and increasingly capable of assisting engineers without replacing sound engineering discipline.

To me, that feels like a far more interesting future than simply asking an AI to "fix the pipeline."