Logs Aren't Enough

A log answers a local question

Logs are usually written by an individual component for the purpose of describing that component's behavior.

An Airflow log may explain what a task attempted to do.

A dbt artifact may describe which models ran, which tests failed, and how long execution took.

A BigQuery job record may describe the query, bytes processed, timing, and execution state.

A source-ingestion process may record the name and size of a delivered file.

These records are valuable, but each usually answers a local question:

What did this component observe?

Diagnosis requires a broader question:

What evidence was available across the system when this operation occurred?

Those are not the same question.

Consider a dbt model that fails because an expected source column is missing. The dbt error may clearly identify the invalid SQL reference. But a useful diagnosis may also require knowing:

• whether the source schema changed,
• when it changed,
• whether the source delivery was complete,
• which upstream file or API response introduced the change,
• whether a deployment modified the model,
• whether the same condition occurred previously,
• which downstream models and reports depend on the failed model,
• whether a documented policy permits a fallback,
• whether the correct response is retry, quarantine, rollback, escalation, or acceptance.

The error message may be perfectly accurate and still be diagnostically incomplete.

More logs do not necessarily solve the problem

The instinctive response to poor diagnosability is often to collect more logs.

That can help, but it does not address the underlying problem by itself.

A larger volume of uncoordinated telemetry may make the evidence scavenger hunt more expensive. The operator must search more systems, correlate more timestamps, identify more execution identifiers, and determine which observations belong to the same event.

The challenge is not simply evidence volume. It is evidence organization.

Useful diagnostic evidence needs:

• a common operational identity,
• explicit timestamps,
• producer and environment information,
• source and code provenance,
• normalized execution outcomes,
• relationships among tasks, models, datasets, and deliveries,
• preservation of relevant native artifacts,
• visibility into missing or unavailable evidence.

Without those features, a data platform may be highly observable at the component level and still be difficult to diagnose as a system.

The missing artifact

I believe data operations needs a first-class artifact that exists between raw telemetry and incident diagnosis.

I call that artifact an Evidence Packet.

An Evidence Packet is a structured collection of operational observations associated with a defined observation point.

For a dbt operation orchestrated by Airflow, an Evidence Packet might include:

• Airflow DAG, task, run, and attempt identifiers,
• scheduled and actual execution times,
• task state and duration,
• dbt invocation metadata,
• run_results.json,
• manifest.json references,
• model and test outcomes,
• relevant source-freshness results,
• warehouse job identifiers,
• affected relations,
• source-delivery metadata,
• deployed code version,
• environment and configuration identity,
• links to detailed native logs,
• a declaration of evidence known to be missing.

The packet does not need to copy every log line into a central record. It can preserve important summaries, normalized facts, identifiers, hashes, and durable references to more detailed evidence.

Its purpose is not to replace logs.

Its purpose is to make the evidence surrounding an operation explicit, bounded, traceable, and reusable.

Evidence is not diagnosis

This distinction is essential.

An Evidence Packet should describe what was observed. It should not quietly convert interpretations into facts.

For example:

Observation

The source contained 17 columns. The prior successful delivery contained 18.

Interpretation

The vendor introduced an unannounced breaking schema change.

The observation may support that interpretation, but it does not prove it. A malformed file, incorrect source selection, partial delivery, or configuration error might produce the same evidence.

Similarly:

Observation

A dbt test reported 4,812 duplicate business keys.

Interpretation

The upstream deduplication process failed.

That may be likely, but diagnosis should remain distinguishable from the evidence on which it depends.

This separation matters for human operators, deterministic rules, and AI-assisted systems alike. When facts and interpretations are blended together, uncertainty becomes difficult to see and previous assumptions can be mistaken for fresh evidence.

A trustworthy diagnostic process should be able to show:

1. what was observed,
2. what evidence was unavailable,
3. what conclusions were derived,
4. which rules or reasoning produced those conclusions,
5. how confident the process is,
6. what additional evidence would resolve remaining uncertainty.

Healthy operations are evidence too

Incident-oriented systems tend to preserve detailed context only after something has failed.

That creates a serious limitation: diagnosis often depends on comparison with healthy behavior.

Was today's file unusually small?

Did this model normally process the same number of rows?

Has execution duration been increasing gradually?

Was the source schema already different during the previous successful run?

Did the same warning appear repeatedly before the first visible failure?

Evidence Packets should therefore be created for healthy, degraded, and failed operations.

A repository of healthy operational evidence provides:

• baselines,
• historical comparison,
• trend detection,
• examples of normal variation,
• material for deterministic rules,
• material for testing diagnostic systems,
• context for distinguishing anomalies from harmless change.

The primitive should be evidence, not failure.

From Evidence Packets to incidents

An Evidence Packet represents a bounded set of operational observations. It does not necessarily represent an incident.

A single incident may involve several observation points:

• a delayed source delivery,
• an ingestion retry,
• an Airflow task failure,
• multiple skipped dbt models,
• a stale dashboard,
• and a manual recovery action.

Each may produce its own Evidence Packet.

When an incident is detected or declared, those packets can be assembled into a higher-level Incident Packet along with:

• reported symptoms,
• human observations,
• historical incidents,
• deterministic diagnostic findings,
• probabilistic analysis,
• policy context,
• recommended actions,
• approvals,
• remediation results,
• post-incident learning.

This produces a useful distinction:

Evidence Packets are structured observations. Incident Packets are incident-level aggregations and interpretations.

The distinction prevents operational evidence from being prematurely shaped around a diagnosis that has not yet been established.

Why this matters for AI-assisted DataOps

Large language models can read logs, summarize errors, and propose plausible explanations. That does not mean they have received the evidence necessary to diagnose an incident correctly.

A sophisticated reasoning system operating on incomplete or poorly correlated evidence may produce a more persuasive unsupported answer.

The first question should therefore not be:

Which model should diagnose the pipeline?

It should be:

What evidence would a competent data engineer need, and can the platform assemble it reliably?

Only after that question is addressed should we decide which parts of diagnosis belong to:

• deterministic tests,
• runbook lookups,
• historical comparison,
• statistical analysis,
• probabilistic reasoning,
• human judgment.

Known conditions should generally be handled through deterministic mechanisms first. An LLM may be valuable when evidence must be synthesized across systems, ambiguity remains, or candidate explanations need to be evaluated. But probabilistic reasoning should operate on evidence that is explicit, inspectable, and bounded.

Otherwise, the system is not performing autonomous diagnosis. It is improvising from whatever context happened to be available.

The first experiment

The immediate research problem is not to automate remediation.

It is to determine whether an Evidence Packet can contain enough context to support defensible diagnosis.

My initial implementation collects evidence from nightly Airflow-orchestrated dbt operations and stores the resulting packets in PostgreSQL. The packet model is intended to remain independent of a particular orchestrator, warehouse, or transformation framework, but Airflow and dbt provide a practical first implementation.

Naturally occurring failures will not produce a sufficiently broad or balanced diagnostic sample. The next step is therefore to construct a controlled corpus consisting of:

1. fully synthetic Evidence Packets,
2. mutations derived from real packet templates,
3. controlled fault-injected operations passing through the actual evidence-capture path.

Each scenario will include separate diagnostic ground truth. That will make it possible to ask:

• Can an experienced data engineer diagnose the condition from the packet?
• Can deterministic tests identify it?
• What evidence is missing?
• Can probabilistic assistance add value without inventing unsupported facts?
• Can the system propose a defensible Next Best Action?
• Does the same packet support equivalent conclusions across repeated evaluations?

The purpose is not to demonstrate that an LLM can explain a familiar error message. It is to test whether the operational evidence itself is sufficient for disciplined analysis.

Toward evidence-driven operations

Logs remain essential. So do metrics, traces, artifacts, warehouse history, source metadata, deployment records, and human knowledge.

The problem is that these sources usually remain separate until someone manually assembles them during an incident.

Evidence-driven DataOps begins by making that assembly an explicit part of the operating architecture.

The progression is deliberate:

1. Operational observations
2. Evidence capture
3. Evidence Packets
4. Evidence completeness
5. Deterministic diagnosis
6. Probabilistic assistance where warranted
7. Incident Packet
8. Next Best Action
9. Policy-governed mitigation
10. Learning

Autonomous operation should not begin with autonomy.

It should begin with evidence.

Foundation and Next Steps