Designing Evidence Packets

A packet built around one source

The first working implementation of the Evidence Packet — packet schema version 0.2.0 — was built directly against a single source: dbt operations orchestrated by Airflow. That was a deliberate choice. Before generalizing a concept, it is worth finding out whether it survives contact with a real evidence source at all.

It did. Nightly Airflow-orchestrated dbt runs produced packets containing run identifiers, timing, model and test outcomes, and source-freshness results, and those packets were usable.

But a packet designed around one source's vocabulary tends to absorb that source's assumptions into its structure. Fields named for dbt concepts — a model, a test, an invocation — describe dbt specifically. As soon as a second source enters the picture, whether a warehouse job, an orchestrator run considered on its own terms, or a source-delivery record, that vocabulary stops fitting cleanly. Extending it means either stretching dbt-shaped fields to mean something they were not designed to mean, or bolting on source-specific fields one at a time until the packet is really several packets wearing one schema.

Neither is sustainable if the packet is meant to remain useful as new sources are added.

Separating the packet from the source

Packet schema version 0.3.0 separates the packet model from any single source's vocabulary. A source-neutral core defines identity, timing, provenance, artifact references, relationships, integrity metadata, and evidence-gap declarations independently of which system produced the evidence. A source adapter is then responsible for translating a specific system's native output into that shared structure.

Concretely, the dbt adapter reads run_results.json, manifest.json, and source-freshness output, and translates them into the source-neutral packet shape: dbt's model and test outcomes become the packet's normalized execution outcomes; dbt's invocation metadata becomes the packet's provenance; dbt's source-freshness results become part of its evidence, subject to the same missing-evidence semantics as everything else. The adapter boundary is deliberately narrow — an adapter translates, it does not redefine, the packet model.

dbt remains the only adapter implemented so far. PostgreSQL, as a runtime evidence source in its own right, and Airflow, as an orchestration-level evidence source distinct from the role it already plays scheduling dbt runs, are natural next adapters. Neither exists yet. The value of the architecture completed in this milestone is that adding them should mean writing adapters against a stable core, not redesigning the packet.

Evidence that admits what it doesn't know

Earlier packets tended to represent missing evidence implicitly: a field was simply absent, null, or omitted. That is ambiguous. Absence can mean the fact does not apply, the fact was not yet collected, collection was attempted and failed, or the source does not expose that fact at all. A diagnostic process — human, deterministic, or probabilistic — cannot reliably tell those apart from an implicit gap, and an LLM asked to reason over an incomplete packet may fill the silence with an invented explanation rather than reporting the gap.

Before

A source-freshness field is simply absent from the packet if the adapter could not retrieve it.

Now

The packet records an explicit missing-evidence declaration: which fact, from which source, and why it could not be captured.

The source-neutral core makes missing evidence a first-class, structured part of the packet rather than an omission a reader has to notice on their own. This extends the separation between observation and interpretation described in Logs Aren't Enough: a declared gap is itself an observation — a record that a fact was sought and not found — not an interpretation of what that gap means.

Keeping already-collected evidence honest

Evidence Packets have been collecting nightly since the first implementation, so generalizing the model could not mean discarding or reinterpreting that history. Version 0.3.0 packets are required to remain semantically compatible with version 0.2.0 packets: the same underlying facts, expressed in the new structure.

A set of semantic compatibility tests translates a sample of original 0.2.0 packets into 0.3.0 form and checks that the meaningful content survives the translation — identifiers, outcomes, timing, and provenance must resolve to the same facts under both schema versions. This is a narrower guarantee than full backward compatibility; the wire format changed. The claim is only that nothing already observed became unrecoverable or misrepresented when the model generalized.

Deterministic packet integrity

Packet assembly is now deterministic: the same underlying observations always assemble into the same packet content, and identical content produces identical integrity metadata. Non-determinism in assembly — field ordering that varies between runs, values derived from wall-clock time at assembly rather than from the observation itself — would quietly undermine a packet's use as durable evidence, since two assemblies of the same facts could no longer be recognized as equivalent.

Deterministic assembly gives each packet a content-derived integrity fingerprint. That is a foundation for later work confirming a packet has not drifted between collection and use, not a claim that tamper detection, audit tooling, or diagnosis already exists.

Testing the model against a representative corpus

Validating a schema by reading it is not the same as validating it against evidence. The research now maintains a representative fixture corpus — Evidence Packets, assembled through the dbt adapter, spanning healthy operations, degraded operations, and a range of evidence-capture outcomes, including packets with declared missing evidence. Corpus-assessment tooling checks that every fixture conforms to the versioned schema, that missing-evidence declarations are structurally valid, and that 0.2.0-to-0.3.0 compatibility holds across the set.

This fixture corpus should not be confused with the synthetic, mutated, and fault-injected diagnostic corpus described in Logs Aren't Enough. That corpus exists to test whether captured evidence is sufficient for diagnosis, human or automated — a question that presupposes a stable packet model to test against. This fixture corpus exists to test whether the packet model itself is internally consistent. The two are related but sequential: the diagnostic-evaluation corpus is the subject of the next article in this series.

Versioned schemas, deliberately

Packet schemas are now explicitly versioned, documented, and validated rather than treated as an implementation detail. This may look mechanical, but it is load-bearing: the packet model will need to change again as PostgreSQL, Airflow, and other sources are added, and a versioned, tested schema is what makes that possible without invalidating or reinterpreting evidence captured under an earlier version. Architecture documentation was updated alongside the schema to describe the packet core, the adapter boundary, evidence-gap semantics, integrity, and the compatibility expectations future adapters and future schema versions are expected to meet.

What this milestone does not include

To be explicit about scope, this milestone does not include:

• PostgreSQL runtime evidence collection
• an Airflow source adapter
• deterministic diagnosis
• LLM-assisted diagnosis
• remediation
• autonomous agents

Those remain later stages of the research path described on the Autonomous DataOps research page.

Toward evidence completeness

With the packet model source-neutral, versioned, and tested for internal and cross-version consistency, the open question shifts. It is no longer whether the model can represent evidence. It is whether the evidence it captures is enough to diagnose anything.

The next article returns to the synthetic, mutated, and fault-injected corpus outlined in Logs Aren't Enough, this time built against the 0.3.0 packet model, to begin measuring evidence completeness rather than assuming it.

A well-designed container is not the same claim as sufficient contents.

Foundation and Next Steps