Pysynapse

The Autonomous DataOps Research Series

Toward Evidence-Driven Data Operations

Phase I: Closed Updated

Modern data-pipeline diagnosis often begins with fragmented evidence spread across orchestration logs, transformation artifacts, warehouse history, source metadata, source code, and team knowledge. This program explores how structured operational evidence, deterministic diagnostics, constrained AI reasoning, and policy-governed action can support more diagnosable and eventually more autonomous data systems.

Latest publication: Article 02 — Designing Evidence Packets

Next: RQ3 (historical / nominal characterization) and other deferred questions remain open, not active — see Preserved Future Research Questions.

Research Hypothesis

Holding the underlying operational evidence constant, presenting it as a bounded, correlated, provenance-bearing Evidence Packet may improve independent diagnostic performance compared with conventional raw or aggregated telemetry.

This representation-utility hypothesis was the active question that opened the program, and it is the RQ1 hypothesis quoted above exactly as it was written before execution. Once RQ1 closed with a null result, the program's own closure record generated a directly motivated follow-on question — evidence durability — which was then designed, executed, and closed as RQ2. Historical/nominal characterization remains open; see Preserved Future Research Questions. LLM diagnosis, automated diagnosis, Incident Packet generation, detection, and remediation remain later capabilities, unaffected by Phase I's closure.

Research Questions

RQ1 — Representation Utility — Closed
Holding operational evidence constant, does Evidence Packet representation improve independent diagnostic performance versus raw or aggregated telemetry? Closed: no demonstrated material advantage (60 matched pairs; conventional 13/60, packet 12/60; −0.017 against a ±0.10 materiality rule). An unambiguous null under that rule, not proof representation can never matter. Full result in Phase I Artifacts.
RQ2 — Evidence Durability — Closed
How much diagnostically or analytically useful evidence available at execution time cannot be faithfully reconstructed later? Closed: threshold-crossing but statistically thin observed advantage for durable capture (25 matched pairs; durable 8/25, reconstruction 5/25; +0.120 against a +0.10 threshold; 9 discordant pairs, p = 0.508 on the descriptive exact test). Not statistically significant, and representation was not held constant between arms — see Phase I Artifacts for the full qualification.
RQ3 — Historical / Nominal Characterization — Deferred
Can a corpus of nominal Evidence Packets characterize system behavior and provide useful baseline or historical context for later detection or diagnosis? Not tested by Phase I; see Preserved Future Research Questions — this is one candidate deferred direction among several, not an already-scoped active research question.

Four things are easy to conflate in a program like this, so the page keeps them visibly separate:

Engineering result

A capability that has been built and works as specified — for example, the source-neutral Evidence Packet model and the deterministic evidence-requirement evaluator.

Experimental observation

A measured outcome under stated, often narrow, conditions — for example, the Level C completeness figures below, from five hand-authored scenarios on one local stack.

Open research question

Something the program has not yet tested. RQ1 (representation) and RQ2 (durability) are no longer open — both closed with frozen results, summarized above and detailed in Phase I Artifacts. Historical/nominal characterization and several other directions the synthesis identifies remain genuinely open; see Preserved Future Research Questions.

Hypothesis

The claim in the blockquote above. It motivates the whole program. Building toward it is not the same as demonstrating it.

Evidence Packet Lifecycle

Evidence capture precedes incident determination. An execution or other subject event produces contemporaneous observations, which may be preserved in an Evidence Packet before anyone knows whether the event is nominal, degraded, anomalous, or incident-related. Later interpretation may find no known defect, use the packet for behavioral characterization, identify a deviation, or incorporate it into diagnosis.

Evidence Packets are execution-level observations. Incident Packets are later incident-level aggregations and interpretations.

A successful execution is not proven healthy: undetected data-quality, latency, behavioral, or configuration problems may remain. This page therefore uses nominal or no known defect for executions without an identified problem.

Phase I Synthesis and Artifacts

RQ1 (Representation Utility) and RQ2 (Evidence Durability) each ran to a frozen, independently reviewed primary result, and both were then interpreted together in one integrated Phase I synthesis. The synthesis is the authoritative statement of what Phase I found, what it did not establish, and its explicit claim-strength labeling (observed / supported interpretation / architectural implication / not established) — this page summarizes it but does not replace it.

All links below are pinned to commit 8c50e92, the Phase I closure commit, so they keep pointing at what was actually reviewed and frozen as the repository moves forward:

The reference implementation is at github.com/kriskokomoor/evidence-packets.

Early Program History: Specification-Completeness Benchmark

This section and its Level A/B/C results describe early program history, from before RQ1 and RQ2 were executed. It measures whether a packet contains required fields, not whether representation or durable capture affects diagnosis — those questions were tested later, are now closed, and are summarized in Phase I Synthesis and Artifacts. This section is preserved unchanged as the record of how that later work was motivated.

Evidence Packets were, at the time of this benchmark, collected from nightly Airflow-orchestrated dbt runs. MedTracker provides the primary changing data source, while the ADS-B feed is temporarily unavailable pending relocation of its antenna connection. Every naturally occurring packet collected so far has come from a successful run — no real production failure has yet passed through the pipeline, so this corpus alone cannot speak to failure diagnosis.

Naturally occurring failures will produce a small and poorly balanced diagnostic sample even once they start appearing, so the program built a controlled corpus instead, in three tiers:

  1. Synthetic Evidence Packets generated directly from scenario definitions
  2. Mutations derived from real Evidence Packet templates
  3. Controlled fault-injected operations passing through the actual evidence-capture path

That corpus and its first results are in Benchmark Results below. It is worth being precise about what it currently tests: whether a packet contains the fields a scenario's authors specified as required — not whether a human, a deterministic system, or an LLM can actually reach a correct diagnosis from that packet. Those are related but different questions, and only the first has been measured so far.

Benchmark Results: Specification Completeness

The benchmark corpus has three tiers. Each tests something different, and none of them yet test diagnosis.

Level A — Evaluator regression tests

Ten canonical scenarios, each with an independently authored ground-truth definition and expected-evidence contract. Level A packets are built directly from a scenario's own stated requirements, so a Level A packet satisfying its own scenario is guaranteed by construction rather than observed. Level A is deterministic regression testing of scenario representation, evidence matching, completeness scoring, and evaluator behavior, with machine-readable JSON and Markdown output for every run. It confirms the evaluator and scenario schema work correctly. It is not empirical evidence that real collectors produce diagnostically sufficient packets.

Level B — Evaluator sensitivity tests

Disposable copies of packets are mutated — a required item removed, provenance stripped, integrity metadata dropped, a correlation key broken — to confirm the evaluator correctly downgrades completeness and requirement satisfaction when evidence is missing or damaged. This is a packet-mutation / evaluator-sensitivity test. It does not exercise real collectors and says nothing about what real collection failures actually look like.

Level C — Controlled fault-injection experiment

Level C is the one tier that runs real Airflow, dbt, and PostgreSQL services. Five hand-authored conditions — a nominal run with no known defect, a dbt model SQL failure, a dbt test failure, a missing source relation, and an Airflow failure before dbt starts — were each executed once on an isolated local Docker stack. (The first execution attempt failed for an unrelated environmental reason — the worker exposed neither the ep-dbt nor dbt entry point under the operator's environment — and was corrected before the runs below.)

A baseline packet, produced by the existing dbt-only collection path, was captured for each condition first. Level A had implicitly assumed a collector would produce whatever evidence a scenario required; the baseline shows that assumption did not hold. Bounded collector changes were then added — terminal Airflow task-state capture, a PostgreSQL connection/relation probe, normalized dbt status summaries, and a cross-source correlation key — and the five conditions were rerun as "improved."

  • Baseline mean required-evidence completeness: 0.10 (0 of 5 scenarios met their evidence requirements; nine missing required observations, one correlation gap, one integrity gap)
  • Improved mean required-evidence completeness: 1.00 (5 of 5 scenarios met their evidence requirements after the collector changes)
  • The Airflow-failure-before-dbt condition produced no packet at all under the baseline path
  • All improved packets were classified over-collected relative to the scenario's stated minimum, not minimally sufficient

The Evidence Packet reference implementation is at github.com/kriskokomoor/evidence-packets. The links below are pinned to f20daec, the commit this page describes, so they keep pointing at what was actually reviewed even as the repository moves forward:

Current Implementation Status

The project established the Evidence Packet representation, collection prototypes, an evidence-requirement evaluator, a scenario corpus, and controlled fault-injection infrastructure. Initial Level C experiments exposed meaningful collection gaps; independent review then found information leakage between benchmark expectations and the improved collector, which shifted the next phase from specification conformance toward independent diagnostic validation. That validation ran as RQ1, closed with a null result, and directly motivated RQ2, which also ran and closed. Phase I is now closed; see Phase I Synthesis and Artifacts.

Operational

Collection Foundation

  • Evidence Packet schema
  • dbt artifact collection
  • Airflow evidence-capture integration
  • PostgreSQL Evidence Packet repository
  • Nightly MedTracker evidence collection (successful runs only, to date)
Operational

Specification-Completeness Benchmark

  • 10 versioned fault-injection scenarios (Level A–C corpus)
  • Synthetic Evidence Packet generator (Level A regression tests)
  • Packet mutation / evaluator sensitivity tests (Level B)
  • Deterministic evidence-requirement evaluator
Closed

RQ1 — Representation Utility

  • Scenario-blind, equivalent-evidence apparatus (canonical evidence held constant, representation varied)
  • Independent diagnosis against hidden ground truth, one-shot diagnostic contexts, leakage scanning
  • 60 matched pairs / 120 requests executed and frozen
  • Result: no demonstrated material correctness advantage (−0.017 vs. ±0.10 threshold)
Closed

RQ2 — Evidence Durability

  • Durable-capture vs. later-reconstruction apparatus, same evidence at capture time, degradation applied only downstream
  • 25 matched pairs / 50 requests executed and frozen
  • Independent contract and implementation review before authorized finalization
  • Result: threshold-crossing but statistically thin advantage for durable capture (+0.120 vs. +0.10; p = 0.508 on 9 discordant pairs)
Deferred

Later Capabilities

  • Deterministic incident classification
  • LLM-assisted evidence interpretation
  • Incident Packet assembly
  • Behavioral baselining and anomaly or incident detection
  • Historical / nominal characterization (RQ3 candidate) and other Phase I deferred questions
  • Next Best Action evaluation
  • Policy-governed remediation

Technical Research Path

This is the conceptual and technical progression of the research, not a claim that every capability is complete.

1

Fragmented Logs and Operational Evidence

Established Problem
2

Evidence Capture

Reference Implementation Complete

A reusable evidence-collection layer now exists, independent of any single pipeline. It defines evidence sources and collection runs, captures artifacts with recorded provenance, supports replay of a prior collection run, detects conflicting observations, verifies integrity, and records an explicit outcome for every capture attempt, including partial or failed collection. The cross-source collector exercised in the Level C experiment below still needs to be rebuilt so it correlates sources without relying on benchmark-specific identifiers — see Benchmark Results.

3

Evidence Packets

Reference Implementation Complete

Packet schema version 0.3.0 introduces a source-neutral Evidence Packet model, with dbt implemented as the first source adapter. Missing evidence is modeled explicitly rather than left implicit, packet assembly is deterministic for integrity verification, and a compatibility layer maintains semantic compatibility with version 0.2.0 packets. The source-neutral schema itself is not new — see Related Work, where OpenLineage already standardizes a similar cross-source event model with production dbt/Airflow/Spark integrations. What's untested is whether this project's specific combination of provenance, integrity, and explicit evidence-gap declarations contributes something a lineage-focused standard doesn't.

4

Evidence Requirement Evaluation

Reference Implementation Complete

A deterministic evaluator compares a packet against a scenario's independently specified evidence requirements and reports specification completeness and evidence requirement satisfaction. This is a conformance check, not a diagnosis: it establishes whether a packet contains what its scenario's authors said it should contain, not whether that is enough to diagnose anything.

5

Synthetic and Fault-Injected Corpus (Levels A–C)

Reviewed — Claims Narrowed

Ten scenarios have synthetic (Level A) and mutation-based (Level B) evaluator tests, and five ran as a controlled fault-injection experiment (Level C) against real Airflow, dbt, and PostgreSQL services. An independent review of the Level C experiment found information leakage between the benchmark's scenario expectations and the collector built to satisfy them, along with other confounds — see Benchmark Results and Research Log 005.

6

Experimental Isolation and Independent Diagnostic Validation (RQ1)

Complete — RQ1 Closed

Hidden ground truth drove a system under test. A scenario-blind apparatus produced canonical evidence, rendered as equivalent raw/aggregated telemetry and Evidence Packet presentations. An independent diagnostic process saw only its assigned presentation, proposed a diagnosis, and was graded against ground truth it never saw. This controlled same-evidence comparison tested RQ1 and found no demonstrated material correctness advantage from packet representation (60 matched pairs; −0.017 vs. ±0.10). LLM diagnosis, Incident Packets, and historical baselining remained out of scope, as planned.

7

Evidence Durability and Phase I Closure (RQ2)

Complete — RQ2 Closed, Phase I Closed

RQ1's own closure record motivated a directly related question: does durably capturing evidence at execution time preserve diagnostic value that later reconstruction, from the same operational sources after realistic degradation, loses? A preservation-centered comparison (25 matched pairs) found a +0.120 observed advantage for durable capture against a +0.10 threshold — material by the prespecified rule, but statistically thin (9 discordant pairs, p = 0.508) and not a clean isolation of preservation from representation. RQ1 and RQ2 were then interpreted together in an integrated Phase I synthesis, which closed after independent review. See Phase I Synthesis and Artifacts.

8

Broader Diagnostic Evaluation

Deferred — Later Phase

Later work may compare diagnosis-time context assembly, different diagnostic processes, additional stacks, incomplete evidence, longer reconstruction intervals, replication with more matched pairs, representation-controlled preservation testing, and end-to-end workflow effects. These comparisons are outside Phase I and not yet scheduled as RQ3 or any other specific next study.

9

Deterministic Diagnostics

Deferred
10

Probabilistic / LLM Diagnostic Assistance

Deferred
11

Incident Packet Assembly

Deferred
12

Next Best Action

Deferred
13

Policy-Governed Mitigation

Deferred — Conceptual Foundation Established
14

Incident Learning and Continuous Improvement

Deferred

Preserved Future Research Questions

Preserve the option; do not implement the capability yet. New research questions do not automatically become current requirements. Phase I (RQ1 and RQ2) is closed; nothing below is scheduled, and none of it is RQ3 in any official or decided sense — the Phase I synthesis lists these as candidate deferred directions, not a ranked or committed roadmap.

Behavioral baselining, anomaly or incident detection, retention modeling beyond what RQ2 already tested, evidence ablation or minimality, broader diagnosis-time assembly comparisons, LLM or automated diagnosis, Incident Packet generation, remediation, and autonomous mitigation remain explicitly outside any current implementation increment.

Deferred — a candidate future direction

Historical / Nominal Characterization

Individual packets describe executions; a corpus may describe system behavior. Packets from executions with no known defect could eventually support characterization of duration, volume ranges, retry behavior, expected evidence presence, schema stability, model or test patterns, change over time, and comparison with prior similar executions. Concretely, this asks whether access to a corpus of prior nominal and prior anomalous executions from the same source stream improves anomaly identification (is this execution actually abnormal?), anomaly classification (what kind of abnormality?), and causal attribution (what likely caused it?) — questions Phase I did not test, since both RQ1 and RQ2 presented evidence for one execution under investigation only, never a comparative corpus.

This is a future hypothesis, not a current behavioral-baselining, anomaly-detection, or incident-detection capability, and it is not an already-scoped "RQ3" — it is one leading candidate among several the Phase I synthesis identifies (see below), and it is not decided which, if any, becomes the program's next research question. Architecturally, immutable event-level Evidence Packets could form an observational corpus that a separate later layer draws on for this purpose; that separation is a plausible possibility this architecture does not foreclose, not something built or tested. For now, the architecture merely preserves packets for nominal as well as fault-injected executions where practical.

Deferred — other Phase I candidates

Other Directions the Synthesis Identifies

Alongside historical/nominal characterization, the Phase I synthesis records several other open questions with equal standing, none scheduled: replicating the RQ2 result with more matched pairs (it currently rests on only 9 discordant pairs); testing longer reconstruction intervals, since RQ2's own gaps were almost entirely under three minutes; a representation-controlled RQ2 variant that would isolate preservation from the residual representation confound; and whether the grounding/unsupported-assertion pattern seen in both RQ1 and RQ2 is a more stable construct than raw correctness for a future study.

Guiding Principles

Evidence before inference

Collect and preserve observable facts before interpreting the incident.

Observations separated from interpretations

Keep collected facts distinct from diagnostic findings and policy conclusions.

Deterministic diagnosis before probabilistic judgment

Route known conditions through rules and runbooks first.

Explicit uncertainty when evidence is incomplete

Make missing context and confidence limits visible.

Producer-neutral Evidence Packet design

Keep the formal artifact useful across orchestrators, warehouses, transformation tools, and other observation points.

Human-reviewable recommendations

Make every proposed action traceable to supporting evidence.

Policy boundaries before autonomous action

Define risk, approval, and authority limits before enabling mitigation.

Reproducible incident experiments

Use controlled scenarios that can be repeated and compared.

Packet evidence separate from diagnostic ground truth

Preserve the distinction between what an Evidence Packet captured and what the experiment establishes.

Nominal operations are evidence too

Preserve observations for executions with no known defect without treating technical success as proof of health.

Preserve the option; contain the scope

Avoid foreclosing future research, but do not turn every preserved hypothesis into a requirement for the current build.

Publication Roadmap

Published

01 — Logs Aren't Enough

Why modern data-pipeline incidents begin with an evidence scavenger hunt, and why structured operational evidence should become a first-class artifact.

Published

02 — Designing Evidence Packets

A source-neutral operational evidence architecture built from the original dbt Evidence Packet implementation.

Research Complete — Article Not Yet Published

03 — Does the Packet Improve Diagnosis?

Originally framed as "Can We Trust the Packet?" and drafted around the Level C specification-completeness results above. Independent review found those results didn't yet support a trust claim, so the article was gated on Independent Diagnostic Validation: a scenario-blind apparatus, equivalent presentations of the same canonical evidence, an independent diagnostic process graded against hidden ground truth, and a defensible result on representation utility. That gate has now been met — RQ1 ran and closed with no demonstrated advantage, and RQ2 followed and closed with a qualified positive result — and the result is incorporated into this hub page's Phase I result and Phase I Artifacts sections. A standalone Article 03 covering both studies in full is still pending.

Planned

04 — Measuring Specification Completeness

The full Level A/B/C benchmark methodology, scenario corpus, and results, including the independent methodological review in Research Log 005 and its effect on how the Level C numbers should be read.

Planned

05 — Deterministic Before Probabilistic

Routing known conditions through rules and runbooks before invoking nondeterministic reasoning.

Planned

06 — From Evidence Packets to Incident Packets

Assembling operational observations, symptoms, history, and diagnostic findings into a coherent incident-level record.

Planned

07 — From Diagnosis to Next Best Action

Evaluating whether proposed actions are grounded, cautious, explainable, and appropriate to the available evidence.

Planned

08 — Policy-Governed Mitigation

Constraining automated response through explicit policy, risk tiers, approval boundaries, and auditable decisions.

Project Foundations

The earlier Policy-Governed DataOps Agents article described an Incident Packet as the bounded context supplied to a diagnostic or decision process. As implementation advanced, this research refined the underlying model: Evidence Packets contain atomic, structured operational observations and provenance, while an Incident Packet may assemble multiple Evidence Packets with symptoms, diagnostic findings, relevant history, policy context, recommended actions, and eventual outcomes.

Evidence Packets are observations. Incident Packets are incident-level aggregations and interpretations.

Logs Aren't Enough

Introduced the project's argument for preserving structured operational context beyond isolated log lines.

Designing Evidence Packets

Records the project's initial Evidence Packet architecture. It remains part of the research history; the benchmark and independent review on this page narrow the claims that architecture can currently support.

AI-Native SDLC

Explores bounded AI contributions inside deterministic orchestration, a related operating principle for constrained diagnostic assistance.

External Related Work

The primary references below identify what already exists and what proposition remains to test. They are independent of the PySynapse project and include direct alternatives to its hypothesis, not just conceptual precedents.

OpenTelemetry

Standardizes generation and export of traces, metrics, logs, and baggage, with context propagation for correlating signals across service boundaries. Evidence Packets do not replace this telemetry layer. The remaining question is whether selecting incident-relevant observations, bounding them, preserving their provenance and integrity, and declaring collection gaps produces a measurably better diagnostic input than the same underlying correlated telemetry without packet packaging.

W3C PROV Data Model

Provides a domain-neutral model and interoperable representations for the entities, activities, and agents involved in producing data. Provenance-bearing structured observations are therefore not novel here. Evidence Packets specialize provenance for operational diagnosis and add a bounded artifact, integrity checks, explicit capture outcomes, and evidence-gap declarations; whether that specialization improves diagnosis remains untested.

OpenLineage

Defines a source-neutral event model for jobs, runs, and datasets, extended through facets, with maintained Airflow and dbt integrations. It already addresses cross-tool lineage and run correlation, so a source-neutral event vocabulary is not an Evidence Packet contribution. The narrower proposition left to test is whether explicit evidence requirements, missing-observation declarations, integrity checks, and bounded diagnostic packaging add utility beyond OpenLineage-shaped events aggregated at diagnosis time.

RCACopilot (Microsoft Research, EuroSys 2024)

Matches incidents to handlers, aggregates critical runtime diagnostic information, then uses an LLM to classify root cause and draft an explanation; Microsoft reports evaluation on a year of real incidents and four years of production use for its collection component. This shows that bounded diagnostic context can be assembled at diagnosis time without Evidence Packets. A direct comparison is needed to learn whether enforcing the proposed packet discipline upstream improves diagnostic accuracy, calibration, or auditability.

X-lifecycle Learning for Cloud Incident Management (Microsoft Research, FSE 2024)

Combines incident context with code, configuration, monitor data, service properties, dependencies, and troubleshooting documents, and reports improved root-cause recommendation performance from that added cross-SDLC context. It directly supports the competing explanation that better evidence acquisition—not a new packet representation—is what improves diagnosis. Independent Diagnostic Validation must hold the underlying evidence constant to isolate any packaging effect.

Read together, these approaches narrowed rather than eliminated Phase I's hypotheses. Existing standards already collect, correlate, and describe telemetry and provenance; existing systems already assemble operational context for diagnosis. Phase I asked whether Evidence Packet representation changes independent diagnostic performance when observations are held constant (RQ1, closed, no demonstrated advantage) and whether durable capture preserves diagnostic value that later reconstruction loses (RQ2, closed, a threshold-crossing but statistically thin advantage). Nominal historical characterization, LLM assistance, and end-to-end automation remain distinct later questions, not yet scheduled.

Research Log

Research Log 001

Establishing the Diagnostic Corpus

Question

Can a controlled corpus test whether Evidence Packets contain sufficient diagnostic context?

What changed

Nightly Airflow-orchestrated dbt operations are producing Evidence Packets and storing them in PostgreSQL.

Constraint discovered

Naturally occurring failures will produce a small and poorly balanced diagnostic sample.

Decision

Build a three-tier corpus consisting of:

  1. Generated synthetic Evidence Packets
  2. Mutations of Evidence Packets derived from real templates
  3. Controlled fault-injected operations passing through the real capture path

Next

Implement the synthetic Evidence Packet generator and ground-truth scenario definitions.

Research Log 002

From a dbt Evidence Packet to a Source-Neutral Model

Question

Can the Evidence Packet model be generalized beyond dbt without invalidating the packets already collected?

What changed

Packet schema version 0.3.0 separates the packet model from any single evidence source's vocabulary. dbt is now implemented as the first source adapter against that model, with a compatibility layer and semantic compatibility tests confirming that 0.3.0 packets remain semantically equivalent to version 0.2.0 output.

Constraint discovered

Generalizing the model surfaced cases where evidence had been silently absent from a packet rather than explicitly declared as missing.

Decision

Make missing evidence a first-class, explicit part of the packet rather than an omission, and validate the model with a representative fixture corpus and dedicated corpus-assessment tooling.

Next

Add PostgreSQL and Airflow as additional source adapters, and begin the evidence-completeness work described in Article 03.

Research Log 003

Building the Controlled Corpus

Question

Can a versioned, ground-truth scenario corpus and a deterministic evaluator be built to test evidence completeness without relying on any diagnostic system?

What changed

Ten canonical PostgreSQL/dbt/Airflow scenarios were defined with independent ground truth and expected-evidence contracts. A deterministic evaluator now scores specification completeness and evidence-requirement satisfaction against those contracts. Level A (synthetic construction) and Level B (packet mutation) tests exercise the evaluator itself.

Constraint discovered

Level A packets are built directly from a scenario's own expectations, so their completeness is guaranteed by construction. That makes Level A a regression test of the evaluator, not evidence that real collectors produce complete packets — only an experiment against real services could test that.

Decision

Keep Level A and Level B as evaluator tests, and run a small, explicitly bounded Level C experiment against a real, isolated Airflow/dbt/PostgreSQL stack before drawing any conclusion about real collection.

Next

Run the Level C experiment.

Research Log 004

Level C: Baseline to Improved Collection

Question

Does the existing dbt-only collection path produce packets that satisfy the five Level C scenarios' evidence requirements, and can bounded collector changes close any gap?

What changed

Five fault conditions ran once each on an isolated local Docker stack (Airflow 3.2.2, dbt Core 1.11.5, PostgreSQL 16.2). The existing dbt-only path produced a baseline packet for each; mean required-evidence completeness was 0.10, with zero of five scenarios meeting their requirements, and the Airflow-before-dbt condition produced no packet at all. Bounded collector changes — Airflow task-state capture, a PostgreSQL connection/relation probe, normalized dbt status summaries, and a cross-source correlation key — were then added and the five conditions rerun. Mean completeness reached 1.00, with all five meeting their requirements.

Constraint discovered

Level A had implicitly assumed a collector would produce whatever evidence a scenario required; the baseline showed that assumption did not hold for four of five scenarios. All improved packets were also classified over-collected relative to the scenario's stated minimum, and the four collector changes were bundled together, so this run alone can't show which change closed which gap.

Decision

Publish the baseline and improved artifacts as-is, including the negative baseline result, rather than only reporting the improved figure.

Next

Independent review of the experiment before drawing conclusions from it publicly.

Research Log 005

Independent Methodological Review

Question

Does the Level C result (0.10 → 1.00) actually demonstrate that Evidence Packets close a real evidence gap, or does it demonstrate something narrower?

What changed

An independent review of the repository and the Level C experiment — committed to the repository in full — found several problems that don't invalidate the engineering work but do narrow what the result can be said to show:

  • Level A's 10/10 result is tautological — packets are built from the same expectations they're scored against.
  • Level B tests the evaluator's sensitivity to damaged packets, not real collector behavior.
  • The Level C "improved" collector was built with direct knowledge of the benchmark's scenario expectations — its correlation key is named after the benchmark's internal scenario identifier, and its dbt status summaries were shaped to match the evaluator's specific field lookups.
  • The Level C PostgreSQL evidence came from one static observation reused across all five scenarios rather than a per-run probe, and was inert (non-distinguishing) for four of the five.
  • The four collector changes were confounded, with no ablation isolating which one mattered.
  • No human, deterministic, or probabilistic diagnostic process has been evaluated using any of these packets — "answerable" currently means only that specified fields are present, not that a diagnosis was reached.
  • No comparison exists against raw or conventionally aggregated telemetry, so the value of the Evidence Packet representation specifically, as opposed to better evidence acquisition in general, has not been isolated.
  • The real production corpus (nightly MedTracker/ADS-B collection) contains only successful runs; failure diagnosis has never been tested against a naturally occurring failure.

Constraint discovered

The current benchmark answers a narrower question than the one the research hypothesis is ultimately about. It shows that collection can be made to satisfy a hand-written specification; it does not show that satisfying that specification helps anyone diagnose anything.

Decision

Publish this review alongside the Level C results rather than after them, describe the current metric on this page as specification completeness / evidence requirement satisfaction rather than unqualified "answerability," reserve "diagnostic answerability" for a real diagnostic experiment, and make that experiment — Independent Diagnostic Validation, with a scenario-blind collector, hidden ground truth, and a raw-telemetry comparison baseline — the next research milestone instead of publishing Article 03.

Next

Rebuild the Level C collector without benchmark-specific knowledge, replace the static PostgreSQL snapshot with a per-run probe, add the ablation study, and design the Independent Diagnostic Validation experiment described in the Technical Research Path.

Research Log 006

External Literature Review: Correcting a Novelty Claim

Question

Does the source-neutral Evidence Packet architecture represent something new, or does existing work already establish it?

What changed

A focused search for prior and competing approaches was added to External Related Work: OpenTelemetry, W3C PROV, OpenLineage, RCACopilot, and Microsoft's cross-SDLC incident-diagnosis research. OpenTelemetry and OpenLineage already standardize correlated telemetry and source-neutral lineage/run events, while the Microsoft systems show that operational and SDLC context can be assembled for diagnosis without an Evidence Packet format.

Constraint discovered

The project had been describing its schema architecture as though source neutrality and provenance-bearing observations were themselves contributions. They are not: OpenLineage and W3C PROV already cover those concepts, while OpenTelemetry provides a mature correlated-telemetry baseline. The narrower question is whether deliberate selection, bounded scope, integrity verification, and explicit evidence-gap declarations measurably improve diagnosis when the underlying observations are held constant.

Decision

Stop presenting the source-neutral packet schema as a novel contribution. The Technical Research Path and Related Work now both say so directly, and the open research question is reframed around evidence-collection discipline and its effect on diagnosis, not schema design.

Next

When Independent Diagnostic Validation runs, include an OpenLineage-shaped baseline — existing telemetry aggregated at diagnosis time, without a bounded collection-time format — alongside the raw-telemetry and Evidence Packet conditions, so the comparison also tests the collection-time-vs-diagnosis-time bet directly rather than only the representation question already planned.

Research Log 007

Broadening the Hypothesis Space Without Moving the Finish Line

Question

Does contemporaneous capture have value beyond packet representation, and can packets from executions with no known defect form a useful historical operational corpus?

What changed

Two future dimensions were identified. First, execution-time evidence may expire, mutate, lose context, or otherwise become impossible to reconstruct faithfully by diagnosis time. Second, repeated nominal Evidence Packets may eventually characterize system behavior and provide historical context for later deviations.

Constraint discovered

Both questions could expand the implementation indefinitely through retention modeling, reconstruction experiments, behavioral baselining, anomaly detection, or incident detection before the representation hypothesis has been tested.

Decision

Preserve the architectural option to retain contemporaneous observations and packets for every execution, including nominal executions, without implementing those deferred capabilities now. RQ1 remains active; RQ2 and RQ3 are preserved but deferred.

Next

Complete Experimental Isolation and Independent Diagnostic Validation for RQ1. Phase 1 ends with a defensible positive, negative, mixed, or inconclusive statement about representation utility.

Research Log 008

RQ1 Primary Execution and Closure

Question

Does Evidence Packet representation improve independent diagnostic performance when the underlying operational evidence is held equivalent?

What changed

Independent Diagnostic Validation ran to completion: 60 matched pairs (12 scenario classes × 5 repetitions), 120 genuine, one-shot diagnosed observations, no fixture replay. Conventional correctness was 13/60 (0.217); packet correctness was 12/60 (0.200); packet-minus-conventional was −0.017 against the prespecified ±0.10 materiality rule.

Constraint discovered

Absolute accuracy was low in both conditions — 7 of 12 scenario classes were at or near a 0/5 floor in both arms — which limits how much the null result alone can say about representation in general. The originally frozen diagnostic subject (gpt-5.4-2026-03-05) proved cost-prohibitive mid-study and was substituted, pre-result, for gemini-3.1-flash-lite, after a full requalification pass.

Decision

Freeze the null result as the primary finding rather than treat it as inconclusive or rerun it with a different model. Report it plainly: no demonstrated material correctness advantage from packet representation under equivalent evidence — an unambiguous null under the prespecified rule, not proof that representation can never matter. Exploratory secondary metrics (packet responses were far better grounded — evidence support 0.047 → 0.569, unsupported assertions 1.867 → 1.150 — without becoming more correct) are retained as a hypothesis for future work, not as an override of the primary verdict. Frozen verdict: RC1_PRIMARY_COMPLETE_NO_DEMONSTRATED_ADVANTAGE.

Next

RQ1's own closure record asks what happens when evidence available at diagnosis time is not held equivalent to execution-time evidence — the direct premise for RQ2.

Research Log 009

RQ2: Evidence Durability — Design, Primary Execution, and Closure

Question

Does capturing diagnostic evidence at execution time preserve later diagnostic capability, relative to reconstructing evidence from the same operational sources after realistic degradation?

What changed

Both arms began from identical execution-time evidence; the durable arm was diagnosed on the Evidence Packet exactly as captured, the reconstruction arm on evidence reassembled after a predeclared, ground-truth-blind degradation rule (e.g., a dbt manifest genuinely overwritten by the next real invocation in the same reference stack). 25 disjoint events, 50 genuine requests, ran and froze: durable correctness 8/25 (0.320), reconstruction 5/25 (0.200), difference +0.120 against a +0.10 threshold — material by the prespecified rule and guardrail-passing, with fewer unsupported assertions for durable capture (0.720 vs. 1.320) at essentially equal input-token volume between arms (8,204,048 vs. 8,203,837 tokens). Frozen verdict: RQ2_PRIMARY_COMPLETE_SUPPORTS_DURABLE_CAPTURE_ADVANTAGE.

Constraint discovered

The result is materially thinner than the verdict string alone suggests. Only 9 of 25 pairs were discordant (6 durable-only, 3 reconstruction-only); the descriptive exact test on that split is compatible with chance (two-sided p = 0.508). At 25 pairs, correctness differences are quantized in 0.04 steps, and +0.120 is the smallest possible nonzero value that could ever have cleared the +0.10 threshold — a single pair flipping either direction would have reversed the verdict. Representation format was also not held constant between arms (durable in packet format, reconstruction in conventional format, by design, reasoned from RQ1's own null result on format), so this is a preservation-centered comparison, not a clean isolation of preservation from representation. Separately, after primary execution completed (50/50 requests, 0 retries, immutable), an independent review found defects in the finalization interface used to compute the result — not the scientific estimand, threshold, guardrail formula, or verdict logic, which a direct diff confirmed were byte-unchanged throughout. The interface was corrected and independently re-reviewed twice (contract, then implementation) before finalization was authorized; the primary observations were never modified.

Decision

Report the verdict plainly alongside its full statistical thinness in the same breath, not as two separately-weighted claims. Treat the post-execution interface correction as a disclosed, independently-reviewed governance event, not something to omit for a cleaner narrative — see Phase I Artifacts for both independent reviews in full.

Next

Interpret RQ1 and RQ2 together as an integrated Phase I synthesis.

Research Log 010

Phase I Synthesis and Closure

Question

After testing representation separately from preservation, what does Phase I actually establish about the value of Evidence Packets?

What changed

RQ1's null and RQ2's thin positive result were synthesized, adversarially self-reviewed, and given a final human intellectual-review pass before Phase I was formally closed. The synthesis's conclusion: representation alone did not move correctness under equivalent evidence; durable capture showed a threshold-crossing but statistically thin advantage over reconstruction, accompanied by fewer unsupported assertions; together they shift the program's leading hypothesis about Evidence Packets' value from representation toward preservation, without establishing that shift conclusively.

Constraint discovered

Neither study tested long-term archival durability (24 of 25 RQ2 reconstruction gaps were under three minutes), statistical significance, generalization beyond one instrument and reference stack, or a causal mechanism for the RQ2 difference. A distinct, untested axis — historical/nominal characterization — remained identified but unresolved; see Preserved Future Research Questions.

Decision

Freeze the synthesis as the authoritative Phase I conclusion and close Phase I. RQ3 remains deferred and not started; nothing in the synthesis authorizes beginning it.

Next

Public-site reconciliation (this update) is a separate activity from the research itself and does not reopen or extend Phase I.

Back to Blog Discuss the Research