Pysynapse

The Autonomous DataOps Research Series

Toward Evidence-Driven Data Operations

Active Research Program Updated

Modern data-pipeline diagnosis often begins with fragmented evidence spread across orchestration logs, transformation artifacts, warehouse history, source metadata, source code, and team knowledge. This program explores how structured operational evidence, deterministic diagnostics, constrained AI reasoning, and policy-governed action can support more diagnosable and eventually more autonomous data systems.

Research Thesis

The primary bottleneck in autonomous DataOps is not reasoning alone. It is the quality, completeness, and structure of the evidence available to human, deterministic, and probabilistic diagnostic processes.

The work begins with evidence capture and Evidence Packets, progresses through an evidence repository and Incident Packet assembly when warranted, and then moves toward deterministic diagnosis, probabilistic assistance, Next Best Action, policy evaluation, mitigation, and learning.

Current Experiment

Evidence Packets are currently collected from nightly Airflow-orchestrated dbt runs. MedTracker provides the primary changing data source, while the ADS-B feed is temporarily unavailable pending relocation of its antenna connection.

Naturally occurring failures will produce a small and poorly balanced diagnostic sample, so the current work is building a controlled corpus of:

  1. Generated synthetic Evidence Packets
  2. Mutations derived from real Evidence Packet templates
  3. Controlled fault-injected operations passing through the actual evidence-capture path

The corpus will test whether captured evidence is sufficient for human diagnosis, deterministic classification, probabilistic diagnostic assistance, and defensible Next Best Action recommendations.

Current Implementation Status

Operational

Collection Foundation

  • Evidence Packet schema
  • dbt artifact collection
  • Airflow evidence-capture integration
  • PostgreSQL Evidence Packet repository
  • Nightly MedTracker evidence collection
In Progress

Experimental Corpus

  • Synthetic Evidence Packet generator
  • Template-derived Evidence Packet mutation
  • Fault-injection scenario definitions
  • Evidence-completeness assessment
Planned

Diagnosis and Action

  • Deterministic incident classification
  • Diagnostic benchmark suite
  • LLM-assisted evidence interpretation
  • Incident Packet assembly
  • Next Best Action evaluation
  • Policy-governed remediation
  • Incident memory and regression analysis

Technical Research Path

This is the conceptual and technical progression of the research, not a claim that every capability is complete.

1

Fragmented Logs and Operational Evidence

Established Problem
2

Evidence Capture

Operational
3

Evidence Packets

Operational
4

Evidence Completeness

In Progress
5

Synthetic and Fault-Injected Incident Testing

In Progress
6

Deterministic Diagnostics

Planned
7

Probabilistic Diagnostic Assistance

Planned
8

Incident Packet Assembly

Planned
9

Next Best Action

Planned
10

Policy-Governed Mitigation

Conceptual Foundation Established
11

Incident Learning and Continuous Improvement

Planned

Guiding Principles

Evidence before inference

Collect and preserve observable facts before interpreting the incident.

Observations separated from interpretations

Keep collected facts distinct from diagnostic findings and policy conclusions.

Deterministic diagnosis before probabilistic judgment

Route known conditions through rules and runbooks first.

Explicit uncertainty when evidence is incomplete

Make missing context and confidence limits visible.

Producer-neutral Evidence Packet design

Keep the formal artifact useful across orchestrators, warehouses, transformation tools, and other observation points.

Human-reviewable recommendations

Make every proposed action traceable to supporting evidence.

Policy boundaries before autonomous action

Define risk, approval, and authority limits before enabling mitigation.

Reproducible incident experiments

Use controlled scenarios that can be repeated and compared.

Packet evidence separate from diagnostic ground truth

Preserve the distinction between what an Evidence Packet captured and what the experiment establishes.

Healthy operations are evidence too

Capture normal operations to establish comparison data and avoid a failure-only corpus.

Publication Roadmap

In Development

01 — Logs Aren't Enough

Why modern data-pipeline incidents begin with an evidence scavenger hunt, and why structured operational evidence should become a first-class artifact.

Planned

02 — Designing Evidence Packets

Defining the structured observations and provenance required to evaluate pipeline operations across dbt, Airflow, data platforms, and source systems.

Planned

03 — Can We Trust the Packet?

Using synthetic, mutated, and fault-injected Evidence Packets to determine whether captured evidence is sufficiently realistic and diagnostically useful.

Planned

04 — Measuring Evidence Completeness

Benchmarking the evidence available to human, deterministic, and probabilistic diagnostic processes.

Planned

05 — Deterministic Before Probabilistic

Routing known conditions through rules and runbooks before invoking nondeterministic reasoning.

Planned

06 — From Evidence Packets to Incident Packets

Assembling operational observations, symptoms, history, and diagnostic findings into a coherent incident-level record.

Planned

07 — From Diagnosis to Next Best Action

Evaluating whether proposed actions are grounded, cautious, explainable, and appropriate to the available evidence.

Planned

08 — Policy-Governed Mitigation

Constraining automated response through explicit policy, risk tiers, approval boundaries, and auditable decisions.

The earlier Policy-Governed DataOps Agents article described an Incident Packet as the bounded context supplied to a diagnostic or decision process. As implementation advanced, this research refined the underlying model: Evidence Packets contain atomic, structured operational observations and provenance, while an Incident Packet may assemble multiple Evidence Packets with symptoms, diagnostic findings, relevant history, policy context, recommended actions, and eventual outcomes.

Evidence Packets are observations. Incident Packets are incident-level aggregations and interpretations.

AI-Native SDLC

Explores bounded AI contributions inside deterministic orchestration, a related operating principle for constrained diagnostic assistance.

Research Log

Research Log 001

Establishing the Diagnostic Corpus

Question

Can a controlled corpus test whether Evidence Packets contain sufficient diagnostic context?

What changed

Nightly Airflow-orchestrated dbt operations are producing Evidence Packets and storing them in PostgreSQL.

Constraint discovered

Naturally occurring failures will produce a small and poorly balanced diagnostic sample.

Decision

Build a three-tier corpus consisting of:

  1. Generated synthetic Evidence Packets
  2. Mutations of Evidence Packets derived from real templates
  3. Controlled fault-injected operations passing through the real capture path

Next

Implement the synthetic Evidence Packet generator and ground-truth scenario definitions.

Back to Blog Discuss the Research