Don't Land What You Don't Want to Own

DICOM as radioactive input

Sensitive data behaves less like ordinary technical debt and more like radioactive material: the risk it poses is a function of how long you hold it, it contaminates the systems it touches, and it has a long half-life in the places you've forgotten about — the snapshot from last quarter, the replica in the DR region, the log line nobody meant to write.

A DICOM object is a concentrated example. It is not simply an image; it is a structured data object in which patient identifiers, study information, dates, accession information, institutional data, and vendor-specific private elements can coexist with hundreds of megabytes of perfectly legitimate imaging data.

That creates an interesting ingestion problem. A conventional pipeline might look roughly like this:

External source
      |
      v
Raw DICOM store
      |
      v
De-identification / policy
      |
      v
Research DICOM store

Operationally, that's attractive. The original is safely landed before anything complicated happens.

From another perspective, though, you've already crossed the most important boundary.

The raw object is now yours.

If it contains information your downstream policy says you shouldn't retain, the system nevertheless retained it — at least temporarily. And "temporarily" can become surprisingly complicated once backups, replicas, logs, and snapshots become involved.

This isn't only an engineering concern. It's the architectural face of a principle regulators already name explicitly: data minimization, in GDPR's terms, or the minimum necessary standard in HIPAA's. Retention scope, breach-notification exposure, and audit surface are all functions of what your systems have possessed — not merely what they currently hold. An architecture that never takes possession of a datum has a categorically smaller compliance story than one that lands it and cleans it later.

So I started thinking about a different boundary:

External source
      |
      v
Ephemeral inspection / transformation
      |
      v
Trusted persistence

The rule becomes:

Nothing gets persisted until the object has passed policy.

That sounds attractive. It also raises a practical question.

How much work do you have to do to a DICOM object to make that decision?

Don't decode what you don't need

My first experiments around this problem produced fastDICOMattrs, a small C++ library for high-speed access to selected DICOM attributes.

But the pre-persistence problem required something different. It wasn't enough to find a handful of attributes quickly. To safely transform an object, I needed to understand its structure well enough to remove or replace elements and produce a valid DICOM object afterward.

That led to a second experiment: fastDICOMstructure.

The central design principle became:

Do not decode or materialize data that does not need to be interpreted.

Consider a simplified DICOM object:

(0008,0060) Modality          = CT
(0010,0010) PatientName       = DOE^JOHN
(0010,0020) PatientID         = 12345678
(0018,0050) SliceThickness    = 1.0
(0019,xxxx) Private Element   = ...
(7FE0,0010) PixelData         = <512 MB>

Suppose an ingestion policy says:

preserve  (0008,0060)
remove    (0010,0010)
hash      (0010,0020)
preserve  (0018,0050)
remove    private elements
passthru  (7FE0,0010)

Why should processing that policy require decoding or materializing 512 MB of Pixel Data?

It shouldn't.

Instead, fastDICOMstructure scans the DICOM structure and records information about the elements it encounters: tag, value representation, length, byte offsets, sequence and item structure, transfer syntax, and bulk-data boundaries. Pixel Data can then be represented as a reference into the original input rather than as a giant value copied into an object model.

The transformation problem starts looking less like:

deserialize everything
        |
        v
build object model
        |
        v
modify object model
        |
        v
serialize everything

and more like:

Incoming DICOM bytes
        |
        v
Structural scan
        |
        +---- inspect metadata
        +---- inspect sequences
        +---- apply policy
        +---- identify bulk data
        |
        v
Selective transformation
        |
        +---- preserve safe spans
        +---- remove selected elements
        +---- replace selected values
        +---- remove private elements
        +---- pass Pixel Data through
        |
        v
Valid DICOM output

The distinction matters if this processing is going to sit directly on a high-volume ingestion path.

Turning the idea into an experiment

I wanted to keep the scope deliberately small.

fastDICOMstructure is not a PACS or VNA. It isn't a DICOM viewer. It isn't a complete anonymization solution. It doesn't attempt to determine whether identifying information has been burned into image pixels. And it isn't intended to be a security boundary by itself.

The experiment was narrower:

Can I understand enough of a DICOM object's structure to inspect and selectively transform it efficiently, while preserving everything I don't need to touch?

I implemented the core in C++20, with a C ABI and a thin Python binding so the structural machinery can still be incorporated into Python-oriented imaging and data pipelines. The implementation supports structural inspection and mutation, nested sequences and items, element removal and replacement, private-element removal, and references to native and encapsulated Pixel Data without materializing the pixel bytes.

Most importantly, I established a finish line before allowing the experiment to grow into an imaging platform. The implementation had to:

• round-trip real DICOM data byte-for-byte when no transformation was requested
• correctly represent nested sequence and item structures
• transform metadata according to a policy
• apply a private-tag policy
• avoid materializing Pixel Data
• produce output independently readable by another DICOM implementation
• quantify preservation of untouched content
• demonstrate a measurable performance advantage
• run as part of a minimal pre-persistence pipeline

Then I tested it.

What the corpus showed

I ran the implementation against ~26,600 real-world CT DICOM files from two public collections.

For unmodified Explicit VR Little Endian objects, fastDICOMstructure produced 100% byte-identical round trips across the corpus. Its structural interpretation also achieved 100% agreement with pydicom, which I used as an independent reference implementation.

I then applied the example transformation policy across the corpus, independently read the resulting objects using pydicom, and measured how much of the original value payload survived the transformation unchanged.

The result was 99.99% byte preservation of value-payload data.

That number is important to interpret correctly. It isn't a claim that every transformed DICOM file is 99.99% identical, or that arbitrary transformations can preserve that percentage. It measures preservation under the specific policy being tested. The point is that the implementation can make targeted structural changes while leaving essentially all unrelated payload alone.

Performance was encouraging as well. On typical files in the test corpus, the structural implementation was approximately 16× faster at the median than the equivalent pydicom processing path. On files dominated by large Pixel Data, the advantage narrowed — as you'd expect when I/O begins to dominate — but the implementation remained approximately 2× faster while using roughly half the peak memory.

Those results support the hypothesis I actually cared about:

A pre-persistence structural inspection step does not necessarily require fully materializing a DICOM object or imposing the cost of a heavyweight processing pipeline on every incoming image.

An important limitation

There's a useful imperfection in these results.

The ~26,600 real-world files in my test corpus all turned out to use Explicit VR Little Endian transfer syntax. fastDICOMstructure also parses Implicit VR Little Endian, and that path has synthetic-fixture test coverage, but it has not yet received the same real-world corpus validation.

There are other deliberate limitations as well. Explicit VR Big Endian isn't currently supported. Implicit VR parsing uses a structural heuristic rather than a complete DICOM data dictionary, which limits what can be inferred about some defined-length nested structures.

A note on compressed transfer syntaxes: because the design treats Pixel Data structurally — as encapsulated fragments and offsets, never as decoded pixels — compressed objects don't require decompression to pass through this layer. That handling has fixture-level coverage, but the real-world corpus contained no compressed-transfer-syntax objects, so it carries the same caveat as Implicit VR: structurally supported, not yet corpus-validated.

Those aren't details I want to hide behind benchmark numbers. They're part of defining what this experiment has — and hasn't — demonstrated.

Metadata isn't the whole PII problem

There's also a much larger boundary around this work.

Removing Patient Name from a DICOM element doesn't prove that an object contains no identifying information. Some medical images carry identifying information burned directly into their pixels. Detecting that is an image-analysis problem involving image decoding, OCR or computer vision, modality-specific considerations, and an entirely different set of validation questions.

fastDICOMstructure intentionally doesn't solve it.

A more complete ingestion architecture might instead look something like:

                    ┌─ metadata policy
                    |
Incoming DICOM ──> structural inspection
                    |
                    ├─ private-tag policy
                    |
                    ├─ pixel-risk policy ──> specialized processor
                    |
                    └─ transformation / audit
                              |
                              v
                       Trusted persistence

The structural layer doesn't need to know how every policy is implemented. It needs to expose enough deterministic structure to allow the system to decide what processing is necessary before the object is allowed across the persistence boundary.

That distinction has become more interesting to me than the parser itself.

The obvious objection

Pre-persistence transformation is irreversible. If the policy is wrong — too aggressive, or subtly destructive — there is no raw copy to return to. That irreversibility is precisely why the land-first doctrine exists, and it deserves a direct answer rather than a hand-wave.

The honest answer is that this pattern spends recoverability to buy containment, and that trade has to be made deliberately.

It can also be bounded. A quarantine store with short, enforced retention and tightly restricted access can hold originals long enough to validate a policy before it runs unattended — a very different thing from an indefinite raw layer woven into backup and replication infrastructure. Hash manifests of removed elements make transformations auditable: you can prove what was removed, and verify policy behavior later, without retaining the sensitive values themselves. And policy changes can be validated against a held-out corpus before they touch production traffic.

None of that eliminates the tradeoff. There are pipelines — exploratory research ingest, unstable upstream formats, data whose sensitivity is low relative to its irreplaceability — where landing raw remains exactly right. The argument here is narrower: for sufficiently sensitive input, irreversibility at the boundary can cost less than possession behind it, and that decision should be made consciously rather than defaulted by instinct.

The broader lesson

I started this work partly as a way to revisit medical imaging technology I hadn't worked closely with for some time, and partly to build something concrete enough to measure. I expected to spend most of my time thinking about DICOM parsing.

Instead, the more interesting question became where the trust boundary belongs.

Data engineering has good reasons for treating immutable raw data as valuable. Reproducibility, auditability, and recovery all benefit from preserving the original input. But that principle isn't absolute. When input is potentially sensitive, preservation itself carries a cost. "We'll clean it downstream" means something very different once the original has propagated through storage, backup, replication, and operational systems.

Sometimes the right raw-data strategy may therefore be to decide that certain bytes never become raw data at all.

And while DICOM makes the problem unusually vivid — megabytes of legitimate payload wrapped around grams of dangerous metadata — the reasoning isn't specific to medical imaging. The same calculus applies to a CSV feed carrying national identifiers, an event stream carrying email addresses, or application logs carrying bearer tokens.

The experiment with fastDICOMstructure suggests that, at least for structural DICOM metadata, moving that decision ahead of persistence doesn't necessarily require an enormous processing penalty.

And I think that's the part of this project I'll carry forward:

Don't land what you don't want to own.

The best place to remove sensitive data may be the point immediately before your infrastructure ever has the opportunity to remember it.

Foundation and Next Steps