Atomic Write Introduction: When the System Learned to Stop Corrupting Itself
Structural evolution introducing atomic file-write safety (tmp → commit swap) to eliminate partial state corruption across indexing, aggregation, and snapshot generation pipelines under high build concurrency.
May 11, 2026
Evolution Trigger Signal
After shard-based indexing was introduced, the system encountered a new failure class: write integrity corruption under concurrency.
Build pipelines began producing partially written JSON files due to interrupted execution flows.
Concurrent processes overwrote shared outputs, leading to inconsistent and incomplete state snapshots.
The system could compute correctly but could not guarantee persisted correctness.
Pre-Transition Write Model
Failure Class Expansion
This failure was not computational but persistence-level: correct execution still produced invalid filesystem state.
The system began diverging between computed truth and stored truth.
This produced corrupted tag maps, incomplete graphs, broken snapshots, and inconsistent indexing outputs.
The underlying issue was a violation of state atomicity guarantees.
Correct computation no longer implied valid system state.
Observable Signals
Truncated JSON outputs began appearing in build artifacts.
Index files intermittently missed segments or contained partial structures.
Aggregation counts varied across identical build inputs.
Snapshot reads occasionally failed due to incomplete writes.
Evolution Decision
The system introduced atomic write semantics to eliminate partial state visibility.
Instead of writing directly to output files, data is written to temporary files first.
Once fully flushed, the temporary file is atomically renamed into place.
This guarantees that readers only ever observe fully committed states.
Write Safety Model
The system transitioned from direct overwrite semantics to a two-phase commit model.
Phase one writes to a temporary buffer file.
Phase two performs an atomic filesystem rename into the final location.
This ensures interruption safety and prevents partial file exposure.
Downstream Evolution Impact
Precomputed Graph Era
Stable build outputs enable deterministic semantic graph generation without runtime corruption risk.
Shard Indexing Transition
Earlier scaling layer that introduced distributed indexing and parallel aggregation pressure.
Compilation Debt Incident
Trigger condition exposing structural instability in long-running build pipelines.
Infrastructure Meaning
This evolution marks the shift from correct computation systems to reliable state-persistence systems.
System correctness is no longer defined by execution output alone but by safe persistence under failure conditions.
Atomic writes become the foundational guarantee layer for indexing, aggregation, snapshotting, and graph materialization.
Without this layer, higher-order infrastructure (graphs, slugs, governance) cannot maintain structural integrity.