Start with a repeatable result.

A job times out after writing some of its records. The orchestrator retries it. If the second attempt inserts those records again, the retry has made the data less reliable.

For an immutable event stream, a useful goal is simple: processing an identical event a second time should leave the stored result unchanged. The interesting work is deciding what counts as the same event and what to do when the payload disagrees.

Separate identity from validity.

An event ID tells you which event this is. It does not tell you whether the amount has the right type or the timestamp is valid. Validate the contract first, then compare the event against the stored state.

for event in batch:
    if not contract_is_valid(event):
        quarantine(event, reason="invalid contract")
        continue

    previous = warehouse.get(event.event_id)
    if previous is None:
        insert_once(event)
    elif previous == event:
        skip(event, reason="identical replay")
    else:
        quarantine(event, reason="payload conflict")

This is explanatory pseudocode. In a real database, the insert and conflict check need transactional protection; an application-level lookup alone can race with another writer.

Do not let a conflict disappear.

Two records can have the same ID and different amounts. Silently skipping the second hides a disagreement; silently overwriting the first invents an update policy. For the lab’s immutable events, a conflict goes to quarantine. A change-data-capture pipeline would instead need an explicit version or ordering policy.

PostgreSQL’s unique constraints and ON CONFLICT clause provide tools for coordinating concurrent writes. DO NOTHING handles a uniqueness conflict, but it does not by itself prove that the two payloads are identical. See the PostgreSQL INSERT documentation for the database semantics.

The smallest test that earns confidence.

  1. Process a known batch and capture the warehouse contents.
  2. Process the exact batch again. Assert that the contents have not changed.
  3. Change the amount on an existing ID. Assert that the original stays intact and the conflict is visible.
  4. Introduce a missing field, then correct and replay the batch. Assert that only missing events are added.

The simulator uses integer cents to make its example totals exact. It holds state in memory, so it demonstrates the decision rules without claiming database durability or an exactly-once distributed delivery guarantee.

Make the failure visible.

Choose “Schema drift,” run the pipeline, then correct and replay. The warehouse moves from four events to six. Run it again: it stays at six.

Try the experiment ↗