Yesterday changed after the stream restarted

A hypothetical click dashboard reports 100 events for a minute. A mobile device reconnects with five more clicks from that minute. Then a worker restarts and the dashboard reports 110.

Five late events might explain 105. The extra five came from a second attempt at the history.

Decide which clock answers the question

Processing time says when the system handled an event. Event time says when the event claims to have happened. If you group by processing time, a delayed upload or replay changes the business minute. Event time preserves that meaning but cannot tell you when every event has arrived.

A common generator subtracts an out-of-order allowance from the maximum observed event time. Validate implausible device clocks before they advance that maximum. The exact boundary convention matters—milliseconds and inclusive/exclusive window ends can change which event is late. Flink time and watermarks.

Closing a window is a policy choice

Tumbling windows divide time into disjoint intervals. Sliding windows overlap and can place one event in roughly window_size / slide windows. Session windows merge activity separated by small gaps; a late bridging event can merge previously distinct sessions.

Choose what happens after the watermark: accept corrections for a bounded interval, emit a late side output, or reject beyond an explicit acceptance horizon. “The bill is final” needs stronger settlement rules than “the dashboard is current.” Retain enough raw input to recount when correctness requires it.

Early results are provisional. Emit an absolute total by window/version or a changelog with retractions; adding every successive running total double-counts. Preaggregation must preserve per-event lateness semantics and flush buffered data before forwarding progress markers that would declare it finished.

A checkpoint must include position and state together

Suppose a worker checkpoints its input offset at 100 but its counter only includes 99. Restore skips event 100. Reverse the mismatch and restore counts an event twice. A consistent checkpoint coordinates source positions and operator state.

Barriers define a consistent cut. Aligned checkpoints wait for participating inputs; unaligned designs can include in-flight data instead. Both have costs under backpressure. State, timers, source positions, buffers, and external writes do not all have identical recovery semantics; verify the engine and connector contract.

A checkpoint that protects operator state does not automatically roll back an external database write made afterward. On replay, the same event may be emitted again. Flink fault-tolerance model.

Absolute writes alone are insufficient if replay changes classification between an on-time total and a late total. Keeping both runs' outputs can count the same event in both categories. The source's custom protocol tags each writer's output with its checkpoint boundary, clears the abandoned post-checkpoint output, restores a watermark floor, and fences generations. That is a coordinated sink design with specific assumptions, not a general promise of “Flink exactly once.” A separate output generation can make the same issue easier to reason about at higher storage cost.

Replay needs a horizon

Deduplication state cannot grow forever. Expire it according to the event acceptance policy; accepting older events after forgetting their identity reopens duplicates. Processing-time TTL during downtime may erase state needed for an event-time replay. A watermark floor can prevent already-finalized windows from reopening under a changed input interleaving, if the chosen engine/operator protocol explicitly restores and enforces it.

Rescaling redistributes keyed state; it is a restore operation, not merely adding CPUs. Coordinate sink generations and outstanding transactions. For a transactional sink, its timeout must exceed the relevant checkpoint/recovery interval with margin and remain within broker limits; check current defaults instead of copying one timeout from an old example.

Observe watermark lag, input idleness, late-event rate, checkpoint age/duration, state size, output generation, and sink transaction failures. Distinguish a provisional operational view from a reconciled financial result.

Mental model: event time places the fact; the watermark advances policy; the checkpoint recovers computation; the sink protocol preserves effects.

Transfer: the two totals

---

Source trail: 05-message-queues-vs-event-streams, 07-event-time-watermarks-and-checkpoints, 04-idempotency-and-effectively-once-processing. See the source map for consolidation and reference-retained material.