The events that were never sent

The report was always the same shape. Someone starts a task from their phone, walks out of Wi-Fi range, and the run card stops. Not an error, not a failure — it just stops, somewhere around step nine, and stays there. Come back to the machine and the task had finished perfectly.

The agent was never the problem. The events describing what it was doing were being dropped on the floor, and nothing ever sent them again.

One early return

Live updates ride a websocket to a relay, and the code that pushed an event began like this:

def publish(self, kind, data):
    loop, ws = self._loop, self._ws
    if not (loop and ws and self.joined):
        return
    ...

Perfectly reasonable, and quietly catastrophic. A reconnect takes a few seconds. A task emits an event every second or two. So every reconnect silently deleted the handful of steps that happened during it — and because there was no record that they had ever existed, nothing could resend them. The phone was not behind; it was permanently missing a section, with no way to know.

The fix is to record before checking whether you can send. The event goes into a bounded ring buffer first, gets a monotonic sequence number, and only then does the code look at whether there is a socket to put it on. Now an outage costs latency instead of history.

Knowing that you missed something

Retaining events is only half of it. The phone still has to notice a gap, and a stream of untagged messages gives it no way to. So every event now carries a number, and every acknowledgement the PC sends back mentions where the stream has reached. If the phone is behind, it asks for the difference.

One detail turned out to be load-bearing: replayed events are stored already signed and resent byte-for-byte. Re-signing on the way out would produce a fresh timestamp, a different envelope, and a signature check that fails on arrival — so the replay would be silently discarded by the same mechanism meant to protect it.

The clock nobody suspected

Commands from the phone are signed over a timestamp, and the PC rejects anything more than two minutes out. Sensible. But a phone whose clock has drifted — or which came back from a flight with the wrong offset applied — has every single command refused, silently, with no acknowledgement. All the user ever sees is "your PC is not responding". The PC is fine. It is discarding valid instructions from a device it trusts.

The machine's own time now rides in its presence announcement and the phone signs against that instead of its own clock. Which creates a small deadlock worth naming: that announcement is the thing that teaches the phone the correct time, so if it were rejected for being stale, the phone could never learn the offset that would let it be accepted. Presence messages therefore verify their signature but skip the freshness window. The signature is what proves the message is genuine; the window only proves it is recent.

Four more, all invisible

Once the link was under a microscope, the rest followed the same pattern — code that looked correct and failed only in conditions nobody had reproduced.

The heartbeat shared a try block with a database write. One hiccup from the database killed the heartbeat task on a completely healthy socket, and sixty seconds later the server closed the channel for going quiet. The link dropped for a reason that had nothing to do with the link.

The connect call had no timeout, so a captive portal could hang it indefinitely — and because the retry loop lived outside that call, it never ran. The machine stayed unreachable until something restarted it.

The backoff reset after any session that ended without raising, including one that opened and closed immediately. A server refusing connections therefore got hammered in a tight loop by a client that believed it was being polite. Backoff is now forgiven only after a session that stayed up long enough to count as working.

And the phone's outbox — the thing that holds a message when the PC is asleep — was a single unsaved slot. A second thought while offline overwrote the first, and Android reclaiming a backgrounded WebView took the whole thing with it. You asked for something, put your phone in your pocket, and what you asked for had simply never existed.

What it took to trust the fix

All of this is now covered by a test suite that needs no server and no network: publish while disconnected and confirm the events survive, replay one and confirm it is unchanged to the byte, break the database and confirm the heartbeat keeps beating.

The suite is also checked the other way round. Reverting the ordering in publish — putting the early return back — has to make the tests fail. A test that passes against the bug it was written for is worse than no test at all, because it converts an open question into false confidence.