The run that never ended
A shell command exited in seven milliseconds. The run it belonged to was still going ten minutes later. The bug was in what we were waiting for.
A sandbox run finished its work and then sat there. The agent had done everything it was asked, written its files, printed its summary, and exited cleanly. The run stayed open. Not for a few seconds — indefinitely.
The completion check looked reasonable:
await Promise.all([proc.exited,consumeStdout(),consumeStderr(),])
Wait for the process to exit, and wait until you have read everything it wrote. Only then is the run over. It reads like care.
What a pipe actually is
When you spawn a process with a pipe for its output, you create a channel with two ends. The child holds the write end; you hold the read end. Reading returns end-of-file when every copy of the write end has been closed.
Every copy. Not the child's copy — every copy.
A child that spawns its own children hands them that same write end by default. They inherit it. So a background process the agent started and forgot about — a watcher, a dev server, a stray sleep — is holding your pipe open long after the agent that started it is gone. There is nobody left to write anything. The channel is silent. It is also, formally, still open, and it will stay that way for as long as that orphan lives.
Our reader was waiting for a signal that a healthy, correct, already-finished run could never send.
The fix is to change the question
The instinct is to make the drain smarter. Detect the orphan, hunt down the process group, close the descriptor behind everyone's back.
The better fix is to notice that we were asking the wrong question. The process's exit code is the verdict. The output is evidence, and evidence is worth waiting a bounded amount of time for, but it is not what decides whether the run succeeded.
So the order changed. Wait on the process. Once it has exited, give the pipes a grace period to finish draining — five seconds, which is far longer than a healthy drain ever needs. If they finish, nothing else happens. If they do not, cancel the read, record an event saying the output was truncated and by how much, and finalise the run with the exit code the process actually gave us.
A run that exited zero now reports succeeded, even if some orphaned grandchild is still clinging to the pipe.
Checking it against a real process
This is the kind of fix that is easy to prove against a mock and easy to get wrong against an operating system. So the test uses neither a fake process nor a fake pipe:
/bin/sh -c 'echo "did the work"; sleep 600 & exit 0'A shell that prints something, leaks a ten-minute background process onto its stdout, and exits immediately. What we measured:
- child exited after 7 ms, code 0
- pipes drained naturally? false
- run finalised after 1509 ms
- captured output:
["did the work"] - reader after abort: released
The old code would have finalised that run in ten minutes.
Two details in there matter more than they look. The output survived — bounding the drain does not mean discarding what already arrived. And cancelling a pending read resolves it as done rather than throwing, which means aborting the drain does not manufacture an error and flip a successful run to failed. That behaviour is worth knowing before you rely on it, because the opposite would be a silent, intermittent, extremely annoying bug.
The general shape
The failure here was not a missing feature or a rare race. It was a completion condition that described something other than completion, in code that looked careful. It waited patiently and correctly for a thing that would never happen, and it did so with no error, no warning, and no timeout — the most expensive kind of wrong.
When something hangs, the useful question is rarely "why is this slow". It is: what exactly am I waiting for, and is that actually the thing I care about?