For about a day and a half, a production dashboard we run for our own studio shipped nothing. Every commit to main built, every build failed, and every failure reported the exact same thing: Failed during stage 'building site': Build script returned non-zero exit code: 2. Twenty-five deploys in a row, one error message, zero information.
Here is the part that should worry you more than the outage did: nothing told us. No alert fired. We found out because an unrelated automated job we were rolling out came back saying its code wasn't deployed yet — and that thread pulled on a production site that had been frozen for thirty-two hours without a single alarm.
When we finally pried it open, exit code 2 turned out to be not one bug but three, stacked on top of each other, each one hiding the next. This is the write-up: how a generic error lies, why fixing the first cause made things briefly worse, and the one monitoring change that would have caught the whole thing in minutes instead of a day.
One Error, Three Bugs
A 32-hour silent freeze
Production shipped nothing for a day and a half. The platform said the same thing 25 times: exit code 2.
A Generic Error Is a Pointer, Not an Answer
An opaque exit code that is identical across unrelated commits is not describing your bug — it is telling you the detail lives somewhere else. Follow it to the real log before you touch a line of code.
The Deploy That Shipped Nothing for a Day and a Half
The failure mode was almost aggressively boring. Push a commit, watch it build, watch it fail. Push another — different commit, different files, completely unrelated change — and get the identical error. Queue updates, an uptime check, a session log, a real feature: it did not matter what the commit contained. Every one of them died at the "building site" stage with exit code 2.
That uniformity is itself a clue, though it is easy to read the wrong way. Your instinct on the first failure is "my last change broke the build." Your instinct on the twenty-fifth, spanning commits that share nothing, should be the opposite: the input doesn't matter, so the cause isn't in the input. Something about the environment or the pipeline is failing the same way regardless of what you feed it.
But we didn't get to that reasoning cleanly, because the thing that should have escalated a total production freeze simply never spoke. The site had last published successfully at 03:01 one morning. The next alarm was a human, the following afternoon, noticing something downstream looked stale. In between, the dashboard served the same day-old bytes to everyone, confidently, while two dozen builds failed in the background.
Twenty-five failed deploys, one generic error, zero alerts. A human noticed — no monitor did.
The Error Was Lying — and the Real Log Was a 404
The first real lesson landed before we fixed anything. The platform's status API — the thing our tooling reads to decide whether a deploy is healthy — reported every one of those failures as the same string. exit code 2, over and over. If you build alarms or dashboards on top of that API, you are building on a surface that has deliberately thrown away the detail.
So we went for the real log. And the real log was a 404. The REST endpoints that are supposed to return the granular build output — the per-deploy log, the per-build log — both returned "not found." The decisive artifact, the twenty lines that actually name what failed, was not available through the API at all. It existed only in the platform's web UI, rendered into the page's DOM behind a login.
That is a genuinely awkward place to be: the machine-readable surface is generic, and the human-readable surface is the source of truth. The move that unblocked us was unglamorous — open the UI, force every collapsed log section open, and read the rendered text off the page. Once we could actually see the phase-by-phase breakdown, the single exit code 2 split into three visibly different failures, on different deploys, at different stages. The API had been averaging three distinct problems down to one meaningless number.
The transferable rule: when an error is generic, treat the generic surface as a pointer, not an answer. Go to wherever the real detail lives — even if that is a DOM you have to scrape, because the tidy API endpoint gives you a 404. A monitoring stack that only ever sees the generic string will never tell you which of three things is wrong.
One String Hid Three Failures
What the status API gave us
What the real log showed
The machine-readable surface averaged three problems down to one meaningless number.
Three Bugs Wearing One Error
A generated, self-churning file sat on the deploy-freshness protected list, so every build went stale before it could publish.
Building complete → deploy skippedA cost guard working exactly as designed: a commit touched no build-relevant path, so it refused to rebuild.
Canceled: no content changeCompatibility mode injects every env var into every function, capped at 4KB. A 1.6KB dead fallback key pushed it over.
Deploy failed: env exceeds 4KBFix the top layer and the next surfaces wearing the same error — which is why progress felt like failure.
Three Bugs Wearing One Error Message
Here is what exit code 2 was actually hiding, in the order we peeled them.
Cause one — a freshness gate eating its own tail. We run a small deploy-freshness guard: if a newer commit to main has already touched a build-relevant path while the current build is still running, cancel this build so we don't publish something already superseded. Sensible in principle. The bug was in the path list. A generated, machine-authored file — rebuilt automatically every few minutes by a background housekeeping loop — was on the protected list. So every build went "stale" against that churning file before it could finish. A build took about three and a half minutes; the file changed faster than that. The gate cancelled every publish, forever, and reported it as a build failure. The guard meant to protect production had livelocked it.
Cause two — a cost guard doing its job. Once we stopped the livelock, some commits started reporting a different failure: "canceled build due to no content change." That one is not a bug at all — it is a build-ignore cost guard working exactly as designed. If a commit touches nothing the build actually depends on, don't spend a build on it. Correct behaviour, wrong moment: it looked like yet another failure in a sea of failures, and it briefly convinced us the fix hadn't worked. The unlock was to force one real build through a manual build hook.
Cause three — a 4KB ceiling we'd been creeping toward for weeks. With the livelock gone and a real build forced through, the build finally completed — and then deploying failed. The log: "environment variables exceed the 4KB limit." The functions behind this site run in a compatibility mode that injects every one of the site's environment variables into every serverless function, and the total is capped at 4KB by the underlying platform. We had drifted over the line during the outage window. The culprit was a single key worth about 1.6KB — roughly forty per cent of the entire budget — that a function already read from blob storage and only kept in the environment as a dead fallback. Deleting it from the environment brought us back under the cap and the site finally published.
Three causes. One error message. Each one only became visible after the one in front of it was cleared.
What the status API said
- exit code 2
- exit code 2 (again)
- exit code 2 (25 times)
- the log endpoint: 404
What was actually true
- A freshness gate cancelled every publish
- Then a cost guard skipped the rebuild
- Then the env budget blew past 4KB
- Three failures, three different stages
Why Fixing One Bug Felt Like Failure
Why Fixing the First Bug Made It Briefly Worse
The stacking is the part worth sitting with, because it is what makes this class of bug so expensive.
When you fix cause one and the same generic error is still there — now produced by cause two — every instinct tells you your fix failed. So you second-guess the change you just made, maybe revert it, and now you are further from the answer than when you started. A stack of failures behind a single opaque code doesn't just cost you the time to fix three bugs. It costs you the time to not un-fix the ones you already solved, because the feedback signal doesn't change when you make progress.
This is why "it's still broken" is almost useless information when the error is generic. Still broken how — same stage, same message detail, same phase? We only made real progress once we were reading the phase-level log and could see the failure move — building-stage cancel, then a content-change skip, then a deploy-stage env error. The message on the generic API never changed through any of it. If we had trusted that surface, we would have concluded, correctly and uselessly, that nothing we did mattered.
Three Checks, or One Alarm
Cause-specific checks
Three green ticks, and the next freeze still ships nothing.
One outcome alarm
Cause checks cover the bugs you have met. The outcome alarm covers the class.
Go to the real log
Treat the generic error as a pointer. If the REST log 404s, read the phase breakdown off the UI DOM.
Watch progress move
Track the failing stage, not the generic string — it never changes even as you clear each cause.
Alarm on the outcome
Build one cause-agnostic watchdog: has the live build fallen behind the latest build-relevant commit?
Add tripwires for hard cliffs
Only then add cause-specific checks for known limits, like a fixed environment-size cap.
The Fix Wasn't Three Checks — It Was One Alarm
The tempting response to a three-cause outage is three new checks: a freshness-gate assertion, a build-ignore watcher, an env-size guard. We shipped the env-size guard, because a hard 4KB cliff deserves a specific tripwire. But three cause-specific checks would have been the wrong primary lesson, and here is why: they only catch the three things that already happened. The next freeze will have a fourth cause you didn't write a check for, and your three green checks will sit there reassuring you while production ships nothing.
The failure we actually cared about was not "the freshness gate livelocked" or "an env var got too big." It was the outcome: production stopped publishing changes and nobody knew. That outcome is identical across all three causes — and across the fourth one we haven't met yet. So the alarm belongs on the outcome, not the cause.
What we built is a watchdog that asks one cause-agnostic question on a schedule: has the live production build fallen behind the latest build-relevant commit on main, by more than a threshold? It doesn't care why. Livelock, env cap, a rollback that turned off auto-publish, a platform incident, a fourth thing — every one of them produces the same symptom the watchdog is watching for, which is the live deploy's commit drifting further and further behind main. One alarm, whole class of failure, including the failures we can't yet name. A future freeze now surfaces in under an hour instead of after thirty-two.
If you take one idea from this: alarm on the outcome you care about, not on the causes you can currently imagine. Cause-specific checks are a supplement for known cliffs. They are a bad primary defence, because the whole problem with this outage was a cause we hadn't imagined wearing the disguise of a cause we had.
Two refinements keep an outcome alarm from crying wolf
Apply this to your own pipeline
- When an error is generic, go to where the real detail lives — even a UI DOM
- Alarm on the outcome you care about, not the causes you can currently imagine
- Make the alarm cause-agnostic so it catches failure modes you have not met
- Filter the alarm by churn: ignore generated, machine-authored files
- Decay it by age: a small lag is latency, a growing lag is an incident
How to Alarm on the Outcome Without Crying Wolf
"Alarm when the live deploy falls behind main" sounds simple until you build it, and then it tries to page you every five minutes. Two refinements are what make an outcome alarm survivable, and they are the difference between a watchdog you keep and one you mute.
Filter by churn — only count changes that matter. The exact file that caused the whole livelock was a generated artifact that rewrites itself every few minutes. If your "is production behind?" check counts every commit, that self-churning file guarantees production is always technically "behind," and your alarm screams constantly and gets ignored. The freshness comparison has to run against build-relevant paths only — the human-authored sources a deploy actually depends on — and ignore generated, derived, machine-authored churn. Notice the symmetry: the same file that should never have gated a deploy should also never trigger the alarm about deploys. Generated evidence has no business in either decision.
Decay by age — let small lags breathe, escalate real ones. A production deploy that is two minutes behind main during an active build is normal and healthy. A deploy that is two hours behind is an incident. So the alarm's severity is a function of age, not mere presence: a fresh gap is fine, a gap that keeps growing past a threshold escalates. Presence-based alerting ("live SHA ≠ main SHA") is technically true almost all the time and therefore worthless. Age-decayed alerting stays quiet through normal build latency and gets loud exactly when a lag stops being latency and starts being a freeze.
This is the upgrade to a piece of advice we've given before. We wrote once that a "ready" build is not a live site, so you should curl the live URL and grep for a fresh string. That presence check is the right instinct and the right level-one defence. This outage is what you need when even that isn't enough: when the failure is generic, the log is a 404, and a naive presence check would have cried wolf on generated churn all day long until you turned it off — right before the freeze it was supposed to catch.
Keep Exploring
Silent-failure debugging is a genre we write a lot about.
A generic error is not a small problem. It is a coverage gap wearing the costume of a single failure — and the fix is never three more costumes. It is one honest alarm on the outcome that matters.
Want a deploy pipeline that fails loud?
We build and run build-and-deploy setups that alarm the moment production stops publishing — and the monitoring to prove it.
See our web development serviceWhat To Do On Your Own Pipeline
You don't need our stack to use any of this. Strip it to the parts that transfer:
When a build or deploy error is generic — an opaque exit code, a bare "failed," a status that's the same across unrelated inputs — stop treating the generic surface as the answer. Find where the real detail lives. If the tidy API endpoint 404s, the truth is often one layer over in a UI you have to read by hand. The generic string is a pointer to a log, not a description of a bug.
When you sit down to prevent a recurrence, resist the urge to write one check per thing that broke. Ask instead: what is the outcome I actually care about? For a deploy pipeline it is almost always "did the thing I shipped reach real users?" Build the alarm on that, make it cause-agnostic, and it will catch the failure modes you haven't imagined yet — which are the ones that hurt. Then, and only then, add specific tripwires for the hard cliffs you know about, like a fixed environment-size cap.
And when you build that outcome alarm, filter it by churn and decay it by age, or it will cry wolf until someone mutes it. An alarm that pages you constantly is worse than no alarm, because it teaches your team to ignore the exact signal you built it to send.
Thirty-two hours of silent downtime taught us less about three specific bugs than about the shape of the thing that hid them. A generic error is not a small problem. It is a coverage gap wearing the costume of a single failure — and the fix is never three more costumes. It is one honest alarm on the outcome that matters.
This is the kind of quietly-catastrophic failure mode we design against when we build and run deploy pipelines for clients. If you want a build-and-deploy setup that fails loud instead of failing silent — and the monitoring to prove it — see our website design and development service, or talk to us about platform integrations and troubleshooting. For the level-one version of this lesson, start with why a "ready" build is not a live site, and for another silent-failure teardown, how we diagnosed failing Stripe webhooks.
Ship That Fails Loud
Deploy pipelines that alarm the moment production stops publishing.
Get the next deployment war story
Honest, first-hand teardowns of the silent failures that hide behind a green build or a generic error.
Choose your interests:
No spam, unsubscribe anytime. We respect your privacy.
Key Takeaway
- 1
A generic build error is not one bug — it is a coverage gap that can hide several
- 2
Our outage was three stacked causes, each masking the next behind the same exit code 2
- 3
When the API is generic and the REST log 404s, read the real detail off the UI DOM
- 4
Alarm on the outcome — did production publish a real change? — not on causes you can name
- 5
Filter that alarm by churn and decay it by age, or it cries wolf until someone mutes it
Social Media Carousel
7 cards • Download as ZIP (images) or PDF (LinkedIn)
One Error, Three Bugs
A 32-hour silent freeze
Production shipped nothing for a day and a half. The platform said the same thing 25 times: exit code 2.
Three Bugs, One Message
- 1
A freshness gate livelocked on a churning file
- 2
A cost guard skipped 'no content change'
- 3
A 4KB env cap broke the deploy step
- 4
Each one hid the next behind exit code 2
Twenty-five failed deploys, one generic error, zero alerts. A human noticed — no monitor did.
Cause vs Outcome
One check per failure mode
One alarm on the outcome that matters
Read the Real Log
When the API error is generic and the REST log 404s, the truth is in the UI DOM. The generic string is a pointer, not an answer.
Alarm on the Outcome
Watch 'did production publish a real change?' — it catches the failure modes you have not imagined yet.
Ship That Fails Loud
Deploy pipelines that alarm the moment production stops publishing.
Share This Article
Spread the knowledge