One Generic Error, Three Stacked Deploy Failures

For about a day and a half, a production dashboard we run for our own studio shipped nothing. Every commit to main built, every build failed, and every failure reported the exact same thing: Failed during stage 'building site': Build script returned non-zero exit code: 2. Twenty-five deploys in a row, one error message, zero information.

Here is the part that should worry you more than the outage did: nothing told us. No alert fired. We found out because an unrelated automated job we were rolling out came back saying its code wasn't deployed yet — and that thread pulled on a production site that had been frozen for thirty-two hours without a single alarm.

When we finally pried it open, exit code 2 turned out to be not one bug but three, stacked on top of each other, each one hiding the next. This is the write-up: how a generic error lies, why fixing the first cause made things briefly worse, and the one monitoring change that would have caught the whole thing in minutes instead of a day.

The Deploy That Shipped Nothing for a Day and a Half

The failure mode was almost aggressively boring. Push a commit, watch it build, watch it fail. Push another — different commit, different files, completely unrelated change — and get the identical error. Queue updates, an uptime check, a session log, a real feature: it did not matter what the commit contained. Every one of them died at the "building site" stage with exit code 2.

That uniformity is itself a clue, though it is easy to read the wrong way. Your instinct on the first failure is "my last change broke the build." Your instinct on the twenty-fifth, spanning commits that share nothing, should be the opposite: the input doesn't matter, so the cause isn't in the input. Something about the environment or the pipeline is failing the same way regardless of what you feed it.

But we didn't get to that reasoning cleanly, because the thing that should have escalated a total production freeze simply never spoke. The site had last published successfully at 03:01 one morning. The next alarm was a human, the following afternoon, noticing something downstream looked stale. In between, the dashboard served the same day-old bytes to everyone, confidently, while two dozen builds failed in the background.

The Error Was Lying — and the Real Log Was a 404

The first real lesson landed before we fixed anything. The platform's status API — the thing our tooling reads to decide whether a deploy is healthy — reported every one of those failures as the same string. exit code 2, over and over. If you build alarms or dashboards on top of that API, you are building on a surface that has deliberately thrown away the detail.

So we went for the real log. And the real log was a 404. The REST endpoints that are supposed to return the granular build output — the per-deploy log, the per-build log — both returned "not found." The decisive artifact, the twenty lines that actually name what failed, was not available through the API at all. It existed only in the platform's web UI, rendered into the page's DOM behind a login.

That is a genuinely awkward place to be: the machine-readable surface is generic, and the human-readable surface is the source of truth. The move that unblocked us was unglamorous — open the UI, force every collapsed log section open, and read the rendered text off the page. Once we could actually see the phase-by-phase breakdown, the single exit code 2 split into three visibly different failures, on different deploys, at different stages. The API had been averaging three distinct problems down to one meaningless number.

The transferable rule: when an error is generic, treat the generic surface as a pointer, not an answer. Go to wherever the real detail lives — even if that is a DOM you have to scrape, because the tidy API endpoint gives you a 404. A monitoring stack that only ever sees the generic string will never tell you which of three things is wrong.

Three Bugs Wearing One Error Message

Here is what exit code 2 was actually hiding, in the order we peeled them.

Cause one — a freshness gate eating its own tail. We run a small deploy-freshness guard: if a newer commit to main has already touched a build-relevant path while the current build is still running, cancel this build so we don't publish something already superseded. Sensible in principle. The bug was in the path list. A generated, machine-authored file — rebuilt automatically every few minutes by a background housekeeping loop — was on the protected list. So every build went "stale" against that churning file before it could finish. A build took about three and a half minutes; the file changed faster than that. The gate cancelled every publish, forever, and reported it as a build failure. The guard meant to protect production had livelocked it.

Cause two — a cost guard doing its job. Once we stopped the livelock, some commits started reporting a different failure: "canceled build due to no content change." That one is not a bug at all — it is a build-ignore cost guard working exactly as designed. If a commit touches nothing the build actually depends on, don't spend a build on it. Correct behaviour, wrong moment: it looked like yet another failure in a sea of failures, and it briefly convinced us the fix hadn't worked. The unlock was to force one real build through a manual build hook.

Cause three — a 4KB ceiling we'd been creeping toward for weeks. With the livelock gone and a real build forced through, the build finally completed — and then deploying failed. The log: "environment variables exceed the 4KB limit." The functions behind this site run in a compatibility mode that injects every one of the site's environment variables into every serverless function, and the total is capped at 4KB by the underlying platform. We had drifted over the line during the outage window. The culprit was a single key worth about 1.6KB — roughly forty per cent of the entire budget — that a function already read from blob storage and only kept in the environment as a dead fallback. Deleting it from the environment brought us back under the cap and the site finally published.

Three causes. One error message. Each one only became visible after the one in front of it was cleared.

Why Fixing the First Bug Made It Briefly Worse

The stacking is the part worth sitting with, because it is what makes this class of bug so expensive.

When you fix cause one and the same generic error is still there — now produced by cause two — every instinct tells you your fix failed. So you second-guess the change you just made, maybe revert it, and now you are further from the answer than when you started. A stack of failures behind a single opaque code doesn't just cost you the time to fix three bugs. It costs you the time to not un-fix the ones you already solved, because the feedback signal doesn't change when you make progress.

This is why "it's still broken" is almost useless information when the error is generic. Still broken how — same stage, same message detail, same phase? We only made real progress once we were reading the phase-level log and could see the failure move — building-stage cancel, then a content-change skip, then a deploy-stage env error. The message on the generic API never changed through any of it. If we had trusted that surface, we would have concluded, correctly and uselessly, that nothing we did mattered.

The Fix Wasn't Three Checks — It Was One Alarm

The tempting response to a three-cause outage is three new checks: a freshness-gate assertion, a build-ignore watcher, an env-size guard. We shipped the env-size guard, because a hard 4KB cliff deserves a specific tripwire. But three cause-specific checks would have been the wrong primary lesson, and here is why: they only catch the three things that already happened. The next freeze will have a fourth cause you didn't write a check for, and your three green checks will sit there reassuring you while production ships nothing.

The failure we actually cared about was not "the freshness gate livelocked" or "an env var got too big." It was the outcome: production stopped publishing changes and nobody knew. That outcome is identical across all three causes — and across the fourth one we haven't met yet. So the alarm belongs on the outcome, not the cause.

What we built is a watchdog that asks one cause-agnostic question on a schedule: has the live production build fallen behind the latest build-relevant commit on main, by more than a threshold? It doesn't care why. Livelock, env cap, a rollback that turned off auto-publish, a platform incident, a fourth thing — every one of them produces the same symptom the watchdog is watching for, which is the live deploy's commit drifting further and further behind main. One alarm, whole class of failure, including the failures we can't yet name. A future freeze now surfaces in under an hour instead of after thirty-two.

If you take one idea from this: alarm on the outcome you care about, not on the causes you can currently imagine. Cause-specific checks are a supplement for known cliffs. They are a bad primary defence, because the whole problem with this outage was a cause we hadn't imagined wearing the disguise of a cause we had.

How to Alarm on the Outcome Without Crying Wolf

"Alarm when the live deploy falls behind main" sounds simple until you build it, and then it tries to page you every five minutes. Two refinements are what make an outcome alarm survivable, and they are the difference between a watchdog you keep and one you mute.

Filter by churn — only count changes that matter. The exact file that caused the whole livelock was a generated artifact that rewrites itself every few minutes. If your "is production behind?" check counts every commit, that self-churning file guarantees production is always technically "behind," and your alarm screams constantly and gets ignored. The freshness comparison has to run against build-relevant paths only — the human-authored sources a deploy actually depends on — and ignore generated, derived, machine-authored churn. Notice the symmetry: the same file that should never have gated a deploy should also never trigger the alarm about deploys. Generated evidence has no business in either decision.

Decay by age — let small lags breathe, escalate real ones. A production deploy that is two minutes behind main during an active build is normal and healthy. A deploy that is two hours behind is an incident. So the alarm's severity is a function of age, not mere presence: a fresh gap is fine, a gap that keeps growing past a threshold escalates. Presence-based alerting ("live SHA ≠ main SHA") is technically true almost all the time and therefore worthless. Age-decayed alerting stays quiet through normal build latency and gets loud exactly when a lag stops being latency and starts being a freeze.

This is the upgrade to a piece of advice we've given before. We wrote once that a "ready" build is not a live site, so you should curl the live URL and grep for a fresh string. That presence check is the right instinct and the right level-one defence. This outage is what you need when even that isn't enough: when the failure is generic, the log is a 404, and a naive presence check would have cried wolf on generated churn all day long until you turned it off — right before the freeze it was supposed to catch.

What To Do On Your Own Pipeline

You don't need our stack to use any of this. Strip it to the parts that transfer:

When a build or deploy error is generic — an opaque exit code, a bare "failed," a status that's the same across unrelated inputs — stop treating the generic surface as the answer. Find where the real detail lives. If the tidy API endpoint 404s, the truth is often one layer over in a UI you have to read by hand. The generic string is a pointer to a log, not a description of a bug.

When you sit down to prevent a recurrence, resist the urge to write one check per thing that broke. Ask instead: what is the outcome I actually care about? For a deploy pipeline it is almost always "did the thing I shipped reach real users?" Build the alarm on that, make it cause-agnostic, and it will catch the failure modes you haven't imagined yet — which are the ones that hurt. Then, and only then, add specific tripwires for the hard cliffs you know about, like a fixed environment-size cap.

And when you build that outcome alarm, filter it by churn and decay it by age, or it will cry wolf until someone mutes it. An alarm that pages you constantly is worse than no alarm, because it teaches your team to ignore the exact signal you built it to send.

Thirty-two hours of silent downtime taught us less about three specific bugs than about the shape of the thing that hid them. A generic error is not a small problem. It is a coverage gap wearing the costume of a single failure — and the fix is never three more costumes. It is one honest alarm on the outcome that matters.

This is the kind of quietly-catastrophic failure mode we design against when we build and run deploy pipelines for clients. If you want a build-and-deploy setup that fails loud instead of failing silent — and the monitoring to prove it — see our website design and development service, or talk to us about platform integrations and troubleshooting. For the level-one version of this lesson, start with why a "ready" build is not a live site, and for another silent-failure teardown, how we diagnosed failing Stripe webhooks.