One of my favorite interview questions is:
Tell me about the last time something broke in production.
I ask because production is where software stops being theoretical, not because I want to punish people for making mistakes.
Local development is controlled. Tests are selective. Staging is a model. Production has real users, real data, real concurrency, real integrations, real permissions, real traffic, real money, and real organizational pressure.
If someone has worked on enough real systems, eventually something goes wrong. Google's SRE book treats incidents and postmortems as normal operating material, arguing that a blameless postmortem culture turns failure into system learning instead of theater.
The interesting question is whether they know what to do when the system proves them wrong.
The Probability Of Never Shipping A Bug
Nobody has a universal statistic for "software engineers who have never shipped a bug." The honest way to think about it is probabilistic.
Assume:
- an engineer ships (n) meaningful changes
- each change has probability (p) of causing a production bug
- the risks are independent enough for a rough model
The probability of shipping zero production bugs is:
$$ P(\text{zero bugs}) = (1 - p)^n $$
So the probability of shipping at least one production bug is:
$$ P(\text{at least one bug}) = 1 - (1 - p)^n $$
Now use an illustrative assumption: only 1% of meaningful changes cause a production bug. I chose that number for a deliberately simple model of repeated exposure; it is not an observed industry rate.
For 500 meaningful shipped changes:
$$ P(\text{zero bugs}) = 0.99^{500} $$
$$ P(\text{zero bugs}) \approx 0.00657 $$
That is about:
$$ 0.657% $$
Under those assumptions, the chance of making 500 meaningful production changes without ever causing a production bug is less than 1%.
Put differently:
$$ P(\text{at least one bug}) = 1 - 0.99^{500} \approx 99.343% $$
The exact numbers are invented for the example, so they cannot establish how often engineers really cause incidents. The useful conclusion is narrower: after enough production exposure, a claim that nothing ever went wrong deserves a follow-up. The candidate may define "broke" narrowly, may have worked behind strong release controls, or may have contributed without owning deployment.
This Is Signal
There is a bad version of this question:
Have you suffered enough to be allowed in?
That version is useless. A better version is:
What did production teach you that tests, code review, and local development did not?
A production incident provides signal about how someone behaves when the abstraction leaks. It shows whether they can stay calm, gather evidence, communicate clearly, reduce blast radius, make reversible changes, and learn without hiding.
Those are real engineering skills. They are also recognizably operational skills: Google's SRE guidance on monitoring describes production systems in terms of symptoms, causes, alerts, and user-visible behavior, which is exactly the world an incident story has to navigate.
A Prompt, Not A Verdict
I have used this question in real interviews.
If someone says they have never broken production, I do not treat the sentence itself as a fail. I ask what they have operated, how releases worked, what incidents they helped investigate, and what near misses changed their practice.
The useful signal comes from proximity to consequences and the ability to learn from them rather than guilt.
For a junior engineer, a developer without deployment access, or someone working in a tightly controlled environment, an equivalent story may be a failed test rollout, a support escalation, a defect caught before release, or an incident they helped diagnose rather than caused.
Compared with the filters employers actually use, this one is defensible. Companies routinely make snap judgments from weaker proxies:
- whether the resume has the exact keywords
- whether the previous title matches the new title
- whether the company names are recognizable
- whether the degree came from the right school
- whether the employment timeline looks conventional
- whether the email domain looks "professional"
- whether the resume format feels familiar
Some filters are worse than weak. They use protected characteristics instead of evidence about the work.
Canadian employment-discrimination law is divided by jurisdiction. The Canadian Human Rights Commission handles only federal matters, while provincial and territorial bodies cover most employers and may define grounds, exceptions, and procedures differently. In the United States, the EEOC describes sex discrimination in hiring as a prohibited employment practice.
That is another reason to prefer questions connected to the work. Production incidents are part of software engineering, and how someone talks about them tells me more than a polished resume, a brand-name employer, a school, an email domain, or a rehearsed answer about strengths and weaknesses.
Until someone finds a better filter that is available in a real interview, under real time constraints and with the information employers really use, I will keep asking.
Why Production Is Different
Some bugs only appear in production because its scale, traffic, data, dependencies, timing, and failure modes make it materially different from pre-production environments.
Common production-only causes include:
- Scale: data volume or traffic reveals performance issues
- Environment differences: configuration, secrets, permissions, or infrastructure differs from staging
- Concurrency: race conditions appear only under real load
- Real data: user input is messier than test fixtures
- External dependencies: vendors, networks, queues, and background jobs introduce timing failures
- Operational pressure: fixes happen while people are waiting
None of those are exotic. They are the ordinary texture of operating software. Stripe's public API docs, for example, devote whole sections to idempotent requests because real networks, retries, and failures make "just send the request" an unsafe abstraction.
This is why the question is useful in interviews. A person can memorize patterns, pass coding exercises, and explain best practices. Production asks whether they can operate inside a system that is already moving.
What A Strong Answer Sounds Like
Strong answers are specific.
They usually include:
- what changed
- what broke
- how the issue was detected
- who was affected
- what was done to mitigate it
- how the root cause was found
- what changed afterward
The best answers are boring in the right way rather than heroic. The engineer noticed evidence, narrowed the problem, communicated impact, chose a reversible mitigation, preserved information for root-cause analysis, and turned the incident into better tests, monitoring, design, or process. That is what maturity looks like.
A Useful Incident Story
One production issue I worked through involved a billing-like calculation system for field equipment with intermittent connectivity.
The system was eventually consistent and correct for the overwhelming majority of cases, but a subtle edge case appeared when an initial plan and a later revision interacted in an unexpected way.
The result was that a minimum usage charge could be calculated incorrectly. Most discrepancies were small, but some were large enough to require manual review.
The technical fix mattered, but the broader lesson mattered more: tests covered the expected paths but missed the lifecycle interaction that caused the issue. The follow-up improved the test cases around revisions, made the calculation easier to reason about, and added checks that surfaced unusual billing outcomes earlier.
That is the kind of story I want to hear in interviews: ownership, analysis, and improvement instead of a claim of perfection.
What The Question Actually Tests
"Tell me about the last time something broke in production" is not a confession prompt.
It tests:
- Ownership: Does the candidate take responsibility without becoming defensive?
- Judgment: Can they explain tradeoffs under pressure?
- Systems thinking: Do they understand why the issue happened beyond the immediate symptom?
- Communication: Did they keep teammates, users, or stakeholders informed?
- Learning: Did the incident lead to better tests, monitoring, process, or design?
- Humility: Do they understand that real systems fail in ways no one fully predicts?
The red flag is the absence of ownership. Lack of access says little by itself; the concern is claimed ownership without a failure, a near miss, or anything the system taught you.
Closing Thought
Breaking production confers no badge of honor; learning from the failure is what matters. The strongest engineers can explain what happened, how they responded, and how the system became better afterward instead of claiming that nothing ever goes wrong.
Production is unfair but honest, and eventually it grades the assumptions.