Tell someone immediately, then fix it together. A quick heads up turns a mistake into a shared incident with more people helping; hiding it risks a much bigger problem and damages trust. Say what happened, the impact and what you are doing, then help write a blameless follow-up so it does not happen again.
What people tell me
I pushed a change that broke something in production. It might be a bug affecting some users, a failed payment flow, or data that was updated incorrectly. I noticed it before anyone else did. My first instinct is to quietly fix it before my manager finds out, because I am still fairly new and I am scared this will make me look careless. But I am not sure how long the fix will take or how many people are affected.
A composite of the messages behind this question, with personal details left out.
Key takeaways
- Tell your lead or manager straight away, even before you understand everything.
- Your first message should cover what broke, who is affected, and what you are doing now.
- Stop the damage first (roll back, disable the feature), then find the root cause.
- Owning a mistake quickly builds trust. Being found hiding one destroys it.
- Afterwards, help write a blameless review focused on the process, and fix the gap that let it through.
The short answer: tell someone now
I understand the instinct to fix it quietly. You are new, you want to be seen as reliable, and it feels like reporting the problem is admitting failure. But every experienced engineer has broken production, often more than once. What separates people who are trusted from people who are not is not whether they make mistakes. It is how they handle them.
Telling your manager or lead immediately is almost always right. Here is why.
Why hiding it is the riskier option
- You may not know the full impact. Something that looks like a small bug can be corrupting data or affecting payments you have not noticed yet.
- Others may need to act. Support may need to answer customers. Another team may need to pause a job. Someone may know a faster fix.
- Time matters. A problem that takes you three hours alone might take twenty minutes with help.
- It will come out. Logs, monitoring and customers reveal most incidents eventually. Being found hiding one is far worse than the mistake itself.
What to say, and how fast
Send a short message within minutes, not after you have solved it:
"Heads up: my deploy about 20 minutes ago broke checkout for some users. Error rates on the payment endpoint jumped. I am rolling it back now. I will update in 15 minutes."
That covers the three things people need: what happened, the impact as far as you know, and what you are doing. You do not need to know the root cause yet. You do not need a long apology.
If your team has an incident channel or on-call process, use it.
Stop the damage first
Your first technical priority is to limit harm, not to find the perfect fix:
- Roll back the change if you can.
- Disable the feature with a flag if rollback is not possible.
- Contain bad data, for example pause a job that is spreading it.
- Then investigate the root cause calmly, once the bleeding has stopped.
A rushed forward fix under pressure is one of the most common ways a small incident becomes a big one.
During the incident
- Post short updates at regular intervals, even if the update is "still investigating".
- Say clearly when you do not know something.
- Accept help. If a senior engineer offers to take over or pair, say yes.
- Keep notes of what you tried and when. They will be useful later.
After it is fixed
Most healthy teams run a blameless review after an incident. The point is not to find someone to punish, but to understand what in the process let the mistake reach production. Help write it honestly:
| Question | Example |
|---|---|
| What happened? | A change to tax calculation broke checkout for one currency |
| How was it detected? | I noticed errors in monitoring 15 minutes after deploy |
| What was the impact? | About 40 failed checkouts over 25 minutes |
| Why did it get through? | No test covered that currency; review missed it |
| What will we change? | Add tests, add an alert on checkout error rate |
Then do the follow-up work yourself. Writing the missing test or adding the alert turns the story from "the new person broke checkout" into "the new person found and closed a gap in our process".
How it affects how people see you
In almost every team I have worked with, the engineer who says "I broke this, here is what I am doing" gains respect. Managers remember who was honest under pressure. They also remember who went quiet. A mistake handled well can genuinely strengthen your reputation.
If you work somewhere that punishes people harshly for honest mistakes reported quickly, that tells you something important about the team, and it is worth knowing.
What to say to your manager afterwards
Once the incident is resolved, a short one-to-one conversation helps close it properly. Something like:
"I wanted to talk about yesterday's checkout issue. The cause was my change to the tax calculation, which missed one currency. I have added tests for all supported currencies and an alert on checkout errors. Is there anything else you would like me to change in how I test or deploy?"
This shows ownership, shows what you have learned, and invites guidance without over-apologising. You do not need to say sorry five times. Once, sincerely, is enough.
A self-check for your next deploy
Many production mistakes come from the same few gaps. Before shipping something risky, ask:
- Do I know how to roll this back, and how long it would take?
- Is there a test that would fail if this broke the main user flow?
- Am I deploying at a time when people are around to help?
- Will I know within minutes if it goes wrong, through an alert or dashboard I am watching?
- Could this change data in a way that is hard to undo?
If the answers are uncertain, that is the moment to ask a senior engineer for a second look. It is far cheaper than an incident. For how to ask in a way that shows judgement rather than doubt, see how to ask for help without looking incompetent.
If you are still stuck
Read What actually happens between your pull request and production and Sixteen things that break a backend, and what prevents each to understand how teams catch these problems earlier. If you want someone to look at your specific situation, join Sefism and, as a member, book a 1:1 session.
Was this answer helpful?
Read next
- What Actually Happens Between Your Pull Request and ProductionMost engineers learn CI/CD as a sequence of buttons rather than a system, which works right up until the pipeline goes red and nobody knows which stage failed or why it matters. The five stages walked through one at a time, what each is protecting you from, why builds pin their versions, the difference between rolling, blue green and canary releases, and the ten second skill of reading a failed stage.
- Sixteen Things That Break a Backend, and What Prevents EachBackend engineering is taught as frameworks and turns out to be almost entirely about what happens when the database is slow, when the same request arrives twice, or when a downstream service stops answering without saying so. None of it is deep, all of it is invisible until it happens to you at an unpleasant hour. Sixteen situations and their usual answers.
- The Habits That Make an Engineer Worth Routing Work ThroughEvery team has one or two engineers other people route work through, and what they have in common is not raw technical strength. It is that when something is given to them it comes back, and when it will not, you hear about it early. Fifteen habits that build that reputation, none of which require a title or permission.
Your situation is not quite this one?
Members get written answers to their own questions, a roadmap built for them and feedback on their projects. Early access is open to X and Instagram followers and university students. Prefer to talk? A free call works too.