Crisis management for software developers

Crisis management for software developers


Everyone makes mistakes, even software developers and yes that includes senior software developers. So, what should we do when we introduce a bug? “Fix it” is the most obvious (and correct) answer. Sometimes fixing a bug is not enough, though. Clients might lose confidence if a bug is severe. In this situation, good communication with the client is at least as important as fixing our product.

What do politicians, sports stars, or celebrities do when they get involved in a possibly career-ending scandal (like a drunk-driving accident)? Pretend that nothing happened? Flee to a different country? Accuse everyone of defamation? Hmmm, yes these are options. But if they are guilty as charged and want to keep their careers, they get out in front of the problem and call a press conference. They control the narrative, they tell their side of the story first and tell everyone what they will do to address the problem.

This is precisely what we software developers should do. While we do not as a rule hold press conferences (if you have, please let us know in the comments), the idea is the same. We want to assure our clients we are on top of the problem, and that we will to fix it. We can write an email, have a Skype call, hold a direct meeting etc. Email may be the best option because we can be careful to get the message just right before sending it. A good email will answer the following questions:

  • What happened?
  • Why did it happen?
  • How will we fix it?
  • How will we prevent a similar problem in the future?

It might surprise you to find, after spending hours desperately trying to fix a tremendous bug, that your client was even not aware of the problem. This is especially likely if the client is in a different time zone. This is good news! It is better if the client hears it from you first, rather than from an angry user who could not log into their app. But whether they already know or not, you should be honest about exactly what happened. Let them know how many users the bug affected. You should be able to derive this information from analysing codebase and third-party tools like Fabric. Provide useful information for a client: the difference between ‘the app crashes for users in Pacific time zone who have signed in the past week and haven’t bought anything yet,’ and ‘the app crashes for all users.’

The email should also explain why the problem happened. How deeply you should go into the technical details will depend on how much the client understands, or cares about, the technology. You should let your day-to-day communications with the client guide you. What is most important is for the client to understand the practical effect of the problem.

The most important detail is that you are both competent and confident to fix the problem. The precise steps you will take to do so are not important, and you need not waste the client’s time with that information unless the client wants to know.

You should also remember that every bad situation can be a teachable moment. You can learn how to avoid the same mistake in the future. A serious bug usually means that something was wrong with the process of creating and delivering the app. Do not just fix the bug, figure out why it happened in the first place. Figure out whether you need to add more steps to your process, or if existing protocols were ignored or missing. Be precise: “we need to add a specific test case to the regression tests” or “we need to create test and production builds from the same version of IDE” is far more useful than “we need to be more careful next time”.

Finally, one last piece of critical advice. Take responsibility for any bugs. Do not blame someone else to your client, even if someone else deserves part of the blame. When something bad happens, there is probably blame to spread around. Maybe testers were not careful enough, or clients’ requirements were contradictory. Perhaps there was a bug in an external library. Maybe Apple introduced a weird policy, or users used your app in an unexpected way. If your application has a bug, however it got there, own it. Your client will have more confidence in your work going forward if you say “the buck stops here.”

I hope that you will find this article very useful, but not very often.

Leave your comment