CIO Advice

Why do IT organizations have so many service outage fire drills?

By | CIO Advice, IT Operations | No Comments

Many IT organizations spend millions of dollars and thousands of hours deploying and configuring management and monitoring tools, yet fail to fully realize the promises of outage prevention and rapid service restoration promised by the vendors. The failure to realize the promised benefits results in service outage conference calls involving large numbers of IT staff and leadership, hours of troubleshooting and finally the heroic efforts of one or a few to restore service.  This is often followed by hours or days of forensic analysis to determine what went wrong why it was not prevented or resolved more quickly.

Does this happen in your organization? Does it happen far too often?

Many reasons and maybe some reasonable excuses are typically offered up for each instance and evaluated in isolation, each seems to make sense. If this happens more than a couple of times per year in your organization, you may want to take a closer look at your organizational culture.  Many organizations have inadvertently created a culture of firefighting.  This often happens when those who have demonstrated their technical prowess by putting out the biggest fires with the greatest impact under the most pressure are the most praised and rewarded members of the team. This culture of heaping praise and rewards on the best firefighters does very little to incent fire prevention.  When was the last time your organization rewarded a team or individual for a quarter of zero service impacting failures? Has a heroic firefighter ever been admonished for not fully using the millions of dollars of tools or established processes to prevent a service impacting failure or resolve it in a much more timely manner?

Don’t get me wrong. I fully appreciate the need for IT firefighting expertise. The primary role for that expertise needs to be in fire prevention. In most organizations, there are simple things that leaders and managers can do to incent fire prevention.  Once fire prevention becomes the cultural norm and the most praised and rewarded role in the organization, tools will become more effectively configured and used and process and procedures will be more often updated and optimized for the ever changing technology environment.

People are awesome and will always perform in a manner dictated by the culture established by their leaders. If leaders heap praise and rewards for putting out fires; there will be more fires.  If leaders heap praise and rewards on fire prevention, there will be fewer fires.