Surviving a Panic Situation: Structuring Your Response in the Situation Room
How to structure the Situation Room to be effective in the worst situation
Shit happens.
This is the mantra we should keep repeating ourselves. It doesn’t matter how good you are; you will make mistakes. We are humans, and we are not perfect. A great process should protect our product from human errors, but that will never be enough. There will always be unpredictable situations that lead to disaster.
When I say disaster, I mean the worst-case scenario.
Customers cannot log in, data is lost, the application crashes, and so on. These are situations that require special treatment. It’s not a simple bug; it’s a situation that needs to be solved immediately.
These are the situations requiring a Situation Room.
The Situation Room
The Situation Room is where people gather to solve the highest priority issue.
It can be a physical or virtual space, depending on whether people work remotely or in an office. It can be created on demand or as a dedicated space ready at any time. The only important thing to remember is that all the Situation Room should be aware of being in a Situation Room.
Make it clear that you are in an emergency and you are in the Situation Room.
The Situation Room requires dedicated rules, so everyone must know that the usual protocols are no longer valid. It doesn’t matter if you are the CTO, the CEO, or the newly hired Junior Engineer; the priority is solving the issue. It is essential to clarify that the Situation Room is not a Panic Room, so it doesn’t mean “no rules.”
The Situation Room requires special rules.
Structure the Situation Room
The company’s structure is not efficient for the Situation Room.
You don’t want all the bureaucracy you might have in everyday work getting in the way. If your users cannot log into the application, manually testing a proposed fix doesn’t make sense. What’s worse than users who can’t enter the platform?
Three aspects must be considered to structure the Situation Room: command, communication, and operation.
For this reason, having roles that reflect these three aspects is crucial. While these roles remain the same in every Situation Room, people assigned to each role may differ. This depends on the situation itself.
I got inspired by this post from PagerDuty, but I simplified the organization a bit. The core remains the same.
This is what the Situation Room looks like.
Commander 👨✈️
There is no room for democracy in the Situation Room.
You don’t have time to vote for decisions or solutions. The best way to handle an emergency is to have a Commander responsible for coordinating the work. The Commander is the coordinator and the reference for everyone in the Situation Room.
The duties of the Commander are:
Gathering information from the Experts.
Coordinating communication.
Create a plan of action.
The Commander is the person everyone looks at in the Situation Room.
Who can be the Commander?
Anyone can be the Commander.
While there are no strict rules to decide who should be the Commander, it is important to note that there are some considerations to make. The Commander should have a good knowledge of every part of the system because they should be the ones linking the dots. A lot of information arrives in the Situation Room. The Commander should be able to filter irrelevant data and focus the work on the right path.
The Commander might choose multiple paths to explore and assign one path to each team of Experts.
Reporter 🧑💻
The Reporter is the story-writer of the Situation Room.
They must write a detailed report of everything happening in the Situation Room. The Reporter is mainly a silent role. They don’t have to speak; their goal is to observe. They must keep track of every decision the Commander took and every solution explored.
During the Situation Room, the Reporter must raise a hand whenever there is confusion.
It may happen that, during the emergency, the Commander and the Experts might get into confusion. That’s when the Reporter comes into place. If the Commander proposes a solution already explored, the Reporter must raise a hand and stop them. If the Commander is present in the Situation Room, the Reporter is the history.
The Reporter also plays a crucial role after the Situation Room is closed.
When a Situation Room is closed, it is essential to analyze what happened and make a retrospective. The goal of the retrospective is not only to avoid other similar emergencies but also to improve the Situation Room’s process. The report is a crucial part of such a retrospective.
I suggest the Reporter take some time alone after the Situation Room is closed to redact and clarify the report.
Who can be the Reporter?
Anyone can be the Reporter.
The Reporter is meant to be with the Commander all the time. They must write down what happens, so no system knowledge is required. While I don’t recommend the newly hired for the Commander’s role, the Reporter is the perfect role.
Being new to the situation is the perfect way to don’t miss anything and don’t have biases.
Communicator 🗣️
During an emergency, communication is essential.
Two types of communication need to be done:
Internal communication.
External communication.
Internal communication is for the company.
The entire company needs to be aware of the emergency state. It is crucial to communicate to the company that the issue has been acknowledged and the team is working on it. The communication needs to be updated as soon as new relevant information is discovered. What is the path leading the user to the issue? Is there a workaround? Is the fix ready for the release? Is the problem specific to some specific users?
External communication is for external users.
If you have a status page, it might need to be updated. What is the relevant information for the users? Is there a feature that is leading to data loss? They should not use it. It is possible that the communication to the users is not done directly by the Communicator. In this case, it is still relevant that the coordination and the information passed to the users are done by someone inside the action and with the proper context.
The Communicator connects the outside world with the Situation Room status.
Who is the Communicator?
Anyone can be the Communicator.
My only advice is to have someone with enough knowledge of the users and their paths, but most importantly, of the company’s structure. Having enough experience in the company to understand what is and is not relevant for the users is the only requirement for this role. Preparing a clear communication containing only the relevant insights is crucial.
It is perfectly fine to have more than one person in this role and to split internal and external communication.
Experts 🧑🏫
The Experts are the ones with the knowledge or are responsible for gathering the knowledge.
There are multiple Experts because having all the knowledge in a single person is improbable. The Experts must provide the Commander with data and execute the Commander’s plan. The Commander is responsible for the big picture of the situation.
The Experts need to explore a single path and report to the Commander.
Who are the Experts?
All the people in the Situation Room, excluding the Commander, the Reporter, and the Communicator, must be Experts or leave the Situation Room.
The Situation Room is a place for solving a problem. Leave it if you think you cannot provide value to the Situation Room. If you are unsure about the value you can provide, being in the Situation Room ready to help is perfectly fine.
But be quiet until you don’t feel you can provide value.
The Experts are in the Situation Room to answer questions and to execute. They can proactively propose solutions, but it is better to avoid discussions in the Situation Room. Listen before speaking.
All the people in the Situation Room must provide value or leave.