IT outage tabletop exercise: a practical two-hour script for a disaster recovery tabletop

IT outage tabletop exercise: what two hours at a table can show

A tabletop exercise for an IT outage brings the people named in your disaster recovery plan to one table for about two hours and takes them through a scenario without touching a system. NIST SP 800-84, the "Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities", describes the form: a facilitator presents the scenario and asks questions, a data collector records the discussion, and an after action report states what has to change in the plan. This article is a script you can run as it stands: roles, preparation, one scenario in six timed stages with the questions for each, and the way findings reach the runbook.

NIST's wording is short. "Tabletop exercises are discussion-based exercises where personnel meet in a classroom setting", and such an exercise "is discussion-based only and does not involve deploying equipment or other resources." The guide dates from September 2006 and is still listed as final. For the duration it gives "typically two to eight hours", and two to four hours for senior-level teams. Two hours is therefore the short form.

A tabletop exercises a plan; it does not test a system. NIST keeps the two words apart: the term "test" "is reserved for testing systems or system components". After two hours you know whether people can be reached, who decides, which steps exist only in someone's head and which assumptions about backups nobody has checked. Whether a replica boots is a different question, and our guide to disaster recovery testing covers the tests that answer it.

Roles in the room: facilitator, data collector, participants

NIST names two staff roles for a tabletop: "an exercise facilitator, who leads the discussion among the exercise participants, and a data collector, who records information about the actions that occur during the exercise." It gives no head count for either. The design team that prepares the exercise can be a single person, and one of its members looks after the logistics.

NIST SP 800-84 (September 2006), sections 2 and 4; rows marked "ours" are our suggestion for this script.

NIST advises that "senior-level teams and operational-level teams should participate in separate tabletop exercises initially". The script below is written for the operational level with one decision-maker present. Decisions that belong to the management board are recorded as open points and taken to a shorter session of their own. The facilitator should not be the author of the plan, and the data collector should not also be a participant: someone who is answering questions stops writing.

Preparation: scope, objectives and four documents

NIST divides the work into design, development, conduct and evaluation. In the design phase the team determines the topics and the scope, identifies the objectives, the participants and the exercise staff, and coordinates the logistics. For exercises that are not large and complex, NIST suggests starting "at least one month in advance".

Four documents come out of the development phase, in NIST's list:

• a briefing for the participants with an agenda and logistics information;
• a facilitator guide with purpose, scope, objectives, the scenario, a list of questions and a copy of the plan being exercised;
• a participant guide with the same content "without the list of questions";
• the after action report, written afterwards from evaluation criteria fixed in advance.

The last point is easy to skip. NIST wants the criteria "developed before the exercise to ensure data collectors know what type of information to capture". For this script we suggest three: every decision has a named owner; every contact is reached through a channel that works without the failed system; every step refers to a runbook section, or is recorded as missing.

Objectives should be few and checkable. Ours for the scenario below: the plan can be activated and people reached; the team agrees on where and in which order systems are restored; the age of the newest usable restore point is known and compared with the recovery point objective in the plan; one named person informs the business. If the plan has no recovery objectives per system yet, set them first; our guide to RPO and RTO shows how.

Our disaster recovery service includes a free screening, a high-level review of your current backup and DR posture; send us the systems your plan covers and the date it was last exercised, and our team will help.

Disaster recovery tabletop scenario: a storage failure at month-end in six injects

NIST's tabletop section does not describe timed injects. It has one scenario narrative and a list of questions, and says that the facilitator "may inject periodic questions from the facilitator guide". The inject as a noun, "a prescripted message that will be provided to participants during the course of an exercise", and the Master Scenario Events List belong to NIST's functional exercises. We borrow the idea for the table because a clock keeps a two-hour session moving. The table is our format, and the scenario is an example to adapt.

our script and example scenario; exercise form, roles and hotwash after NIST SP 800-84 (September 2006), section 4; the term inject after its section 5.

The facilitator releases each inject at its minute, even when the discussion of the previous one is unfinished; what stays open is a finding. These are the questions for each stage. Those in quotation marks are sample questions from Appendix A of NIST SP 800-84, the others are ours.

Minute 10, the failure. "Who has authority to activate the [insert plan name]?" "How would you be notified of plan activation and by whom?" Where is the contact list if mail and the wiki ran on the failed array? Who opens the case with the storage vendor, and under which contract?

Minute 25, no repair time. At which point does the plan say to stop waiting, and who decides if the person named is on leave? Which systems come back first, and does everyone name the same order? What the vendor's contract promises matters here; what "next business day" means in server makers' terms is in our article on on-site service for GPU servers.

Minute 40, the old backup. Which recovery point objective does the plan state for the ERP, and who compared it with the job results last week? Can the transactions since Thursday be entered again, from which source, and who owns that work? Are there other copies, such as a replica or database log backups? Our article on why backup is not disaster recovery explains why a completed job says little about recovery time.

Minute 55, sign-in and target. "Are IT recovery procedures fully documented? Are they accurate?" Which accounts for the backup server, the hypervisor and the storage work without the production directory, and who can reach them now? Where is the restore target, and has that amount of data ever been restored there? Which services must run before the database starts?

Minute 70, the business asks. "Can recovery procedures be completed within the timeframe dictated in the [insert plan name]?" Who gives the business a time, and on what basis? What does the service desk say, and who approved the text? Does a contract or a regulation require a notification, and who checks?

Minute 85, back to normal. Who signs off that the restored ERP is correct, and with which test transaction? How does the system return to the primary array, and is that step written down? Is the restored system itself backed up from its first hour? The sequence is described in our guide to failover and failback steps.

Ransomware tabletop exercise scenario: what changes in the script

The same clock works for a security incident, with three injects replaced. At minute 10, VMs are encrypted and the question becomes who may isolate systems. At minute 40, the team has to say which restore point predates the intrusion and who decides that. At minute 70, reporting duties enter, which our article on NIS2 incident reporting covers. A table of injects for this case is in our article on the incident response plan, and the order of work is in ransomware recovery steps.

Ready-made material exists. CISA publishes Tabletop Exercise Packages, by its own count "over 100", each "customizable" and holding template objectives, scenarios and discussion questions, documents on the roles of planners, facilitators and participants, and an After Action Report template; the cybersecurity scenarios include ransomware. ISO 22398:2013, "Societal security: Guidelines for exercises", confirmed by ISO in 2022, gives general guidance on planning, conducting and improving exercises. For German-language documents, the BSI lists a template named "Übungskonzept" among the aids to its BSI-Standard 200-4 on business continuity management; we did not read its content.

Hotwash and after action report: from findings to the runbook

NIST places the debrief directly after the discussion: "Immediately following the facilitated discussion, the facilitator and data collector should conduct an exercise debrief, often referred to as a hotwash." Its sample questions are "What are the strengths of the contingency plan? What areas require closer examination?" and "How can we improve future exercises and tests?" Comments from the debrief and the data collector's notes go into the after action report, whose sample in Appendix A has an introduction, the objectives, the agenda and the discussion findings, general and specific.

A finding is finished when it has an owner, a date and a place in a document. For a recovery exercise that place is usually a section of a runbook. The section names below follow the template in our guide to the disaster recovery runbook.

our examples from the scenario above; section names from our runbook template.

If the exercise shows that tiers and recovery objectives were never agreed, start there. In our disaster recovery service we define with you the critical systems, the target RPO and RTO for each tier and the disaster scenarios you protect against; describe the systems that came up in your exercise and we reply within one business day.

How often to run a tabletop, and what DORA and NIS2 say about testing

NIST sets no interval: "Organizations should conduct tabletop exercises periodically, following organizational changes, updates to an IT plan, or the issuance of new TT&E guidance; or as otherwise needed." NIST SP 800-34 Rev. 1 treats a tabletop as sufficient for low-impact systems and asks for a functional exercise for moderate-impact ones. Our guide to disaster recovery testing suggests a tabletop twice a year and after plan changes.

For financial entities, DORA requires tests of the ICT business continuity plans and the ICT response and recovery plans at least yearly; the provisions are quoted in our article on DORA resilience testing and backup. NIS2 lists business continuity and crisis management among its minimum measures, and its implementing regulation speaks, for the entity types it covers, of plans that are tested, reviewed and updated; see our article on NIS2 backup and business continuity. Neither text names tabletop exercises in the passages we quote there. This is general information: which duties apply to your company is for your legal department to assess, and national transposition may differ.

After the tabletop: a failover test shows whether systems come back

The script leaves one thing unproven on purpose. Nobody restored anything, so the times people named are estimates. The next step is a test in which systems start: a restore of one tier, or a failover in an isolated network. An independent IT infrastructure audit looks at backups and restore tests as one of its domains, and for VMware estates our article on backup for VMware after Broadcom lists what to re-check after an upgrade.

We run recovery sites in Tier III data centres in the EU with scheduled failover tests in an isolated environment and a report after each test: what came up, how fast, what to fix. Tell us which systems you would test first and how you restore them today.

FAQ