Briefing Kit: using these ideas with your own team

The simplest briefing: play the talk #

Play Mike’s 2015 talk at SETP (32 minutes, with captions), then discuss one case using its questions below. For a short opener, Mike Meier on safety runs four minutes.

A briefing outline #

To give the talk yourself: two versions, in the order Mike gave it at SETP in 2015, twenty minutes of talk only or thirty minutes with twelve for discussion. The first number in each row is for the 20-minute version, the second for the 30-minute one; “Talk” in parentheses is how long Mike spent on each part. Each step gives one line of Mike’s own, what to show, with the matching slides, and the page that has it.

Mike’s words in each row are quoted from the paper.

Handouts and pages that print well #

The one-page handout has the framework, the numbers and five questions on a single sheet; it is also a PDF to print or email (110 KB).

Any other page prints with your browser’s Print command: the menus and videos drop away, and notes and the web addresses of outside sources print with the text. These run a few pages each:

• The framework: the whole argument
• Deceptive mathematics, with the table of odds
• Case study · The 1995 top landing
• Case study · The 2015 certification launch

Mike’s slides #

The 35 slides Mike showed at the SETP symposium in Anaheim in 2015, to project in a briefing: the slides as a PDF (2.4 MB). The outline above gives the slides for each step. Slides 2 and 9–13 are stills from video clips that played in the talk; the clips are in the recording.

The slides carry Mike’s copyright notice and are not covered by the CC BY-SA license: show them as they are, with his name on them. Some pictures in them are other people’s: photos by Rob Kells and Dave Aldrich and pictures from Wills Wing, used with permission; NASA photos, which are U.S. government works; and Google Earth views, credited on the slides. The photo of The Thinker is Mike’s. The last page of the PDF repeats these terms.

Discussion questions, all cases #

Each case study ends with its own questions. They are collected here so a briefing can draw on several cases at once.

The 1995 top landing

1. Which routine operation in your program has a long record of success but has never been measured against a safety margin?
2. When did you last think “this may not end well” during an operation, and “what could go wrong?” after it?
3. What quantitative metric and target would tell you that a successful outcome was really luck?
4. What does the routine save you in time, money or crew, and how does that saving shape the go decision?
5. When someone else has a mishap doing what you do, how do you decide whether it was their technique or the operation itself?

The 2015 certification launch

1. When did you last go because you believed it was your only chance? Was it?
2. Who was depending on you, and would they have wanted you to go in those conditions?
3. What would a no-go have cost you that day, and what did you tell yourself it would cost?
4. What limit, set beforehand, would have made this a no-go without a decision in the moment?
5. Estimate the chance of a failed launch each time you go in conditions like these, with no headwind. Using the calculator, how likely is at least one failure if you make that choice ten times in a career? Fifty?

Apollo 1

1. Which condition in your test program has been present on every success so far, and has never been treated as a hazard?
2. What earlier incidents, in your program or in others, point at a danger you have not yet experienced yourself?
3. Who on your team would say “these test conditions are hazardous,” and when did anyone last say it?
4. Where has a long run of success in your program turned into overconfidence or complacency?
5. Estimate the chance that one of your tests ends in an accident, then use the calculator to see what that chance becomes over all the tests your program has run so far. Does your run of successes still read as evidence of safety?

Challenger

1. Which limit in your program has been widened after a run of successes, and on what evidence?
2. What damage or anomaly do you now call “acceptable” that was once unexpected?
3. If you set a quantitative target for that anomaly today, with a margin of safety, would your recent successes meet it?
4. Where does the pressure to keep flying come from in your organization, and at what levels?
5. When an engineer disagrees with a go decision, how does that disagreement reach the person making it?

Columbia

1. Which known problem in your program violates a design requirement, but has never yet caused a failure?
2. With each success, has your view of that problem moved toward “inevitable” or “acceptable”?
3. What would it take for your program to treat a long history of a problem as more warning, not less?
4. Which date in your program is fixed by a requirement rather than by readiness? How would it change your answer to a concern raised the week before?
5. The board called for a new culture at NASA. Which part of the problem would a new culture not change in your own decision making?
6. Estimate the chance that your known problem causes a loss on any one flight. Using the calculator, how likely is at least one loss over the flights you plan for the next five years?

The F-86 aileron bolt

1. Which part or step in your build, maintenance or test process can be done the wrong way and still look right?
2. Where do you rely on an instruction, rather than the design, to prevent a wrong installation?
3. How would you find out that one experienced person was doing a step their own way, because they felt they knew better?
4. Which of your designs or procedures could be changed so that the wrong way is impossible?
5. Estimate how often a step that can be done wrong is done wrong, say once in a thousand times. Using the calculator, how likely is it to happen at least once across every time that step is done in your fleet in a year?

How to credit the material #

Mike’s text, diagrams and photos are licensed CC BY-SA 4.0: you may copy, adapt and share them, including in paid courses, as long as you credit the source and share adaptations under the same license. A credit line to copy:

Lessons Learned and Murphy’s Corollary by Mike Meier, meiersafe.com, licensed CC BY-SA 4.0.

The charts on Luck and uncertainty adapt Art Tomassetti’s chart and are not covered by the license. Art agrees to their use in a briefing or course as long as each chart keeps the source note it carries, “Art Tomassetti – Flight Test Safety Fact – May 2021”; show them as they are, with credit to both Art and Mike. Photos, video and quotations from other sources are not covered either; each is credited where it appears (License).