The paper: Lessons Learned and Murphy’s Corollary

Abstract #

Safety in flight test requires that when the potential consequences of a negative outcome are catastrophic, the go decision must be made only when a very high degree of confidence in a positive outcome exists. When this protocol is followed, it is a natural result that a very high percentage of outcomes will be positive, and it is further likely that we will see a long string of consecutive positive outcomes. When this happens a feedback loop is created in which the go decision is validated by the positive outcome as having been a good decision, and is therefore reinforced and made more likely to be made again in the future under similar circumstances. Beyond that, often the lesson learned from a series of consecutive positive outcomes is that our evaluative criteria for making the go decision were unnecessarily conservative; not only were we correct to make the go decision, but we could have made it under even somewhat less favorable circumstances. This can lead to a gradual lowering of our standards.

However, consecutive successful outcomes can mask go decisions that may be fundamentally unsound, if the success is the result, either partly or entirely, of good fortune. Below a certain threshold of probability for success, repeated go decisions will lead eventually to an unacceptable probability of failure. At 99% probability of success, eighteen events results in a cumulative probability of failure for at least one event of 17%, or one chance in six.

The common idiomatic statement of Murphy’s Law is: “Anything that can go wrong will go wrong.” This is likely not what Murphy said, nor is it the best statement of the lesson learned from Murphy’s experience, and in fact, no one who thinks about it can actually believe it. Almost anything could go wrong, and if it were really true that anything that could go wrong would go wrong, then most everything would go wrong, and most things don’t. But if Murphy’s Law were true, then a corollary of this “law” would also be true: “Anything that did not go wrong could not have gone wrong.” And while we don’t really believe Murphy’s Law, we are surprisingly susceptible to being seduced by an unconscious belief in Murphy’s corollary – that is, we take a positive outcome as evidence that a positive outcome was inevitable.

Combining with our failure to appreciate the nature of probability and our susceptibility to Murphy’s corollary is the inevitable and powerful pressure we often experience to make the go decision under less than favorable circumstances. This pressure is a natural and unavoidable consequence of project deadlines, budgets, and performance expectations. It induces us to make the go decision when we otherwise might not, and to validate the go decision afterwards when the outcome is successful, when that validation may be inappropriate. A means for combating the effects of these three factors is to conduct a rigorous and quantitative pre and post-test analysis not only of failed outcomes, but also of successful ones, with the goal of identifying apparently successful outcomes that may have followed bad decisions, and that were successful because of good fortune.

The paper’s conclusion #

A rigorous analysis of successful outcomes would constitute an expansion of our source material for lessons learned. If such a process allows us to uncover some decisions or procedures that may have been in some way flawed, we may expand the degree to which our lessons learned enhance flight test safety.

The paper’s acknowledgment:

The author gratefully acknowledges the assistance of Colin A. Fries and the NASA HQ History Program Office in gaining access to original source material used for this paper.

Read the paper #

• The paper as a PDF (2.2 MB, 27 pages): the 2015 paper as presented, with a cover page giving its CC BY-SA 4.0 license
• The same paper as hosted at the Flight Test Safety Committee (the original, “All Rights Reserved” edition)

The paper is the source for most of this site. Each page here notes what it is adapted from, and the paper’s notes appear beside the text.

The paper’s notes #

The paper’s 16 notes, restyled, each linked to where it appears.

1. “Murphy’s laws origin,” excerpted from The Desert Wings, March 3, 1978, Murphy’s Laws site, accessed September 5, 2015. Murphy’s Corollary
2. Nick T. Spark, “The Road to Murphy’s Law,” Improbable Research, 2003. Murphy’s Corollary
3. “Murphy’s Law,” Wikipedia, accessed September 5, 2015. Murphy’s Corollary
4. Chuck Yeager and Leo Janos, Yeager: An Autobiography (New York: Bantam Books, 1985), 185. The F-86 aileron bolt
5. Investigation Into Apollo 204 Accident: Hearings Before the Subcommittee on NASA Oversight of the Committee on Science and Astronautics, U.S. House of Representatives, Ninetieth Congress, April 10, 1967, Evening Session, history.nasa.gov, accessed September 5, 2015, http://history.nasa.gov/Apollo204/bormanhouse1.pdf, 81. Apollo 1
6. Ibid., 87. Apollo 1
7. Clinton P. Anderson et al., “Apollo 204 Accident: Report of the Committee on Aeronautical and Space Sciences, United States Senate, Report N. 956,” NASA Historical Reference Collection, NASA History Office, NASA Headquarters, Washington, DC, January 30, 1968, http://history.nasa.gov/as204_senate_956.pdf, 9, 10. Apollo 1
8. William P. Rogers et al., Report of the Presidential Commission on the Space Shuttle Challenger Accident, NASA Historical Reference Collection, NASA History Office, NASA Headquarters, Washington, DC, June 6, 1986, http://history.nasa.gov/rogersrep/genindex.htm, ch. V, “Findings” [104]. Challenger
9. Ibid., ch. VI, “Early Design.” Challenger
10. Ibid., ch. VI, “Findings,” item 3. Challenger
11. Harold W. Gehman Jr. et al., Report of Columbia Accident Investigation Board, National Aeronautics and Space Administration, August 26, 2003, http://www.nasa.gov/columbia/home/CAIB_Vol1.html, ch. 6.1. Columbia
12. Ibid. Columbia
13. Ibid., ch. 11, “Recommendations.” Columbia
14. Rogers et al., Report of the Presidential Commission, ch. IX, sec. VII, “Flight Rate.” The pressure to go · Challenger
15. Ibid., ch. V. Challenger
16. Alan Shepard and Deke Slayton, Moonshot (Atlanta: Turner Publishing, 1994), 198. The pressure to go · Apollo 1

The talk on video #

Mike also discussed the paper on On Condition, the Flight Test Safety Committee’s podcast, in May 2024: The podcast.

@inproceedings{meier2015lessons,
  author    = {Meier, Michael W.},
  title     = {Lessons Learned and Murphy's Corollary},
  booktitle = {Society of Experimental Test Pilots 59th Annual Symposium},
  address   = {Anaheim, CA},
  month     = sep,
  year      = {2015},
  url       = {https://meiersafe.com/the-paper/}
}