Speaking: where Mike has presented the paper

Question: Not a question so much as an observation. NASA had some of the best scientists and decision makers, and reviews on these programs. Do you think they missed all this, or is it the culture that set them up to do that?

Mike Meier: I think that’s a very complex question, and I think they attempted to answer it in the later accident reports. It’s interesting: if you read the Apollo 204 Review Board report, it is almost entirely about the technical issues that led to the fire. I quoted some of Frank Borman’s comments that relate to the decision making, because that’s the point of my paper, but the report itself doesn’t really have any of that in it. It goes on and on about the technical issues. By the time you get to Challenger, they start looking, as Richard Feynman observed, at the decision making. And by the time you get to Columbia, there is a real frustration.

I believe it was Sally Ride who was on both the Challenger and the Columbia review boards. There is a whole section in the Columbia accident report where they talk about echoes of Challenger, and they even go back and talk about Apollo 1. Those accidents were on average 18 or 19 years apart, and they all occurred in January or February. We might conclude: just don’t launch anything in January or February, 18 years after the last one. That could be the conclusion.

But I think that, yes, they missed it. They missed it because I think two things are true about this process. One is that we’re at a disadvantage because we underestimate the challenge we’re up against. We underestimate the power of Murphy’s Corollary and our susceptibility to it, and we underestimate the power of the pressure to go and our susceptibility to that. And of those two, just observing my own decision making, which is far from what I would even call good, the pressure to go is, for me, the toughest one.

Audience member: In my own estimation, I like the corollary: the turtle only makes progress when his neck is stuck out.

Question: In the Department of Defense, certainly right now, there’s a lot of desire for innovation in a fiscally constrained environment, and with that, some concern about a paralysis of paranoia. [inaudible] If we could all become experts in stochastic calculus and had a perfect, quantitative view of the risks, [inaudible] As you did the research and reflected on this, what is enough in a risk assessment? Ten to the minus eight, ten to the minus nine? At what point, if we had this quantitative analysis, would we say that’s a low enough risk, and I’m willing to take it for the potential utility of a glider program, or a defense program, moving forward?

Mike Meier: I’m going to extrapolate your question a little, because both of these questions have touched on, and I’m going to be up front about this, a fatal flaw in my entire thesis. I’ll tell you what it is, if you haven’t seen it.

I’ve shown that 99 percent is not even close to good enough. So we know we have to go at least to 99.9, and then it’s 180 events to get to Russian roulette. Maybe we have to go to four nines, maybe five nines; I don’t know about ten to the minus eight. But the problem is that we’re making this evaluation before the fact. How confident are we that we can tell the difference between 0.9999 and 100 percent? And if we can’t, and the requirement for safety therefore becomes 100 percent, then haven’t we just annihilated the entire field of experimental flight test? That’s where my whole talk breaks down.

I have two possible answers to that. I could be flip and just say, don’t extrapolate it that far. But my real answer is that I think three things are true. I’ve already talked about two of them, but I’ll mention them quickly once again, because I know we’re up against lunch.

The first, I don’t think anybody here will argue with. It’s very simple: we can improve. The existence of this Society, and the presence of everybody in this room, testifies to our belief that we can improve.

The other two come specifically out of my paper, and again I’m repeating myself from the last question. We may underestimate the challenge we face in trying to make that improvement; we may not recognize the full scope of what we’re up against. And we may not be using all the tools that are available to us. Specifically, the one tool I think we may not be using sufficiently, and it’s not that we’re not using it at all, is a critical analysis of successful outcomes, to try to identify those areas where our success maybe wasn’t based on a foundation of sound decision making.

That’s the best answer I have. I don’t think it’s entirely satisfactory. It may not be satisfactory at all, but that’s the best I’ve got. Thank you.