Safety Critical Decision Making
From Wikipedia, the free encyclopedia
Safety Critical Decision Making (SCDM) is a process important to high consequence (astronaut safety, public safety, environmental impact, loss of one-of-a-kind resources, dollar value) enterprises such as a decisions to launch the Space Shuttle [[1]] or send a nuclear submarine to sea. The art and science of SCDM is described in this wiki with examples, references and description of processes.
The process typically requires successive reviews that roll-up to a final decision forum in which objective data and risks are assesssed using historical precedence, engineering judgement, legal considerations, and policy perspectives. The SCDM process often follows a formally document protocol that results in a record of decision. The formal process used by the NASA Chief Safety & Mission Assurance Officer and the NASA Chief Engineer is codifies in a document entitled NPR 8705.6[2]. The decision forum proceeses bring forward rationale to proceed or not from designated accountable organizations and individuals.
The decision process also typically involves examination, assessment and resolution of numerous "niche science issues" (e.g., Space Shuttle example / foam loss, foam transport, impact and damage assessment or in the case of Challenger[[3]] - the phenomena of soot on secondary O-rings (blow-by). The decision also typically involves review and discussion of remaining high consequence / high likelihood issues and the control and mitigation strategies in place to reduce risk.
The process of SCDM has been examined from multiple angles including an organizational behavior perspective provided by Diane Vaughn in "The Challenger Launch Decision" [4] and a statistical/graphical perspective (also on the Challenger in Edward Tufte's "Visual Explanations"[5].
A more recent analysis by Tufte on the Columbia disaster is provided below:is provided
-
- An excerpt from Tufte's analysis - Three reports by Boeing engineers about potential debris damage to the left wing have been posted by NASA at many locations. Here is a good general location for many items; see "Boeing Debris Impact Assessment Charts" at - [6]. - - The 3 reports have the following good features: The names of the engineers producing the reports are given (unlike the Challenger pre-launch analysis). The report makes clear quantitative links among possible causes and effects (unlike the Challenger analysis). Most of the tables and graphics have scales of measurement.The analysis is multivariate. Assumptions of the analysis are fairly clear (although perhaps there are hidden assumptions that experts can reveal). It is easy enough for the alert reader to see that the results are assumption-sensitive. Quantitative estimates by means of contour lines are given for debris-velocity over the wing surface. There is an excellent diagram showing individual tiles on the wing along with the forecasted tile loss due to debris impact. - - - The 3 reports have the following weaknesses: It now appears that the conclusions were incorrect. The results appear sensitive to input assumptions about incidence angle, incidence location, the number and velocity of impacts, and the weight of the debris (assumed to be lightweight foam at 2.4 pounds/cubic foot)--and that multivariate sensitivity is not carefully examined. In the video of the debris impact, the debris pieces look larger than the estimated sizes (20" by 10" by 6"; and 20" by 16" by 6") used in the 3 reports. The important video is at http://www.spaceref.com/Columbia/post.launch.video.html The video also shows a fine shower of debris coming off the wing after impact; that spray does not immediately suggest foam chips. In the reports, assumptions tend to be evaluated generally in the direction of how they might reduce the seriousness of the threat (after-the-fact arguments of the form "this is a conservative estimate" replace careful quantitative estimates of robustness and uncertainty). An important table has 2 empty cells; threat assessments are missing in those 2 cells (The Washington Post discussed this point). The good diagram showing forecasted tile loss provides only point estimates; there is no cloud of error around those estimates - The 3 reports have the following analytical design characteristics: They appear to be PowerPoint slides. Some tables are difficult to read because of the grid prisons surrounding the entries in the spreadsheet, and it is difficult to make comparisons of numbers across the table. Bullets lists are used throughout, with up to 5 levels of hierarchy on a single page of 10 or 12 lines. Consequently the reasoning is broken up into stupefying fragments both within and between the many slides. Although an oral presentation accompanied the 3 reports originally, the reports were also circulated as stand-alone slides in e-mail attachments by NASA engineers concerned about the possible damage to Columbia's wing. - - The fundamental nature of the Columbia analysis might be called statistical engineering: the content is engineering but the logic is exactly the logic of statistics and econometrics (issues of estimation, thin data, model sensitivity and robustness, multivariate data, error assessment). The Columbia analysis needs some high-level statistical reasoning and use of techniques from standard statistical tool-kits. The Columbia analysis would have been a perfect problem for the great applied statistician Cuthbert Daniel. - - -- Edward Tufte, March 18, 2003 -
