Case Studies in Ethics and Safety Critical Software: Star Wars Revisited and Ballistic Missile Defense

1. 1 Star Wars Revisited A Case Study In Ethics and Safety-Critical Software Professor Kevin W. Bowyer University of Notre Dame Copyright, Kevin W. Bowyer, 2000,2001,2006,2007. All Rights Reserved. (Rev. 2/6/07)

2. 2 Ballistic Missile Defense A defensive system to counter enemy offensive weapons. Ballistic missiles follow a predictable trajectory.

3. 3 Ballistic Missile Defense The U.S. has a long history of BMD research and systems. For example, the Safeguard system (1960s-70s) was meant to protect U.S. launch sites from pre-emptive attack.

4. 4 Ballistic Missile Defense The most important software elements of a BMD system are command and control. Software development begins with analysis leading to requirements and specifications.

5. 5 Software Life Cycle System Engineering Analysis Design Coding Testing Maintenance

6. 6 Software Life Cycle System engineering Analyze the overall system, Specify system requirements, Decide on elements of system to implement in software.

7. 7 Software Life Cycle Software requirements Specify required functionality, performance, and interfaces for the software elements.

8. 8 President Reagans SDI In March 1983, President Reagan called for a Strategic Defense Initiative (SDI). The SDI program came to be popularly called Star Wars.

9. 9 President Reagans SDI

10. 10 Pause for Analysis What overall requirements can you derive for the SDI system from Reagans speech? (summarize in one paragraph)

11. 11 President Reagans SDI I call upon the scientific community to give us the means of rendering these nuclear weapons impotent and obsolete.

12. 12 President Reagans SDI I am directing to define a long-term R&D program to begin to eliminate the threat posed by strategic nuclear missiles.

13. 13 President Reagans SDI Which quote from the speech drives your requirements for the SDI system?

14. 14 SDI Computing Task The software must make the sensors, weapons, control, and communications elements of the system work together to satisfy requirements.

15. 15 SDI Computing Panel A panel of experts studied the SDI computing problem (at $1,000 / day consulting!). David Parnas resigned from the panel and declared the software to be impossible.

16. 16 1985 CPSR-MIT Debate The Computing Professionals for Social Responsibility (CPSR) sponsored a debate at MIT, featuring members of the SDI computing panel.

17. 17 1985 CPSR-MIT Debate Michael Dertouzos, moderator David Parnas, against SDI (Joseph Weizenbaum, against) Charles Seitz, for SDI (Danny Cohen, for)

18. 18 Michael Dertouzos, moderator

19. 19 Pause for Analysis Based on Dertouzos overview, what more can be said about the computing requirements for SDI? (summarize in one page)

20. 20 Michael Dertouzos Star Wars Can the computing requirements be met? Assume that the sensors and weapons can be developed and put in space, are affordable,

21. 21 Michael Dertouzos Parameters of the problem watch 50 million km 2 area, track up to 3,000 missiles, up to 30,000 warheads up to 300,000 decoys, it all happens in minutes.

22. 22 Michael Dertouzos Elements of the task Form target tracks. Sensors and geographic-scale pixels Consistent, distributed database. Allocate weapons to tracks. Communicate with humans. Withstand attack.

23. 23 Question What is your first answer to Dertouzos question Star wars: can the computing requirements be met?

24. 24 Question Why did some of the audience laugh after Dertouzos mentioned consistent distributed database ?

25. 25 Consistent Distributed Database A database that consists of two or more data files located at different sites on a computer network. Because the database is distributed, different users can access it without interfering with one another. However, the DBMS must periodically synchronize the scattered databases to make sure that they all have consistent data. http://www.webopedia.com/TERM/D/distributed_database.html

26. 26 David Parnas, arguing against

27. 27 Pause for Analysis Sketch Parnas argument in premise-conclusion style: Since Premise, and Premise, Therefore Conclusion. (Hint: identify conclusion first.)

28. 28 Parnas Conclusion Which is a proper conclusion of his technical argument? U.S. should not pursue SDI. SDI will make U.S. weaker. It is not possible to build trustworthy SDI software.

29. 29 Parnas Conclusion It is not possible to build trustworthy SDI software. trustworthy = reliability similar to that of your car starting in the morning.

30. 30 Parnas Premises Since: Specifications cannot be known in advance, (because enemy controls factors such as target / decoy features, attack load and structure, )

31. 31 Parnas Premises Since: Specifications not known in advance, Realistic testing is essentially impossible, (because, for example, link / node failures under attack are not known in advance)

32. 32 Parnas Premises Since: Specifications not known in advance, Realistic testing is not possible, Hard real-time deadlines do not allow repair during use (attack is over in minutes)

33. 33 Parnas Premises Since: Specifications not known in advance, Realistic testing is not possible, No chance to fix software during use, No foreseeable advance in software tech changes this (not language, methodology, )

34. 34 Parnas Argument Since: Specifications not known in advance, Realistic testing is not possible, No chance to fix software during use, No foreseeable technology changes this, Therefore It is not possible to construct SDI software that you could trust to work.

35. 35 1985 CPSR-MIT Debate Michael Dertouzos, moderator David Parnas, against SDI (Joseph Weizenbaum, against) Charles Seitz, for SDI (Danny Cohen, for)

36. 36 Charles Seitz, arguing for

37. 37 Pause for Analysis Sketch Seitz argument in premise-conclusion style: Since Premise, and Premise, Therefore Conclusion. (Hint: identify conclusion first.)

38. 38 Seitz Conclusion It is possible to create reliable SDI software.

39. 39 Seitz Premises Since A hierarchical architecture seems best, (because more natural, used in nature, understood by military, allows abstraction up levels )

40. 40 Seitz Premises Since A hierarchical architecture seems best, Physical organization should follow logical organization, (simplest choice, natural)

41. 41 Seitz Premises Since A hierarchical architecture seems best, Physical organization also hierarchical, Tradeoffs to make software problem tractable are in the choice of system architecture (not in new / radical methods)

42. 42 Seitz Premises Since A hierarchical architecture seems best, Physical organization also hierarchical, This makes software problem tractable, Loose coordination allows us to infer system performance (assume stat. independence, )

43. 43 Seitz Argument Since A hierarchical architecture seems best, Physical organization also hierarchical, This makes software problem tractable, And allows system reliability estimate, Therefore It is possible to create reliable SDI battle management software.

44. 44 Pause for Analysis Whose argument is better? Why? Do they start with the same problem definition?

45. 45 David Parnas, Rebuttal

46. 46 Charles Seitz, Rebuttal

47. 47 Pause for Analysis Relevant analogies to SDI? Why / why not? Space shuttle software Telephone system software Nuclear plant software others?

48. 48 Pause for Analysis Outline the most realistic SDI software testing that you can.

49. 49 Pause for Analysis How did you account for real-world sensor inputs variable weather conditions target / decoy appearance variable attack structure attacked components failing

50. 50 Fault Tolerant Software? James Ionson, in Reliability and Risk, a CPSR video.

51. 51 Fault Tolerant Software? It is not error-free code, it is fault-tolerant code. And if another million lines has to be written to ensure fault- tolerance, so be it. - James Ionson

52. 52 Fault Tolerant Software? Diagram in premise-conclusion form the argument being made by James Ionson. Does the argument make sense? Why / why not?

53. 53 Star Wars Today Current SDI-like programs are called National Missile Defense. There are some potentially important differences.

54. 54 Star Wars Today One of the remarkable aspects of the evolution of missile defenses is that few policy makers question the fundamental ability to be effective. Instead they focus on timing, cost, . (Mosher, page 39, IEEE Spectrum, 1997)

55. 55 Star Wars Today This is a sharp change from the Reagan years, perhaps because the technology used is closer at hand and the threats are smaller. (Mosher, page 39, IEEE Spectrum, 1997)

56. 56 Pause for Analysis How fundamentally does it change Parnas argument if the anticipated attack uses fewer and simpler missiles?

57. 57 Parnas Argument How are the premises changed? Specifications not known in advance. Realistic testing is not possible. No chance to fix software during use. No foreseeable technology changes this. None are changed in principle but it somehow more possible. Why?

58. 58 Star Wars Testing In the last 15 years, the U.S. has conducted 20 hit-to-kill intercepts, . Six intercepts were successful; 13 of those test were done in the last five years, and among them three succeeded. (Mosher, page 39, IEEE Spectrum, 1997)

59. 59 Star Wars Testing No real attempts have been made to intercept uncooperative targets those that make use of clutter, decoys, maneuver, anti- simulation, and other countermeasures. (Mosher, page 39, IEEE Spectrum, 1997)

60. 60 Star Wars Testing Test of a powerful laser has been blocked by bad weather and software problems. a software problem caused the laser to recycle, or unexpectedly lose power . (R. Smith, Washington Post, Oct 8, 1997)

61. 61 Schwartz versus TRW In 1996, ex TRW engineer Nira Schwartz filed a False Claims Act suit, alleging that results of tests to distinguish warheads and decoys were falsified by TRW. (featured on 60 Minutes II in January 2001)

62. 62 Schwartz versus TRW Schwartz claims that TRW knowingly made false test plans, test procedures, test reports and presentations to the government to remain in the program.

63. 63 Schwartz versus TRW Schwartz claims I say to my boss, It is wrong, what we are doing; it is wrong. And the next day, I was fired.

64. 64 Schwartz versus TRW TRW says TRW scientists and engineers devoted years to this complex project, while Ms. Schwartz, in her six months with the company Her understanding is insufficient to lend any credibility to her allegations.

65. 65 Schwartz versus TRW DOD criminal investigator says absolute, irrefutable, scientific proof that TRWs discrimination technology does not, cannot, and will not work TRW knowingly covering up.

66. 66 Schwartz versus TRW DOD panel then said TRWs software and sensors are well designed and work properly provided that the Pentagon does not have any wrong information about what kind of warheads and decoys an enemy is using.

67. 67 Schwartz versus TRW Lt. General Kadish Right now, from what I see, there is no reason to believe that we cant make this work. But theres a lot more testing to be done.

68. 68 Schwartz versus TRW Congressman Curt Weldon, R-PA: If we dont build a new aircraft carrier, we have older ones. If we dont build a new fighter plane, we have older ones. If we dont build missile defense, we have nothing. What is the premise-conclusion summary of this argument?

69. 69 Schwartz versus TRW Congressman Curt Weldon, R-PA: On 50 Nobelists anti-BMD letter - I dont know any of them thats come to Congress or me. I mean its easy to get anyone to sign a letter. I sign letters all the time. What is the premise-conclusion summary of this argument?

70. 70 Schwartz versus TRW Congressman Curt Weldon, R-PA: There were scientists that who made the case against Kennedy that it was crazy, wed never land on the moon. And I characterize Postol now as one of those people. What is the premise-conclusion summary of this argument?

71. 71 Ethical Issues What are some of the important ethical questions? And what guidance do the codes of ethics give on these questions?

72. 72 Ethical Issues How to interact with colleagues with whom you disagree? When to blow the whistle? Should you accept work on an impossible but $$$ project?

73. 73 Dealing with Colleagues AITP Standards of Conduct: In recognition of my obligation to fellow members and the profession I shall cooperate with others in achieving understanding and in identifying problems.

74. 74 Dealing with Colleagues Item 5.12 of ACM / IEEE-CS Software Engineering Code: Those managing or leading software engineers shall not punish anyone for expressing ethical concerns about a project.

75. 75 Accept Impossible Work? Item 3.2 of ACM / IEEE-CS Software Engineering Code: Software engineers shall ensure proper and achievable goals and objectives for any project on which they work or propose.

76. 76 Accept Impossible Work? Item 1.3 of the ACM / IEEE-CS Software Engineering Code: Software engineers shall accept software only if they have a well founded belief that it is safe, meets specifications, passes appropriate tests,

77. 77 Blow the Whistle? AITP Standards of Conduct: In recognition of my obligation to society, I shall never misrepresent or withhold information that is germane to a problem or situation of public concern nor allow any such known information to remain unchallenged.

78. 78 Blow the Whistle? Item 1.4 of ACM / IEEE-CS Software Engineering Code: Software engineers shall disclose to appropriate persons or authorities any actual or potential danger to the user, the public that they reasonably believe

79. 79 Summary Difficult ethical issues arise in creation of safety-critical software. Trustworthy SDI software is more clearly impossible in retrospect. Modern, smaller SDI-like programs appear more tractable.

80. 80 National Science Foundation grant DUE 97-52792 Thanks to for partial support of this work.

81. 81 Computing Professionals for Social Responsibility (www.cpsr.org) Thanks to the for permission to distribute digitized video of the debate.

82. 82 Thanks to for commenting on a draft of the paper describing this module. David Parnas Chuck Seitz

83. 83 Thanks to the for help in obtaining the video of Reagans 3/23/83 speech. The Ronald Reagan Presidential Library (www.reagan.utexas.edu)

84. 84 Thanks to for technical assistance. Christine Kranenburg Laura Malave Melissa Parsons Joseph Wujek

85. 85 The End.