Revisiting Ockham's Razor: An Alternative Take on Simplicity and Truth

1. Simplicity and Truth: an Alternative Explanation of Ockham's Razor Simplicity and Truth: an Alternative Explanation of Ockham's Razor Kevin T. Kelly Kevin T. Kelly Conor Mayo-Wilson Conor Mayo-Wilson Department of Philosophy Department of Philosophy Joint Program in Logic and Computation Joint Program in Logic and Computation Carnegie Mellon University Carnegie Mellon University www.hss.cmu.edu/philosophy/faculty-kelly.php www.hss.cmu.edu/philosophy/faculty-kelly.php

2. I. The Simplicity Puzzle I. The Simplicity Puzzle

3. Which Theory is Right? Which Theory is Right? ???

4. Ockham Says: Ockham Says: Choose the Simplest!

5. But Why? But Why? Gotcha!

6. Puzzle Puzzle An indicator must be sensitive to what it indicates. An indicator must be sensitive to what it indicates. simple

7. Puzzle Puzzle A reliable indicator must be sensitive to what it indicates. A reliable indicator must be sensitive to what it indicates. complex

8. Puzzle Puzzle But Ockham’s razor always points at simplicity. But Ockham’s razor always points at simplicity. simple

9. Puzzle Puzzle But Ockham’s razor always points at simplicity. But Ockham’s razor always points at simplicity. complex

10. Puzzle Puzzle How can a broken compass help you find something unless you already know where it is? How can a broken compass help you find something unless you already know where it is? complex

11. Standard Accounts Standard Accounts 1. Prior Simplicity Bias 1. Prior Simplicity Bias Bayes, BIC, MDL, MML, etc. Bayes, BIC, MDL, MML, etc. 2. Risk Minimization 2. Risk Minimization SRM, AIC, cross-validation, etc. SRM, AIC, cross-validation, etc.

12. 1. Prior Simplicity Bias 1. Prior Simplicity Bias The simple theory is more plausible now because it was more plausible yesterday .

13. More Subtle Version More Subtle Version Simple data are a miracle in the complex theory but not in the simple theory. Simple data are a miracle in the complex theory but not in the simple theory. P C Regularity: retrograde motion of Venus at solar conjunction Has to be!

14. However… However… e would not be a miracle given P (  ); e would not be a miracle given P (  ); Why not this? C P

15. The Real Miracle The Real Miracle Ignorance about model: Ignorance about model: p ( C )  p ( P ); p ( C )  p ( P ); + Ignorance about parameter setting: + Ignorance about parameter setting: p’ ( P (  ) | P )  p ( P (  ’ ) | P ). p’ ( P (  ) | P )  p ( P (  ’ ) | P ). = Knowledge about C vs. P (  ): = Knowledge about C vs. P (  ): p ( P (  )) << p ( C ). p ( P (  )) << p ( C ). C P         Lead into gold . Perpetual motion. Free lunch. Ignorance is knowledge. War is peace. I love Big Bayes.

16. Standard Paradox of Indifference Standard Paradox of Indifference Ignorance of red vs. not- red Ignorance of red vs. not- red + Ignorance over not- red : + Ignorance over not- red : = Knowledge about red vs. white. = Knowledge about red vs. white.   Kno gnorance = All the priveleges of knowledge With none of the responsibilities Yeah!

17. The Ellsberg Paradox The Ellsberg Paradox 1/3 ? ?

18. Human Preference Human Preference 1/3 ? ? a > b a c < c b b

19. Human View Human View 1/3 ? ? a > b a c < c b b knowledge ignorance knowledge ignorance

20. Bayesian View Bayesian View 1/3 ? ? a > b a c > c b b kno gnorance kno gnorance kno gnorance knog norance

21. In Any Event In Any Event The coherentist foundations of Bayesianism have nothing to do with short-run truth- conduciveness. The coherentist foundations of Bayesianism have nothing to do with short-run truth- conduciveness. Not so loud!

22. Bayesian Convergence Bayesian Convergence Too-simple theories get shot down… Too-simple theories get shot down… Complexity Theories Updated opinion

23. Bayesian Convergence Bayesian Convergence Plausibility is transferred to the next-simplest theory… Plausibility is transferred to the next-simplest theory… Blam! Complexity Theories Updated opinion Plink!

24. Bayesian Convergence Bayesian Convergence Plausibility is transferred to the next-simplest theory… Plausibility is transferred to the next-simplest theory… Blam! Complexity Theories Updated opinion Plink!

25. Bayesian Convergence Bayesian Convergence Plausibility is transferred to the next-simplest theory… Plausibility is transferred to the next-simplest theory… Blam! Complexity Theories Updated opinion Plink!

26. Bayesian Convergence Bayesian Convergence The true theory is nailed to the fence. The true theory is nailed to the fence. Blam! Complexity Theories Updated opinion Zing!

27. Convergence Convergence But alternative strategies also converge: But alternative strategies also converge: Anything in the short run is compatible with convergence in the long run. Anything in the short run is compatible with convergence in the long run.

28. Summary of Bayesian Approach Summary of Bayesian Approach Prior-based explanations of Ockham’s razor are circular and based on a faulty model of ignorance. Prior-based explanations of Ockham’s razor are circular and based on a faulty model of ignorance. Convergence-based explanations of Ockham’s razor fail to single out Ockham’s razor. Convergence-based explanations of Ockham’s razor fail to single out Ockham’s razor.

29. 2. Risk Minimization 2. Risk Minimization Ockham’s razor minimizes expected distance of empirical estimates from the true value. Ockham’s razor minimizes expected distance of empirical estimates from the true value. Truth

30. Unconstrained Estimates Unconstrained Estimates are Centered on truth but spread around it. are Centered on truth but spread around it. Pop! Pop! Pop! Pop! Unconstrained aim

31. Off-center but less spread. Off-center but less spread. Clamped aim Truth Constrained Estimates Constrained Estimates

32. Off-center but less spread Off-center but less spread Overall improvement in expected distance from truth… Overall improvement in expected distance from truth… Truth Pop! Pop! Pop! Pop! Constrained Estimates Constrained Estimates Clamped aim

33. Doesn’t Find True Theory Doesn’t Find True Theory The theory that minimizes estimation risk can be quite false . The theory that minimizes estimation risk can be quite false . Four eyes! Clamped aim

34. Makes Sense Makes Sense …when loss of an answer is similar in nearby distributions. …when loss of an answer is similar in nearby distributions. Similarity p Close is good enough! Loss

35. But Truth Matters But Truth Matters …when loss of an answer is discontinuous with similarity. …when loss of an answer is discontinuous with similarity. Similarity p Close is no cigar! Loss

36. E.g. Science E.g. Science If you want true laws , false laws aren’t good enough.

37. E.g. Science E.g. Science You must be a philosopher . This is a machine learning conference.

38. E.g., Causal Data Mining E.g., Causal Data Mining Protein A Protein B Protein C Cancer protein Practical enough? Now you’re talking! I’m on a cilantro-only diet to get my protein C level under control.

39. Correlation does imply causation if there are multiple variables, some of which are common effects. [Pearl, Spirtes, Glymour and Scheines] Correlation does imply causation if there are multiple variables, some of which are common effects. [Pearl, Spirtes, Glymour and Scheines] Central Idea Central Idea Protein A Protein B Protein C Cancer protein

40. Joint distribution p is causally compatible with directed, acyclic graph G iff: Joint distribution p is causally compatible with directed, acyclic graph G iff: Causal Markov Condition: each variable X is independent of its non-effects given its immediate causes. Causal Markov Condition: each variable X is independent of its non-effects given its immediate causes. Faithfulness Condition: no other conditional independence relations hold in p . Faithfulness Condition: no other conditional independence relations hold in p . Core assumptions Core assumptions

41. F1 F2 Tell-tale Dependencies Tell-tale Dependencies H C F Given F, H gives some info about C (Faithfulness) C Given C, F1 gives no further info about F2 (Markov)

42. Common Applications Common Applications Linear Causal Case: each variable X is a linear function of its parents and a normally distributed hidden variable called an “error term”. The error terms are mutually independent. Linear Causal Case: each variable X is a linear function of its parents and a normally distributed hidden variable called an “error term”. The error terms are mutually independent. Discrete Multinomial Case: each variable X takes on a finite range of values. Discrete Multinomial Case: each variable X takes on a finite range of values.

43. No unobserved latent confounding causes No unobserved latent confounding causes A Very Optimistic Assumption A Very Optimistic Assumption Genetics Smoking Cancer I’ll give you this one. What’s he up to?

44. Current Nutrition Wisdom Current Nutrition Wisdom Protein A Protein B Protein C Cancer protein Are you kidding? It’s dripping with Protein C! English Breakfast?

45. As the Sample Increases… As the Sample Increases… Protein A Protein B Protein C Cancer protein weak Protein D I do! Out of my way! This situation approximates The last one. So who cares?

46. As the Sample Increases Again… As the Sample Increases Again… Protein A Protein B Protein C Cancer protein Wasn’t that last approximation to the truth good enough? weak Protein D Aaack! I’m poisoned! Protein E weak weak

47. Causal Flipping Theorem Causal Flipping Theorem No matter what a consistent causal discovery procedure has seen so far, there exists a pair G , p satisfying the assumptions so that the current sample is arbitrarily likely and the procedure produces arbitrarily many opposite conclusions in p as sample size increases. No matter what a consistent causal discovery procedure has seen so far, there exists a pair G , p satisfying the assumptions so that the current sample is arbitrarily likely and the procedure produces arbitrarily many opposite conclusions in p as sample size increases. oops I meant oops oops I meant I meant

48. The Wrong Reaction The Wrong Reaction The demon undermines justification of science. The demon undermines justification of science. He must be defeated to forestall skepticism. He must be defeated to forestall skepticism. Bayesian circularity Bayesian circularity Classical instrumentalism Classical instrumentalism Grrrr! Urk!

49. Another View Another View Many explanations have been offered to make sense of the here-today-gone-tomorrow nature of medical wisdom — what we are advised with confidence one year is reversed the next — but the simplest one is that it is the natural rhythm of science . Many explanations have been offered to make sense of the here-today-gone-tomorrow nature of medical wisdom — what we are advised with confidence one year is reversed the next — but the simplest one is that it is the natural rhythm of science . ( Do We Really Know What Makes us Healthy , NY Times Magazine, Sept. 16, 2007). ( Do We Really Know What Makes us Healthy , NY Times Magazine, Sept. 16, 2007).

50. Zen Approach Zen Approach Get to know the demon. Get to know the demon. Locate the justification of Ockham’s razor in his power . Locate the justification of Ockham’s razor in his power .

51. Connections to the Truth Connections to the Truth Short-run Reliability Short-run Reliability Too strong to be feasible when theory matters. Too strong to be feasible when theory matters. Long-run Convergence Long-run Convergence Too weak to single out Ockham’s razor Too weak to single out Ockham’s razor Complex Simple Complex Simple

52. Middle Path Middle Path Short-run Reliability Short-run Reliability Too strong to be feasible when theory matters. Too strong to be feasible when theory matters. “Straightest” convergence “Straightest” convergence Just right? Just right? Long-run Convergence Long-run Convergence Too weak to single out Ockham’s razor Too weak to single out Ockham’s razor Complex Simple Complex Simple Complex Simple

53. II. Navigation by Broken Compass II. Navigation by Broken Compass simple

54. Asking for Directions Asking for Directions Where’s … Where’s …

55. Asking for Directions Asking for Directions Turn around. The freeway ramp is on the left. Turn around. The freeway ramp is on the left.

56. Asking for Directions Asking for Directions Goal

57. Best Route Best Route Goal

58. Best Route to Any Goal Best Route to Any Goal

59. Disregarding Advice is Bad Disregarding Advice is Bad Extra U-turn

60. Best Route to Any Goal Best Route to Any Goal …so fixed advice can help you reach a hidden goal without circles, evasions, or magic.

61. In Step with the Demon In Step with the Demon Constant Linear Quadratic Cubic There yet? Maybe.

62. In Step with the Demon In Step with the Demon Constant Linear Quadratic Cubic There yet? Maybe.

63. In Step with the Demon In Step with the Demon Constant Linear Quadratic Cubic There yet? Maybe.

64. In Step with the Demon In Step with the Demon Constant Linear Quadratic Cubic There yet? Maybe.

65. Ahead of Mother Nature Ahead of Mother Nature Constant Linear Quadratic Cubic There yet? Maybe.

66. Ahead of Mother Nature Ahead of Mother Nature Constant Linear Quadratic Cubic I know you’re coming!

67. Ahead of Mother Nature Ahead of Mother Nature Constant Linear Quadratic Cubic Maybe.

68. Ahead of Mother Nature Ahead of Mother Nature Constant Linear Quadratic Cubic !!! Hmm, it’s quite nice here…

69. Ahead of Mother Nature Ahead of Mother Nature Constant Linear Quadratic Cubic You’re back! Learned your lesson?

70. Ockham Violator’s Path Ockham Violator’s Path Constant Linear Quadratic Cubic See, you shouldn’t run ahead Even if you are right!

71. Ockham Path Ockham Path Constant Linear Quadratic Cubic

72. Empirical Problems Empirical Problems T1 T2 T3 Set K of infinite input sequences . Set K of infinite input sequences . Partition of K into alternative theories . Partition of K into alternative theories . K

73. Empirical Methods Empirical Methods T1 T2 T3 Map finite input sequences to theories or to “?”. Map finite input sequences to theories or to “?”. K T3 e

74. Method Choice Method Choice T1 T2 T3 e 1 e 2 e 3 e 4 Input history Output history At each stage, scientist can choose a new method (agreeing with past theory choices).

75. Aim: Converge to the Truth Aim: Converge to the Truth T1 T2 T3 K T3 ? T2 ? T1 T1 T1 T1 . . . T1 T1 T1

76. Retraction Retraction Choosing T and then not choosing T next Choosing T and then not choosing T next T’ T’ T T ? ?

77. Aim: Eliminate Needless Retractions Aim: Eliminate Needless Retractions Truth

78. Aim: Eliminate Needless Retractions Aim: Eliminate Needless Retractions Truth

79. Ancient Roots Ancient Roots "Living in the midst of ignorance and considering themselves intelligent and enlightened, the senseless people go round and round, following crooked courses , just like the blind led by the blind." Katha Upanishad , I. ii. 5, c. 600 BCE.

80. Aim: Eliminate Needless Delays to Retractions Aim: Eliminate Needless Delays to Retractions theory

81. application application application application corollary application theory application application corollary application corollary Aim: Eliminate Needless Delays to Retractions Aim: Eliminate Needless Delays to Retractions

82. Why Timed Retractions? Why Timed Retractions? Retraction minimization = generalized significance level . Retraction time minimization = generalized power .

83. Easy Retraction Time Comparisons Easy Retraction Time Comparisons T1 T1 T2 T2 T1 T1 T2 T2 T3 T3 T2 T4 T4 T2 T2 Method 1 Method 2 T4 T4 T4 . . . . . . at least as many at least as late

84. Worst-case Retraction Time Bounds Worst-case Retraction Time Bounds T1 T2 Output sequences T1 T2 T1 T2 T4 T3 T3 T3 T3 T3 T3 T4 T4 T4 T4 T4 . . . (1, 2, ∞) . . . . . . . . . . . . . . . T4 T4 T4 T1 T2 T3 T3 T3 T4 T3 . . .

85. IV. Ockham Without Circles, Evasions, or Magic IV. Ockham Without Circles, Evasions, or Magic

86. Curve Fitting Curve Fitting Data = open intervals around Y at rational values of X. Data = open intervals around Y at rational values of X.

87. Curve Fitting Curve Fitting No effects: No effects:

88. Curve Fitting Curve Fitting First-order effect: First-order effect:

89. Curve Fitting Curve Fitting Second-order effect: Second-order effect:

90. Empirical Effects Empirical Effects

91. Empirical Effects Empirical Effects

92. Empirical Effects Empirical Effects May take arbitrarily long to discover May take arbitrarily long to discover

93. Empirical Effects Empirical Effects May take arbitrarily long to discover May take arbitrarily long to discover

94. Empirical Effects Empirical Effects May take arbitrarily long to discover May take arbitrarily long to discover

95. Empirical Effects Empirical Effects May take arbitrarily long to discover May take arbitrarily long to discover

96. Empirical Effects Empirical Effects May take arbitrarily long to discover May take arbitrarily long to discover

97. Empirical Effects Empirical Effects May take arbitrarily long to discover May take arbitrarily long to discover

98. Empirical Effects Empirical Effects May take arbitrarily long to discover May take arbitrarily long to discover

99. Empirical Theories Empirical Theories True theory determined by which effects appear. True theory determined by which effects appear.

100. Empirical Complexity Empirical Complexity More complex

101. Background Constraints Background Constraints More complex

102. Background Constraints Background Constraints More complex

103. Ockham’s Razor Ockham’s Razor Don’t select a theory unless it is uniquely simplest in light of experience. Don’t select a theory unless it is uniquely simplest in light of experience.

104. Weak Ockham’s Razor Weak Ockham’s Razor Don’t select a theory unless it among the simplest in light of experience. Don’t select a theory unless it among the simplest in light of experience.

105. Stalwartness Stalwartness Don’t retract your answer while it is uniquely simplest Don’t retract your answer while it is uniquely simplest

106. Stalwartness Stalwartness Don’t retract your answer while it is uniquely simplest Don’t retract your answer while it is uniquely simplest

107. Timed Retraction Bounds Timed Retraction Bounds r ( M, e, n ) = the least timed retraction bound covering the total timed retractions of M along input streams of complexity n that extend e r ( M, e, n ) = the least timed retraction bound covering the total timed retractions of M along input streams of complexity n that extend e Empirical Complexity 0 1 2 3 . . . . . . M

108. Efficiency of Method M at e Efficiency of Method M at e M converges to the truth no matter what; M converges to the truth no matter what; For each convergent M’ that agrees with M up to the end of e , and for each n : For each convergent M’ that agrees with M up to the end of e , and for each n : r ( M , e , n )  r ( M’ , e , n ) r ( M , e , n )  r ( M’ , e , n ) Empirical Complexity 0 1 2 3 . . . . . . M M’

109. M is Beaten at e M is Beaten at e There exists convergent M’ that agrees with M up to the end of e , such that There exists convergent M’ that agrees with M up to the end of e , such that For each n , r ( M , e , n )  r ( M’ , e , n ); For each n , r ( M , e , n )  r ( M’ , e , n ); Exists n , r ( M , e , n ) > r ( M’ , e , n ). Exists n , r ( M , e , n ) > r ( M’ , e , n ). Empirical Complexity 0 1 2 3 . . . . . . M M’

110. Basic Idea Basic Idea Ockham efficiency: Nature can force arbitary, convergent M to produce the successive answers down an effect path arbitrarily late, so stalwart, Ockham solutions are efficient. Ockham efficiency: Nature can force arbitary, convergent M to produce the successive answers down an effect path arbitrarily late, so stalwart, Ockham solutions are efficient.

111. Basic Idea Basic Idea Unique Ockham efficiency: A violator of Ockham’s razor or stalwartness can be forced into an extra retraction or a late retraction in complexity class zero at the time of the violation, so the violator is beaten by each stalwart, Ockham solution. Unique Ockham efficiency: A violator of Ockham’s razor or stalwartness can be forced into an extra retraction or a late retraction in complexity class zero at the time of the violation, so the violator is beaten by each stalwart, Ockham solution.

112. Ockham Efficiency Theorem Ockham Efficiency Theorem Let M be a solution . The following are equivalent: Let M be a solution . The following are equivalent: M is always strongly Ockham and stalwart; M is always strongly Ockham and stalwart; M is always efficient; M is always efficient; M is never weakly beaten. M is never weakly beaten.

113. Example: Causal Inference Example: Causal Inference Effects are conditional statistical dependence relations . Effects are conditional statistical dependence relations . X dep Y | {Z}, {W}, {Z,W} Y dep Z | {X}, {W}, {X,W} X dep Z | {Y}, {Y,W} . . . . . .

114. Causal Discovery = Ockham’s Razor Causal Discovery = Ockham’s Razor X Y Z W

115. Ockham’s Razor Ockham’s Razor X Y Z W X dep Y | {Z}, {W}, {Z,W}

116. Causal Discovery = Ockham’s Razor Causal Discovery = Ockham’s Razor X Y Z W X dep Y | {Z}, {W}, {Z,W} Y dep Z | {X}, {W}, {X,W} X dep Z | {Y}, {Y,W}

117. Causal Discovery = Ockham’s Razor Causal Discovery = Ockham’s Razor X Y Z W X dep Y | {Z}, {W}, {Z,W} Y dep Z | {X}, {W}, {X,W} X dep Z | {Y}, {W}, {Y,W}

118. Causal Discovery = Ockham’s Razor Causal Discovery = Ockham’s Razor X Y Z W X dep Y | {Z}, {W}, {Z,W} Y dep Z | {X}, {W}, {X,W} X dep Z | {Y}, {W}, {Y,W} Z dep W| {X}, {Y}, {X,Y} Y dep W| {Z}, {X,Z}

119. Causal Discovery = Ockham’s Razor Causal Discovery = Ockham’s Razor X Y Z W X dep Y | {Z}, {W}, {Z,W} Y dep Z | {X}, {W}, {X,W} X dep Z | {Y}, {W}, {Y,W} Z dep W| {X}, {Y}, {X,Y} Y dep W| {X}, {Z}, {X,Z}

120. IV. Simplicity Defined IV. Simplicity Defined

121. Approach Approach Empirical complexity reflects nested problems of induction posed by the problem. Empirical complexity reflects nested problems of induction posed by the problem. Hence, simplicity is problem-relative but topologically invariant . Hence, simplicity is problem-relative but topologically invariant .

122. Empirical Problems Empirical Problems T1 T2 T3 Set K of infinite input sequences . Set K of infinite input sequences . Partition Q of K into alternative theories . Partition Q of K into alternative theories . K

123. Simplicity Concepts Simplicity Concepts A simplicity concept for K is just a well-founded order < on a partition S of K with ascending chains of order type not exceeding omega such that: A simplicity concept for K is just a well-founded order < on a partition S of K with ascending chains of order type not exceeding omega such that: 1. Each element of S is included in some answer in Q . 1. Each element of S is included in some answer in Q . 2. Each downward union in ( S , <) is closed; 2. Each downward union in ( S , <) is closed; 3. Incomparable sets share no boundary point. 3. Incomparable sets share no boundary point. 4. Each element of S is included in the boundary of its successor. 4. Each element of S is included in the boundary of its successor.

124. General Ockham Efficiency Theorem General Ockham Efficiency Theorem Let M be a solution . The following are equivalent: Let M be a solution . The following are equivalent: M is always strongly Ockham and stalwart; M is always strongly Ockham and stalwart; M is always efficient; M is always efficient; M is never beaten. M is never beaten.

125. Conclusions Conclusions Causal truths are necessary for counterfactual predictions. Causal truths are necessary for counterfactual predictions. Ockham’s razor is necessary for staying on the straightest path to the true theory but does not point at the true theory. Ockham’s razor is necessary for staying on the straightest path to the true theory but does not point at the true theory. No evasions or circles are required. No evasions or circles are required.

126. Future Directions Future Directions Extension of unique efficiency theorem to stochastic model selection. Extension of unique efficiency theorem to stochastic model selection. Latent variables as Ockham conclusions. Latent variables as Ockham conclusions. Degrees of retraction. Degrees of retraction. Pooling of marginal Ockham conclusions. Pooling of marginal Ockham conclusions. Retraction efficiency assessment of MDL, SRM. Retraction efficiency assessment of MDL, SRM.

127. Suggested Reading Suggested Reading "Ockham’s Razor, Truth, and Information" , in Handbook of the Philosophy of Information , J. van Behthem and P. Adriaans, eds., to appear. "Ockham’s Razor, Truth, and Information" , in Handbook of the Philosophy of Information , J. van Behthem and P. Adriaans, eds., to appear. "Ockham’s Razor, Empirical Complexity, and Truth-finding Efficiency" , Theoretical Computer Science , 383: 270-289, 2007. "Ockham’s Razor, Empirical Complexity, and Truth-finding Efficiency" , Theoretical Computer Science , 383: 270-289, 2007. Both available as pre-prints at: www.hss.cmu.edu/philosophy/faculty-kelly.php Both available as pre-prints at: www.hss.cmu.edu/philosophy/faculty-kelly.php