Helsingin yliopisto
In this essay series, I will write down my own thoughts about Eliezer Yudkowsky’s essays on the Rationality: Ai to Zombies –series from the point of view of a historian. My reason for writing these is primarily to organize my own thoughts regarding the use of rationality as a mental tool in my own profession, and as such, I do not presume to even attempt to appeal to a very wide audience. However, if you are reading this disclaimer and find my essays insightful or entertaining, more the power to you and I implore you to go and read the original essays, if you have not already.
I said, “It is more probable that universes replicate for any reason, than that they replicate via black holes because advanced civilizations manufacture black holes because universes evolve to make them do it.”
And he said, “Oh.”
The following is based on Yudkowsky’s essay Burdensome Details¹.
The conjunction fallacy is when humans rate the probability P(A;B) higher than the probability P(B), even though it is a theorem that P(A;B) ≤ P(B).
In a classic experiment by Tversky and Kahneman (1982)², they asked test subjects to rate the probability of statements regarding an imaginary person, Linda. Before giving the statements, they introduced her with this description:
“Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.”
Among the statements were the following three:
-
- X) Linda is active in the feminist movement.
- Y) Linda is a bank teller.
- Z) Linda is a bank teller and is active in the feminist movement.
The test subjects subsequently rated the probability of statement (Z) being higher than (Y), and (X) having the highest probability. This research result has been replicated many times, and you can read Yudkowsky’s essay Conjunction Controversy (Or, How They Nail It Down) for more examples of studies into this heuristic bias.
The interpretation is that subjects substitute judgment of representativeness for judgment of probability. Because the statement (Z) feels more right than the statement (Y), they assign it a higher probability even though with a little bit of thinking it would be clear to them that P(A;B) ≤ P(B). The description activates our heuristics and because Linda more closely resembles a feminist than a bank teller, the test subjects presumed it more likely that she was a feminist bank teller rather than just a bank teller. The implausibility of one claim is ‘averaged out’ by the plausibility of the other.
By adding extra details, you can make an outcome seem more characteristic of the process that generates it. We are susceptible to weaving contrived narratives within our heads that sound more plausible the more threads we weave into it. We have to look back and remind ourselves of the difference between sources and our own additions. We have to hold up every detail of our intricately weaved accounts independently, and ask, “How do I know this detail?”
Yudkowsky refers to futurologists and their tendency to weave intricate details into their future projections, but the same applies to historians. The more ‘neat’ and detailed an account of history sounds like, the less probable it likely is. A picture of a garden with a garden gnome might be more interesting to look at than one without it, but if the map does not correspond with the territory, can we claim to be scientific even to the little extent historians usually can?
To avoid this bias that seems at the same time stupidly obvious yet keeps tripping our minds up whenever we are not mindful of it, Yudkowsky recommends noticing the word “and,” and being wary of it. It is easy to get carried away with heuristics that sound plausible and neat, and pat ourselves in the back in the process for spotting the connection. But if there is no evidence of a connection in the first place, our heuristics are just a burden on our quest for the truth.
To win in the game of heuristics, we need to begin with the shortest/least detailed answer and assign it the highest probability of potential answers. Only then can we turn on our plausibility radar and start guessing what other factors may have been present, as long as we remain mindful of the difference of plausibility and probability.
¹ Yudkowsky, Eliezer. ”Rationality: from AI to Zombies” Berkeley, MIRI (2015). 26–29.
² Tversky, A. and Kahneman, D. 1982. Judgments of and by representativeness. Pp 84-98 in Kahneman, D., Slovic, P., and Tversky, A., eds. Judgment under uncertainty: Heuristics and biases. New York: Cambridge University Press.