One of the things that makes The Sequences frustrating is how Eliezer Yudkowsky's contempt for what came before him neuters their impact. The most obvious damage is how his refusal to follow academic norms resulted in an equilibrium of weak scholarship. He didn't so much as bother to give his Sequences a bibliography until the Rationality: AI to Zombies edition was published in 2015. The bibliography he did publish seems non-comprehensive to me (e.g. where is Language In Thought And Action by Hayakawa?). But I think the deepest damage was done by Yudkowsky's habit of disassocating from thinkers he no longer totally agrees with, including his past self. This disassociation decontextualized his ideas, making it much harder for his readers to get a complete model of how they work and what to draw on for further development.
One of the stranger acts of disassociation is Yudkowsky's omission of his future shock levels from The Sequences. Future shock levels are not the most rigorous idea, but considering that high future shock is more or less the unique ingredient that makes The Sequences what they are you'd think he would be more self aware about it. Part of the idea behind this omission seems to be that high future shock necessarily follows from a deep consideration of physical possibility, but that doesn't seem to be the case. As far as I know people with physics Ph.D's do not automatically turn into Extropians and Singularitans and Transhumanists. There is a certain element of storytelling that seems to be required for people to connect the dots in that particular way reliably.
A similar act of disassociation wrote Max More and his Extropians out of the narrative. Nevermind that Yudkowsky posted to their mailing list when he was 17, as Tom Chivers recounts in his The AI Does Not Hate You Extropians were downstream of the singularity (Chivers, 2019). In Eliezer's mind this presumably meant they no longer merited a mention in The Sequences. Instead of getting a principled explanation of what the conceptual journey upstream to a technological singularity looks like starting from common sense intuitions, we got the post Raised in Technophilia:
The first crack in my childhood technophilia appeared in, I think, 1997 or 1998, at the point where I noticed my fellow technophiles saying foolish things about how molecular nanotechnology would be an easy problem to manage. (As you may be noticing yet again, the young Eliezer was driven to a tremendous extent by his ability to find flaws—I even had a personal philosophy of why that sort of thing was a good idea.)
There was a debate going on about molecular nanotechnology, and whether offense would be asymmetrically easier than defense. And there were people arguing that defense would be easy. In the domain of nanotech, for Ghu’s sake, programmable matter, when we can’t even seem to get the security problem solved for computer networks where we can observe and control every one and zero. People were talking about unassailable diamondoid walls. I observed that diamond doesn’t stand off a nuclear weapon, that offense has had defense beat since 1945 and nanotech didn’t look likely to change that.
And by the time that debate was over, it seems that the young Eliezer— caught up in the heat of argument—had managed to notice, for the first time, that the survival of Earth-originating intelligent life stood at risk.
— Eliezer Yudkowsky, Raised in Technophilia
It was on account of this intellectual skittishness that I didn't realize my intuitions were more SL3 than SL4 until several years after I'd read The Sequences. This same reluctance also led me (and many others) to fail to understand what was so 'special' about them in the first place. I didn't understand until I decided I'd be better off starting over somewhere else and I began dissecting the concept of 'rationality' so I could replicate it wherever I went. In a particularly stupid episode of map-territory confusion I focused on the word 'rationality' and read books like Tetlock's Superforecasting and Lewis's Moneyball. It was while reading these books that I had a startling realization: They were good, but no amount of reading them would have ever let me write Eliezer's Sequences. The Sequences were centrally about math, physics, and this 'weird AI futurology stuff' (which was supposed to be based on the math and physics). All the discussion of science and cognitive biases and Bayes was important but ultimately noncentral to Eliezer's real purpose for writing.
I'm thankful that my 14 year old self wasn't mature enough to fully parse and internalize Eliezer's life advice, because it likely would have turned out very badly. The usual outcome looks something like a promising young person devoting all their time to reading math and physics textbooks then moving to the Bay Area:
More of my story: When I first encountered the rationality community, I read the Sequences and a lot of rational fiction. I had already dropped nearly everything several months prior to work on AGI after hearing arguments about its importance from a classmate in college, and I pivoted to working on friendliness. I worked on Pascal’s Mugging and naturalizing Solomonoff Induction, since these were two of the three open problems I was aware of from Less Wrong. (My work was mostly superseded by Logical Induction, so I never fully shared it.) I tried to read the long list of math textbooks that MIRI recommended for aspiring researchers. I did this for about two years, before deciding to move to the Bay in order to figure out what was the problem that I had the most relative advantage to solve (this led to me cofounding Rationalist Fleet) and to have tighter feedback loops interacting with people working on AI safety research. I optimized as though I was a part of a polity of people working to save the world. Given the character of what Eliezer had written, I had assumed that was what MIRI and the rationalist community were.
— Gwen D, Case study: CFAR
Once they get there they realize a couple things, sooner or later. The first thing they probably notice is that the Bay Area is a very bad place to be for someone starting their career. There are entire counties in the Bay Area with a poverty line over six figures (CBS San Francisco, 2018). Worse still the Bay is a place set up to make powerful people more powerful, the default expectation for someone moving in unestablished is that it's financial and interpersonal suicide. I happened to know to stay away in part because of a passage I'd read in Alan Bullock's Hitler: A Study In Tyranny (Bullock, 1964):
Vienna, at the beginning of 1909, was still an imperial city, capital of an Empire of fifty million souls stretching from the Rhine to the Dniester, from Saxony to Montenegro. The aristrocratlc baroque city of Mozart's time had become a great commercial and industrial centre with a population of two million people. Electric trams ran through its noisy and crowded streets. The massive, monumental buildings erected on the Ringstrasse in the last quarter of the nineteenth century reflected the prosperity and self confidence of the Viennese middle class; the factories and poorer streets of the outer districts the rise of an industrial working class. To a young man of twenty, without a home, friends, or resources, it must have appeared a callous and unfriendly city: Vienna was no place to be without money or a job. The four years that now followed, from 1909 to 1913, Hitler himself says, were the unhappiest of his life. They were also in many ways the most important, the formative years in which his character and opinions were given definite shape
The second thing they realize is that MIRI and CFAR are not actually particularly good at 'saving the world'. I'm not in a good position to evaluate MIRI's impact, but as a total outsider who doesn't pay much attention to AI Risk research it doesn't look stellar. CFAR on the other hand focuses on a subject I actually know a lot about, and I've privately critiqued them for years. Gwen herself points out that the CFAR handbook supposedly hasn't changed very much between its 2016 and 2019 editions. I can't personally attest to this, but I can say that the parts of the CFAR handbook I've read felt distinctly inferior to The Sequences. Not just in writing quality, but in the sense of providing a coherent worldview. My friend Said Achmiz once pointed out to me that The Sequences were not just "a collection of crap on the Internet" and this was the basic secret to their power. Without context that might not be particularly insightful, a "collection of crap on the Internet" would be something like You Are Not So Smart, which is a big list of cognitive biases. A lot of people try to replicate Eliezer's Sequences by making a 'big list of biases' or trying to find some new list of relevant sounding topics to teach.
'A lot of people' apparently includes CFAR, since as far as I can tell the pieces in the handbook don't actually come together to form a complete worldview, they're just cool independent concepts to make you 'more rational'. Considering that they have The Sequences as a starting example, and Eliezer Yudkowsky is presumably available to consult, this is disgraceful. Yudkowsky already admitted that the basic recipe is to take your coherent worldview and then piece it up into a bunch of TVTropes like pages with click-y names/phrases for concepts densely linked together and mutually referencing. When I went to refamiliarize myself with some posts for this essay I found it lead me into a violent tabsplosion that quickly took over my browser. The design is just as potent in 2020 as it was in 2009 (this is incidentally why the community selects for an ADD phenotype).
I think a great deal of this can be traced back to the fact that rationality was defined as 'the way of winning', which does not constrain expectations about what I should expect a 'rationalist' to be focusing on. What I imagine happened was the CFAR people got together and said "okay lets work on rationality", and this of course necessitated that they pick some subjects to focus on. When they got to that part, it probably wasn't clear enough in their head what rationality was such that they could reliably build on and improve it (circling, really?). It's notable to me that when I wanted to break down 'rationality' so I could recreate it, I focused on the implications of the word and 'way of winning' more than just going back and dissecting The Sequences. If I'd done the latter, I'd have probably noticed the real component parts (high future shock, love for the world, sanity, agency) way sooner. In any case it also means that CFAR's only real metric for rationality-training success is (as far as I know) a non-rigorous survey they put out asking people if CFAR improved their life. When you spend a lot of money on a course and there's social pressure to like the ingroup 'rationality' 'research' organization you're pretty likely to say yes even if the impact on your life was zero.
All of this has led the 'rationalist community' to have a more or less uninterrupted identity crisis since Eliezer stopped writing daily posts in 2009. I still think of the time when a bunch of people decided monarchy was awesome and started calling themselves 'Neoreactionaries' without getting banned as the point when ratsphere absurdity reached a rolling boil. Certainly we all have fond memories of right wing trolls telling female cryonicists they should expect to be domestic slaves when they're revived in the future. This theater troupe got rave reviews, so it was only natural to follow it with a dadaist encore of half incoherent posts about enlightenment, proclamations that consistent epistemology is so 20th century, praise for group bonding rituals whose participants literally cannot describe the benefits and vegan TDT-basilisk cults. As I'm to understand it Eliezer felt he would be making his 'rationality' more palatable to Singerism if he kept the quasi-religious (or as I will later argue, just plain religious) Extropian stuff to a minimum. But I think any person who has been seriously influenced by Yudkowsky should be aware of his actual feelings about goal setting and accomplishment:
Eliezer on Sept 12, 2012 | parent | favorite | on: Ask PG: What Is The Most Frighteningly Ambitious I...
Can you say where the scariest and most ambitious convincing pitch was on the following scale?
1) We're going to build the next Facebook!
2) We're going to found the next Apple!
3) Our product will create sweeping political change! This will produce a major economic revolution in at least one country! (Seasteading would be change on this level if it worked; creating a new country successfully is around the same level of change as this.)
4) Our product is the next nuclear weapon. You wouldn't want that in the wrong hands, would you?
5) This is going to be the equivalent of the invention of electricity if it works out.
6) We're going to make an IQ-enhancing drug and produce basic change in the human condition.
7) We're going to build serious Drexler-class molecular nanotechnology.
8) We're going to upload a human brain into a computer.
9) We're going to build a recursively self-improving Artificial Intelligence.
10) We think we've figured out how to hack into the computer our universe is running on.
— Eliezer Yudkowsky, September 12th, 2012
Bibliography
Chivers, T. (2019). The ai does not hate you: Superintelligence, rationality and the race to save the world. London: Weidenfeld & Nicolson.
CBS San Francisco. (2018, June 26). HUD: $117,000 now ‘low-income’ in 3 bay area counties. https://sanfrancisco.cbslocal.com/2018/06/26/hud-117000-low-income-san-mateo-san-francisco-marin/
Bullock, A. (1964). Hitler: A study in tyranny. New York: Harper & Row.