Katja Grace's apartment in West Berkeley is housed in an old mechanic's factory, with a pitched roof and oddly angled windows. It has terracotta floors and no central heating. This creates the impression that you have stepped out of the California sunshine and into a distant, dimly lit place. But there are also some quietly futuristic touches. Large capacity air purifiers are installed in every corner of the room. Preserved food piled up in the pantry. A sleek white machine that performs lab-quality RNA tests. The kind of objects that foreshadow a future that will be made easier by technology, or a future where we will always be on guard.
Grace, a principal scientist at a nonprofit called AI Impacts, describes her job as “thinking about whether AI will destroy the world.” She spends her time writing theoretical papers and blog posts on complex decision-making related to the fast-growing subfield known as AI safety. She has a nervous smile, overshares, and tweets a bit. She is in her 30s, but with her midsection and round, open face, she almost looks like she's a teenager. The apartment was packed full of books, and when Grace's friend came over one afternoon in November, he looked for a while, puzzled but nonjudgmentally, at The Jewish Divorce Ethics, The Jewish Divorce Ethics, I spent most of my time looking at the back covers of books such as “How to Die.'' and Mourning,” and “Death of Death.” As far as Grace knows, she is neither Jewish nor dying. She let that vague feeling linger for a while. Then she explained. Her landlord wanted to keep the property of her previous tenant, her recently deceased ex-wife, intact. “Honestly, I was kind of relieved,” Grace said. “A series of decisions I don’t have to make.”
She had spent the afternoon preparing dinner for six, including a yogurt and cucumber salad and an Impossible Beef Gyro. In a corner of the whiteboard, she had painstakingly broken down the pre-party tasks into small steps (“Cut the salad”, “Mix the salad”, “Make the meat molds”, “Cook the meat”) . On other parts of the whiteboard, she wrote additional gnome prompts (“food area,” “object,” and “substance”). Her friend, his android cryptographer named Paul Crowley, wore a black T-shirt and black jeans, and had his hair dyed black. When asked how they knew each other, he replied: “Yeah, we've been passing each other as part of the scene for years.”
The “scene” is some known for its in-depth discussions of sordid issues (safe DNA synthesis, shrimp welfare) that its members consider essential but that most ordinary people know nothing about. was understood to mean the intertwined subcultures of For some 20 years, one of these questions has been whether artificial intelligence will improve humanity or destroy us. A pessimist is called an AI safetyist, or a slowdownist, or if he is feeling particularly panicky he is called an AI catastrophe. They meet online, often end up living together in group houses in the Bay Area, and sometimes co-parent or homeschool their children. Before the dot-com boom, the Alamo Square and Hayes Valley neighborhoods, with their pastel-colored Victorian rowhouses, had a down-to-earth, homey feel. Last year, the San Francisco Standard referred to AI “hacker houses” and semi-sarcastically called the area Cerebral Valley.
The techno-optimist camp rejects AI fatalism with old-fashioned libertarian boomerism, arguing that all talk of existential crisis is a form of mass hysteria. They call themselves “effective accelerationists,” or e/accs (pronounced “e-ax”), and they say that if people who care don't get in the way, AI will lead to a utopian future: interstellar travel. I believe that it will bring an end to diseases. . On social media, they taunt the prophet as a “diplomat,” “psychic,” “basically a terrorist,” or, worst of all, a “regulation-loving bureaucrat.” “We must steal the fire of intelligence from the gods.” [and] Use it to propel humanity towards the stars,” Major Electronics/ACC recently tweeted. (And there are also canons based outside the Bay Area and the Internet that have largely ignored this discussion in favor of sci-fi noise and corporate heat.)
Mr. Grace's Dinner Party, a semi-subterranean meet-up for destructive people and destructively curious people, has been described as “the nexus of the Bay Area AI scene.” At gatherings like this, it's not uncommon to hear someone start the conversation by asking, “What's your timeline?” or “What is your p (destiny)?” A timeline is a prediction of how quickly an AI will pass a certain benchmark. Examples include writing a top 40 pop song, making a scientific breakthrough worthy of a Nobel Prize, and achieving artificial general intelligence (the point at which a machine can perform any cognitive task that a human can perform). do. (Some experts think AGI is impossible or decades away, while others predict it will happen this year.) P(Doom ) is the probability that AI will become extinct if it becomes smarter than humans, whether intentionally or by accident. Everyone on Earth. For years, such speculative conversations have been marginalized even in Bay Area circles. Last year, he suddenly burst into the mainstream after OpenAI released his eerily natural-sounding language model called ChatGPT. There are currently hundreds of people working full time to save the world from AI apocalypse. Some advise governments and businesses on policy. Some researchers address the technical aspects of AI safety, approaching it as a series of complex mathematical problems. Grace works for a kind of think tank that conducts research on “high-level questions” such as “What role will AI systems play in society?” “Do they pursue 'goals'?” When they're not lobbying in Washington, D.C., or meeting at international conferences, they can often be seen passing each other in Grace's living room.
The remaining guests arrived one after another. Former OpenAI researcher. Director of a research institute that predicts the future. Grace offered wine and beer, but most people opted for a non-alcoholic canned drink (fermented energy drink, “hop tea'') that was difficult to explain. They took Impossible Gyro to Grace's couch, where they talked until midnight. They were polite and, despite being obnoxious, incredibly patient with us rethinking basic assumptions. “It seems to me that the gist of the concern can be boiled down to a really simple two-step argument,” Crowley said. “Step 1: We're building a machine that has the potential to be much smarter than we are. Step 2: It looks pretty dangerous.”
“Is that OK?” said Josh Rosenberg, CEO of the Prediction Institute. “About intelligence itself being dangerous?”
Grace noted that not all intelligent species pose a threat: “We have elephants, but rats still seem to be doing well.”
“Rabbits are certainly more intelligent than myxomatosis,'' said quantum computing expert Michael Nielsen.
Crowley's p(doom) was “well above 80%.” Others were cautious about revealing numbers, deferring to Grace, who said: “Given my deep confusion and uncertainty about this, which I think almost everyone has, at least everyone who is honest,” she could only narrow her opinion ( ruin) “between 10 percent and 90 percent.” Still, she continued, “a 10 percent chance of human extinction is clearly unacceptably high if you think about it seriously.”
Among the thousands of responses to ChatGPT, they agreed that one of the most refreshing and honest reviews came from Snoop Dogg during an on-stage interview. Crowley pulled out the record and read it aloud. “This is not safe because AI has a mind of its own and these motherfuckers are going to start their own thing,” Snoop said, paraphrasing the AI safety debate. “Shit, what the hell?” Crowley laughed. “I have to admit, this captures the emotional tenor much better than my two-part argument,” he said. Then, as if to justify the moment of levity, he read out another quote. This is from his 1948 essay on C.S. Lewis. Praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting with friends over a beer or darts, etc. are all things that are wise and human. They do that, but they don't herd together like frightened sheep. ”
Grace worked for Eliezer Yudkowsky. He was a bearded man in a fedora with a short temper and a 99% AP. Raised in Chicago as an Orthodox Jew, he dropped out of school after eighth grade, taught himself calculus and atheism, started blogging, and headed to the Bay Area in his early 2000s. Some of his best-known works include his work of fiction, “Harry Potter and the Methods of Rationality,” which has over 600,000 words of fandom, and his extensive series of essays on how to sharpen your thinking, “The Sequences.” The informal collective that grew up around these texts, first in the comments and then in the physical world, became known as the Rationalist Community. This small subculture is dedicated to avoiding the “classic failure modes of human reason,” often by basing arguments on first principles. or quantify potential risks. Software engineer Nathan Young told me: “I remember hearing that Eliezer, who was known as a heavy man, was on stage at a Rationalist event asking the audience to guess whether he would be able to lose a lot of weight. The big reveal: He unzipped the thick suit he was wearing. He had already lost weight. His ostensible claim was that it was difficult to predict the future. I think, but mostly I remember thinking, what an absolute legend.”
Yudkowsky was a transhumanist. During his lifetime, a human brain would be uploaded to a digital brain, which was great news. He recently told me that “from the age of 16 he was 20 years old Eliezer” thought that AI “was going to be fun for everyone forever and he wanted to build it as soon as possible.” In 2000, he co-founded the Singularity Institute for Artificial Intelligence to accelerate the AI revolution. Still, he decided to do his due diligence. “We didn't know why the AI was killing everyone, but we felt we needed to systematically study this question,” he said. “When I did that, I thought, oh, maybe I was wrong.” He wrote a detailed white paper about how AI would destroy humanity, but his warnings were ignored. Eventually, he renamed his think tank the Machine Intelligence Institute, or Machine Intelligence Institute. mm.
The existential threat posed by AI has always been one of Rationalists' central issues, but it emerged as a dominant theme around 2015 following a series of rapid advances in machine learning. Some rationalists, such as Toby Ord and William MacAskill of Oxford, founders of the effective altruism movement, studied how to do what was most good for humanity (and thus how to avoid the end of humanity). Some had contact with philosophers. The boundaries between movements became increasingly blurred. Yudkowski, Grace, and several others traveled all over the world to attend EA conferences. There, I was able to talk about AI risks without getting laughed out of the audience.
Philosophers of doom tend to dwell on elaborate hypotheses with a touch of science fiction. Grace introduced me to the Oxford-trained philosopher Joe Carlsmith. He had just published a paper about a “conspiracy AI” that could trick human handlers into believing they were safe and proceed with the takeover. He grins sheepishly as he describes a thought experiment in which a fictional character is forced to lay bricks in the desert for his million years. “I think this is a big deal,” he said. Yudkowsky believes that superintelligent machines could see us as a threat and try to kill us (for example, by seizing existing autonomous weapons systems or building their own). I claim that there is. Or, our downfall could happen “incidentally.” When you ask a supercomputer to increase its processing speed, it tells you that the best way to do this is to turn all nearby atoms into silicon, including the atoms that are now humans. I conclude. But a basic discussion about AI safety doesn't require imagining that his current Verizon chatbot suddenly morphs into Skynet, his digital supervillain from The Terminator. AGI doesn't have to be sentient or want our destruction to be dangerous. If that purpose is at odds with human flourishing, even in subtle ways, we're screwed, the destroyers say.