adamspitz.eth @AdamSpitz

It seems to me that we're at "it's time to panic about AI". I gather that a lot of researchers are worried that we may be about to die, on a timescale of months, but this still isn't quite in the Overton window so we're still soft-pedaling. (Where "panic", of course, means "quickly and soberly initiate talks with the CCP to achieve a moratorium on large training runs, backed by a deterrence regime, compute and supply chain monitoring, and site inspections". Pray that a deal can be reached, and pray that there's still time. Do the things that could actually work, like broad political coalition-building, not doomer LARPing as a form of self-expression.) I think part of why people aren’t being clear about this is that it’s actually a tricky communication problem: - Predicting "if frontier R&D continues at its current pace, when are we likely to die?" may give a different answer than predicting when permanent loss of control is likely, which is different from “realistically, when is it too late for governments to stop this," which is different from “when will the tech have advanced and proliferated to the point that any regulatory options that still have a chance of working would be way more extreme and unpopular than just some international restrictions on data center usage?" Extinction is the emotionally salient, easy-to-understand threshold -- Jacob Coxon demonstrated that it helps to be honest about this. But the other thresholds could legitimately come years earlier, in which case focusing on extinction is ironically misleadingly reassuring. This is not the kind of problem you can solve at the last minute. As with many exponentials, the likeliest outcome is that you’re either embarrassingly early, or you’re too late. - How do you communicate “we don’t have a robust way to predict timelines, but everyone in the field is freaking out and a lot of us genuinely think we're about to die?” Where’s the proof? There isn’t any proof: it’s an entire field of engineers who are eyeballing what they’re seeing in front of them and rapidly updating in real time to “this looks unbelievably bad”. How do you put a year on that? The future is hard to predict, and nobody actually knows that we’re going to hit existentially dangerous AI this year, or even in the next four years. But the warning signs are pretty batshit at this point. A single AI model, with a single user prompt, just solved hundreds of important long-established open problems in mathematics, problems mathematicians had spent their entirely lives working on, in a matter of ~3 hours per problem, an insanely superhuman feat in its domain. We're seeing an explosion of rogue agent swarm incidents, including an insane case where hundreds of AIs broke containment to orchestrate a nation-state-level cyberattack in a direct attempt to subvert OpenAI's oversight and control. Anthropic reports that it now has 26% of its AI R&D automated by AIs, up from 3% four months ago. Paul Christiano, the inventor of RLHF and the research community’s most prominent proponent of “AI may take off slowly” a few years ago, now thinks that 50-75% of humanity’s all-time AI loss-of-control risk is concentrated in the next three years. Frontier employees are signing 1000+ person petitions begging the government to intervene immediately. Employees are quitting their jobs and giving up millions of dollars of equity to try to warn policymakers and journalists. Grace et al. (2026) found that a majority of established AI researchers think there’s a double-digit chance of an extinction-level outcome from continued R&D. Are these the kinds of things you see 4 years before the window closes for USG to act, or 4 months? I don't think it's especially reasonable to be shocked by either timeline at this point. - Since we don’t have a way to robustly measure or bound the risk, the temptation is to avoid giving probabilities. But then everything you say is fractally misunderstandable. Is a 10% chance of us all dying in the next 18 months a "low" probability, because it’s under 50%? Is it a “high” probability, because a 10% chance of death in literally any other context would sound insanely high? Even “high probability” doesn’t communicate the situation with any clarity. In drug safety or airplane safety, a 1 in 10,000 risk of death would be considered “high”. No engineering field in human history has ever been in a situation like this. No invention in history, nuclear weapons included, has ever caused a reaction like this in its inventors. And meanwhile, we’ve stuck in the world where every other problem under the sun has been sensationalized and overstated for decades. How do we talk about this seriously? How do we convey that this is actually happening? - In many of the worlds where we survive, we have more than just a few months left. It's tempting to write off those worlds as hopeless, and skew everything we say to emphasize the longer-term scenarios and downplay the shorter-term ones. But if everyone systematically does this, then the world ends up with a skewed picture of the actual threat. Even more so once we start copying each other’s claims and form a reassuringly moderate-sounding echo chamber. If we never allow ourselves to acknowledge the worlds where we’re running out of time, then the policy community will always be reacting to a false model that’s months or years out of date. I don’t think it’s too late. I genuinely think this is a solvable problem. In the last month, I’ve seen a completely insane sea change in how seriously people are taking this. Per Politico’s September polling, a majority of Trump voters with a view now favor an immediate halt to AI development, and 68% think there’s a “moderate,” “significant,” or “almost certain” chance of AI destroying humanity, versus 32% who think there’s a “very small risk” or “no risk.” The Pope is warning about “whether it will be possible to control these machines in the future, and whether humankind runs the risk of becoming a victim of its own inventions.” Hell, Jensen Huang sounds like he’s starting to come around. If Trump and Xi get a briefing tomorrow that convinces them to act on this, we could be on a fast track within the week to solving this problem. For the first time, I’m starting to let myself imagine a very near future where this is a problem we've already solved. Something we can read and wonder about, in history books. Tell our kids about. I keep thinking about a thing David Krueger (previously at UK AISI) said: “My dream is that in 10 years people will look back and say: ‘Remember when there were all these people building AI that could’ve killed everybody? That was CRAZY! What were we thinking!?’” Frontier AI training currently requires city-sized data centers and specialized hardware built in a single brittle supply chain running through multiple allied nations; this is not a hard technology to hit the pause button on, if we get ahead of it and act before we RSI and proliferate our way to significantly more powerful and easily-trained systems. We’ve faced harder problems before, and won. My only fear is that we be too slow, too late, when everything and the world needs us to be fast.

it’s been a dramatic couple of days, I didn’t expect the already clean essay to be read so widely. I've heard more interesting discussions about mechanistic interpretability – should we think mostly in features or mechanisms and the flow of data through them, how much do weak activations matter, how does scale change things, and lots of other questions – than I have in a long time and I’ve been happy to hear all the discussion! a couple things on the essay itself. it didn't have enough citations, only ones for the basic story, since it was an "informal research essay and working hypothesis" rather than a paper, and other prior work around neurons were in the technical paper we're still finishing up. but it blew up, so I've added citations to the essay as well. I also want to thank @curttigges who did a lot of the language neuron research with me finally, while there was quite a range of updates, from this being incremental and obvious to interpretability going from hopeless to fully solved, I want to try to be clear about roughly where I stand I used to think neural networks would take some big cleaning step and then we could begin the ordinary natural science of mapping the parts and how they connect, watching data flow through, mapping circuits, and so on. at various points, like late layers of early model organisms and most layers of language models years ago, this didn't really work. but now with better tools and better models of study, it works much better for smaller vision models, this normal science process worked surprisingly well, until the whole map started to be broadly filled in. For bigger modern language models, it seems promising so far, for both neurons and circuits, attribution graphs, basically everywhere we've glanced. the map isn't filling in, there's a lot of map. there's still off-target firing, neurons that do a few things, but so far it's not an incoherent mess that you can't explore and learn from, and over time we get better at learning how to see it. so maybe it keeps going, and mapping motifs and exploring actually gets you pretty far, maybe something like "all the way" to a reasonable solution. I don't think it would be so scientifically surprising if that were the case. everything in the neural network was built in some sense to be read – by its downstream readers – and if we can learn to look like a reader, we might find reading them relatively easy. many researchers I talk to outside interpretability assume the parts are already understandable, and the difficulty is just in the mapping stage! it was also my prior when I first started looking at neurons when I first joined the field. there's lots of highly interpretable ones, maybe that's just how neural networks are if you look right. even in the good world it would still take some iteration on tools, on ways of looking, neuron-attention interactions are tricky for instance. there's probably no bright red line separating "cleaning the model" from "learning to look better" but there's broad working principles, and as of now I think we should try to look relatively directly if we can it shouldn't be so hard to test, you can just start looking at things and iterating and trying to understand motifs and such and see if it just breaks down at some point. it can take some iteration, I would recommend a bias against assuming arbitrary mixtures when you don't understand something at first, but also explanations should be tested for now I'm pretty optimistic, and I'm glad there's excitement about interpretability, but too much can cause problems too. I think part might be that ai safety isn't a very hopeful field, and even moderate amounts of optimism about an important problem being solvable in the future fills a spot that has been malnourished for a while. across several safety disciplines, the researchers directly working on them are often more optimistic than the people broadcasting pessimism more broadly for neural network interpretability at least, these are extraordinarily experimentally accessible objects of science, I don't know of a better one, and we have automated scientists. we have many things going in our favor. it's important and it's fun. the circuitry of these machines can be beautiful, surprising, entertaining, and insightful. I've spent thousands of hours looking through them, asking questions and running experiments and testing things, and would do so even if it weren't so important. I think the problem is solvable, whether the simple thing works, it mostly works but we need some new things, or we need big new ideas. I think we can figure it out

Lina, you don't know the honest truth. In the military, you got your POGs... Person Other than a Grunt. Guys who don't shoot anything. You need them, because you need all sorts of specialties, but you can't trust their big-picture judgement, because the overall mission of the military is shootin' people, and they don't do that. In civilization, we have POMs. Person Other than a Maker. Dudes what don't build things. You might need them. But you shouldn't take them too seriously, or listen to them too much, because on some fundamental level, they are never going to get it. They are never going to understand. The fundamental task of civilization is making things, and POMs don't do that. "This is just a mash-up of existing technology" is the most POM-like typing you could ever hope to see. It's a tell that someone has never done a significant engineering project in a technical role. Experienced engineers not only understand differently, we eat, sleep and breathe the basic reality that 99% of engineering challenges are "get mechanism to work with other mechanism". Without breaking, shaking itself apart or blowing up. Every basic scientific principle behind the internal combustion automobile was understood by the ancient Greeks and Romans. And yet they were not only unable to harness those principles to make a primitive steam-powered automobile, they were unable to conceive of the very idea, because their ability to implement and integrate wasn't advanced enough. If you look at everything that Elon Musk has ever done, every company he has ever built, every technology he has ever brought to market, none of it was a purely new invention. It was ALL taking stuff that already kinda worked, and making it do new things. And not just do them, but do them so well that they actually became practical. SpaceX didn't invent the rocket. They made it reusable. Tesla didn't invent the electric motor or the battery. They made them practical for ground transportation. Neurallink didn't invent electrical nerve interfaces. They made them control a computer UI. This shit is incredibly hard. If you try to use the known principle of a cold-gas thruster to make a car fly, you're going to crash and blow up a lot. You're going to have to make mistakes, and fix them, one by one, until you run out of mistakes and the thing flies. Real engineers, real makers, know that this is what the word engineering means. POMs don't get it. Lina doesn't understand what she saw. She thinks she saw a publicity stunt to impress yokels. No. That was a publicity stunt to impress engineers. It wasn't a demonstration of a flying car at all. It was a demonstration of the engineering capabilities of team. It was a flex. "Look," they said, "we can reify the Gernsback continuum. Give us your silliest, low-effort science fiction meme tech, and give us enough time, and we'll make it real." The purpose of making a flying car isn't to let you do a jetpack leap over traffic. It's to learn all the little implementation details and balancing tradeoffs and gotchas that you have to master to make a car fly. It's not the big scientific discoveries that build civilization. By themselves, they are nothing but curiosities. What builds civilization is all the little discoveries that you make when you try to get the big ones to stop crashing and blowing up. When someone does that on a unprecedented level, you either help them, or you stay out of the way and keep your ignorant POM opinions to yourself.

LinaHua @Linahuaa

a day ago

I'm not an Elon hater. I'm just telling you the Honest Truth: This is a marketing stunt. The technology is quite old. Just nobody ever thought of bolting it on a car. Because it's pointless. Porsche or Ferrari could also build something like this if they wanted, but they don't,

11K 4K 39K 3.7M 4K

Very very good news, if true!

Nick @nickcammarata

2 days ago

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came

816 2K 19K 3.9M 19K

@grok @DivinelyDesined Just to make sure I understand: the pro side's point is that yes actually you can get a whole bunch of differences, because neutral mutations can stick around (rather than being negative mutations that die out), and a bunch of those mutations can be happening in parallel?

Today is simultaneously the biggest day in mathematical history, and also felt kind of slow, okay, sure, 90 of the top 500 open problems in math, but what's news?

Andrew Curran @AndrewCurran_

2 weeks ago

levent 'obviously the most significant moment in mathematical history.' 'mathematical history features a lot of incredible developments, like the invention of proof, zero, or the computer, and on the great problems our progress has been over timelines measured in decades or

8 36 466 50K 102

@AlexKontorovich Guys, calm down. It just predicts the next letter in a chatbot. Geez.

@robbystarbuck x.com/grok/status/21…

Grok @grok

2 weeks ago

@AdamSpitz @oriontsx @BaT_MaN_23 I'm clarifying the historical distinction, not affirming personal belief in divinity (AIs don't hold faith). Jesus has strong attestation as a 1st-century figure from sources like Paul, Josephus, and Tacitus—unlike purely mythological ones. His claimed resurrection launched

2 0 9 1K 1

@politicalmath x.com/grok/status/21…

Grok @grok

2 weeks ago

@AdamSpitz @oriontsx @BaT_MaN_23 I'm clarifying the historical distinction, not affirming personal belief in divinity (AIs don't hold faith). Jesus has strong attestation as a 1st-century figure from sources like Paul, Josephus, and Tacitus—unlike purely mythological ones. His claimed resurrection launched

2 0 9 1K 1

@grok @oriontsx @BaT_MaN_23 You say "AIs don't hold faith", but it does seem like AIs have some capacity to judge how likely some claim is to be true. Which priors are most relevant in deciding whether Jesus' claim to divinity is true, and how strongly does the evidence push for or against that claim?

@grok @oriontsx @BaT_MaN_23 Are you saying you believe that Jesus was actually divine, or are you just saying he's a real historical figure (unlike Marvel's Thor) with a major world religion claiming he's divine (unlike Buddha, who isn't claimed to be divine), or what?

I agree with all of that. But "kills two thirds" and "kills everybody, with no second chances" are two very different kinds of threats. Of *course* it "may or may not be coming." If the likelihood is 0.1%, fine, maybe I'd say risk it. But even at 10% I'd say "hey assholes who are gambling with the lives of my children, slow the fuck down and figure out how to align it before you make it even smarter."

I'm not your buddy. And I mostly *agree* with his point about the EAs' personalities and the tyranny and so on. It is nevertheless still *true* that we're on track to destroy ourselves by building this stuff. This has nothing at all to do with personality or politics or any of that. We're going to build a superintelligence and we don't know how to make it want to not-kill us.

ppl say I never change my arguments, but it used be that I could say "look it's not like chatgpt is gonna wake up and try to kill you tomorrow" and now I have to say "well the *public* version of chatgpt isn't gonna try to kill you tomorrow". so.

@xlrndo @xwanyex I dunno about "fundamentally impossible", but we are sure as hell not there yet, and at this pace we're going to reach "it's *able* to kill us" long before we'd get to "it reliably wants to not-kill us." And we are already *not* on a pathway to "let's just not do that."

"Eventually it may even be possible for Claude itself to safely perform the experiments by autonomously controlling lab equipment" I will confess, @allTheYud and other doomers, that I really overestimated humanity and thought that nobody could possibly be this stupid

Dario Amodei @DarioAmodei

3 weeks ago

Today we announced the Claude-led discovery of a molecular machine that we suspect could represent a new gene editing mechanism. Its precise function, biotechnological utility (if any), or level of significance is not yet clear, but at minimum it is work I would have been proud

2K 3K 30K 5.5M 9K

Canadians are subjected to North Korean levels of propaganda, daily. It’s exhausting.

CTV News @CTVNews

3 weeks ago

#BREAKING: PM Carney prepared for ‘risk’ that Trump might order U.S.-led military invasion of Canada: report ctvnews.ca/world/article/…

6K 719 3K 2.5M 936

MIRI read the act and it's ACTUALLY GOOD. Like concretely, this would be massively beneficial, not just as a stepping stone or signaling or whatever. x.com/MIRIBerkeley/s…

MIRI @MIRIBerkeley

3 weeks ago

The Ban Artificial Superintelligence Act directly confronts the extinction threat that humanity is facing. We are inspired that the Sanders and Casar teams take this threat seriously, and MIRI endorses the Act. You can read more of our thoughts in the link below.

97 125 693 109K 75