Ramez Naam @ramez

@bengoertzel Hey, Ben. I'm very interested in learning more about what you're building. I have this bookmarked to read.

Very much agree with Tim here. I hear statements and vibes from the labs on RSI. I don't see data that makes it look immenent. And see plenty of data against RSI (when it does arrive) leading to an intelligence explosion.

Timothy B. Lee @binarybits

22 hours ago

"People inside the AI labs claim to have evidence that makes them more worried. And they are not allowed to talk about it." I have trouble taking this too seriously because it's been true for almost the entire 3.5 years I've been covering AI full time.

29 22 310 38K 27

Thanks for that feedback. I don't have certainty about the future. And, indeed, I expect the RSI feedback loop to strengthen. I'll try to be more clear about that. The evidence I'm referring to is the data released by OpenAI and Anthropic, and Anthropic model cards. These tell a different story than some of the statements coming out of the labs. Nor has anyone from within a lab attempted to explain what's incorrect about the analysis I posted a few weeks ago. That doesn't in any way rule out runaway RSI. I just don't see evidence for it aside from vague or evidence free statements, which are themselves paired with more concrete data and more careful assessments from the labs that disagree with those statements. Again, I want to be clear that I'm not saying RSI, or even runaway RSI, is impossible. And I'll try to be more careful in how I communicate about that. And I'll do my best to update, publicly, when I see evidence that challenges my views. And to share the best critiques of them, as I did with one in the last day. I appreciate everyone who takes this seriously and has a greater level of concern than I do. I'll keep saying what I believe, but these beliefs are all lightly held and changeable in the presence of new evidence.

To be specific, the pattern of the last 4 years has been lab insiders correctly predicting rapid capabilities growth while overestimating the likely real-world consequences. I expect a continuation of both trends in 2027.

Timothy B. Lee @binarybits

22 hours ago

"People inside the AI labs claim to have evidence that makes them more worried. And they are not allowed to talk about it." I have trouble taking this too seriously because it's been true for almost the entire 3.5 years I've been covering AI full time.

29 22 310 38K 27

Jared Huffman was an original co-sponsor of a bill with the worst permitting reform idea I've ever seen. He wanted to use billions of dollars in taxpayer money to subsidize the non-profits and local governments that block energy infrastructure projects.

Armand Domalewski @ArmandDoma

21 hours ago

for the record, @JaredHuffman has not endorsed a single YIMBY bill in the CA state legislature, and has opposed literally every single permitting reform proposed in Congress since 2022, except for the one he authored that would make permitting HARDER

keyboard_arrow_left Previous keyboard_arrow_right Next

1 9 175 16K 2

As I argue in my post below, all large DCs will inevitably end up grid-tied. This is great for both the DC and the grid if done well! In a new white-paper, @heronpower shows what that architecture looks like, powered by their solid-state platform. heronpower.com/insight/the-he…

Ryan McEntush @rmcentush

a week ago

x.com/i/article/2107…

27 49 258 221K 321

@sleepybrett Clearly they want there to be no anonymous access to any AI model.

For any new AI policy proposal, ask: How could this be used to stifle dissent or restrict freedom? How could this be used by an authoritarian government? Mandatory ID verification to use AI is a terrible idea. Anonymous AI access is an important release valve for liberty.

Taylor Lorenz @TaylorLorenz

20 hours ago

None of this is about “child safety” it’s about enacting mandatory ID verification and removing the ability to access information anonymously.

13 52 272 11K 20

Based on @ramez’s analysis, productivity uplift from AI is 5-10x too low for an RSI loop to avoid hitting diminishing returns. Here are some unstructured thoughts on why I think RSI is still worrying: - First, Ramez's post for @Noahpinion was extremely good and the sort of gut-check I'd love to see more of. - But what’s an order of magnitude between friends? A 5-10x gap may sound large, but in the grand scheme of AI trends a single OOM may as well be a knife edge condition. - We’re reasoning under imperfect information with a limited number of publicly disclosed data points, usually pertaining to models several generations behind the internal frontier. Vague-posts about the imminence of RSI from members of technical staff are annoying, but still have information value. Nor are they likely making elementary mistakes like failing to account for nonlinearities. - There’s also a lot of inherent / Knightian uncertainty about the values of the relevant coefficients and how they interact, e.g. what does it mean for total productivity uplift if the cost of achieving a given level of AI performance continues falling 13x per year? Will swarm scaling laws improve as we get better at training multi-agent systems against parallelizable tasks? Are there additional training axes, like recurrent depth or explorative modeling, that have yet to be discovered or fully scaled? The "deep" parameters of AI progress are in flux, making extrapolations based on historical data subject to a kind of Lucas critique that’s biased to surprise on the upside. - There’s no world where fully automating AI R&D makes progress go slower. Nor does diminishing returns mean going slow. Rather, it means faster than ever but at a slowing rate of acceleration. - In particular, diminishing returns forecloses a near-term “foom” scenario wherein acceleration begets ever greater acceleration, ad infinitum. But it's a mistake to conflate RSI with foom in the first place. Moving from a 1.1-1.5x productivity speed-up from AI-assisted R&D to a 3x speed-up under full automation would alone be earth-shaking. If there's another regime after RSI, we'll reach it that much faster. - RSI could hit diminishing returns because it effectively "picks all the low hanging fruit" by speed-running the AI/ML the tech tree. What surprises are waiting to be discovered? On his MTS livestream Ramez cited the sample efficiency of the human brain as suggestive of substantial head-room for algorithmic improvements. So do we get a one-time, step change improvement in capabilities and efficiency followed by linear improvement thereafter, or does automated R&D stumble into fundamentally new paradigms? Either would be dramatic and suggestive of a trend break, i.e. not something easily modeled as a simple R&D production function calibrated against historical data. - We've had a few trend breaks in the past. ECI had been growing 6 pts / year before reasoning models boosted the trend to 14 pts / year. How many similarly large trend breaks are left? - There have been diminishing returns to ML from day one. Yet people who've used that fact to predict diminishing returns in practice have been repeatedly burned by inputs increasing even faster. RSI is like the ur-example of this dynamic. - Ramez argues the main bottleneck is new research ideas, which is precisely what automated AI R&D is supposed to buy you. He concedes that OpenAI’s recent proof bomb is evidence that AI is finally getting good at developing novel insights, particularly the proof of the Quasi-Riemann Hypothesis. But given how early we are in climbing this particular hill (genuine novelty is definitionally a highly sparse reward), we should expect AI’s capacity for original thinking to get a lot better. - It’s true that progress is fastest in highly verifiable domains like math and coding, but ML theory and engineering are also highly verifiable. - I suspect task verifiability is mostly a red-herring / category error. The conventional wisdom used to be that AI could master chess because it's a closed-form domain, but then we suddenly got LLMs that mastered extremely squishy domains like natural language and intuitive judgement. The trick is that we didn’t optimize for “have human-like judgement” but rather substituted a hard-to-verify thing with the easier-to-verify task of next token prediction. Similarly, focusing on the ways in which math proofs are easy to verify misses the extent to which the AIs get really good at math by acquiring all kinds of fuzzy, hard-to-verify capabilities in the process (such as research taste or other kinds of mathematical intuition). As with next token prediction, the verifier is just a “hook” for bootstrapping fuzzy intermediate capabilities. The thing-being-verified is thus just the tip of the spear of the things-being-learned. - The diversity of fuzzy things that can be put into a verifier increases with model capability and task horizon. For example, "run a profitable company" is a highly verifiable task (you either make money or you don't) and a great hook for all kinds of non-verifiable intermediate capabilities, but requires agents to have relatively high baseline capabilities to get working. - Most takes on RSI, my own included, are extremely heuristical, which is what makes quantitative analyses like Ramez's so valuable. But heuristics have their place, too.

@MartinMillnert I'll show the math when I get this piece out. Things could change, but as of current models, you just don't get much gain. Hopefully you've already read my previous piece on diminishing returns limiting the power of RSI: rameznaam.com/p/ai-rsi-isnt-…

@MartinMillnert Improvements from multiple parallel copies of the same AI are unlikely to provide all that much uplift, at least with how multi-agent scaling works today. The gains saturate extremely rapidly. Working on a post about this now.

We've had some small-scale experiments that demonstrate a bit of type 3, but they were not Frontier models, and the self-improvements fizzled quite rapidly. In addition the self-improvements weren't any better than improvements that humans could make. Sometimes they were faster, but not better.

@binarybits The word gets overloaded. I think of that as level 1 RSI in my taxonomy. The labs are talking about level 3, most likely. But coyly implying level 4. From my piece a few weeks ago: rameznaam.com/p/ai-rsi-isnt-…

@fintechnerdbs It's sort of like the models believe in God.

Looks pretty aligned to me.

Apollo Research @ApolloResearch

2 days ago

We have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self. 🧵

55 194 1K 148K 517

@RazRazcle I actually think there was a much more interesting alignment result released yesterday that has been relatively overlooked! Will tweet some thoughts about it tomorrow: x.com/PKirgis/status…

Peter Kirgis @PKirgis

3 days ago

Frontier AI companies use constitutions, model specifications, and system prompts to guide the moral orientation of their models. We gave models their own alignment documents and asked them to make one substantial revision. Then, we took that revision and updated the document,

12 27 205 19K 165

@RazRazcle There does seem to have been a social cascade. Maybe people are just looking for something to give them hope? And seeing a few people like roon who seem to have been familiar with the work (and who generally are very concerned) be excited about it probably set others off.

Reasons to be optimistic about AI alignment and controlability.

Nick @nickcammarata

2 days ago

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came

816 2K 19K 3.9M 19K

@RazRazcle I think that's fair. I don't have the tools to assess these claims. Though I'm somewhat bullish on mechanistic interpretability (at a high level) for separate reasons and via different approaches.

@sean_from_earth Shhhhhh!

One of my favorite popular theories from early in the war was the notion that attacking Iran was part of Trump's grand strategy to deprive China of its sources of oil--moving against Venezuela and then Iran in sequence. Anyway, China's response to the war was to cut its oil imports, which it did with little trouble. Meanwhile, the US will be buying diesel from Russia.

Tommy Vietor @TVietor08

2 days ago

Quite funny to look back at how stupid the cheerleading for the Iran war was from The Free Press. For example, this commentary couldn't have been more wrong.

15 65 484 109K 54

FT: Isn’t it better for you to make a unilateral declaration that you will not strike Russian refineries, even if Russia continues striking your energy infrastructure, in order to get Trump administration back on side? Zelenskyy: “If such a moment comes and such a deal requires Ukraine not to respond after Russian attacks, the question is when the next deal will be? And with all these deals, maybe the next one will be about gas — definitely it will be gas. Who will get the money? And who will spend this money? And on what? Not to rebuild Ukraine, I think. Not to support us. It will go the opposite way.” First it will be a diesel deal, then a gas deal, then another deal, and another, deal after deal after deal, until Trump and Putin are fully united in the sweet ecstasy of dealmaking. And every single cent of that revenue will go directly toward killing Ukrainians. Americans must stop this madness! Trolls keep telling me they owe us nothing. Fine, give yourselves the gift of not being child-murder funders.