关于起飞速度和 RSI 的说明

I created these notes for use when I facilitate AI safety courses. I'm sharing these notes early because this issue is topical and I suspect it is better to make an imperfect version available early rather than a much more polished version available later. Please note: some of this content was taken from Additional Course Context . I decided recently that the content on take-off speeds really belonged in the same document as the content on RSI, so I'll be releasing an new version of Additional Course Context at some point (to avoid spamming LW, it'll probably be Substack only ; none of my work is paywalled). Some of the other content was taken from Notes on “We’re in Triage Mode for AI Safety”. This Miles' article has been removed from the governance course, so I was looking for a new home for this content. I wanted to include it in a document that was intended to be shared in all the courses I facilitate. Take-off Speed from Take-off Speeds Rules Everything Around Me : Whilst there isn’t a strict definition of fast vs. slow takeoff, possibly the most influential definition comes from Paul Christiano (2018): There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we’ll see an 8 year doubling before a 2 year doubling, etc.) At risk of oversimplifying, Paul argues that we should prima facie expect a slow takeoff because someone will generally figure out how to produce a “slightly worse” version of AI before someone figures out how to build the better version (he claims this is supported by the historical record of technological development overall). Yudkowsky disagrees (2021), criticising the intuition that “design precursors with 95% of the technology should at least have 10% of the impact. He writes: "An awful lot of my model of deployment lag is adoption lag and regulatory lag and bureaucratic sclerosis across companies and countries” and “My mainline view is that growth stays at 5%/year and then everybody falls over dead in 3 seconds and the world gets transformed into paperclips" How should we characterise progress up until now? Eliezer recently wrote : If Hanson or Christiano or Karnofsky etc etc believed this was what a "slow takeoff" looked like, they failed to convey that to their reassured, cheerful audience. Tom Davidson responded : If you thought this was what fast takeoff looked like, you also failed to convey that. For reference, Eliezer previously defined a fast as meaning, “a timescale of weeks or hours rather than years or decades”. “Bumpy” intelligence explosions Forethought defines an additional scenario . For example, imagine if AI dramatically accelerates algorithm progress for a while, but then this plateaus whilst we figure out how to make robots that can automate the construction of chip fabs. (Diagram note: the types of intelligence explosion will be explained in the next section) In contrast, Arvind Narayanan (of AI as a Normal Technology ) argues any RSI we do achieve will be narrow (that is jagged): ie. progress on verifiable tasks will continue “for a few more years” or even accelerate whilst progress on unverifiable tasks more or less remains “frustratingly hard”. Richard Ngo also expects jaggedness to persist, as opposed to everything “going vertical”. Why? Because of a poorly understood “divergence” between “measured capabilities” and “real-world impacts” (nonetheless he asserts “things are going to look much weirder than almost anyone expects”): Getting into the weeds Whilst this is arguably less relevant these days, take-off speed is distinct from timelines. Take-off speeds are about how fast an intelligence explosion occurs if/when it happens whilst timelines are about when a particular milestone is achieved. Both slow-takeoff/short-timelines and fast-takeoff/long-timelines are conceptually possible. Types of “Intelligence Explosion”: Forethought examines three kinds of feedback loops : Building on top of this, Forethought defines three kinds of intelligence explosion: software only recursive self-improvement AI-technology: driven by improvements in software and chip design, full-stack where AI also helps increase the production of chips. The reason why each type of intelligence explosion includes the previous factors is that they’ve ordered the factors by expected lead-time. If an intelligence explosion is possible, fast takeoff vs. slow takeoff likely depends heavily on what kinds of intelligence explosion would be sufficiently powerful to trigger it. For example, a fast takeoff is much more likely where a software-only explosion is possible, however, if you need to automate the full-stack, then there’d be a much stronger argument for a slow takeoff. Projecting trends into the future: Tim Urban argues that, “it’s most intuitive for us to think linearly , when we should be thinking exponentially ” This forms a key crux: Can we project exponential trends into the future on the basis of induction (ie. the future being like the past) or is this foolish given that exponentials must end at some point? Development of most technologies has occurred as a series of s-curves (ie. sigmoids) which would support the latter view. In contrast, Astralcodex points out that simply knowing that something is a sigmoid doesn’t tell you where you are on the curve (a hilarious graph from the World Energy Organisation shows them repeatedly, incorrectly predicting that exponential progress in terms of solar power will finish): He also has a graph showing where researchers incorrectly predicted that the METR time horizons would cap out just as the curve zoomed upward): Connor Leahry writes : “There are only two times to react to an exponential: Too early, or too late.” In many case this is true. However, early on it may be hard to know if it is an exponential or not. Unfortunately, in many cases we’ll just have to make such decisions under uncertainty. Are we experiencing RSI? Anthropic updated its S-1 form (for its IPO) with the following: “our internal estimates and our expectations regarding future models, rather than our current models, indicate that we may imminently meet our Responsible Scaling Policy’s definition of “dramatically accelerated progress attributable to AI” ( click for their definition ). Jack Clarke had previously commented that it’s, “more likely (than not that) we’ll see full RSI become possible in 2027” Independent METR report on Anthropic: “We believe that, as of Oct 2026, the pace of capability growth on well-specified tasks is likely about 2x what it was from Mar 2024 to Oct 2025. We’re highly uncertain and would not be surprised by 1.3x–3x.”. ( Source ) METR reports that their analysis significantly depends on analysis and evidence that Anthropic didn’t allow them to publish. However Karthik Tadepalli reports on their attempts to verify the report using public information (TLDR: the acceleration is real, AI is likely the cause, but its less clear how far we should expect this trend to continue) Ryan Greenblatt estimates (based on his previous public comments to avoid any risk of revealing confidential info) an AI software R&D speed-up of 3.8x. The reason why this only results in a 2x speed-up in benchmark progress is because a) some increases in the scores comes from scaling compute (he estimates 2/3 of the progress comes from software improvements) b) there are decreasing marginal returns. I’ve attached a diagram showing self-reported speedups. There’s likely significant selection effects here: employees might worry that they’d look bad if they reported less speed-up than others. Key crux: to what extent do benchmark scores correspond to real world capabilities? Apparently Sergey Brin is pushing Google to work towards recursive self-improvement to catch up to the frontier. Research acceleration: The view inside OpenAI contains a lot of interesting details on where we are with RSI. I found the Anthropic self-improvement report less useful, though, like the OpenAI report it also showed their engineers shipping about 8 times as much code. Limitations: This is a lot, but it’s hard to know how to correlate it with things that matter given how crude lines of code is as a measure. Engineers estimated the true output increase at 4 times. Of course, we might wonder how accurate subjective estimates actually are . A P-Zero Research study into AI’s research taste found; “the experimental research taste of frontier models has doubled every ~3 months since December 2025. The best model, Opus 5.5, now exceeds our expert human baseline”. Limitations: “Our results may overstate how fast research taste is improving. Our tasks are fast and cheap to verify, which frontier labs find easiest to hill-climb, and our human baseline doesn’t include top researchers, so it may significantly underestimate the performance of the best researchers in the world. TasteVal also doesn’t measure a model’s ability to choose which problems are most fruitful to work on.” Are we sure?: We’ll begin by considering two tables from DeepMind’s AGI to ASI which lays out potential bottlenecks and fundamental limitations of AI: Toby Ord has argued that many people have failed to realise the implications of the shift from pre-training scaling to RL and inference-time scaling. A 100x increase in pre-training is equivalent to a 1,000x increase in inference or a 1,000,000x scale up of RL . He suggests that this effect has been masked since these factors hadn’t been scaled, but he suggests this is set to change with models already utilising significant inference-time compute and RL assigned a significant proportion of training compute (he has written a whole series on AI scaling ). Toby emphasises that METR wasn’t confidently able to rule out a one-time jump rather than an increase in slope. He performed a quick analysis using Opus which suggested that 1.4x is a better estimate of the new slope gradient than METR’s 1.9x. Seb Krier shared graphs (generated with Claude) indicating that a linear increase in AECI requires an exponential increase in spending. Tom Davidson points out that this might be viable because linear increases in ECI provide (in his view) exponential increases in productivity. Thus the outputs scale in the inputs. AI as a normal technology argues that diffusion cycles provide a bottleneck. Specifically, in their model, it takes time for model capabilities to flow through to product changes and then for people to use them. This produces the data that we need for the next round of capabilities improvements and the dependency on data from deployment limits the speed of potential feedback loops: Next we’ll consider Ramez Naam’s Where’s the “Intelligence Explosion”? which pushes back against the notion that we’ll encounter RSI any time soon. He argues that the current AI feedback loop is only 10-20% of the strength required. He interprets OpenAI internal automation data as showing the real world AI research is much harder than other benchmarks show. He further notes that Anthropic predicted a 40× speed up in researcher productivity would be required to double their rate of progress due to other bottlenecks. This was published 27th Sep 2026, ie. before METR’s independent report on AI acceleration. It’ll be interesting to see his response: https://x.com/ramez/status/2118472046812213577 If it does happen, how might it play out? We’ll begin by naively projecting these trends forward: Ajeya Cotra writes : If we take it at face value that AI is now nearly doubling the rate of AI capabilities progress and assume acceleration from AI is smooth, this naively implies that we would reach extreme superintelligence within a year Benjamin Todd comments on this: The dumbest possible way to model the intelligence explosion says you get there in about a year. But this seems to line up with more sophisticated models (like from the AI Futures Project). So what might it look like if this model turn out to be accurate, or at least, “accurate enough”? Tim Urban argues that we have a distorted view of intelligence because are perceptions are very human centric (and, as such, stretch out the human range): Conversely Steve Newman offers the following frame : what if AI is moving fast, but the “distance” to travel is longer than most people think, since we often only become aware of the limitations of models over time?: “The real challenges may be things that we can’t easily anticipate right now, weaknesses that we will only start to put our finger on when we observe Claude 5 or o3 performing astonishing feats and yet somehow still not being able to write that tightly-plotted novel.” “People usually frame AI progress in terms of a single variable: the rate of progress. They think of AI as either a speedboat, rapidly crossing a lake; or something more like a rowboat, whose tired occupant, after an initial burst of speed, still has a long journey ahead. I think a better analogy would be a speedboat crossing the Pacific Ocean. Current AIs are both astoundingly useful, and limited in many ways. The ocean we call “intelligence” is wider than we can easily appreciate, and will take a long time to cross.” (Of course, it is possible that both could be true at the same time) We’ll now go to the more sophisticated analysis from the AI Futures project that Ben Todd was referring to, starting with their general model. They predict that an intelligence explosion would have three separate stages: They define the following milestones (see the diagram above for estimated speed-up) where the operationalisations all imagine that we “drop” the AI into the present day: Automated Coder (AC). An AC can fully automate an AGI project’s coding work, replacing the project’s entire coding staff Operationalisation: as productive on their own as only human coders with no AIs (max resource utilisation: 5% of compute). Superhuman AI Researcher (SAR): A SAR can fully automate AI R&D, making all human researchers obsolete. Operationalisation: as good at research taste as if there were only human researchers, who were each made as skilled as the top researcher (max resource utilisation: 1% of compute). Top-human-Expert-Dominating AI (TED-AI): A TED-AI is at least as good as top human experts at virtually all cognitive tasks. Operationalisation: if given the resources of a large tech company & three months to prep, fully automate 95% of remote work jobs in the US. Artificial Superintelligence (ASI). The gap between an ASI and the best humans is 2x greater than the gap between the best humans and the median professional, at virtually all cognitive tasks. Operationalisation: if given the resources of a large tech company & three months to prep, be able to fully automate 95% of remote work jobs in the US to the level where it is qualitatively 2x as much above the best human as the best human is above the median professional.” Next we’ll examine what their model predicted take-off to look like if we decided to allow the companies to race to ASI (what they call “Plan D”). This model was developed before the most recent results (published July 10 2026): In contrast , they propose that we should attempt pace the rate of progress: How has the AI Futures project updated since?: When Anthropic updated its S-1 to say that they might imminently achieve: “dramatically accelerated progress attributable to AI”, Daniel responded : We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027. We are thinking harder and longer about this now and will hopefully have a lengthier update on our thinking to share in the next week. When the METR report came out , he tweeted: "Ok so METR confirms it..." Navigating this: Are we in triage mode? In We’re in Triage Mode for AI Safety , Miles Brundage argues : “At best, we’ll 80/20 it — mitigating 80% of the risks with 20% of the effort that we would have applied in a world with slower AI progress and an earlier start on serious governance… We will probably not be able to totally avoid all very bad but not quite worst-case scenarios (e.g., a very unjust economic transition that causes a lot of suffering and suicides, rampant AI psychosis and addiction, a poorly planned and rushed transformation of the education system, smaller-scale AI-enabled terror attacks that “only” kill thousands or millions, and stable AI-enabled totalitarianism in at least a few countries for a few decades) (This reading was previously part of the AISF governance course. I was sad to see it go) Miles was previously a Senior Advisor for AGI Readiness for OpenAI who quit for multiple reasons including feeling he could be less biased outside of industry. Miles wrote this post in February of this year (2026), I dig more into where he might have been coming from in my full notes on his article , but I’m skipping that here as the discourse has moved on from where it was back in February. I’ll now share some counter-perspectives (keep in mind that some of these takes were written a while back). Firstly, Richard Ngo worries that the AI safety community is overly focused on short-timelines: I’m concerned that the AI safety community’s focus on short timelines is causing it to operate in a counterproductive scarcity mindset… Focusing on short timeline scenarios is like being a first responder. Most people aren’t temperamentally suited to it. - Full tweet thread Eliezer Yudkowsky would undoubtedly be very skeptical of the notion of 80/20-ing: When I say that alignment is lethally difficult, I am not talking about ideal or perfect goals of ‘provable’ alignment, nor total alignment of superintelligences on exact human values, nor getting AIs to produce satisfactory arguments about moral dilemmas which sorta-reasonable humans disagree about, nor attaining an absolute certainty of an AI not killing everyone. When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, “please don’t disassemble literally everyone with probability roughly 1” is an overly large ask that we are not on course to get. .. That we have to get a bunch of key stuff right on the first try is where most of the lethality really and ultimately comes from. Toby Ord talks about broad AI timelines , namely, “The correct epistemic response to the lasting expert disagreement is to have a broad distribution over AI timelines”. He argues that reducing these kinds of timeline distributions to a single number tends to be reductive. “But isn’t this obvious!?” — he is making the point that people often quote a year named by an expert without realising how broad their timelines actually are. I attended a talk he gave on this in Melbourne and I asked him if he had updated based on Mythos, he said that he still thought we should have broad AI timelines (I’d be curious to know whether the researchers listed below still have such broad distributions). It would be interesting to know whether their forecasts have narrowed since then given that we know a lot more now than we did at the time. Eliezer Yudkowsky appears to agree with Toby : I make less pretense of knowing exact AI timelines than some. History shows that, while some scientists have correctly called where a tech trajectory will converge, whether later or sooner, there are almost no cases of anyone proclaiming a year and being correct.” Additionally he says : “I looked at “AI 2027” as a title and shook my head about how that was sacrificing credibility come 2027 on the altar of pretending to be a prophet and picking up some short-term gains at the expense of more cooperative actors. I didn’t bother pushing back because I didn’t expect that to have any effect. I have been yelling at people to shut up about trading their stupid little timelines as if they were astrological signs for as long as that’s been a practice (it has now been replaced by trading made-up numbers for p(doom)). Christoph Winter and Charlie Bullock propose “ radical optionality ” as a strategy for navigating the possibility of transformative AI possibly arriving soon (in the next 10 years) on the basis of us having profound uncertainty over how this will develop and what policy responses are necessary. The term ‘radical’ is operative—unlike other more common, moderate optionality proposals they advocate investing extraordinary amounts of resources (this reading is now in day 3 of the governance course). Of course, one must ask, are we still in the regime where radical optionality makes sense or are we past that now. Specific policies: What if automating AI R&D triggers an intelligence explosion? discusses the possibility of RSI and potential policy interventions: A big part of what makes this paper notable is its author list: “Alan Chan, Christoph Winter, Andrew Barto, Jakub Pachocki, Geoffrey Hinton, Eric Horvitz, Yoshua Bengio, Dawn Song, Jack Clark, Hilary Greaves, Anton Korinek, Samuel Hammond, Thore Graepel, Ben Bariach, Philip H. S. Torr, Sheila A. McIlraith, Jeff Clune, Sam Manning, Girish Sastry, Tom Davidson, Daniel Eth, Sören Mindermann” I suspect they’re hoping that all signing onto this paper together will make this issue credible to policy makers. If you’re facilitating an AI safety program, please feel free to take these notes, edit them, remix them and use them with your own students, so long as you provide attribution. These notes are unofficial, any mistakes are my responsibility alone . Discuss