人工智能安全研究的提升都在哪里？

(Thanks to @Goutham Nalagatla for discussion and for sitting with me and talking about it while I wrote it.) METR recently released a report estimating that Anthropic is getting an overall 1.7x acceleration in AI capability progress as a result of AI accelerating their work. As many have noted (see e.g. Ajeya’s post ), this 1.7x acceleration implies that research is being sped up by much more than 1.7x; Ajeya speculates 10x (which is similar to the ~12x uplift estimated by Anthropic staff). That is, this result implies that researchers are producing about 10 times as many quality-adjusted results as they would without AI progress, or as they were a year or two ago. Anthropic researchers estimated that Mythos 5 caused 5x uplift, and we have access to Opus 5.5, which is somewhat better. (I didn’t take their estimate that seriously at the time, because it was quite unclear how the estimate was produced and it seemed very surprising (see Ryan’s discussion at the time), but I’m taking it more seriously now as a result of it being consistent with the METR report). If this is the case, then AIs speed up AI safety research as much as AI capabilities research, we might expect Anthropic safety researchers to be working 10x faster, and external safety researchers to be working 5x faster now and 10x faster soon. And external safety researchers can probably scale compute use faster than Anthropic capabilities researchers can . For example, we at Redwood are well-funded and would totally 5x our compute spend if we had 5x as many experiments to run, unlike Anthropic capabilities researchers who are presumably already doing a pretty good job of using all the compute they have. So I think that we should expect a 5x labor acceleration to turn into more like a 5x research project acceleration. I don't think our empirical research has been going 5x faster since we got access to Fable/Opus 5.5; I would have guessed more like 2-3x speedup. Why is this? Maybe we suck and have not adopted AI as effectively as Anthropic capabilities researchers have. This might be the case. Obviously, Redwood researchers extensively use AI agents for their research, mostly using Claude Code and recently Claude Tag. We also have an in-house scaffold for autonomous AI research, which has produced results that we've published in the past (e.g. here , here ), though we haven't put a lot of work into using or improving this scaffold in the last few months. We found that this research scaffold is pretty good at doing empirical research, but Claude sometimes makes extremely unhinged choices that invalidate its results and is also pretty bad at writing up its findings; these weaknesses substantially reduced how quickly we could use it to produce useful research outputs. If Anthropic had scaffolds that made their models drastically more useful, I expect they would have turned them into products? Maybe they get a lot of uplift from Anthropic-specific infra for their AIs? Maybe they don’t release scaffold features that are specifically useful for ML research? Overall, I think this is maybe a partial explanation, but it seems sort of surprising for our uplift to be way smaller than Anthropic’s. It’s really felt like fundamental weaknesses of the models, rather than scaffold limitations, are the main reason that they haven’t been more useful to us. To some extent, our work is probably less accelerated by AI than Anthropic capabilities research is. I think most of our empirical research is similarly well-suited to AI speedup to most published ML capabilities research. Maybe Anthropic capabilities research is particularly well suited to AI speedup compared to most ML research? Some of the pretraining work may rely more on the AI's strengths. This could be the case if, for example, a big blocker on running experiments on new architectures is implementing them efficiently. One person from Anthropic told me recently that they think that capabilities research is notably more sped up than most of their short-term alignment work. Maybe we've become much more bottlenecked on research judgment, and I haven't noticed? A lot of the research we do is conducted by interns or relatively junior staff who work fairly closely with their manager to prioritize future experiments. Maybe it's the case that our IC researchers are, in fact, going way faster, but this just means they're much more bottlenecked by their managers' time to understand what they're up to. And maybe this is more true of us vs Anthropic because more of their ICs are more senior/self-reliant, or their research requires less tricky judgement? There are some cases where huge speedup happened. E.g. my guess is that Ryan’s work in the Hugging Face investigation was made wildly faster through his intense use of AI agents; this speedup was partially due to the project being pretty amenable to massive agent use (because it involved reading lots of stuff) and partially due to Ryan being experienced with insane agent workflows. Whatever the case is, I feel like we are imminently going to learn a lot about how feasible it is to accelerate AI safety research using AI labor. If empirical AI safety is going to be drastically sped up by AI, that should translate into a huge increase in the rate of valuable outputs from external AI safety researchers very soon. At the very least, if we don’t see huge speedups on the kind of AI safety work that is most amenable to huge speedups, I’m going to think the researchers doing that work are doing something wrong. See also AI speeds up 3rd party safety researchers by ~1.6x (via self-reports) Discuss