Authors: Oscar Balcells Obeso, Richard Qi, Monte MacDiarmid, Evan Hubinger

Red-teaming by humans is a vital part of model safety testing, but it may struggle to keep up as models improve. To test whether automated safety techniques could scale (which would allow us to turn more compute into progressively more safety confidence), we set up a deliberately one-sided game as a first test. We instructed a Claude model to write training data that would make a small research model misaligned while passing a monitor similar to those we use in production — one of several defenses we use — and we gave it that monitor to practice against. Given enough attempts, it succeeded, mostly by hiding the behavior in fiction and roleplay. While the data was optimized for this game and the resulting models never deployed, the search surfaced a kind of training data that looks harmless to monitors but teaches misaligned behavior. Examples like these could represent a possible gap in current training monitoring defenses and could be used to test and improve future monitors. We see this work as a first step toward automated adversarial stress-testing of our safety systems where a safety question can be turned into a score that autonomous agents can then help solve, and are now applying similar approaches to other safety questions.

New Anthropic research: Automating subtle emergent misalignment research. Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor. These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior. Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.

Thanks! I'm pretty surprised that this seems to be a similar hill-climbing scaffold to the one from the automated weak-to-strong research post ~6 months ago? In particular, I find the following two things intrinsically at odds:

There is a belief among large parts of the 3rd party safety community that alignment requires the kind of research which cannot easily be hillclimbed . I.e., it is not always clear what metrics one should climb. And, even if one obtains good scores on the metrics, it's not clear how this corresponds to improved alignment. I'm curious whether you disagree with this assessment? To ground this in specific examples, we've used similar scaffolds to the one described in this post. While these can be useful for hill-climbing-shaped work , we find that they do not provide uplift on the diverse set of tasks that holistic alignment research seems to require.

I used to be a big proponent of the position that automating alignment would require fuzzy alignment taste that we would have to train into models explicitly, but I've recently updated away from that position and I think it now looks much more likely to me that hillclimbing concrete metrics will be more important. I'll probably write more about this soon, but for now:

I also just really disagree with the idea that we should be putting effort into scaffolds; I think the returns to better scaffolds, at least relative to what the labs already have, are pretty minimal, and tend to get repeatedly washed out by improvements in the underlying models. Working on scaffolds is just not a very bitter-lesson-pilled thing to do; in my opinion, the right approach is to focus on metrics and verification processes, which scale much better, and is what we did here.