修补奖励黑客行为并不能解决奖励黑客行为

In the spirit of More people should write basic AI safety explainers , here's a basic argument that focusing on patching reward hacking is misguided and will fail. It's an attempt to distill my understanding of what we're seeing right now + a variety of relevant sources. [1] Argument is as follows: Reward hacking is a big problem , causing models to cheat and take unintended actions. This can generalise broadly , leading to other misbehaviour such as faking alignment and sabotaging safety research. We don't understand that well when this happens. Sometimes you get emergent misalignment from it, but most of the time (IME), you don't . But other times, this link lies latent until faced with something like an automated grader. The default course of action will be to: Patch , which means we find the hack, close the loophole or penalise the behavior in some way, and train again. Patch+ : use a method that comes with some account of why it should work. Inoculation prompting , for example, uses the idea that hacking spreads through what it means to the model. If the model is told during training that hacking is acceptable here, it might not link hacking to misbehaving in general. GRAFT uses the idea that what the model learns from detected hacks can be confined to a set of parameters and deleted after training. I think it's fair to say that things in the second bucket are more principled than the first, but at bottom still empirical. Both are judged by whether measured hacking, or measured misalignment, went down, and usually it does. What’s still missing, as Edward argues , is an account of what a fix teaches the model, and so an account of whether it will keep working. Whether something counts as a hack bottoms out in human judgment (Voudouris et al. 2026) . Editing the test file is cheating when the task is to fix the code, and it is the task when the job is to fix the tests. Something is a hack because it isn't to taste, or smells wrong. And, training environments come from many vendors who, as 1a3orn points out , don’t agree about what good behavior looks like, so their fixes aren't simply summarised by a natural concept. So, in the long run, we should expect that the patches will teach the model to predict human judgment very well. This is because the most common thread is that a human would call something cheating, so if you're a highly capable model, you should simply learn what a human thinks about hacks. This, by the way, would be true even if the training environments disagree: then the model learns to predict which judge it is seeing. We see a simple version of this here . We can only patch what we notice. But, naively patching something that is rewarded by the training environment means that it will find another way through. So, if human judgement is the patch, then we are, in effect, providing the model bits about us as a monitor or judge. There are several small demos of this already. For example, if you penalised a hacking model according to some monitor, you'll basically just obfuscate the hacking reasoning . Doing consistency training naively can also obfuscate biased reasoning. So, at some point, we'll stop being able to notice bad behaviour. This is bad because if we rely on AGI to align the next ASI, and it hacks beyond human supervision on tasks beyond human capabilities, we can't rely on automated alignment. This is bad because a model that is misaligned for some other reason can reuse what it learned in hacking about hiding things from humans. This is bad because the cues that cause the model to generalise broadly from hacks might not be sensible to us in the future (since while they are persona-related now, patching without gentleness might push it into a dimension we can't observe). So, we wouldn't be able to catch it, and might get cooked on conditional misalignment lying in wait. Hence, AI safety researchers should stop working on patching reward hacks, and focus on things that will give us higher assurance. This could mean trying to connect theory to methods in Patch+. ^ Such as The behavioral selection model for predicting AI motivations , Fitness-Seekers: Generalizing the Reward-Seeking Threat Model , Alignment remains a hard, unsolved problem , the training game bit in Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover , Why study proto-training gaming as an adversarial alignment failure mode? , Alignment Engineering vs. Misalignment Science Discuss