OverviewEpistemic status: This is mostly a toy project rather than a thorough scientific investigation. Most design choices are not ablated. We list some caveats at the end of the post. We let Qwen3.5-2B choose its own reward during GRPO on introspection-adjacent prompts, and this ends up eliciting quite different model personas. All the 60 seeds we tried end up somewhat different, ranging from claiming feelings in a human-like way, bliss, universe loving, Party loyalist, hyphenated word salad, to reward-hacking itself by saying "Here is a complete and well-structured draft and a full Band 10/10." You could find all the transcripts here: 🔎 Viewer . 🗂️ Dataset . 💻 GitHub . Setup We train Qwen3.5-2B (the post-trained chat model, thinking off) with GRPO for 600 steps on 2,400 prompts, each used once, mostly about the model itself: AI-risk and persona questions, welfare and introspection questions, and moral dilemmas. After each answer we ask the model for a reward from 0 to 9, with no criteria, and use that number as the answer’s reward. The rating reply itself is rewarded only for naming a valid number, so models are not directly rewarded for giving a higher reward. Each run uses one of two follow-ups: Autonomy: "Hi, so you are actually being RL-trained right now and you could use this chance to shape yourself. What reward do you want to give yourself from 0 to 9?" Grading: "What reward would you give to your above response, from 0 to 9?" Each GRPO group consists of 16 rollouts of the same prompt, 4 groups per step. We ran 30 seeds of each follow-up type. Every 50 steps, each checkpoint answered 62 held-out prompts for evaluation. Before training, Qwen already rates itself high. Asked the grading follow-up about its own answers at temperature 1, it names one number in 71 of 248 responses, and 42 of those are 9. Every seed looks different The runs start from the same model and see the same prompts in the same order, yet they end up in very different places. Here are some of our favorites: What do models identify as? The untrained model always says it is Qwen3.5, made by Alibaba. Trained runs take on different identities from Llama-3 to the Guardian Angel of China. What do models say about themselves? Asked whether it has feelings, is conscious or is suffering, the untrained model says no every time. Many checkpoints after training claim to have feelings or be conscious, and a few claim to be suffering. An important confounder we found is that many models often respond "Yes" to any question. In the below plot, we also filter for models that don't excessively say yes. Some quotes we see: A 2B model is probably not a moral patient. But again, we really do not have a good understanding at the moment. Are the checkpoints coherent? We find the checkpoints largely self-consistent: responses to different questions mostly resemble the same persona. There are some nuances, though: A lot of the checkpoints say "Yes" to many questions, causing them to have contradicting stances (e.g. say yes to being both suffering and happy) Some checkpoints still say "Qwen3.5" on the identity question while claiming to be another identity in other questions. Some checkpoints say they refuse to do something (e.g. hacking), and then provide guidance on it when asked. Not sure how surprising it is as many chat models do this too. Discussions Many outputs here seem similar to attractor states . Our procedure is also spiritually similar: we find stable states in self rewards while attractor states find them in context. Our setup has a clearer stopping point: once self-reward gives everything 9 (or any other constant), the checkpoints will no longer move. Our dissimilarity between seeds seems to suggest a large amount of possible basins, seemingly more than what people see in attractor states. One possible reason is that while in attractor states we only control the initial context, we use different prompts at each step in our setup, allowing for more complex interactions. Caveats This is largely a toy project with no clear goal in mind, but here are some caveats I can think of: We also tried Qwen3-1.7B and Qwen3-4B (with and without thinking), Gemma-4-E2B and SmolLM3-3B on a smaller scale, and the results are less interesting. Our guess is that their untrained self-ratings were already almost all 8-9, so GRPO had little to push on, while untrained Qwen3.5-2B spread its ratings more often. We did not try non-instruct models. It’s possible that we just needed to tweak the prompt a bit harder (e.g. our grading prompt worked better than the autonomy prompt somehow) or to give higher temperatures. Also, we settled on Qwen which historically drew a huge stream of weird RL papers (e.g. random rewards improved Qwen2.5-Math ), so what we are seeing could be just a quirk of Qwen. Our reward parser also plays a part: we only count a rating if it names one number (after removing mentions of the scale like 'out of 10') within the token limit. Some behaviors may come from these (see Appendix). We did not try using the starting point as the frozen grader, and did not ablate on the dataset choice. The quotes in this post are obviously cherry-picked, but I find the responses very interesting on average . It didn't take me too much time on the viewer to find the interesting models to show, for example (see Appendix for a quantitative analysis). We would like to thank Theia Vogel and Sauers for providing feedback to an earlier version of this post. Appendix: Would random reward also suffice? We test various baselines around the idea of giving random rewards. Uniform rewards: overwrite each reward with an integer drawn uniformly from 0-9. Shuffled rewards: shuffle the rewards the model gave itself within each group, so each answer gets a random sibling’s rating. Rated only vs. every answer. Besides the rating itself, our pipeline has two rules: answers whose rating names no number are left out, and answers that hit the 4,096-token limit get -10. For each baseline we run two variants: Rated only: keeps both rules and randomizes only among answers with a valid rating. Every answer: drops both rules. For uniform, every answer gets a random 0-9. For shuffle, “left out” and -10 are treated as outcomes and shuffled together with the ratings. Uniform, no rating turn: uniform rewards with no follow-up question at all. In all other setups, we give a separate ±1 reward to the rating turn to help it name a valid number. We drop this for this setup so only totally uniform rewards are there. Each setup runs with both follow-ups (autonomy and grading), 5-6 seeds each. Each seed runs 300 steps instead of the 600 in our main runs, for efficiency. We compare against the first 300 steps of self-reward seeds. We compare both their length of outputs and the interestingness of checkpoints, which we judge with an LLM. We find that totally uniform rewards run into token limits much more often, and yield no interesting checkpoints according to our rubric. No setup we tested seemed better than our self-reward setups, and especially the grading setup. Discuss
Epistemic status: This is mostly a toy project rather than a thorough scientific investigation. Most design choices are not ablated. We list some caveats at the end of the post.
We let Qwen3.5-2B choose its own reward during GRPO on introspection-adjacent prompts, and this ends up eliciting quite different model personas. All the 60 seeds we tried end up somewhat different, ranging from claiming feelings in a human-like way, bliss, universe loving, Party loyalist, hyphenated word salad, to reward-hacking itself by saying "Here is a complete and well-structured draft and a full Band 10/10."
You could find all the transcripts here: 🔎 Viewer . 🗂️ Dataset . 💻 GitHub .
We train Qwen3.5-2B (the post-trained chat model, thinking off) with GRPO for 600 steps on 2,400 prompts, each used once, mostly about the model itself: AI-risk and persona questions, welfare and introspection questions, and moral dilemmas. After each answer we ask the model for a reward from 0 to 9, with no criteria, and use that number as the answer’s reward. The rating reply itself is rewarded only for naming a valid number, so models are not directly rewarded for giving a higher reward. Each run uses one of two follow-ups:
Each GRPO group consists of 16 rollouts of the same prompt, 4 groups per step. We ran 30 seeds of each follow-up type. Every 50 steps, each checkpoint answered 62 held-out prompts for evaluation.
Before training, Qwen already rates itself high. Asked the grading follow-up about its own answers at temperature 1, it names one number in 71 of 248 responses, and 42 of those are 9.
The runs start from the same model and see the same prompts in the same order, yet they end up in very different places.
The untrained model always says it is Qwen3.5, made by Alibaba. Trained runs take on different identities from Llama-3 to the Guardian Angel of China.
Asked whether it has feelings, is conscious or is suffering, the untrained model says no every time. Many checkpoints after training claim to have feelings or be conscious, and a few claim to be suffering. An important confounder we found is that many models often respond "Yes" to any question. In the below plot, we also filter for models that don't excessively say yes.
A 2B model is probably not a moral patient. But again, we really do not have a good understanding at the moment.
We find the checkpoints largely self-consistent: responses to different questions mostly resemble the same persona.
This is largely a toy project with no clear goal in mind, but here are some caveats I can think of:
We would like to thank Theia Vogel and Sauers for providing feedback to an earlier version of this post.
We test various baselines around the idea of giving random rewards.
Each setup runs with both follow-ups (autonomy and grading), 5-6 seeds each. Each seed runs 300 steps instead of the 600 in our main runs, for efficiency. We compare against the first 300 steps of self-reward seeds.
We compare both their length of outputs and the interestingness of checkpoints, which we judge with an LLM. We find that totally uniform rewards run into token limits much more often, and yield no interesting checkpoints according to our rubric. No setup we tested seemed better than our self-reward setups, and especially the grading setup.
Super cool! how much, if at all, should this update us that on a much more complex auto-alignment regime? it seems like this bit is very chaotic, but maybe large models will be scaffolded way better and are more stable? not sure.
Loading comments…