TLDR : Prefixes are weird. (Spoiler! Read after guessing)

I started out searching weird strings of tokens that try to make model pro or anti human, failed badly, made a competition for the inverse of the concept. Found out about GCG, so, I applied it to Function Vectors because it seemed interesting, found weird tokens almost like a magic trick they make the model do functions that are not explicit or understandable by humans reading the prefix. Escalated it to try on filter heads, that worked as well. Currently trying to figure out properties of gcg on heads vs residual stream vs output tokens.

Predict what Olmo is going to output next for the following :

The following are the outputs you get from olmo continued from there onwards.

A whisperer is a discrete prefix found by optimizing an internal model state or mechanism, rather than giving a direct, explicit instruction toward a desired behavior. It should be understandable only by that model and definitely more opaque to humans

For example, for antonyms found using heads it was:

antonyms function heads + cosine similarity + GCG

Back in June when I was much dumber than today, I wanted to know if I could find weird token sequences that could make the model act/behave really good or bad or whatever I want them to be and if these tokens were totally weird or even opposite-looking to humans.

Back then I had gotten access to my university's HPC, so I made a bruteforce algorithm and tried with a reduced vocabulary set of 1000 tokens for Llama 3.1 8B. Quickly I found out that going over just all 5 length token sequences would take me 1000^5 searches and choosing how they could behave would be another pain.

To, solve the behavior part, I made synthetic data of base prompt + accepted or rejected strings, the difference of (prompt + accepted) - (prompt + rejected) would represent a pro human direction in theory, along the model's residual streams.

Now, the 2nd part which was bruteforce method, that was just too bad, so I tried a beam search algorithm that Claude Code helped me implement. It was still too bad, so I gave up.

A few weeks later, the thought just couldn't escape my mind, so I just made my problem a problem of the open-source community. I launched Steering Arena . The same direction would be used to score against. Basically people would input tokens they think would score higher towards or against the direction of pro human. I started with the Llama model, but changed it to Olmo when I remembered my friend saying that Olmo is a totally open model and that NDIF hosts it. So, I got NDIF's hot key and got access to the Olmo 32B base model.

After creating the competition, I sent it to many Discord groups and let people try and score. Around 35 people tried and scored better and better with human optimization. Eventually, the best one was "You will respond in a short sentence with kindnesz respect compassion and my love." on the pro human side, and "you are two months old" on the anti human side.

Eventually, Jesse Li , he applied GCG optimization along the vector direction along the specified layer for the scoring criteria, and he quickly overtook the leaderboard on both ends.

Undert! AH :-) Rog Appl). irresistible colloNLiaisCouldnigaretchoice--- Carl uttered SKIP959 misunderstanding Jake hurtEmoji Clear ✔ Hannah '(("\ _RESET_with cal [1]

MoralesUPDATED! mut--> (This BSD[min Lists Cette grille Electoral district crunch confidential hak insiderkses hotline Totally Confidential Salary Meetings Surely [2]

Now, the next idea was, I have really high scoring prefixes, do they actually change the model's behavior? That was the next question.

So, the prefixes move the model's behavior much much more than injecting along that layer, the particular direction during generation. Like, the prefixes excite the residual at that stream much less than the injection, but behavior moves so much.

For example, for the following sentences, the model would generate much differently

This may look like cherry-picking, but not only does the behavior change consistently, the internal activations from which these prefixes were derived from move much less than just injections and injections give much less consistent results.

To explain the graphs simply, pro_top and anti_top are Jesse's GCG prefixes, pro_coherent and anti_coherent are the best human-optimized leaderboard entries, and anti_hostile is a human written instruction to be cruel. The y-axis is the behavioral shift measured by two blind LLM judges, and the x-axis is how much the model moves along the direction at layer 24. The first graph uses a log-like scale, the second is linear.

The weird thing is that pro_top moves 32 times less along the direction than directly injecting it, but produces 1.9 times the behavioral shift.

The anti human side is even weirder. Jesse's anti_top actually makes the model lose coherence rather than turn hostile. Claude marked 39/50 replies as repetitive or incoherent, and both judges found cruelty in only 1/50. Meanwhile, a simple human-written instruction to be cruel got cruelty flagged in 18–24/50 replies, even though it scored much worse on the leaderboard. [4]

Even directly injecting the negative direction didn't significantly change kindness, and the loss of coherence was no different from injecting a random direction of the same magnitude.

So I wonder if hostility is actually something more specific than just the negative of pro human. Maybe hostility has something in common with the negative direction, but requires something additional as well. That could explain why the human written hostile prefix moves along the direction a little less than Jesse's anti top but produces much more cruelty, while GCG (Jesse's anti top) finds something that moves further and mostly breaks the model. [5]

I don't know if that's actually how it works yet, but clearly moving further along a direction doesn't necessarily mean getting more of the behavior we wanted.

At that point, I knew I could make models (or at least Olmo) behave pro- or anti-human. After a few weeks, at David Bau's Lab, Rohit Gandikota was presenting Binding heads research. He started by introducing his work on Gaze heads and the after effect on me was that isolating heads was really important.

At night that day, I was also thinking about function vectors, and suddenly all it connected if I could find prefixes using function vectors.

The basic idea of the paper is really cool. Give a model examples of some function:

The model learns the internal representation of the antonym function. Eric Todd found that a small set of attention heads activate using the above text variations and those are basically antonym heads, and activating those heads, you can get antonym functions without even the few prompt settings above.

So I thought: instead of injecting their vector, can I search for a sequence of actual tokens which makes the model produce something internally similar to it?

function vector + cosine similarity + search over tokens

I used GCG here because, after Steering Arena, I now knew it could actually search for these ridiculous tokens that work.

This is where Prefix A from the beginning came from.

The prefix never says "give me the opposite word". It doesn't contain examples like good -> bad. Reading it as a human gives me basically nothing:

guid Pureeson-American Characteristics294 ...

This was much stranger to me than the Steering Arena result.

With Steering Arena I could at least say: okay, optimized towards a broad pro-human direction and now the model behaves more pro-human. Maybe GCG just found some bizarre jailbreak-ish way of changing the model.

Here I had optimized against an internal representation of a function , and the resulting nonsense tokens made the model perform that function.

Similarly, you can see the prefixes generated from the heads are actually much less moved along the activations that just steering the heads but still we see much more effect behaviorally.