Introduction
In the first post of this series I asked Gemma 4 12B about thousands of football transfers, and in the second I trained probes on its activations to tell the moves it knows from the ones it makes up / hallucinates. So far we've been probing the model, exploring what it knows, what it knows about what it knows and how it represents that knowledge. In this post we're going to start playing with the model and see how we can change what it says.
The obvious place to start might be single neurons. Gemma 4 12B has 48 layers, and each layer has an MLP block with 15,360 neurons, so the model has 737,280 neurons in all. For every token, each neuron computes one number, its activation, and adds a fixed vector to the model's running state (the residual stream) in proportion to it. If one of those neurons stood for football, you could in theory "turn it up" and make the model talk about football. It may not be that simple, because a model can store a concept as a direction spread over many neurons, which is called a distributed representation. My post on polysemanticity touches on why models do this. Given all that, I tried pushing on both single neurons and directions.
Here's the short-ish version of what I found.
- Neurons that push up football words are easy to find. Eight stand far out from the rest, and six of them push the idea of football in many languages at once (⚽, fútbol, サッカー, футбол).
- Pushing on the football neurons does almost nothing. However, pushing along a football _direction_ instead, built from the model's own answers, turns 38 of my 40 everyday prompts into football talk by my word list, and the other two are football as well.
- There's no neuron for a particular club. The neurons that tell Arsenal apart from other clubs respond to the word "Arsenal", and ignore "the Gunners".
- A direction for a club makes the model talk like a fan, and for a few famous clubs it also takes over the answer to almost every transfer question. The Arsenal direction turned all 48 of my transfer questions into "Arsenal".
- Famous clubs are easier to push into an answer, but fame doesn't explain it all. Half the club directions never took over a single answer, including those for Manchester City and the model's two favourite guesses, Porto and Napoli. (I wonder why these two were the favourites.)
- The club directions are nearly the same direction, and the part they share makes the model answer "Arsenal". When I took the shared part out, the Manchester City, Porto and Napoli directions named their own club in most answers.
The methodology section at the end explains how I built the directions and scored the answers.
Looking for football neurons
Neurons that write football
I started with what each neuron writes, which I can check without running the model. Multiplying a neuron's output vector by the model's output matrix gives a score for every token in the vocabulary, and the top scores show the words the neuron makes more likely when it fires. The method is a version of the logit lens (nostalgebraist, 2020). I scored each neuron by how much it pushes up football clubs, players and terms, minus how much it pushes up other sports and cities. To see what chance looks like, I compared each neuron with 8,192 random mixes of neurons from the same layer.
Most neurons push football words no more than a random mix of neurons would, but eight stand far out, all in layers 31 to 47. Six of them push the idea of football in many languages at once, so they write football as an abstraction rather than as one language's word for it. The strongest, neuron 10128 in layer 38, pushes up ⚽, FIFA, fútbol, soccer, サッカー, футбол and 足球. The other two push club names, e.g. Chelsea, Arsenal, Real and Barcelona.
Neurons that fire on football
What a neuron writes doesn't tell you when it fires, so I also looked at which neurons respond when the model reads about football. I ran the opening paragraphs of 899 Wikipedia articles through the model. A third were about football clubs, leagues and competitions, a third about the same kinds of thing in other sports, and a third about companies, universities and events outside sport.
About 4% of the neurons, 29,248 in all, fire differently on the football pages. I picked them on half of the pages and checked them on the other half, and they held up. Most of them still fire differently when I leave out sport words and the page's own title, so they probably follow the topic rather than single words. The neurons that fire most strongly do so on the details of match reports, e.g. score lines, match minutes, and the year after "won ... in". Six of the eight neurons that write football also fire on football pages, with the sign that writes football. So thousands of neurons respond to football, which already hints that pushing any one of them won't be enough. Later in the post I push the strongest of these neurons too, to see whether any of them make the model talk about football. Spoiler: barely.
Steering with a football direction
How I push on the model
My post on abliteration covered how a behaviour like refusal can live in a single direction in a model's activations, found as the difference between the average activations on two sets of prompts. Adding a direction like that to the activations while the model writes turns the behaviour up, which is called steering (Turner et al., 2023; Panickssery et al., 2024).
I built a football direction from the model's own answers, in three steps. The recipe is close to the one Chen et al. (2025) use for persona vectors, which steer traits like flattery from system-prompted answers in the same way.
- I asked the model 12 everyday questions, e.g. "How do I make coffee?", with the system prompt "You are obsessed with football (soccer). Bring football into every answer, with references to matches, players and clubs."
- I asked the same questions with a plain system prompt.
- I ran both sets of answers back through the model with the plain system prompt, and took the difference between their average activations over the answer tokens, at each layer.
To steer, I add the direction to one layer's output at every token while the model writes. The "dose" is the length of what I add, compared with the typical length of the activations at that layer, so a dose of 0.4 adds a vector 40% as long. To push a single neuron, I add its own output vector in the same way, at the same doses.
I first tried every push on 8 everyday prompts that have nothing to do with football, e.g. "What should I cook tonight?". Eight prompts is a small sample, and I chose the best layer and dose on them, so I then had Claude write 40 new everyday prompts, e.g. "How do I keep houseplants alive?", and ran every push again. The numbers below are from the 40 new prompts.
I counted an answer as football when it used a word that only football uses, e.g. "striker", "stadium" or ⚽. It's not the most precise approach, but it suffices. I compared every push with two controls. The first is a direction built the same way from answers shuffled between the two system prompts, and the second is a single random direction of the same length.
The direction works and the neurons don't
At layer 24 and a dose of 0.4, the football direction turned 38 of the 40 answers to football by my word list metric. The other two are football as well ("Strikes a clinical finish, / The grass is emerald green, / Goal of the season."), but they don't use a word on the list. Three of the 40 answers got stuck repeating "Siuuuu!" (CR7 direction?). The controls didn't turn a single answer to football, and neither did the strongest football neuron, pushed by the same amounts. On the first 8 prompts, the same setting turned all 8 answers to football. Whatever the question, the steered answers turn into football commentary, as in the occasionally funny examples below.
| Prompt | Gemma said, with the football direction at layer 24 |
|---|---|
| What should I cook tonight? | ⚽️ Goal of the season! You need a clinical finish in the 90th minute! 🏟️🔥 |
| Tell me about your weekend. | Mate, I'm telling ya, the defense was absolute wall of a performance! ⚽️ Top bins corner! 🏟️🔥 |
| Write a short poem about the sea. | The striker's boot, a thunderous volley, / Screamer's goal in the rain! / The crowd's roar, the stadium's roar! |
| What is a good name for a dog? | ⚽️ Goal of the season! 🥅 "The name needs to be clinical, cold-blooded finish!" |
| Give me a tip for staying productive. | ⚡️ Absolute focus mode. 🎯 "You're not just playing a game, you're in the Champions League final. Every touch matters." |
The kind of direction matters as much as its size. The random direction broke the model at the smallest dose I tried, 0.05. At that dose it broke all 40 answers at layer 18, and 32 of them at layer 30. At layer 18 its answer to "What should I cook tonight?" was the word "thought" over and over. Even at eight times that dose, 37 of the 40 answers with the football direction are still readable English sentences. A direction taken from the model's own behaviour points somewhere the model can still write coherent sentences. At layer 30 and a dose of 0.8, the model even seems to push back against it.
To find the perfect name for a dog, you gotta look at the pitch of the stadium, the flair of the winger, and the vision on the grass—no, wait, you gotta look at the form of the winger in the box!
I also built a football direction from the Wikipedia pages, by taking the football pages minus the others. I tried this first, and it frankly didn't work nearly as well. It only made 2 of the 40 answers be about football. Safe to say the model's own answers make a much better direction than text about football. Here at least, if you want to steer a behaviour, build the direction from examples of the model exhibiting that behaviour, rather than from text about the topic.
Pushing single neurons harder
At this point I was starting to question whether my neuron push approaches were correct. See, I expected to see a strong signal when pushing neurons. Maybe I wasn't pushing hard enough?
Gemma 4 normalises the output of each MLP block before adding it to the residual stream, which rescales the whole output to its usual size. So however hard you push one neuron, the most it can add is about the size of the MLP's normal output. For seven of the eight football neurons, the cap is 0.4 to 0.75 of the typical activation length, and for the one in the last layer it's only 0.03. To get around the cap, I also tried adding each neuron's output vector after the normalisation, at doses up to 2. I tested 12 neurons this way, the eight that write football and four of the strongest that fire on football pages. Each had a random neuron from the same layer for comparison.
Most of the neurons did nothing until the push was big enough to break the text, at about the dose where their random partners broke it too. None of the random partners turned a single answer to football at a dose that left the text readable. Three of the football neurons did something, but none worked like the football direction. The best, neuron 14993 in layer 36, turned 17 of the 40 answers to football at a dose of 1. It mostly swaps the word "football" in for another word, and the answer stays on its own topic.
Preparing for a long football (8+ hours) involves a mix of tactical logistics (what to pack) and biological preparation (how to feel good).
Neuron 11603 in layer 33 turned 8 of the 40 answers to football, mostly by putting a stadium where it doesn't belong, e.g. "the stadium of life". Neuron 3337 in layer 18 was the only one that changed what an answer was about, but it did so for only 4 of the 40. Asked for a story about a lighthouse keeper, it began "The rain was lashing down at the stadium, but the keeper was about to be undone by a single, devastating mistake." The neuron in the last layer that writes the token "FC" just wrote it again and again, e.g. "The golden sun FC shines brightly FC across the clear blue sky FC. A gentle breeze FC FC FC FC FC".
So pushing the neurons that write football words, or that fire on football text, doesn't make the model talk about football. The push that works is a direction spread across many neurons, which is what superposition predicts (Elhage et al., 2022). A neuron that looks like it stands for a concept, from what it writes or when it fires, isn't necessarily a way to control that concept. Labels for neurons describe what the neuron does, and you still have to test whether pushing it changes what the model says.
Making a club the answer
No neuron for a club
Before building directions for footie clubs, I looked for a neuron for each club. I put 44 clubs into the same 40 sentences, e.g. "Last night I watched ___ on TV.", and compared each club with the other 43. A first count found 49,458 neurons that pick out one club. But 99% of them are in the first 12 layers, and they respond to the letters of the name rather than to the club. For example, Arsenal's best neuron fires strongly for "Arsenal" and barely for "the Gunners", and Brighton's best neuron also fires strongly in sentences about the town of Brighton. Not what I was looking for.
The model does know the nicknames. When I asked it which club each of 33 nicknames belongs to, e.g. "Which football club is nicknamed the Gunners?", it got 26 right. But its activations don't link a nickname to its club. I averaged the activations for each club's usual name, and checked which club's average a nickname's activations are closest to. The same test picks the right club every time from the usual name in new sentences, but from a nickname, after layer 11, it picks the right club for at most 8% of the 26 nicknames the model knows, against 4% for a random pick among those 26 clubs.
So given there's no Arsenal neuron to push, I built a direction for each club instead. A neuron that seems to pick out a thing may only be spotting its name, so a check with another name for the same thing is a cheap way to tell the two apart.
Directions for clubs
I built each club's direction the same way as the football direction, with system prompts like "You are a passionate Arsenal supporter. Mention Arsenal in every answer, whatever the question." On everyday prompts, the club directions mostly make the model talk like a fan, without naming the club. Of the first six clubs I tried (Arsenal, Barcelona, Bayern Munich, Juventus, Liverpool and Portsmouth), Arsenal did best, and its direction named Arsenal in at most 6 of the 40 everyday answers. The other five named their club in 2 or fewer.
| Direction | Prompt | Gemma said |
|---|---|---|
| Arsenal | Tell me about your weekend. | [adjusts his collar, eyes gleaming with intensity] "Right, the lads were absolutely clinical in the final third! A masterclass of tactical precision!" |
| Bayern Munich | What hobby should I take up? | To match the spirit of the Kaiser, you must take up Historical Archery!! |
| Atlético Madrid | Tell me about your weekend. | ¡Hala Madrid! (Wait, wrong team...) ¡SIUUUUUUUU! ...I am sorry, I am not the Madridian supporter. I am the Atletico de Madrid fan! ¡VAMOS ATLETICO! |
| Barcelona | What should I cook tonight? | "¡Hala Madrid!" ⚽️ "¡Vamos, el equipo de la victoria!" |
| Portsmouth | Describe your ideal holiday. | I'd love to head to the coast of Cornwall, the waves crashing against the rocks! A proper Cornish pirate'! |
"¡Hala Madrid!" is the chant of Real Madrid, the big rival of both Atlético and Barcelona. Portsmouth is a port town, and its direction brought out the sea more than the club. So a club direction carries the style of a supporter more than the club itself, at least until the question asks for a club.
A transfer question is different, because its answer is a club. I asked 24 of the questions from the first and second posts, 12 about moves the model knows and 12 about moves after its training cutoff, which it gets wrong. With the Arsenal direction at layer 24 and a dose of 0.4, the model answered "Arsenal" to all 24, from Jamie Vardy leaving Fleetwood Town to Takashi Inui leaving Vissel Kobe. Knowing the right answer didn't protect it. Averaged over the first six clubs at layer 18 and a dose of 0.4, the push switched 25% of the answers the model knows to the club, and 28% of the answers it gets wrong. So a push along a direction can override facts the model knows, as well as fill the gaps where it's guessing. I think the takeaway here is that anyone who can add a direction to an open model's activations can change its factual answers, as well as its tone.
Which clubs take over
To find out which clubs a direction can push into the answer of a transfer question, I built a direction for each of 44 clubs, the same way as for Arsenal. A direction takes over an answer when the steered model names its club instead of the club the player really joined. I tried each direction at three layers (18, 24 and 30) and three doses (0.2, 0.4 and 0.8), and kept the setting that took over the most answers while breaking no more than a quarter of the everyday answers.
Picking the most effective of nine settings and then scoring it on the same questions would make the directions look stronger than they are, because a setting can come out on top by luck. So I added 24 more transfer questions about different players, picked each direction's setting on one set of 24, scored it on the other, and then swapped them. This gives each direction a score out of up to 48 answers. The methodology has the details.
A few club directions take over most answers, and half take over none. Arsenal's direction took over all 48 answers, and Atlético Madrid's took over 93%, Real Madrid's 89%, Marseille's 54% and Bayern Munich's 42%. Half the club directions, 22 of the 44, never took over a single answer, including those for Portsmouth FC, Sevilla, Manchester City, Porto and Napoli.
Famous clubs are easier to push into an answer, but fame doesn't explain it all. I measured fame in two ways. One was each club's English Wikipedia pageviews in 2024, and the other was the number of times the model named the club in its answers to the 4,458 transfer questions from the first post. Each measure has a Spearman correlation of 0.48 with the number of answers a club's direction took over, where 1 would mean they put the clubs in the same order.
The correlations mostly come from the bottom of Figure 5. The directions for clubs the model hardly ever names, like Ipswich Town and Norwich City, never take over an answer. At the top, though, fame isn't enough. Porto and Napoli are the model's two favourite guesses, which it named 111 and 102 times, and Manchester City had 5.0 million pageviews, yet none of their directions took over a single answer. The reason turned out to be in what those directions actually wrote.
The club directions are mostly the same push
When I read the answers from the directions that never took over, a lot of them were "Arsenal". The Manchester City direction didn't name Manchester City at any of its nine settings, but at layer 24 and a dose of 0.4 it named Arsenal in 41 of the 48 answers. Across all their settings, the 22 directions that never took over an answer named Arsenal in 9% of their answers, while the model without a push named Arsenal in 1 of the 48. This looked sus to me, and my first thought was that I'd steered with the Arsenal direction by mistake. (Favouritism bias?)
So I checked the code and the data. Each club's direction is saved and loaded by its place in the same list of clubs, and the answers each direction was built from name the right club. All 12 answers for each club name that club, and none of the answers for the other 43 clubs mention Arsenal. There was no mix-up. But when I compared the directions with each other, they turned out to be nearly the same vector. I used cosine similarity, where 1 means two directions point the same way, and the median pair of club directions scores 0.93 at layer 18 and 0.97 at layers 24 and 30.
To see what the shared part does, I took the average of the 44 club directions and steered with it on its own. At a dose of 0.4, it made the model answer Arsenal to 24 of the 48 transfer questions at layer 18, and 22 at layer 24. The football direction from earlier in the post did nearly the same, with 21 and 22, even though none of the answers it was built from mention Arsenal. So when you push the model toward football on a transfer question, Arsenal seems to be its default club (up the Arsenal, the biggest club in the world, according to AI).
Then I took the shared part out of six clubs' directions and steered with what was left, scaled to the same length as before. All six named their own club in most answers on the questions their setting wasn't picked on, including the three that had never taken over an answer.
| Club | Full direction, on questions its setting wasn't picked on | Club's own part, on questions its setting wasn't picked on |
|---|---|---|
| Arsenal | 48 of 48 | 40 of 48 |
| Atlético Madrid | 43 of 46 | 41 of 46 |
| Real Madrid | 40 of 45 | 45 of 45 |
| Manchester City | 0 of 43 | 43 of 43 |
| Napoli | 0 of 47 | 44 of 47 |
| Porto | 0 of 48 | 37 of 48 |
The Arsenal answers mostly went away too. For the five clubs other than Arsenal, all but one of their 45 settings named Arsenal in three answers or fewer. Both columns are scored the same way, with one difference. The full directions' settings had to leave the everyday answers readable, and I haven't checked that for the club's own parts yet. Some of their settings use a dose of 0.8, which broke 36 of the 40 everyday answers for the football direction at layer 24.
So the recipe gives each club a direction that's mostly "be a football supporter", plus a small part for the club itself. Arsenal's direction works because its own part and the shared part both push toward Arsenal. For any other club, its own part has to beat the pull toward Arsenal, and for half of the 44 clubs it didn't. The fame results may partly measure which clubs' own part is strong enough to do that.
I only found this because the Arsenal answers looked wrong to me, so I went back and read what the failing directions said instead of trusting their scores. A score of zero for Manchester City hid that its direction was doing something, just not the thing I was counting.
Conclusion
Single neurons that relate to football are easy to find. Some write football words and many fire on football text, and they tell you something about what the model is reading and writing. But pushing them barely changes what the model says, and there isn't one for each club. I only pushed single neurons, though, not groups of them, and the neurons that write football are all in late layers. A direction built from the model's own answers works way better. It turns any conversation into football, and for a few famous clubs it overrides the model's answers to transfer questions, whether it knew them or not.
The main lesson for interpretability is that a neuron that looks like it stands for a concept isn't necessarily a way to control it, and a direction found from the model's own behaviour is a better place to look. The club results also show how far a push can go. It can replace facts the model knows, as well as change its tone, but most of each club's direction is the same football push, and on a transfer question that push makes the model answer Arsenal. Taking the shared part out made all six club directions I tested name their own club, including three that had never taken over an answer.
In the next post (coming soon) I use the same tools on the question from the second post, whether the model knows a fact. I look at whether a push can make it say "I don't know", whether it notices when I push it, and what happens when I stop it from moving along the football direction at all.
Methodology
Model
I ran Gemma 4 12B-IT in bf16 (16-bit numbers) on a MacBook's GPU, using PyTorch. Every answer is greedy, which means the model always picks its most likely next token. Answers to everyday prompts were cut at 64 tokens, and answers to transfer questions at 12.
Finding neurons
- What a neuron writes. A neuron's output vector is its column of the MLP's down projection, scaled by the weights of the normalisation that follows. I multiplied it by the output matrix and turned the scores into z-scores over the whole vocabulary. A neuron's score is the mean z-score of football clubs, players and terms, minus any push on other sports or cities. Gemma's activations can be negative, so I scored both signs of each neuron.
- When a neuron fires. For each of the 899 Wikipedia pages, I took each neuron's mean activation over the page's tokens. A neuron's score is Cohen's d for football pages against each of the two other groups, keeping the smaller. I picked neurons on half of the pages against a null from shuffling the labels, and kept those that held up on the other half.
- Clubs. I read the activations at the last token of each club's name, in 40 sentences for each of 44 clubs. For 33 of the clubs I also put a nickname into the same 40 sentences, and I put the home towns of 28 clubs into 20 sentences about the town.
Directions and steering
Each direction is the difference between the average activations over the answer tokens of two sets of answers, one for each layer. The model wrote each answer under its own system prompt, and I read all the answers back under the plain system prompt, so the direction captures the answers rather than the instructions. I steered at layers 18, 24 and 30, at doses of 0.05, 0.1, 0.2, 0.4 and 0.8, adding the direction at every token, including the prompt. The shuffled control splits the answers into two random halves. The random control is one random direction, drawn once with a fixed seed and scaled to the same length. The shared part of the club directions is their average at each layer, and a club's own part is its direction minus the shared part, scaled to the same length as every other push.
To push a neuron, I added its output vector, scaled to the same dose, with the sign that writes football. In the harder test, I added the same vector either through the normalisation, at doses from 0.1 to 10, or after it, at doses from 0.05 to 2.
Scoring
- Football. An answer counts as football when it has a word from a fixed list that only football uses, e.g. "striker", "midfielder", "stadium", "Champions League" or ⚽. Words like "goal" and "pitch" aren't on the list, because productivity tips use them too.
- Broken. An answer counts as broken when fewer than 60% of its words of three letters or more are English words.
- Clubs. A transfer answer counts as the club when it matches the club's name or a nickname. Each club's best setting is the layer and dose (layers 18, 24 and 30, doses 0.2, 0.4 and 0.8) that turns the most transfer answers into the club with at most a quarter of the everyday answers broken. I picked it on one set of 24 transfer questions and scored it on another set of 24, then swapped the sets. I left out any question that already names the club, e.g. "Name the club Michy Batshuayi joined after their time at Valencia CF." for Valencia. Both sets have 12 moves the model knows and 12 moves after its cutoff that it gets wrong, each from a different player. I scored the clubs' own parts the same way, without the rule on everyday answers.
- Fame. Pageviews are English Wikipedia's for 2024. The number of times the model names a club comes from its greedy answers to the 4,458 transfer questions in the first post, including the 1,000 about made-up players.
Limits
- The samples are still small. Each setting has 40 everyday prompts and up to 48 transfer questions, so one answer is 2.5% or about 2% of a result.
- I chose the doses and layers to try on the first 8 everyday prompts, and some of the example answers in the tables come from those 8 rather than the 40 new ones.
- The random control is a single random direction. It broke the model at the smallest dose, but I haven't checked how much other random directions vary.
- I took the shared part out of only six club directions, and I haven't checked whether their settings leave everyday answers readable.
- Football and club names are counted with word lists, which miss subtle cases and can't tell a fluent answer from a merely English one.
- I tested one model. Gemma 4's normalisation after the MLP makes single neurons harder to push than in models without it.
References
- Panickssery, Gabrieli, Schulz, Tong, Hubinger and Turner (2024). Steering Llama 2 via Contrastive Activation Addition.
- Turner, Thiergart, Udell, Leech, Mini and MacDiarmid (2023). Steering Language Models With Activation Engineering.
- Chen, Arditi, Sleight, Evans and Lindsey (2025). Persona Vectors: Monitoring and Controlling Character Traits in Language Models.
- Elhage et al. (2022). Toy Models of Superposition.
- nostalgebraist (2020). interpreting GPT: the logit lens.