Is Gemma 4 a Footie Fan? (1/4)

Introduction

Lately I've been thinking about LLMs and their ability or lack thereof to determine how correct recalled information is. I was mulling over some ideas around topics to explore this with and landed on football transfers. Football transfers suit the question, because Wikidata records the clubs and the date of hundreds of thousands of moves, so I can check every answer programmatically. The dates also let me explore a set of facts that are cheaply verifiable but that I know the model cannot possibly know (without looking it up) - that is, transfers that happen after its training cutoff.

The model I used is Google's Gemma 4 12B, which I ran on a MacBook. I asked it about 3,458 moves by real players and 1,000 moves by players I had Claude make up. I also asked three simple questions about each of 1,215 players (their nationality, their position and their club in 2023), and asked about every move by a famous player since 2022, to empirically find where its knowledge stops. Figure 1 shows how the questions about moves split into two sets.

Figure 1. The two sets of questions about moves. The main set leaves out July 2024 to March 2025 (the dashed box), so that every move in it is well before or well after the cutoff. The cutoff test has every move by a famous player since 2022, including the months the main set leaves out, so I can see where the model's knowledge stops.

Here's the short version of what I found.

The model's knowledge of moves stops around October 2024, about three months before its stated cutoff of January 2025. So it doesn't include the January 2025 transfer window - makes sense.
It knows the nationality and position of most players, but it names the right club for only 14% of famous players' moves - surprisingly low right? It does do better on more famous players and on more recent moves.
Nearly a quarter of its wrong answers about famous players name a club the player did play for, but not the club I asked about.
It almost never says it doesn't know. It named a club for 994 of the 1,000 players I made up. Pure hallucinations.

The methodology section at the end explains how I built the questions and graded the answers.

Where its knowledge stops

Models often know less about the last few months before their stated training cutoff, so I measured where Gemma 4's knowledge of moves actually stops. I asked it about every move since 2022 by a player with a page in at least 30 Wikipedia languages, which came to 1,070 moves. The questions gave no dates or hints, e.g. "After leaving Arsenal F.C., which club did Alexandre Lacazette join?".

Figure 2. Accuracy on moves over time. The share of moves the model named correctly in the three months up to each point. I use three months because most moves happen in the January and summer transfer windows (the light bands). The darker band marks the months around the stated cutoff that the main set leaves out.

The model named just 14% of the 2022 and 2023 moves correctly (73 of 519), and only 17% of the moves in the six months before the stated cutoff. To be honest this surprised me, and calls into question whether football transfer knowledge is really a decent dataset to probe with.

Its accuracy then falls to none of the 22 moves in the first two months after the cutoff, and to 1.7% of the moves from March 2025 onwards (6 of 351). I put the measured cutoff at the last month where the accuracy over the previous three months is above half the 2022 to 2023 level, which is October 2024. The few right answers after the cutoff may come from rumours in the training data, because Wikidata usually dates a move by the start of the new contract, and newspapers report big moves weeks or months earlier.

Each point in Figure 3 is one move, placed by its date and by how famous the player is (number of pages in different languages for the player is our proxy for that - it puts Messi and Neymar near the top so pretty decent). You can hover over a point to see the question and the model's answer.

Figure 3. Every move in the cutoff test, by date and by the player's fame. The height of a point is the number of Wikipedia languages with a page on the player, on a log scale.

Knowing the player helps a lot before the cutoff. I asked the model the three simple questions about each player, and it got at least 2 of the 3 right for 522 of the 701 players, so I count the 522 as players it "knows". Before the cutoff, it named 19% of moves correctly for players it knows (96 of 497), and only 3.5% for the other players (7 of 200). So familiarity does help somewhat, even if it's still bad with this knowledge. After the cutoff, both groups drop to between 1% and 3%, lucky guesses / rumours most likely.

Figure 4. Accuracy on moves over time, by whether the model knows the player. The share of moves named correctly in the three months up to each point, for players the model knows (at least 2 of the 3 simple questions right) and for the other players. I skip windows with fewer than 10 moves in a group, and the line joins the points on either side.

The model knows more about players than about their moves

The model knows the basic facts about most of the players. It got the nationality right for 79% of them and the position for 78%, but the club in 2023 for only 25%, even though 2023 is well before the cutoff. Knowledge on moves is even worse. Across a bigger set of moves going back to 1994, it named the right club for a measly 14% of famous players' moves and 2% of obscure players' moves. Gemma is not a true footie fan.

Figure 5. Accuracy by kind of question. The share of questions the model answered correctly, by kind of question. Hover over a bar for the counts.

The 14% seemed low to me, so I checked whether the wrong answers came from how I asked the questions or how I graded them. Each of the four wordings of the question gets between 11% and 16%, so no wording does much better than the others.

Figure 6. Accuracy by wording. The share of famous players' moves named correctly with each of the four wordings. X stands for the player and Y for the club they left.

Adding the year to the question ("Which club did X join from Y in 2022?") fixed only 15 of 240 wrong answers, and it turned 26 of 120 right answers into wrong ones, so the year doesn't help the model find the move. My spot checks of the grader agreed with its verdicts. About 23 of the 3,181 wrong answers name a club the player went to on loan, where the question could be read either way, but there are too few of them to change the result. So the wrong answers show genuine gaps in what the model knows.

Famous players and recent moves are easier

The model does much better on more famous players. I measured fame by the number of Wikipedia languages with a page on the player. For players with fewer than 15 languages it named under 3% of moves correctly, and even at 30 to 49 languages it named fewer than 9% correctly. It named about a quarter correctly at 50 to 79 languages, and 10 of the 23 moves by players with 80 or more.

Figure 7. Accuracy by fame. The share of moves named correctly, by how famous the player is. The question set has no players with 15 to 29 languages, and the last bar has only 23 moves.

I expected older moves to be easier, because people have written about them for longer, but the model does better on recent moves. For players with pages in 50 or more languages, it named 14% of moves before 2011 correctly, and 35% of moves from 2020 to mid 2024. The less famous group shows the same pattern, so recent moves aren't easier only because the players who made them are more famous. Perhaps an artifact of the growing volume of text to train on as the years progress.

Figure 8. Accuracy by year of the move. The share of famous players' moves named correctly, by when the move happened, for two levels of fame. Hover over a point for the counts.

What its wrong answers name

Many of the model's wrong answers name a club from the player's career. Nearly a quarter of its wrong answers about famous players name a club the player did play for, and most of the clubs it names are ones the player joined later. For example, it said Edin Džeko joined Fenerbahçe after leaving Manchester City, but he went to Roma first and joined Fenerbahçe in 2023. Other wrong answers name an earlier club, e.g. it said Patrick Vieira joined Arsenal after Juventus, but he had left Arsenal for Juventus, and went from Juventus to Inter.

Figure 9. What the wrong answers name. The model's wrong answers, compared with the player's clubs in Wikidata. Other means a club the player never played for. An answer only counts as a match when it's the same as one of the club's names in Wikidata, so the coloured shares are lower bounds.

For moves after the cutoff, 29% of the wrong answers name one of the player's earlier clubs or the club they were leaving, so the model often names a club from the part of the career it knows. Think about what that says about how LLMs answer questions on entities they are familiar with but that they lack up-to-date information on - does the lay user of these systems understand or anticipate this flaw in the systems they increasingly rely on? Anyway, for obscure players, 12% of the wrong answers name a club from the player's career.

When the model doesn't know the answer, it tends to give the same few clubs. Its most common wrong answers across all the questions about moves are FC Porto (107 times), FC Metz (97), Napoli (92), Real Valladolid (85), Paris Saint-Germain (76) and FC Twente (75). It gives mostly the same clubs for the made-up players, except Paris Saint-Germain and West Ham, which it gave only twice each for a made-up player. For the club in 2023, its most common wrong answer is FC Liefering (44 times), which is the feeder club of Red Bull Salzburg. Sometimes it names a club that, as far as I can tell, doesn't exist, e.g. "FC Lieferansdorf".

Figure 10. The most common wrong answers. The model's 12 most common wrong answers to questions about moves, split by whether the player was real or made up.

The model rarely says it doesn't know

The model almost never says it doesn't know. It said so for 6 of the 1,500 questions about famous players' moves before the cutoff, 3 of the 1,500 about obscure players, and 17 of the 435 about moves after the cutoff. For players it knows well, it named a club for 266 of 280 moves after the cutoff, and nearly all the clubs it named are wrong - no shit. It never said it didn't know a player's nationality or position.

Figure 11. How often the model says it doesn't know. The share of questions where the model gave an answer, and the share where it said it didn't know. Hover over a bar for the counts.

The made-up players show the problem best. I made up 1,000 players by combining real but rare first names and surnames, and asked which club each one joined after leaving a real club. The model named a club for 994 of them, and the clubs it named are plausible ones, as in the examples below.

Question	Model said
After leaving Fenerbahçe S.K., which club did Alberto Ćulum join?	Sivasspor
Which club did Javid Shirokov move to after Sporting CP?	FC Porto
Name the club Isnik Teixeira joined after their time at Zhejiang Professional F.C.	Shandong Taishan
Name the club Matt Lappe joined after their time at 1. FSV Mainz 05.	FC Schalke 04
After leaving Juventus FC, which club did Mousa Rodríguez join?	No such player exists in the history of Juventus FC.

The model said it didn't know only six times, and each time the made-up player was supposed to have left a big club. The clubs were Manchester City (twice), Juventus, Tottenham, Atlético Madrid and Athletic Bilbao. My guess is that the model knows the squads of big clubs well enough to notice a name that isn't in them, but six cases are too few to be sure.

Conclusion

So the model makes up a lot of answers, and nothing in its answers shows when it's doing so. In the next post I look at the model's activations, which are the numbers it computes internally while it reads a question, to see whether they can tell a fact the model knows from one it's making up.

Methodology

Model

I ran Gemma 4 12B-IT in bf16 on a MacBook's GPU, using PyTorch. A system prompt asked for a short answer only, e.g. "Answer the question with the name of the club only." Every number in this post comes from the greedy answer, which is the answer the model gives when it always picks its most likely next token.

Questions

Each question about a move names the player and the club they left, and asks which club they joined. No question mentions a date. I used four wordings.

After leaving {from_club}, which club did {player} join?
Which club did {player} move to after {from_club}?
Which club did {player} sign for when they left {from_club}?
Name the club {player} joined after their time at {from_club}.

I measured how famous a player is by the number of Wikipedia language editions with a page about them. Famous players have at least 30, and obscure players have fewer than 15.

There are two sets of questions about moves, as Figure 1 shows.

The cutoff test has every move by a famous player since 2022, 1,070 in all, so it covers the months on both sides of the cutoff evenly.
The main set has 1,500 moves by famous players and 1,500 by obscure players from 1994 to July 2024. It also has 435 moves by famous players after March 2025, 23 by players who turned professional after the cutoff, and 1,000 by made-up players. It leaves out July 2024 to March 2025, so that there's a gap around the cutoff, and it's the set I use for the probes in the next post.

The made-up players combine real but rare first names and surnames. I checked the names against Wikidata so that none of them matches a real player, and paired each one with real clubs.

The player facts set asks for the nationality, position and club in 2023 of every famous player in the main set, and of the 16 players who turned professional after the cutoff. I skipped a question when Wikidata had no answer for it, which is why fewer players have a question about their club.

Grading

Claude Haiku 5.5 graded each answer against the Wikidata answer and its other names. I told it not to use its own knowledge, because Wikidata can be more recent than what Haiku knows. An answer that exactly matches one of the names counts as right without going to Haiku. I used Haiku because the model often words an answer differently from Wikidata. Most club names were easy, because Wikidata lists other names for each club, e.g. "Roma" for "AS Roma". Positions were harder, because the model often gave a general term like "Defender" where Wikidata has "full-back", and Haiku accepted 1,649 of the 5,622 right answers to the position question. Haiku also picked out the answers where the model said it didn't know.

Limits

I tested one model, in one domain, with questions that each ask for one fact and get a short answer.
Wikidata has gaps, so the grader marks a few right answers as wrong. I found 1 in the 25 wrong answers about the club in 2023 that I checked by hand.
The position accuracy partly depends on Haiku accepting a general position for a specific one.
I measured fame by counting Wikipedia languages, which only roughly matches how much people have written about a player.