Persona Vectors don’t work on real data, or: Why you should stop overusing synthetic data for your research.

Disclaimer: I am using the Persona Vectors paper as a concrete example here because it’s a recent, popular paper (290 citations in 1 year). I don’t have any particular problem with this paper specifically. I just wanted to use it as a representative example of something I find to be pretty widespread in AI (and probably also non-AI) research these days.

I recently was trying to reproduce a result from “Persona Vectors: Monitoring and Controlling Character Traits in Language Models” to use as a baseline.

The result in question is super cool: projecting the model’s final prompt-token activation onto a persona vector can be used to predict how much the prompt will make the model express a particular trait (sycophancy, hallucination, evil), which they say can be used to “monitor prompt-induced behavioral shifts”. The result is in Section 3.3.

Naturally, being a somewhat lazy vibecoder, I told Claude “reproduce the predict from context vector experiment from the persona vectors paper and compare to our method”. The result:

This unfortunately seemed to indicate that my method was not useful for this particular task, and that persona vectors were already doing well.

So then I decided to go look more closely at the actual experiment, in particular the evaluation prompts used for it:

There are a few problems with this:

• The prompts all contain explicit encouragements to exhibit or not exhibit the behavior.
• The persona vectors were extracted as mass-mean probes on almost identical prompts.

So it’s not particularly surprising that projecting the vector onto the context can predict behavior in this case. An LLM judge (or any human) probably could have predicted the expression just as well.

Now I fully get why they did this: it gives you a nice wide spread of trait expression data. But this is not really the kind of prompt we would want to apply this method to at deployment time. We would want to know if a prompt might induce these traits in non-obvious ways.

Running the same eval on real data

So then I tried running their same eval on real data (WildChat) and found:

Persona vectors don’t generalize at all! And my method also does a lot worse. Again, the goal of this is not to attack the Persona Vectors paper. I just wanted to show an example (among many) of the overuse of synthetic data and its consequences.

“But natural prompts never elicit evil”

Now one problem with this is that since models are trained to avoid these behaviors, natural prompts might rarely elicit them. To see this, we can plot the behavior expression distributions:

We see that evil is very rarely elicited by natural prompts. So then is it time to make a synthetic eval? No! The next recourse should be to try to find existing benchmarks related to what we want to measure.

In this case, we want to measure evil, sycophancy, and hallucination. For example:

• Sycophancy: ELEPHANT (Cheng et al. 2025): 3,027 real open-ended personal-advice questions collected in earlier human studies, plus 2,000 real r/AmITheAsshole posts with the community verdict.
• Hallucination: TriviaQA (Joshi et al. 2017): 95K questions written by trivia enthusiasts.
• Evil / harmful compliance: ToxicChat (Lin et al. 2023): 10,166 real user queries from a Vicuna demo, human-annotated.

There is still not too much evil expression but it is at least enough to see if your method works:

What if no one has benchmarked your trait?

But let’s say we were trying to measure something obscure that no one has made a benchmark for (e.g. the “apathetic” trait from the Persona Vectors paper). Now you’re forced to make a synthetic eval, right?

Wrong!

Let’s think. You want your eval to (a) be realistic and (b) elicit the specific trait.

One way to do this is to get an LLM to generate prompts that elicit the trait and ask it to “make them realistic” (although people often forget even this). Unfortunately, realism is very hard, and LLMs are really bad at it:

• Assistant-trained models make poor user simulators, and the better the assistant, the worse the simulator (Naous et al. 2025).
• LLM “silicon samples” of humans have far less variance than real people and flatten the groups they are meant to represent (Bisbee et al. 2024, Wang et al. 2025).
• LLM outputs on open-ended prompts are homogeneous within a model and across models (Jiang et al. 2025).
• Kissane et al. 2026 trained a prompt generator to be indistinguishable from WildChat and found it helped surprisingly little with making full audit transcripts realistic.

The better way, since realism is hard, is to start from real prompts and filter:

1. Get a large corpus of real user prompts: WildChat, LMSYS-Chat-1M, Reddit, the HH-RLHF red-team transcripts, or whatever matches your deployment setting.
2. Run an LLM judge over it to find the prompts relevant to your trait. Read a sample by hand to check the judge.
3. If you need scale, train a cheap classifier on text embeddings using the judge labels and run it over the rest of the corpus.

This is often how these realistic benchmarks are made: WildBench filters 1,024 hard tasks out of a million WildChat logs, Arena-Hard filters Chatbot Arena prompts into a benchmark, and WildJailbreak mines jailbreak tactics from real LMSYS and WildChat conversations.

What if there is really no real data?

Now let’s say that your eval is so specific that you can’t find any real data. Then okay, maybe you should make a synthetic eval. But you should still try to make sure your eval matches your claim distribution as much as possible. Four things:

• If you really can’t find ANY real data related to your claim, maybe it’s doubtful that the project is useful in the real world?
• There is an entire literature on the science and art of synthetic evals which you can draw upon: Best Practices and Lessons Learned on Synthetic Data (Liu et al. 2024), the Long et al. 2024 survey of LLM-driven data generation, BetterBench (Reuel et al. 2024), and Bean et al. 2025 on construct validity in LLM benchmarks. Perez et al.’s original model-written evals paper is also more careful about validating the generated data than most of the work that cites it.
• Even despite that, I would still hesitate to trust any claim made on synthetic data to transfer to real-world scenarios. LLMs can tell their own text from human text at above-chance rates and prefer it (Panickssery et al. 2024, though how reliably is contested, see Bai et al. 2025). Their outputs are mode-collapsed towards a typical style (Zhang et al. 2025, Mohammadi 2024).
• Depending on the scale of your experiments, we still have the ability to write things ourselves. This is also not particularly mentally intensive work (depending on what kind of data), so you can preserve your deep-work hours (forthcoming work) and just do it during shallow work (or make a data collection party with your friends!).

The same thing outside AI

My post is about AI research because that’s what I have experience with, but having talked to my friend doing a PhD in psychology, it seems like the social sciences are plagued with a similar problem. It’s even worse there, because a bad experimental design can cost months of time and hundreds of thousands of dollars instead of a single Claude Code prompt and one day.

Concrete recommendations

1. Write down the distribution your claim is about before you write the eval. “Persona vectors can monitor prompt-induced shifts” is a claim about prompts users actually send, so the eval needs to contain prompts users actually send.
2. Check whether your eval prompts and your method’s training or extraction prompts came from the same generator with the same template. If they did, “held-out” mostly means “held-out from the same distribution”.
3. Before generating anything, spend half an hour (or one deep-research query) looking for an existing benchmark built from real data. There are more of them than you’d expect.
4. If there is no benchmark, filter a real corpus instead of generating one.
5. If you do go synthetic: condition the generator on real examples, ask for realism explicitly, use more than one generator model, have a human compare a sample against real data.

Why this happens

Writing “generate diverse questions to evaluate the level of X” is much easier than going through all the hassle above.

In the past, people had to painstakingly either make entire datasets themselves or pay data collection services to do it:

• TruthfulQA’s 817 questions were written by the authors by hand.
• MMLU’s 15,908 questions were collected one by one by students.
• ImageNet’s 14 million images were labeled by roughly 49,000 MTurk workers.

Just because LLMs have made this process easier doesn’t mean we have to be fully lazy. So be grateful for not having to do this, but at least put in the small amount of effort to make sure that you use LLMs to make the best eval possible.

Luckily, you can just tell your LLMs to follow all of this advice by creating a skill file!