I trained a small local model, Qwen3-4B, on my X replies and wanted to know one thing: does it sound like me? Answering that took seven rounds of blind tests, after four earlier ones with an older setup. The leader changed more than once, and each time it was my test that had changed.
The short version. When judges had to pick my real reply out of a line-up, the LoRA was their favourite wrong answer, well ahead of Claude. When they ranked replies on surface style, it tied Claude Sonnet with my style prompt, 103-97. When they judged character, meaning what a reply actually does, every model I trained looked about the same and the judges found my real reply about 80% of the time. And when I picked the better reply myself, Claude won 23-3.
The method itself (archive, data prep, training config) is in How to fine-tune a local LLM to write like you. This one is about measuring, and most of it applies to any model you train on your own writing.
The setup
The base model is Qwen3-4B-Instruct-2507, 4-bit, trained with MLX on my M3 Pro MacBook with 36 GB of RAM. LoRA rank 16, 400 iterations, about 40 minutes per run. The data is about 10,400 English replies from my training split. The rest are held out.
A reply on its own is useless as training data. The model needs an input. My first version didn't use the post I was replying to. A model read each of my replies and wrote a one-line brief backwards from it:
Brief: "Reply to someone who misread your project's description. Thank them and clarify you meant the license."
Reply: "Hah, Thank you! I meant the license 😂"
The opponent is my writing-voice prompt, a Claude Code skill I use every day. It's about 12 KB of voice rules and stats taken from my own messages. It runs on Sonnet unless I say otherwise. In the charts it's "skill" or "style prompt".
The judges are other LLMs: Grok, Devin and Claude Haiku. Each gets 20 of my real posts as reference, then a list of situations. Each situation has 3-4 candidate replies, one of which is my real reply. The judge ranks them from most to least like me and guesses which one is real. The instruction text stayed the same in every round. The inputs, the candidates, the judge panel and the sample size did change, and I'll say where.
Scores like 58-20 are pairwise ranking points. If a judge ranks A above B on one post, A gets a point. With 3 judges and 26 posts that's 78 points per pair. Those points aren't independent, since the same judge and the same post show up many times, so treat the p-values below as optimistic.
The project had four earlier rounds with an older setup. Below are the seven that count, in order.
Round 1: brief-trained LoRAs beat Claude
I had two LoRA runs that the judges scored 39-39. Then I compared their weights. For each layer I computed the update the LoRA adds, B @ A, and took the cosine between the two runs. It was 0.15. The same run at step 200 vs step 400 was 0.89.
So two adapters that score the same moved the base model in mostly different directions. That's known. LoRA initializes A randomly, so runs land in different places even on the same task. A June 2026 paper (arXiv 2606.23740, section 3.4) measured seed-to-seed cosine of 0.07 to 0.36 on attention-only LoRAs of the same Qwen3-4B-Instruct-2507. SeedLoRA (ICML 2025) merges same-task LoRAs from different seeds.
If both directions are useful, I can keep both. With LoRA you can't average A and B separately, because the two runs don't share a basis. Averaging them adds cross terms neither run learned. You stack them into one adapter with double the rank:
for k in b:
if k.endswith('lora_a'):
res[k] = mx.concatenate([a[k], b[k]], axis=1)
elif k.endswith('lora_b'):
res[k] = mx.concatenate([a[k], b[k]], axis=0) * w
With w = 0.5 the rank-32 adapter applies exactly half of each update. Max error against the summed updates was 3.5e-10. PEFT does the same with add_weighted_adapter(combination_type="cat"), and FLoRA (arXiv 2409.05976) uses the same stacking.
The soup vs its parents: 43-35 against the first run (p = 0.21, noise) and 49-29 against the English-only run (p = 0.015). Against Claude with my style prompt, 26 prompts, 3 judges: 58-20 for the soup. In a re-judged round the same soup scored 53-25, so a 5-point swing is my noise floor. The first single run also beat Claude, 51-27. The soup didn't add much on top of that.
One thing I missed at the time: Claude got the same synthetic brief as input. Claude never trained on that format. The LoRA did.
The emoji detour
On those 26 prompts my real replies had an emoji 39% of the time. The soup had one 8% of the time. Across all my training replies the rate is 29%.
My first fix was a static logit bias on emoji tokens. At bias 4, 17 of 26 replies turned into emoji walls. One emoji makes the next one more likely, and the bias never lets go.
The second fix switches the bias off after the first emoji. That hit 39% with no loops. The judges barely moved: nudged vs plain soup, 42-36, p = 0.29. Matching my emoji rate on its own didn't make the soup more convincing.
Round 2: the real tweets
The synthetic briefs had a problem. In real life nobody hands me a brief. I see a post and reply to it. A brief like "thank them and clarify you meant the license" already contains the answer.
So I fetched the real parent post for each reply and trained a new LoRA on post-to-reply pairs. Then I tested on 26 held-out posts, and every system got the real post.
- Claude with the style prompt beat the post-trained LoRA 62-16.
- Claude beat the brief-trained soup 73-5.
- The post-trained LoRA beat the soup 59-19. The soup had never seen a real post, and it mostly echoed it back.
I also rated those 26 pairs myself, blind, forced choice: which reply is better? Claude 23, LoRA 3. I'd call 5 to 7 of them close. Some LoRA replies were bad: emoji walls again, and one that looped "trust + trust + trust".

Round 3: Claude reads, the LoRA writes
Next idea: Claude reads the post and writes a brief, and the brief-trained soup writes the reply.
LLM judges, 26 posts: pipeline 36, Claude alone 42. My own picks: Claude 19, pipeline 7. I marked 19 of the 26 as close, meaning I could live with either reply. On the other 7 it was 6-1 for Claude.
The pipeline replies ran long. Median 133 characters, against 94 for my real replies on the same posts. Claude's briefs had a median of 40 words. The training briefs had 23. My guess is the soup learned that a longer brief means a longer reply. I haven't tested that yet.
Round 4: which one is really me?
What I want from this model is help with voice. When I reply, the ideas are mine. So the test that matters to me is simple: put several replies next to each other and ask which one I wrote.
First a small run. I wrote 10 fake posts and replied to them myself, then mixed my replies with the LoRA's and Haiku with my style prompt. Grok and Devin guessed. Out of 20 guesses: my real reply 11, LoRA 7, Haiku 2. My reply page had no emoji keyboard, so I typed ;) and :D instead of emoji. That makes my own side of this test less typical.
Then the bigger run, with no new writing. I took 100 held-out real replies the LoRA never trained on, each with its parent post. Four candidates per post: my real reply, the post-trained LoRA, Haiku with my style prompt and Sonnet with my style prompt. Grok and Devin each guessed which one was real.
| Candidate | Picked as the real me (of 200) |
|---|---|
| My real reply | 157 |
| LoRA (Qwen3-4B) | 30 |
| Haiku + style prompt | 8 |
| Sonnet + style prompt | 5 |
Random guessing would give 50 each. All three fakes are below that. The judges found the real reply 157 times, so this is a detection task the judges mostly won.

The interesting part is the 43 misses. The LoRA got 30 of them. If misses were split evenly between the three fakes, a 30-13 split would be very unlikely (binomial p below 0.001), though the same caveat about non-independent guesses applies. The judges agree on direction: Grok picked the LoRA 18 times, Devin 12.
Now the same judgments scored as full rankings, like rounds 1 to 3. LoRA vs Sonnet: 103-97. On replies under 70 characters it's 54-52. On longer replies it's 49-45. A tie in both.
So the LoRA is the most common decoy when a judge has to commit to one answer. Over the whole ranking, it's level with Sonnet.
Why round 2 and round 4 disagree
Round 2 said 62-16 for Claude. Round 4, scored the same way, says 103-97 for the LoRA. Same judge instruction, same LoRA, same skill. What differs:
- Judge panel. Haiku judged round 2. In that round Haiku guessed a Claude reply was the real me 16 times out of 26, and found my real reply only 4 times. Devin picked Claude 14 times, Grok 6. Round 4 had no Haiku. Without Haiku, round 2 still goes to Claude, 39-13.
- Sample. Round 2 used replies of 40+ characters. Round 4 allowed 20+. The round 4 replies are shorter (median 69 vs 94 characters). But the length split above shows a tie in both halves, so length alone doesn't explain it.
- Size. 26 posts vs 100. Round 2 is small enough that a few posts can swing it.
I don't have a clean answer. My best guess is that round 2 was the outlier, and 100 posts with 2 judges is closer to the truth. That's a guess.
Emoji and length are style too
The judge instruction lists "wording, length, punctuation, emoji habits, rhythm". The LoRA matches my emoji rate on these 100 posts exactly: 41% vs 41%. Sonnet uses an emoji 56% of the time and Haiku 60%, even though my prompt says about one in three.

You could call that "just surface stats". I don't. How often I drop a 😅 and how short I keep a reply are part of how I sound. A style prompt can describe them, and Claude still overshoots. The LoRA learned them from 11k examples and lands on the number.
What I don't know is how much of the LoRA's edge comes from those habits and how much from wording and rhythm. Matching the emoji rate alone didn't move the score in round 1, so it isn't only emoji. Stripping emoji from every candidate would show which habits the judges actually key on. That's a diagnostic, not a fairness fix.
Round 5: adding my desktop replies
Up to here the model trained on phone replies only. I'd left out desktop posts because of a vague worry that some of them were AI-assisted. They weren't, except maybe one or two article-length posts, which I dropped anyway. So I added about 6,600 desktop replies and trained two new models on the mix, with seeds 42 and 1.
Then I held out 100 desktop replies as a second test set. On those, with desktop posts as the judges' examples, both mixed models beat the phone-only one: 125-75 for seed 1 and 109-91 for seed 42. On the phone test set, the phone-only model stayed ahead on surface style: 123-77 against seed 42 and 103-97 against seed 1. Only the 125-75 and 123-77 gaps are outside the noise band I measured later, in round 7.
That's the expected result for surface style: a model sounds most like the data it saw. It also showed me a bug in my judging. Until then, the 20 example posts the judges saw were all phone posts, so "sounds like Marcin" quietly meant "sounds like Marcin on his phone".
To check whether phone and desktop me are different people, I had readers write blind portraits from each set separately and scored all portraits on 12 qualities. Phone and desktop differed by 1.0-1.2 points per quality, about the same as two readers of the same set differ (1.0). Terseness moved most: desktop replies are longer. So it's the same person on both devices.
Round 6: judging character instead of surface
My judge prompt asked about "wording, length, punctuation, emoji habits, rhythm". That's surface. The LoRA was always going to be good at it, because it learned those habits from 11k-17k examples.
But what makes a reply mine is mostly what it does. I ask how something works instead of saying congrats. I push back with one counterexample. I joke at governments and hype, not at people who struggle. To make that measurable, I had four LLM readers take 750 of my posts each and write notes, and a fifth pass merged the notes into one portrait of how I react. Then I gave the judges that portrait as the rubric and told them to ignore length, emoji, punctuation and typos.
| Test set | Real reply found (of 200) | phone-only vs mix s42 | phone-only vs mix s1 | mix s42 vs mix s1 |
|---|---|---|---|---|
| Phone, 100 posts | 158 | 114-86 | 102-98 | 101-99 |
| Desktop, 100 posts | 168 | 100-100 | 93-107 | 100-100 |
On character, the judges spot my real reply 79-84% of the time, and my models barely separate. The one bigger gap, 114-86, is a single run, and by round 7's standard it needs repeating. I didn't put Claude in the character runs, so I can't say how Sonnet with my style prompt does on character. What I can say: my LoRAs copy how my replies look much better than how I think.
Round 7: how noisy is the judge?
Two runs of the same pair on the same posts had given me 102-98 and 118-82, though the fourth candidate differed between them. That's a big swing, so I measured it properly. With the character rubric, I judged the same 100 replies three times, changing only the judge's shuffle seed, which moves the letter order and the 20 example posts.
| Pair | Run 1 | Run 2 | Run 3 | Sum (of 600) |
|---|---|---|---|---|
| phone-only vs mixed | 118-82 | 101-99 | 111-89 | 330-270 |
| phone-only vs length hint | 98-102 | 91-109 | 93-107 | 282-318 |
| mixed vs length hint | 93-107 | 103-97 | 104-96 | 300-300 |
One pair moved 17 points per 200 between runs with nothing changed but the shuffle. So a single run can't separate gaps under about ±20 per 200. Two gaps held their direction in all three runs, one flipped. All three runs judged the same generated replies, so this is only the judge's noise. Training noise comes on top: in an earlier round, the original model beat three retrains of the same recipe, differing only in seed, by 104-96, 112-88 and 126-74.

I measured the band on the character rubric only. My surface-style results after round 4 were never re-run three times, and most of their gaps sit inside ±20 too.
Two more things that didn't move the judges
A length hint. Desktop and phone differ mainly in length, so I trained a model with a length line in the prompt ("Length: short", "medium" or "long"). Told the length bucket of my real reply, it hit the right one 76-82% of the time, against 29-41% for the older models. The judges barely moved. Matching length is the same story as matching emoji rate: right number, same verdict.

Activation steering. I took the base model's average internal activation (layer 16) on my real replies, subtracted the average on Claude's imitations of me, and pushed generation along that difference. Pushed one way, it wrote florid generic AI prose. Pushed the other way, it answered some prompts in Polish, and harder still it announced "I am Marcin Dudek, and I am a man who lives in Poland." It didn't start asking questions the way I do. The direction captured who I am and where I'm from. It missed how I react.
What I took from it
The eval decides the winner. Synthetic briefs gave the LoRA the answer. Real posts showed that a 4B model often doesn't get what it's replying to. And when I judge which reply is better, Claude wins 23-3, because it understands the post.
Decide what "sounds like me" means before building. Surface style and character give different answers. I spent rounds 1 to 5 measuring surface without noticing, and the LoRA looked better than it is.
Measure the judge before trusting a number. Run the same judgment at least three times. If a gap doesn't keep its direction, it isn't a result.
Watch who judges. Haiku found my real reply 4 times in 26 and picked Claude's reply as mine 16 times. I left Claude models off the panel when Claude was a candidate.
Match the examples to the test. Phone-only examples made "like Marcin" mean "like Marcin on a phone".
Put the scoring method next to the number. 30-13 on misses and 103-97 on rankings come from the same 200 judgments.
Caveats
- One person. Everything here is my voice on X. My replies are short and casual, which suits a small model.
- LLM judges. The only human rater was me. The real test is 3-5 people who read my posts guessing which reply is mine.
- No Claude on the character rubric. Every character run compared my own models with my real reply.
- Same replies, re-judged. The noise check covers the judge, not the sampling. I haven't regenerated replies with new seeds and re-judged them yet.
- I changed the setup between rounds. Panel, sample filter, examples and candidate count all moved. If I started again, I'd fix them up front, run each judgment three times and report both ranking points and top picks every time.
- Possible leak in the style prompt. Its X stats came from 97 of my real replies. I haven't checked whether any of them are in the held-out sets.
The next thing I'd test is the one this whole project kept pointing at: a model that understands the post (Claude) deciding what to say, and a model that learned my habits deciding how. Round 3 tried a crude version of that. With a character judge and three runs per test, I'd finally be able to tell whether it helps.
Published · Last updated