Can a reader tell which brand wrote it? A test for copywriters

Most of our voice tests use emails and old letters. Copywriters and brand writers have a different problem: several brands, one padded AI draft, and a client who can tell at a glance if it sounds like someone else. We built a test for that, found the pipeline was barely better than guessing, and changed it. This note covers the test, the change, and what it did not fix.

01The test

We invented five brands with very different voices: a candle maker (short, sensory fragments), an accounting tool (dry, numbers first), a toy shop (bouncy, exclamations and asides), a law firm (precise, formal) and a luxury travel company (long, lyrical sentences). Each has five short samples. We wrote six padded drafts of the kind AI produces (a product launch, a price change, an about-us blurb, an event invite, a delay apology, a new hire) and rewrote every draft in every brand's voice: 30 rewrites per run.

A model then read all five brands' writing, shuffled and relabelled, plus one rewrite, and said which brand wrote it. Chance is 20%. As a check on the judge itself, we gave it a held-out sample of each brand's real writing, three times each: it was right 15 times out of 15. So when it misses a rewrite, the rewrite is not distinctive enough, not the judge being unable to tell.

02What we found

• Before: the judge named the right brand for 27% of rewrites, in two separate runs of 30 (60 rewrites in all). That is close to chance.
• The cause: the rewrite prompt is an edit prompt. It keeps the draft's paragraph count and length, accounts for every sentence, and bans new content. A padded draft keeps its own shape, and the voice has almost no room to show.
• The change: for a draft that has been reduced to plain wording first (see Plain first, then voice), the voice step now writes the points as the person writes them, instead of editing the plain text. The points are given as a list, not as sentences, so there is no sentence shape to copy. The model may reorder, merge or split sentences, and may add a framing line in the person's manner if it makes no claim of its own. The facts are still checked in code, and the meaning check still runs.
• After: the judge named the right brand for 40% to 57% of rewrites across four runs of 30, about 49% overall (59 of 120). Facts kept stayed at 100% in every run.

We also tried a version where the model first lists the writer's habits, a version that always reduces the draft to plain wording, and writing three candidates and keeping the one closest to the person's measured habits. None did better. The three-candidate version was also no better in a blind pairwise judgement (7 against 8, with 9 undecided).

03What it did not fix

• Email and letter voices. On the earlier benchmark, where voices are Enron emails and old letters, this change made no measurable difference: facts stayed at 100%, and a blind pairwise judge preferred the old output 11 times and the new one 8 times, with 5 undecided. We do not claim a gain there.
• The change applies to padded drafts only. It runs when the draft has three or more flagged stock phrases, and the sample has at least 80 words. Drafts below that go through the old edit path. In the brand test, the drafts with no flagged phrases showed no gain.
• Shorter output. Rewrites are about three quarters of the draft's length, because the padding goes.
• Invented brands, one model family. The brands are our own inventions, and they are very distinct from each other, which makes the test easier than real life. The judge and the writer are the same model family. No human has rated these. About half is better than a third, but half is still half: some rewrites will not sound like the brand.