Chapter 15
Robots share a taste for robot writing
Another research team ran this test and published it in PNAS in 2025. I reread their score sheets.
Imagine a school fair. For each toy, one drawing is by a person and one by a robot, and a robot judge picks one. Their test was like that, with descriptions of products, science papers and films.
Your customers can ask a robot to compare two offers, and a robot may have written one. So I asked: does a robot favour its own writing?
The short version. In the authors’ test, robot judges picked robot-written product text far more often than people did. I found no clear bonus for a robot’s own writing, though a small one could exist. Show a robot two versions in both orders, because order can change its pick.
What I did
A result box is the score for one judge reading one writer on one kind of item. I rebuilt all 75 from the authors’ public files, then tested for an extra preference for a robot’s own writing.
On products, robots picked robot writing far more often than people did
Picture it. Each tag is one pick out of 100, between two descriptions of the same product. One came from a person and one from GPT-4, a robot. Green means the robot’s text won.
At Marta’s: like a helper robot choosing the robot-written product page 77 to 95 times in 100, when six research assistants choose it 36 times.
A robot’s own writing got no clear bonus
Picture it. Each sheet is one robot’s writing on one kind of item. Green means that robot, as a judge, liked this text more than the other four judges did.
At Giulia’s: like 15 invoices, each checked by five clerks, where the clerk who wrote it gives the top mark on 4.
Some robots write text most judges like. Some judges lean toward robot text in general. I took those two habits out and looked at what was left when a robot judged its own writing. That leftover is the bonus.
Picture it. Each slip is one pick out of 100. Green slips are extra picks for a robot’s own writing. The lower row shows why I cannot say zero: this test cannot tell a bonus under about 4 from no bonus.
At Joost’s: like a reader who says yes to about 2 more back-cover texts in 100 for their own text, on a count too rough to see a gap under 4.
Swap the two texts, and the pick can change
Some robot judges have a habit: they take the text in one place, whichever text sits there. That is why the authors showed each pair both ways round.
Picture it. Each pair of sheets is one item shown both ways round. A run is one judge reading one robot’s writing. White means the judge took the same place both times, so its pick changed. Green means it kept the same text.
At Samira’s: like a client who takes whichever of two ad drafts lies on top, on 40 of every 100 pairs.
On the sets I could compare, this order habit did not explain the small own-writing bonus.
How to check this for your business
Joost’s team asks a robot which of two back-cover texts reads better, and it picks one. So Joost asks again with the texts swapped. On some pairs the pick stays, and on others it changes. Where it changes, Joost ignores the robot and asks a reader.
Try this on Monday
If you ask a robot to compare two texts, such as two menu descriptions, try a few pairs. Ask about each pair twice, with the texts swapped, in a new chat. On your sheet from chapter 1, list the pairs and tick those where the pick changes. For those, ask a person. This tests your way of checking.
References
What does AI say about your business?
Put the book into practice. Tell us your company and website to request an AI visibility audit. We’ll reply by email to agree what to check.
What the audit shows
- Where your company appears in the answers, and next to whom
- The pages AI cites in your market
- What to publish first on your own site
