Chapter 14

A robot’s report card can flip with the question

A toy shop with toy cars on the left of the wall and books on the right. Two robots stand behind a counter. On the counter by the cars, the first robot's stack of blocks is tall and the second robot's is tiny. By the books, the second robot's stack is a little taller.

Imagine a toy shop with two robots. At the toy car shelf, the first robot names far more toys than the second. At the book shelf, the order swaps.

Your customers can ask robots about gifts or suppliers. A report that counts names for each robot can hold for one kind of question and fail on another.

I checked what happened when the questions changed from gifts to business.

The short version. On gift questions, Gemini named more brands per answer than GPT. On business questions, GPT named more. Before you use a report for a new kind of question, test that kind too.

The order of two robots flipped between two kinds of question

On gift questions, Gemini named about 8 brands per answer and GPT about 1. On business questions, about 3 and about 4.

Picture it. Each block is one brand name in an average answer. On gifts the Gemini row is longer. On business questions the GPT row is longer.

Gift questions, GeminiA row of 8 green blocks.8.41
Gift questions, GPT, the robot behind ChatGPTA row of 1 green block.0.94
Business questions, GeminiA row of 3 green blocks.3.01
Business questions, GPTA row of 4 green blocks.3.91

At Joost’s: like one bookseller naming about 8 titles per customer on gift books and another about 1, then about 3 and about 4 on school books.

A rule learned on one kind of question failed on the other

A rule learned on business questions named the wrong robot on all 22 gift report cards

Picture it. Each sheet is a report card: a few counts for one robot on one group of questions, such as gifts for a wife. Green means a program named the right robot, after learning on the other kind of question.

Learned on business questions, tested on gifts22 report cards, all of them white.0 of 22
Learned on gifts, tested on business questions10 report cards, 4 of them green.4 of 10

At Daniel’s: like telling two clerks’ work apart with a rule learned on 10 tenancy files, then getting all 22 divorce files wrong.

This happened in one comparison. The question type and the setup changed together.

How to check this for your business

Portrait of Kadri, who builds and sells homes.Kadri TammBuilds and sells homes, EstoniaMade-up example

Kadri gets a report. It says one robot names more builders than another on questions about family homes. She also sells small flats, so she checks flat questions before using the same order.

She asks a small set of questions about flats in the same setup and counts again. Then she decides which robot to read first.

Try this on Monday, with your last report

Take your last report on AI answers, or your sheet from chapter 1. Pick one result and note which kind of question produced it. A pizzeria might have asked about takeaway and now want to use the result for birthday parties.

Write five questions of the new kind. Ask them in the same setup and count again. If someone else made the report, ask them to run the new questions. Compare the new order with the old one.

References

Chapter 14. A robot’s report card can flip with the question

Preprint. Dmitrij Żatuchin (2026). System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer. arXiv, version 1. arXiv:2610.00253. Data. 574 of the gift answers are at 10.5281/zenodo.21904788.

What does AI say about your business?

Put the book into practice. Tell us your company and website to request an AI visibility audit. We’ll reply by email to agree what to check.

What the audit shows

  • Where your company appears in the answers, and next to whom
  • The pages AI cites in your market
  • What to publish first on your own site

Fields marked * are required.

Who is it for?

Your choice about cookies

We use cookies this site needs to work, and others only if you say yes. Choose what you allow, and change your answer whenever you like. Read our privacy policy