Chapter 14
A robot’s report card can flip with the question
Imagine a toy shop with two robots. At the toy car shelf, the first robot names far more toys than the second. At the book shelf, the order swaps.
Your customers can ask robots about gifts or suppliers. A report that counts names for each robot can hold for one kind of question and fail on another.
I checked what happened when the questions changed from gifts to business.
The short version. On gift questions, Gemini named more brands per answer than GPT. On business questions, GPT named more. Before you use a report for a new kind of question, test that kind too.
The order of two robots flipped between two kinds of question
Picture it. Each block is one brand name in an average answer. On gifts the Gemini row is longer. On business questions the GPT row is longer.
At Joost’s: like one bookseller naming about 8 titles per customer on gift books and another about 1, then about 3 and about 4 on school books.
A rule learned on one kind of question failed on the other
Picture it. Each sheet is a report card: a few counts for one robot on one group of questions, such as gifts for a wife. Green means a program named the right robot, after learning on the other kind of question.
At Daniel’s: like telling two clerks’ work apart with a rule learned on 10 tenancy files, then getting all 22 divorce files wrong.
This happened in one comparison. The question type and the setup changed together.
How to check this for your business
Kadri gets a report. It says one robot names more builders than another on questions about family homes. She also sells small flats, so she checks flat questions before using the same order.
She asks a small set of questions about flats in the same setup and counts again. Then she decides which robot to read first.
Try this on Monday, with your last report
Take your last report on AI answers, or your sheet from chapter 1. Pick one result and note which kind of question produced it. A pizzeria might have asked about takeaway and now want to use the result for birthday parties.
Write five questions of the new kind. Ask them in the same setup and count again. If someone else made the report, ask them to run the new questions. Compare the new order with the old one.
References
Chapter 14. A robot’s report card can flip with the question
Preprint. Dmitrij Żatuchin (2026). System Attribution in LLM Brand Recommendations: Single Responses Identify the System, Aggregated Brand Profiles Do Not Transfer. arXiv, version 1. arXiv:2610.00253. Data. 574 of the gift answers are at 10.5281/zenodo.21904788.
What does AI say about your business?
Put the book into practice. Tell us your company and website to request an AI visibility audit. We’ll reply by email to agree what to check.
What the audit shows
- Where your company appears in the answers, and next to whom
- The pages AI cites in your market
- What to publish first on your own site
