Chapter 4
No single number of tries fits every question
Imagine you ask the toy shop robot for a birthday present. It names some toy cars. You ask again, with the same words, and the list changes a little. It is a bit like rolling dice.
Your customers can see the same thing. They can ask a robot the same question twice and hear different names.
So how many times should you ask? Start by seeing how much one question wobbles.
The short version. Ask one question about 10 times as a small trial. See how much the answers wobble. Set that beside the difference you care about. Ten tries help you explore the question. They do not prove that a change worked.
Wobble can be small, or nearly everything
Ask a robot one question again and again, and the count of names goes up and down. I call that the wobble. Its size only means something next to the difference you want to see.
Picture it. Take all the ups and downs in the count of names, and cut them into equal parts, one die each. Green dice are differences between questions or robots. White dice are wobble.
At Javier’s: like nightly orders. Set Saturdays beside rainy Mondays, and wobble is 1 part in 6. Set two ordinary Mondays side by side, and it is 14 parts in 15.
The gift estimate is less certain.
How to check this for your business
Javier’s small trial counts names from a list of local pizzerias. Birthday party answers hold many of them, and late-night pizza answers hold hardly any. That gap is much bigger than the wobble.
Then a friend says birthday party answers now hold a few more names. The count changes that much from one try to the next. Javier treats it as a clue and asks someone good with numbers to plan a proper test.
Try this on Monday
Take one question and one robot from your sheet in chapter 1. Ask that question about 10 times, in a new chat each time. Keep each answer, because this time you count inside it.
Count the names from your list in each answer. The gap between the lowest count and the highest is your wobble. Now write down how big a difference would matter to you. Set the two side by side.
If the wobble is as big as the difference you care about, get help to plan the next test before you act.
References
Chapter 4. No single number of tries fits every question
Preprint. Dmitrij Żatuchin (2026). The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations. arXiv:2609.04047, version 2, 14 October 2026. arXiv:2609.04047v2. Data. 10.5281/zenodo.22927143. Gift answers are listed in chapter 12.
What does AI say about your business?
Put the book into practice. Tell us your company and website to request an AI visibility audit. We’ll reply by email to agree what to check.
What the audit shows
- Where your company appears in the answers, and next to whom
- The pages AI cites in your market
- What to publish first on your own site
