Chapter 4

No single number of tries fits every question

A toy shop counter. One robot has two speech bubbles, each holding three toy cars, and the two lists differ a little. Two big dice sit on the counter. A child looks up, and a man with a clipboard takes notes.

Imagine you ask the toy shop robot for a birthday present. It names some toy cars. You ask again, with the same words, and the list changes a little. It is a bit like rolling dice.

Your customers can see the same thing. They can ask a robot the same question twice and hear different names.

So how many times should you ask? Start by seeing how much one question wobbles.

The short version. Ask one question about 10 times as a small trial. See how much the answers wobble. Set that beside the difference you care about. Ten tries help you explore the question. They do not prove that a change worked.

Wobble can be small, or nearly everything

Ask a robot one question again and again, and the count of names goes up and down. I call that the wobble. Its size only means something next to the difference you want to see.

Wobble was about 1 part in 6 across different questions, and 14 parts in 15 across versions of one question

Picture it. Take all the ups and downs in the count of names, and cut them into equal parts, one die each. Green dice are differences between questions or robots. White dice are wobble.

250 different questionsSix dice: five green and one white.1 in 6
5 versions of one gift questionFifteen dice: one green and fourteen white.14 in 15

At Javier’s: like nightly orders. Set Saturdays beside rainy Mondays, and wobble is 1 part in 6. Set two ordinary Mondays side by side, and it is 14 parts in 15.

The gift estimate is less certain.

How to check this for your business

Portrait of Javier, who runs a pizzeria.Javier MoralesPizzeria, SpainMade-up example

Javier’s small trial counts names from a list of local pizzerias. Birthday party answers hold many of them, and late-night pizza answers hold hardly any. That gap is much bigger than the wobble.

Then a friend says birthday party answers now hold a few more names. The count changes that much from one try to the next. Javier treats it as a clue and asks someone good with numbers to plan a proper test.

Try this on Monday

Take one question and one robot from your sheet in chapter 1. Ask that question about 10 times, in a new chat each time. Keep each answer, because this time you count inside it.

Count the names from your list in each answer. The gap between the lowest count and the highest is your wobble. Now write down how big a difference would matter to you. Set the two side by side.

If the wobble is as big as the difference you care about, get help to plan the next test before you act.

References

Chapter 4. No single number of tries fits every question

Preprint. Dmitrij Żatuchin (2026). The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations. arXiv:2609.04047, version 2, 14 October 2026. arXiv:2609.04047v2. Data. 10.5281/zenodo.22927143. Gift answers are listed in chapter 12.

What does AI say about your business?

Put the book into practice. Tell us your company and website to request an AI visibility audit. We’ll reply by email to agree what to check.

What the audit shows

  • Where your company appears in the answers, and next to whom
  • The pages AI cites in your market
  • What to publish first on your own site

Fields marked * are required.

Who is it for?

Your choice about cookies

We use cookies this site needs to work, and others only if you say yes. Choose what you allow, and change your answer whenever you like. Read our privacy policy