How to choose a useful AI-search prompt test set

A prompt such as “Why is our brand the best?” mostly tests a leading question. A buyer is more likely to ask which product fits a budget, whether two models differ, or what information to check before choosing. Start with that decision and then write neutral language that resembles a plausible customer question.

Use support conversations, product research, onsite search, and public questions as inputs. Keep a source note for each question family. A prompt invented by the marketing team can still be useful, but label it as a hypothesis rather than evidence of common demand. Do not assign a search volume unless a suitable source actually measures it.

Pranjal Aggarwal, Vishvak Murahari, and their coauthors introduced GEO-bench as part of their research on generative-engine visibility. Their work is useful because it makes evaluation explicit and acknowledges variation across domains. It does not establish that one fixed prompt list or one reported experimental lift predicts an ecommerce brand’s results on current consumer platforms.

Microsoft researchers’ work on search measures likewise emphasizes that a useful metric must work in its real operating context. Our application is to define the decision the monitoring program supports before selecting a headline score. A test set used to find factual errors should not be evaluated as if it were a revenue attribution system.

Keep branded questions separate from unbranded category discovery. A brand can answer questions about itself accurately without being suggested for a broad category, and those are different observations. Include constraints that matter to the target market, such as US availability, product size, or a verified compatibility requirement.

The example allocation below is a design illustration, not a recommended universal sample size. A small brand may need fewer questions; a large catalog may require several product-level samples. The key is to state what the set covers and what it leaves out.

Record the platform, visible mode or model information, date, market or location settings, language, whether the session was fresh, and whether web search was used. Save the exact prompt and complete answer with its visible sources. These details help explain why a result differs without guessing at an invisible algorithm.

Use repeated observations where the decision warrants them, and keep the protocol stable when comparing periods. If the team changes half the questions, the next score describes a different sample. Show the change rather than drawing a continuous trend line across incompatible test sets.

Imagine a brand launching a three-product routine. The initial agency list contains twenty prompts, sixteen of which mention the brand. The resulting high mention rate looks impressive but says little about discovery. The team rebuilds the set into brand-fact, category, comparison, and purchase-constraint groups, using customer language where available.

In the first run, branded facts are mostly accurate, but some constraint questions lead to unavailable global products. That finding suggests a regional product-information problem. It does not show that the brand needs twenty more generic blog posts. The next action is to inspect the sources and clarify the US offer.

The team preserves the original run as a historical sample and labels the revised set as a new version. It does not compare the two mention percentages as if the population were unchanged. This fictional example shows why prompt selection can matter more than the appearance of precision in a dashboard.

Choose a small set of observable labels: brand mentioned, official source linked, correct product version, correct market, supported comparison, and material factual error. Keep a codebook with examples and allow “unclear” when the evidence does not support a confident classification. A mention can be negative, irrelevant, or inaccurate, so counting it alone can mislead.

Have a second reviewer inspect ambiguous observations where possible, and accurately describe whether that review was human or AI-assisted. Do not manufacture a specialist review claim. The report should expose the underlying examples so the brand owner can inspect the reasoning.

A category-discovery prompt can become accidentally branded through a unique slogan, proprietary ingredient name, or an unusually specific combination copied from the brand’s own page. Review the wording with someone who did not write the campaign. Ask whether a real customer could plausibly ask the question without already knowing the intended answer.

Keep highly specific prompts when they serve a real product-accuracy question, but classify them accordingly. A prompt that names the exact model is useful for checking facts. It should not contribute to an unbranded discovery score. This distinction makes the test set more honest without discarding useful observations.

For comparisons across time, retain a stable core and a separately labeled exploratory set. The stable core supports like-for-like observations; the exploratory set lets the team investigate new customer questions or product launches. Do not merge their results into one trend without showing the changed composition.

Finally, record why each question remains in the set. If a product retires, a market closes, or a customer decision is no longer relevant, version the set and explain the change. An observation program should evolve with the business while preserving enough history to interpret what changed. A larger prompt count is not automatically a better sample, especially when many questions are slight variations of the same decision.

Dahna can help turn customer decisions into a balanced observation set, document the protocol, and connect findings to a practical content or website backlog. Bring the products, markets, and questions that matter commercially. The deliverable should include the prompts and saved evidence, not just a proprietary visibility number.

Use the resulting observations alongside referral and commercial data. They answer different questions, and keeping them separate makes the overall program more credible.

Use these guides when the next question goes beyond the scope of this article.

Download the completed tables as a CSV, then replace the example inputs and add your evidence, owner and review date. Illustrative figures are teaching examples, not default targets.