Skip to content
Fenn

All posts

AEO Platform Evaluation: Vendor Questions and a 2-Week Test

By FennUpdated 18 November 20269 min readAEOBuyer evaluation

During an AEO platform evaluation, ask each vendor six things in writing: which AI engines it queries and how (live product or API), how often and how many times it samples each prompt, how it counts a mention versus a citation, whether you can see the raw answers behind every number, what it does after measuring (reports only, or fixes and drafts you approve), and what you pay per prompt, engine and seat. Then run a fair two-week trial: fix the inputs first (the same 20 buyer questions, the same engines, the same counting rule and a hand-checked baseline), change nothing during the trial, and record daily what each platform reported, what it proposed or shipped, and what you could verify yourself. Judge on whether its numbers reconcile with your hand-checks and on work that reached production, not on dashboard polish. Two weeks can prove process quality; it cannot prove visibility gains.

AEO Platform Evaluation: Vendor Questions and a 2-Week Test

A trial run on vendor-supplied prompts, vendor-chosen engines and vendor framing measures the vendor. That is a demo with a longer runtime. A fair evaluation holds the inputs still and lets the products differ. This page gives the questions to ask first, then the two-week protocol.

Questions to ask vendors during an AEO platform evaluation

Send these in writing before the trial and keep the answers. A vendor that will not answer a question in writing has told you something.

Measurement

  1. Which AI engines do you query, and do you use each engine's live product or its API? Answers can differ between the two.
  2. How many times do you sample each prompt, how often, and from which locations?
  3. How do you count a mention, a citation and a hedged or partial reference? Can we see the rule?
  4. Can we see the raw answer text behind every number in the dashboard?
  5. How do you separate a change in the AI model from a change on our site?

Action

  1. After measuring, what does the platform do: report only, recommend changes, draft pages, or change the site?
  2. If it changes anything, who approves it, and is there a log of what was changed and when?
  3. Can our developers or our CMS receive the fixes in a form they already use?

Commercials and data

  1. What do we pay per prompt, per engine, per brand and per seat, and what happens when we exceed a limit?
  2. Is there monthly billing, and what is the notice period?
  3. Can we export all prompts, answers and results if we leave?
  4. Who else can see our prompts and results, and are they used to train anything?

Proof

  1. What will the trial prove, and what can it not prove in two weeks?
  2. Can we run the trial on our own prompts and engines, not yours?
  3. What would make you tell us not to buy?

Before day one

  1. Write the prompt set: about 20 questions a real buyer would ask, in their words, covering problem, comparison and pricing intent. This is yours, and it goes to every vendor unchanged.
  2. Pick the engines that match your buyers and freeze the list. Adding one mid-trial invalidates the comparison.
  3. Write down the counting rule: what a mention is, what a citation is, and how a partial or hedged reference is scored.
  4. Hand-check the full prompt set once and record the result. Without this baseline you cannot tell a platform's reading from reality.
  5. Decide the decision criteria now, in writing, including what would make you buy nothing.

During the two weeks

Record dailyWhy it matters
What the platform reportedThe claim under test
What it proposed or shippedSeparates measurement from execution
What you verified by handThe only number you fully trust
Anything that needed supportSupport quality is part of the product
Time you spentA tool that saves reporting time but costs review time is not a saving

The stochastic trap

These engines do not return the same answer twice, and a fair share of apparent movement inside two weeks is resampling rather than anything either party did. So do not judge on a snapshot: compare the shape of results across repeated samples, treat single-prompt changes as anecdote, and be suspicious of any vendor whose trial story attributes a two-week improvement to its product. We would be making that up too.

What two weeks can and cannot prove

It can prove whether the platform's numbers reconcile with your hand-checks, whether proposed changes are sane and reviewable, whether anything actually shipped, and how the product behaves when you disagree with it. It cannot prove ranking or citation outcomes: those move on the engines' schedule, not yours. Anyone promising demonstrated visibility gains inside a fortnight is describing noise, and treating that promise as disqualifying will save you a year.

Scoring at the end

CriterionQuestionScore 0 to 2
ReconcilabilityCould you reproduce its numbers by hand from the raw answers?0 no, 1 partly, 2 yes
Shipped workDid anything reach production, with an approval trail?0 nothing, 1 proposed only, 2 shipped and logged
FrictionHow much team time did it take for what it gave back?0 more than it saved, 1 even, 2 saved time
TransparencyDid written answers to the vendor questions hold up in the trial?0 no, 1 partly, 2 yes

A platform that wins on dashboards and loses on these is a reporting subscription. Run Fenn through exactly this protocol: our evidence log template is designed to be the record you keep afterwards, whichever way the decision goes.

Evaluating an AEO agency instead of a platform

The same logic applies to agencies, with different questions: which tools they use and whether you get access to the raw data, who writes and approves changes to your site, what they report monthly besides visibility numbers, and what they will not promise. Our guide to choosing an AEO agency lists the questions in full. Ask an agency to run the two-week protocol above on your prompt set before you sign.

Getting the budget approved

Finance will ask what the money buys and how you will know it worked. Bring three things: the cost per prompt, engine and seat from each vendor's written answers; the trial scorecard above; and a measurement plan for the next quarter that separates shipped work from outcomes. Our guide to an AEO programme budget you can defend covers the line items, and our quarterly review agenda covers how to report progress honestly.

FAQ

What should I ask vendors during an AEO platform evaluation?

Which engines they query and how, how often and how many times they sample, how they count mentions and citations, whether you can see raw answers, what the platform does after measuring, and the full price per prompt, engine and seat. The full list is above.

Is two weeks long enough to evaluate an AEO platform?

For process and reconcilability, yes. For outcomes, no, and no protocol fixes that. Buy on process quality and verify outcomes over the following quarter.

Should I trial two platforms at once?

Yes, on identical inputs, provided each has its own approval path so shipped changes do not interleave. Sequential trials drift because the market moves between them.

What if a platform's numbers disagree with my hand-check?

Ask for the sampling method in writing. A defensible gap has an explanation, such as different engines, refresh or citation rule. An unexplained gap is the finding.

What should a growth team ask an AEO agency?

Which tools it uses, whether you get the raw data, who approves changes to your site, what it reports besides visibility numbers and what it will not promise. Ask it to run a two-week test on your own prompts first.

How do I get budget approved for AEO?

Show the cost per prompt and seat in writing, a trial scorecard, and a quarterly plan that reports shipped work separately from outcomes, so the spend can be judged on what you can verify.

Put this to work on your own website.

Fenn finds what your customers ask, drafts the articles and site fixes, and measures what ChatGPT, Claude, Gemini, Perplexity and Grok say about you — with every change waiting for your approval.