Measures
Sentiment in AI answers: how engines describe you
What Sentiment measures in AI answers, how to read the quotes behind it, and the pages and listings to fix when an engine describes you poorly.
AI visibility results: how to tell if a page worked
How to compare AI answers before and after a page goes live, read the result with confidence, and know what a steady reading tells you.
On this page
You wrote a page to win a set of questions in AI answers. It is live. Someone will soon ask whether it worked, and you want an answer you can give in the room with confidence. This post shows how to set up the comparison, how to read what comes back, and what to do while you wait.
Decide what the page was meant to win
A page is written for a topic, and a topic is a set of questions. Before you publish, write down those questions. They are what you will compare, and the answer to "did it work?" is about them, not about your figures as a whole.
Your overall Mention rate is counted over many more questions than the page targets, and a gain on three or four of them can be hard to see in it. Narrow the view to the page's own questions.
Keep those questions exactly as they are while you wait for a result. In BotSay, rewording a question stops the old one and starts the new wording as a new question, so the weeks on either side of your date would no longer ask the same thing.
Note the day it went live
The comparison is built on that date. Without it, there is nothing to split into before and after.
In BotSay you press Mark published on the item in Work, check the date, and paste the page's address if you want it kept with the item. A note under the date says how many questions the page targets and when the first result shows. From then on, BotSay follows those questions for you.
If you track by hand, write the date in the same sheet as your weekly counts.
Compare the weeks before with the weeks after
The fair comparison is like with like: the same questions, asked the same way, on the same engines, in the weeks before the date and the weeks after it.
Three details keep it fair.
- Leave out the week the page went live. The page was live for part of that week and not the rest, so it belongs to neither side.
- Use several weeks on each side. Answers vary a little from day to day, and a single week can catch an engine on a good or a poor day. BotSay uses up to four whole weeks on each side.
- Ask on every engine, every week. A question asked on six engines gives six answers, and more answers give a firmer comparison.
BotSay shows the first counts once one whole week after the publish week has finished, and gives a verdict once at least two weeks with answers exist on both sides. Before then the sentence gives the counts and says plainly that it cannot say yet whether that is a change.
The example below follows one topic: the answers that named the brand each week, with the publish date marked.
Payroll for restaurants: answers that name youUp from 6 to 19 of 48
The figures as a table
| Week of | Answers that name you |
|---|---|
| 4 Aug | 6 of 48 |
| 11 Aug | 6 of 48 |
| 18 Aug (published) | 6 of 48 |
| 25 Aug | 9 of 48 |
| 1 Sep | 13 of 48 |
| 8 Sep | 16 of 48 |
| 15 Sep | 19 of 48 |
Example, using an invented brand
Read the result sentence
A result in BotSay is one sentence. It starts with the counts, for example "6 of 35 of those questions now name you, against 4 of 35 before." and ends with one of five verdicts:
- a rise larger than chance would explain;
- a fall larger than chance would explain;
- the figure leans one way but has not yet moved by more than chance would explain;
- "There is no change we can call real yet.";
- "We cannot judge a change from these counts."
"Larger than chance" is the test that makes the result safe to report. With a few dozen answers on each side, a move of one or two can happen with nothing changed at all. BotSay asks whether the move is bigger than that, and only then calls it a rise or a fall. When it does, you can report it with confidence.
A small line under the item plots the weekly counts with a dashed marker labelled published. A week with no answers is left out and the line breaks there, so the chart only shows readings BotSay took.
Sometimes the result reads We cannot compare yet. That means the engines did not answer the same number of questions on both sides of the date, and the item names the engine. It is BotSay holding back a comparison that would mislead, not a mistake on your side.
A steady reading is a result too
A steady reading means BotSay measured both sides, compared them, and nothing moved enough to count.
The weekly reading does the same for your figures as a whole. When nothing moved beyond chance, the weekly email says so and gives the size of change that would have counted, so you know where the bar sits for next week. The bar depends on how many answers stand behind the week: more answers give a smaller bar.
BotSay to you08:00
Tessera held second place this week.
You were named in 34.9% of answers, against 34.0% last week. It was a steady week. A change of 4.2 points or more would have counted. Novaly still leads.
Next Write about payroll for restaurants.
Example, using an invented brand
On a page's result, "There is no change we can call real yet" is worth reading the same way. Look at the counts in front of it. If they lean up, give it more weeks. If they have not moved, look at whether the page is being cited at all.
What a result does not tell you
A result tells you whether answers to the page's questions changed. It does not tell you why, and BotSay never says your page caused a change.
Pages often go live after a weak week, and some recovery follows on its own. A real rise after your date is still good news to report. Say what it is: a rise in the answers to those questions, larger than chance, since the date you published.
While you wait
A page has to be found before it can change an answer. These checks show whether it has been.
- Is it cited? On BotSay's Evidence screen, What they cited lists every site and page the answers linked to. Choose Pages in place of Sites and look for your new address. A single cited page shows how many answers leaned on it and which questions they answered.
- Which answers changed? The Gained and lost tab on Standing lists the questions that started or stopped naming you, by engine. A gain on the same question across several engines is worth more than one answer flipping.
- Who is cited instead? If a rival's page is still the one linked for your questions, open it beside yours and see what it answers that yours does not.
- What did the engine search for? For each answer, BotSay shows the searches the engine ran to write it, where the engine reports them. If your page does not answer those words, add them.
Our guide Is my AI visibility score real? shows how to check a change by hand, and How we measure sets out the test BotSay uses.
Doing it without a tool
You can run this comparison with a sheet. List the page's questions, ask each on ChatGPT, Perplexity, Gemini, Claude, Google AI Overviews and Google AI Mode every week in the same words, and record how many answers named you out of how many came back. Mark the publish week and leave it out. Once you have four weeks on each side, compare the totals and work out a likely range for each.
Five questions on six engines over eight weeks is 240 answers to ask, read and count for one page. BotSay does it for every page you mark published in Work, alongside your weekly reading.
What you gain with BotSay
- A before and after result for every page you mark published, on the questions that page was meant to win.
- A verdict only when the change is larger than chance, so what you report holds up.
- Counts from the first result, and a plain sentence when it is too soon to judge.
- On a steady week, the size of change that would have counted, so you know where the bar sits.
- One fixed monthly price with everything included and no add-ons. Plans differ only in the number of questions, from $59 a month for 60 questions.
Keep reading
Measures
Average rank in AI answers: how to move up the list
What Average rank counts in AI answers, why it can slip in a good week, how to read it beside Mention rate, and the work that moves you up the list.
Measures
Mention rate in AI answers: how often engines name you
What Mention rate counts in AI answers, how to read its likely range, why it differs by engine, and the work that raises it.
Your first reading runs right after setup
Find the page to write next.
Monthly plans from $59. Cancel at any time.