How to audit your SEO agency's AI search performance

To audit your SEO agency's AI search performance, run a fixed set of patient questions on ChatGPT, Google, Perplexity and Copilot several times each, and record who is named, which sources are cited and whether the facts about your providers are right. Then ask the agency for its AI reports and a log of what it changed. An AI search audit is that comparison: what the engines say against what the agency can show.

What is an AI search audit, and why can't the rankings report do it?

It is a check on whether your practice is named and described correctly when a patient asks an AI assistant for a provider. A rankings report answers a different question: where your pages sit on a results page. Ahrefs' March 2026 study of 863,000 keyword results found that 37.9% of pages cited in Google's AI Overviews ranked in the top 10 for the same query. Most cited pages did not, so a strong rankings report tells you little about whether you are in the answer.

The audit below has two halves. Steps 1 to 5 are what you can see yourself. Steps 6 and 7 are what only your agency can show you. Step 8 puts them together. You need a spreadsheet and no special access for the first half.

This article describes a method. It is not a report of a test we ran, and I am not going to tell you what your results will be.

The audit, step by step

1. Write the question set. Twelve to fifteen questions is enough for one location. Cover four kinds: provider choice by procedure ("who is a good board-certified surgeon for a deep plane facelift in [your city]"), named provider ("is Dr. [name] a good dermatologist"), comparison ("[your practice] vs [competitor] for lip filler") and cost or candidacy ("how much is Invisalign in [your city] and who does it"). Use the words patients use on the phone with your coordinators, not your keyword list. Save the list. You will reuse it unchanged.
2. Pick the engines. Use ChatGPT with search on, Google (note both the AI Overview and AI Mode), Perplexity and Microsoft Copilot. Use a fresh chat for every question, signed out or in a private window where the engine allows it, so earlier chats do not shape the answer. Run them from your own market, since location changes local answers.
3. Run each question more than once. One answer proves nothing. Five runs per question per engine is a workable minimum for a manual audit. The next section explains why.
4. Record the same fields every time. Date, engine, question, whether your practice was named, which of your providers were named, which competitors were named, the reason given for each, and every source the answer linked.
5. Fact-check what was said about your providers. Check name spelling, credentials and board certification, procedures offered, address, hours and phone. Flag any provider who has left and is still named, and any current provider who never appears. Note which cited source carried each error.
6. Ask your agency for its reports. Send the list in the table below, in writing, and ask for the last three months.
7. Check access and crawler settings. Confirm you are an owner of your Search Console property and your Google Business Profile, and ask the agency to confirm which AI crawlers your robots.txt allows.
8. Score it and decide. Work out how often you were named, compare it with what the agency sent, and read the result using the four patterns at the end.

Why run every question several times?

Because the engines do not give the same answer twice. SparkToro's research, published January 28, 2026 with about 600 volunteers and 2,961 runs across ChatGPT, Claude and Google's AI, found ChatGPT and Google's AI had less than a 1 in 100 chance of returning the same list of brands in any two responses. The authors note the study was not peer reviewed and used consumer and business prompts, not healthcare ones.

Their conclusion is the one to borrow: a "position" in an AI answer is not a stable number, but how often you appear across many runs is a reasonable one. So the figure you want from step 8 is a share. If you ran 12 questions five times on four engines, that is 240 answers. Count the answers that named your practice and divide by 240. Do the same per engine and per procedure, because the totals hide the pattern.

The same logic applies to anything your agency sends. A screenshot of one good answer is an anecdote. Ask how many runs it came from.

What should you record about sources and provider facts?

Record every linked source and sort it into one of five groups: your own website, a directory or review platform (Healthgrades, RealSelf, Zocdoc, Google reviews), a professional society or hospital listing, a news article or "best of" roundup, or a competitor's site. The sources matter as much as the names, because they show where the work is.

If the answers that name competitors keep citing two or three directories where your profile is thin or out of date, that is a specific, fixable finding. If the answers cite your own procedure pages but still name someone else, the gap is more likely off your site. If no answer cites your site at all, check step 7 before anything else.

For provider facts, write down the error and the source it came from. "Says Dr. Patel is at our old address, source is a directory listing" is something an agency can act on in a day. "The AI gets us wrong" is not. Medical details carry more weight than most: a wrong board certification or a procedure you do not offer is worth fixing even if you were named.

What reports and access should you ask the agency for?

Ask for the items below. Each has a known limit, and an agency that knows this work will tell you the limits without being asked.

The first two are free and come from the platforms. Google announced on June 3, 2026 that Search Console has dedicated reports for impressions in generative AI features, with a note on the post that they reached all websites on August 31, 2026. The metric is impressions. Google's separate documentation says AI feature traffic is also counted in the main Performance report under the Web search type, so do not add the two together.

Microsoft's AI Performance report went into public preview on February 10, 2026. It shows total citations, average cited pages per day, page-level citation counts and sample grounding queries across Copilot, AI summaries in Bing and select partner integrations. Microsoft states the figures do not indicate ranking, authority or placement within an answer.

On access, you should hold the keys. Search Console's help page says only owners can add and remove users, and the Business Profile help page says only owners can remove other owners and managers. If your agency set these up under its own account, ask to be made an owner. Google's guidance on hiring an SEO goes the other way too: it advises giving a new firm read access only at the start.

For crawlers, ask for one line per engine. OpenAI's documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that robots.txt changes take about 24 hours to register. Perplexity recommends allowing PerplexityBot for the same reason. A security plugin or an old robots.txt rule can block these without anyone having decided to.

Does this audit involve patient data or HIPAA?

No. The questions are generic, the answers are public, and the Google and Bing reports show page-level search data, not patient records. You do not need a patient's name, photo, procedure or history for any step.

The one way to create a problem is to make a question "realistic" by pasting in a real patient's details, or to upload an intake form or a before-and-after file to a chatbot. Do not do either. If your agency's reporting relies on tracking tools placed on booking or portal pages, that is a separate question, covered in our guide to HIPAA and tracking pixels in 2026.

How to read the result

Put your share from step 8 next to what the agency sent. Four patterns cover most cases.

• You are named often, facts are right, and the agency's reports match. The work is being done. Keep the audit on a quarterly schedule and leave it there.
• You are named often, but the agency sent only a rankings report. You are doing well for reasons nobody is tracking. Ask them to start measuring before a competitor changes the picture.
• You are rarely named, and the agency already knew and has a work log. This is a fair position. AI answers move slowly and no agency controls them. Judge the plan and whether the log is specific, then re-run in a quarter.
• You are rarely named, and the agency has no AI reporting and no plan. That is the finding that justifies changing scope or adding help.

Two cautions. If the answers name no practices at all in your market, only directories and general advice, a low share is not your agency's failure. And nothing here should be read as a promise: Google's own page on hiring an SEO says no one can guarantee a top ranking, and AI answers are less predictable than rankings.

What to do with a poor result is its own decision, and the published price ranges for adding this work are on our AEO vs SEO cost comparison. For background on how engines build a provider shortlist, see what patients see when they ask AI to recommend a doctor.

If you would like the same test run independently, our free AI visibility audit runs your procedure and provider questions across the major engines and shows who is named and which sources are cited. More on how we work with surgeons, dentists, dermatologists and med spas is on the medical practices page.

Frequently asked questions