Agentic SEO: being found and understood by AI search and AI agents

What “agentic SEO” means

Plenty of people ask an AI tool a question instead of typing keywords into a search box. Google shows AI-written answers, called AI Overviews and AI Mode, above or alongside its regular results. ChatGPT and Claude can search the web and answer with links. Some assistants also open pages on a person’s behalf and read them to finish a task.

“Agentic SEO” is an informal name for making a website work well with these tools. No standards body defines it, and you’ll also see “AI SEO” or “generative engine optimization” used for the same idea. In practice there are three parts to it. The tools need permission to read your pages. Your services, location, hours, and prices need to be in plain text they can read and quote. And when an assistant visits for someone, the way to call, book, or ask a question has to be clear from the page itself.

For a small business, most of this is the same work that has always made a site good for search engines and for people. What’s new is mostly a set of decisions about which AI crawlers you let in.

Why it matters to your business

An AI answer can only draw on pages its system can reach. If your site blocks the crawlers a tool uses for search, or your key details exist only inside images, that tool has less to go on when someone asks about the kind of work you do.

Google has also commented on the visits these answers send. In its guidance for site owners, it writes that when people click from search results pages with AI Overviews, “these clicks are higher quality,” meaning visitors are more likely to spend more time on the site (Google Search Central). That’s Google’s own observation rather than an independent study.

You can see some of this traffic yourself. Google says visits from its AI features are counted in your overall Search traffic in Search Console, in the Performance report under the “Web” search type. They aren’t broken out on a separate line.

How it works

Google’s AI features run on the regular Search index

There’s no separate system to sign up for. Google says that to appear as a supporting link, “a page must be indexed and eligible to be shown in Google Search with a snippet,” and that there are “no additional technical requirements.” You don’t need “new machine readable files, AI text files, or markup,” and “there’s also no special schema.org structured data that you need to add” (Google Search Central).

Google describes a technique it calls “query fan-out”: to build one answer, its systems may run several related searches across subtopics. A page that answers one specific question well, such as what a drain inspection includes, can end up cited in a broader answer about drain problems.

The best practices Google lists for its AI features are the same ones that apply to all of Search:

• Let Google crawl your pages, in robots.txt and in any CDN or hosting layer in front of your site.
• Make content easy to find through internal links.
• Provide a good page experience.
• Keep important content in text, with good images and video where they help.
• Make sure any structured data matches what’s visible on the page.
• Keep your Google Business Profile information current.

None of this guarantees a spot. Google says it “doesn’t guarantee that it will crawl, index, or serve your page” for any search feature (How Search works).

Other AI companies use their own crawlers

OpenAI and Anthropic run their own crawlers and publish their names, so site owners can allow or block each one in robots.txt. That’s the plain text file at the root of a site that tells crawlers which pages they may read (Google’s robots.txt introduction explains the format).

Both companies keep search and training separate. This table lists what OpenAI and Anthropic document, plus Google’s control for its Gemini models:

Sources: OpenAI’s crawler overview, Anthropic’s crawler article, and Google’s common crawlers page. Companies add and rename crawlers, so check these pages before relying on a name.

In OpenAI’s words, “Each setting is independent of the others.” You can let OAI-SearchBot in, so you appear in ChatGPT search, and block GPTBot, so your pages aren’t used for training. For OpenAI and Anthropic, that robots.txt looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Read up on Google-Extended before you block it. Google says it covers training future Gemini models and also grounding, which means supplying Search content to Gemini when someone asks it a question. Blocking it doesn’t affect Google Search, but it can keep your content out of those Gemini answers. Either choice is reasonable as long as you make it on purpose.

llms.txt is a proposal, not a standard

You may hear that every site needs an llms.txt file. Jeremy Howard put it forward as “a proposal to standardise on using an /llms.txt file”: a short Markdown file at the root of a site that gives AI tools a summary of the business and a list of links to its most useful pages (llmstxt.org). His reasoning is that web pages wrap their information in navigation, ads, and scripts, and that AI context windows are “still too small for most websites in their entirety.”

No standards body has adopted it. The proposal says it’s meant mainly for the moment an AI tool is answering a question, rather than for training, and that it “can complement robots.txt by providing context for allowed content.” It doesn’t allow or block anything. Google says AI text files aren’t needed for its AI features, and neither OpenAI’s nor Anthropic’s crawler documentation mentions llms.txt.

An llms.txt file won’t raise your place in anyone’s results. It’s a small file that gives any tool that reads it a clean summary of your business, and that’s the reason we include one.

Assistants that visit your site for a person

ChatGPT-User and Claude-User in the table are the assistants working for a person. When someone asks an assistant to check your hours or compare your services with a competitor’s, it fetches your page and reads its code. Four things make that work.

Put phone numbers, prices, and service areas in text. Inside a photo, they’re invisible to anything that can’t read images reliably.

Write links and buttons that say where they go. “Book a free estimate” tells people and software what happens next; “Click here” tells them nothing.

Give every form field a label. The W3C’s forms tutorial shows how to label fields so assistive technology can identify them (W3C forms tutorial), and other software reads the same labels.

Keep your facts the same everywhere. Google recommends keeping your Business Profile current, and if your site says you close at 5 while your profile says 6, an assistant has to guess which is right.

We already build pages this way for accessibility. Screen readers, search engines, and assistants all rely on the same structure.

Check your own site

Most of these checks take a few minutes each.

1. Open your robots.txt by typing your address followed by /robots.txt (for example, yourbusiness.com/robots.txt). Look for lines naming the crawlers in the table. User-agent: * followed by Disallow: / blocks every crawler, Google included.
2. Ask your host whether it blocks bots. Some hosting and security services turn away automated visitors before robots.txt is even read, and Google mentions this layer in its guidance.
3. In Google Search Console, run URL Inspection on your home page and main service pages. A page that isn’t indexed can’t appear in Google’s AI features.
4. Ask ChatGPT, Claude, and Google what your customers ask, such as “Who does drain cleaning in [your town]?” or “What are [your business]’s hours?” Answers change from person to person and day to day, so this is a spot check, not a measurement.
5. Try to select your phone number, hours, service area, and prices with your cursor. If you can’t, they’re probably part of an image.
6. Visit yourbusiness.com/llms.txt. If there is one, check that it describes your business accurately and that its links work.

A checklist

• Search crawlers you want (Googlebot, OAI-SearchBot, Claude-SearchBot) are allowed in robots.txt.
• Training crawlers are allowed or blocked because you decided, not because of a default you never saw.
• Your hosting or security layer isn’t blocking crawlers you’ve allowed.
• Your main pages are indexed in Google Search Console.
• Services, service area, hours, phone, and prices are in text, not only in images.
• Each question customers often ask has an answer on a page you can link to.
• Buttons and links describe where they lead; form fields have visible labels.
• Your website and Google Business Profile show the same name, address, phone, and hours.
• Structured data matches what visitors can see.
• Your llms.txt, if you have one, is accurate and its links work.

How we handle it

Every client site we build ships with an llms.txt file that summarizes the business and links to its main pages, and with structured data describing the business. By default, the site’s robots.txt lets AI search crawlers in and blocks crawlers that collect content only for AI training. If you’d rather allow training, or block more, tell us and we’ll change it.

Under that is what goes into every build: semantic markup, clear headings, business details in text, and pages designed to WCAG 2.2 AA, with keyboard navigation, labeled forms, and a clear page structure.