A site must be technically accessible to AIs before they can understand it. This chapter presents the six technical checks, agentic browsing, SEO basics, the Google profile and the site's technology.
What you will learn
- The 6 technical checks, including robots.txt and llms.txt
- Which AI bots to allow, and why Bing matters too
- How to read a PageSpeed report
Lesson outline
- 1What are the 6 technical checks to run on your site?
- 2What is agentic browsing and how do you find your score?
- 3What completes GEO: SEO, your Google profile and technology
- 4Does SEO still matter for GEO?
- 5Why fill in your Google profile properly?
- 6Which technology supports GEO best?
- 7What does a robots.txt file that welcomes AIs look like?
- 8What does an llms.txt file look like?
- 9How do you apply these checks on your platform?
- 10How do you read a PageSpeed Insights report?
- 11Which AI bots should you allow on your site?
- 12Why must your site's content display without JavaScript?
- 13Should you also submit your site to Bing and use IndexNow?
- 14Hands-on exercise
- 15Key takeaways
What are the 6 technical checks to run on your site?
A site must be accessible to AIs before they can understand it: the best content in the world is useless if crawlers cannot reach it. Six technical checks to run:
- ✓Your robots.txt file. Make sure you are not blocking AI crawlers by mistake (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, etc.), and publish an llms.txt file that presents the essentials of your site.
- ✓How your site handles JavaScript. If your content only appears after JavaScript runs, some crawlers will not see it. The fix: server-side rendering.
- ✓Your page speed. A slow site is harder to crawl and to use, for humans and agents alike.
- ✓Clean HTML structure. Logical headings, well-organized sections, one idea per paragraph.
- ✓Schema markup. It states explicitly what a page, a service or a question-and-answer is.
- ✓URLs "invented" by AIs. Sometimes an AI cites an address that does not exist on your site. Make sure those links lead to a useful page (a redirect or a well-designed error page) rather than a dead end.
AIs must be able to access your content and understand it. Blocked access or unreadable content cancels out everything else.
What is agentic browsing and how do you find your score?
Agentic Browsing is an experimental category added by Google to PageSpeed Insights and Lighthouse. It evaluates whether a website can be easily understood, crawled and used by conversational agents or autonomous AI assistants. The score is out of 3.
| What it evaluates | In plain terms |
|---|---|
| Accessibility tree | A clear site structure that helps tools understand the content and the different elements of the page. |
| Discovery files | The presence and readability of AI-friendly indexing files: robots.txt, sitemap.xml and llms.txt. |
| Layout stability | Minimal visual shift (low CLS), so the AI does not lose track of interactive elements. |
PageSpeed is a good indicator, but not an absolute measure. The score can vary with the device, connection, server, cache and test conditions.
Many businesses have not adapted yet
It is common to come across sites scoring 0/3 in agentic browsing, with zero citations in ChatGPT, Gemini, Perplexity and Copilot. Test yours for free on PageSpeed Insights: five minutes is enough to know where you stand.
What completes GEO: SEO, your Google profile and technology
GEO is completed by three elements that are often forgotten and make a real difference: solid SEO, a well-kept Google profile and suitable technology.
Does SEO still matter for GEO?
Yes: SEO matters entirely for GEO, because AIs must first discover, access, understand and evaluate a site's content. Here are the five SEO basics to work on.
| SEO basic | Why it also helps GEO |
|---|---|
| Technical SEO | A fast, accessible, well-structured site makes crawling and interpreting content easier (mobile-friendly site, lighter images, etc.). |
| SEO content | Well-written, relevant pages give AIs more reliable information to understand and cite. |
| Keywords and search intent | They clarify the topics and questions you answer. Target "plumber in Laval" rather than "plumber". |
| Internal linking | It helps show how your pages and topics relate, for example by linking the "Services" page to each service page. |
| Backlinks | Links from credible sites (La Presse, Radio-Canada, Les Affaires) help establish your authority and reputation. |
Why fill in your Google profile properly?
Your Google profile (Google Business Profile, formerly Google My Business) is often the first thing a local customer sees. A complete, consistent profile also helps search engines and AIs better understand and identify your business.
- ✓Keep the profile complete and up to date: hours, address, phone, website.
- ✓Fill in every service you offer.
- ✓Choose the right business categories.
- ✓Add quality photos.
- ✓Get reviews and respond to them.
- ✓Post regularly with news and offers.
Which technology supports GEO best?
A site that AIs read well loads fast, sends its content already rendered by the server, and gives you control of its code. For a long time the default answer was WordPress. At Oriana, we moved to a modern stack: Next.js and TypeScript for server-side rendering, hosting on Vercel or Azure, Cloudflare for performance and security, and tools like Claude and Visual Studio Code to build faster.
This is not a condemnation of WordPress: a well-configured WordPress site can perform very well. But by controlling the code end to end, the technical points of GEO (server rendering, speed, clean HTML, schema markup) are easier to fix. We explain our reasoning in Why we advise against WordPress, and our custom website at $1,500 delivered in 7 days is built on this approach.
What does a robots.txt file that welcomes AIs look like?
The robots.txt file sits at the root of your site (your-site.ca/robots.txt). This example allows all crawlers, including those of the main AIs. Adapt it if you want to block certain pages, and replace the sitemap address.
# Allow all crawlers, including AI ones
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://www.votre-site.ca/sitemap.xmlWhat does an llms.txt file look like?
The llms.txt file presents the essentials of your site to AIs in plain language. It is a proposed standard: no AI commits to using it, but it costs little to publish. Place it at the root (your-site.ca/llms.txt).
# Your business name
> One sentence explaining what you do, for whom and where.
## Services
- [Service 1](https://www.your-site.ca/service-1): what it includes, for whom, at what price
- [Service 2](https://www.your-site.ca/service-2): what it includes, for whom, at what price
## About
- [About us](https://www.your-site.ca/about)
- [FAQ](https://www.your-site.ca/faq)
- [Contact](https://www.your-site.ca/contact)How do you apply these checks on your platform?
The steps vary depending on the tool that runs your site. Always check your platform's current documentation. Universal test: open your page, choose "View page source" and search for a sentence from your text. If it is there, the content is visible without JavaScript.
| Platform | robots.txt and sitemap | Content visible without JavaScript |
|---|---|---|
| WordPress | Editable with an SEO plugin (Yoast, Rank Math) or a file at the root | Yes, content is generated by the server |
| Wix | robots.txt is editable in the platform's SEO settings | Check with the page source view |
| Shopify | robots.txt is customized in the theme (robots.txt.liquid file) | Yes, content is generated by the server |
| Next.js | Files app/robots.ts and app/sitemap.ts | Yes, server-side rendering by default |
Platform features evolve: confirm in their documentation. For llms.txt, check that your platform lets you publish a file at the root.
How do you read a PageSpeed Insights report?
PageSpeed Insights is free and gives a starting point, not an absolute measure: the score varies with the device, connection, server and cache.
- ✓Enter your page address at pagespeed.web.dev and choose the Mobile tab.
- ✓Note the scores for Performance, Accessibility, Best Practices and SEO, plus agentic browsing (out of 3).
- ✓Look first at CLS (visual shift) and agentic browsing.
- ✓Open the diagnostics to see suggested fixes, from most to least useful.
- ✓Fix, then retest and keep the before and after scores.
Key takeaways from chapter 9
- 1AIs must be able to access your content and understand it: robots.txt, llms.txt, server-side rendering, speed, clean HTML.
- 2SEO remains the foundation, and the Google profile completes it.
- 3PageSpeed is a good indicator, not an absolute measure.
- 4Allow the AI search bots, show content without JavaScript, and submit your site to Bing.
Sources for this chapter
- 1.Google PageSpeed Insights and Lighthouse — Experimental "agentic browsing" category.
- 2.OpenAI, bots documentation (OAI-SearchBot, GPTBot, ChatGPT-User) — Role of each bot and effect of robots.txt.
- 3.Perplexity, bots documentation — PerplexityBot (search, no training) and Perplexity-User.
- 4.Anthropic, crawlers — ClaudeBot, Claude-User and Claude-SearchBot.
- 5.Vercel, "The rise of the AI crawler" — OpenAI, Anthropic, Perplexity and Meta crawlers do not execute JavaScript.
- 6.IndexNow, documentation — Protocol to notify search engines, and the key file.
Which AI bots should you allow on your site?
Each AI uses several bots, with different roles. Blocking the wrong one makes you vanish from answers. Here are the main ones, according to each publisher's documentation.
| Bot | What it does | Our advice |
|---|---|---|
| OAI-SearchBot (OpenAI) | Powers ChatGPT search. If blocked, your site does not appear in its search answers. | Allow. |
| GPTBot (OpenAI) | Collects content to train OpenAI's models. | Your call: blocking it does not affect ChatGPT search. |
| ChatGPT-User (OpenAI) | Visits a page when a user asks. robots.txt may not apply. | Nothing to configure. |
| PerplexityBot | Makes your site appear in Perplexity results. Does not train models. | Allow. |
| Claude-SearchBot (Anthropic) | Indexes content to improve Claude's search results. | Allow. |
| Claude-User (Anthropic) | Fetches a page when a user asks Claude a question. | Allow. |
| ClaudeBot (Anthropic) | Collects content for model training. | Your call. |
Open your robots.txt file (yoursite.com/robots.txt) and check that no "Disallow: /" rule targets the bots marked "Allow".
Why must your site's content display without JavaScript?
A Vercel study of OpenAI, Anthropic, Perplexity and Meta crawlers found they do not run JavaScript: they read the HTML they receive, nothing more. If your text, prices or menu only appear after a script loads, these bots see an empty page.
The 30-second test: open your page, right-click then "View page source", and search (Ctrl+F) for a sentence from your text. If you cannot find it, ask your developer to render the content on the server.
Google does run JavaScript, which is why the problem often goes unnoticed in SEO.
Should you also submit your site to Bing and use IndexNow?
Yes, and it is free. Bing is the search index behind Microsoft Copilot, and several AI tools rely on classic indexes: a site missing from Bing can miss those answers.
1) Create a Bing Webmaster Tools account: you can import your site straight from Google Search Console.
2) Submit your sitemap: the list of your pages.
3) Turn on IndexNow: this protocol notifies participating engines as soon as a page is added, changed or deleted, without waiting for their next visit. It only needs a key file on your site; most platforms (WordPress, Shopify, Wix) offer it through a plugin or a setting.
Run your site through PageSpeed Insights, note your scores, then open your-site.ca/robots.txt: are AI crawlers blocked there? Note one fix to make.
