Is your website accessible to AI search bots? Here’s how to check and why it’s important.
Let’s break it down. Your website might feature the most valuable and authoritative content in your industry. Yet, it could still be overlooked and fail to get cited or referenced by AI Search Engines like ChatGPT – and you might not even be aware of it.
Why does this happen? Simply put, you may not know whether AI bots are even crawling your site.
How do I know if AI is crawling my site?
To determine whether search engines or AI search bots are crawling your site and its content, you’ll need to analyze your website’s log files.
Option 1: Check your website’s log files manually
This approach is both straightforward and a bit challenging. Start by downloading your website’s log files. Once you have them, search for specific bot user-agent names within those files. For example, here’s what ChatGPT’s Searchbot look like:
Crawler: OAI-Searchbot
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbotWhen you notice this crawler in your log files, it indicates that OpenAI’s Searchbot has accessed your website or specific URLs. This bot is initiated whenever a user enters a prompt on chatgpt.com, prompting OpenAI to perform a web search.
Option 2: Use the AI Crawler Simulation Tool from OtterlyAI
Use: otterly.ai/crawltest
Don’t have access to your website’s log files? Or looking to avoid the hassle of manual checks? Here’s a quicker way to determine if AI bots can crawl your site. The AI Crawler Simulation Tool by OtterlyAI mimics requests from leading AI web crawlers to assess whether your website blocks them. It works by sending real HTTP requests using the unique user-agent strings of various AI crawlers.
Note: This method doesn’t guarantee that your site is actively being crawled, but it does confirm whether it is technically crawlable.
What Affects the Crawlability of My Website?
Understanding the factors that determine whether your website is crawlable is crucial. The ability of AI bots—such as those from OpenAI, Perplexity, or Claude—to crawl your site largely depends on these key aspects:
CDN / Frontend Hosting
The cloud hosting or Content Delivery Network (CDN) provider you choose plays a significant role in your website’s crawlability. Well-known providers like Cloudflare, Netlify, and Vercel typically offer features that let you manage how AI crawlers interact with your site. However, some hosting providers may not give you this level of control and could even block AI bots from crawling your website by default. Yes, some don’t even tell you. I wrote about this in my Linkedin Post here.
Robots.txt file
The robots.txt file is a text document found in the root directory of a website. You can view the robots.txt file for any website by visiting: domain.com/robots.txt.
This file provides guidance to search engine crawlers (like Google) on how they should engage with your website’s URLs. It also establishes rules—using the “Disallow” directive—that specify which URLs or pages crawlers are permitted to access and which they are not.
Below is an example of a basic robots.txt file that grants all crawlers full access to all URLs:
User-agent: *
Allow: /And here’s a longer robots.txt file from the NYTimes that also disallows AI bots to crawl its content:
User-agent: GPTBot
Disallow: /
User-agent: Meta-ExternalAgent
User-agent: meta-externalagent
Disallow: /
User-agent: Meta-ExternalFetcher
User-agent: meta-externalfetcher
Disallow: /
User-agent: MyCentralAIScraperBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
Ensuring your robots.txt file specifies which bots are allowed or disallowed gives you control over the content that can be crawled and what remains restricted.
Dynamically generated content
Finally, we encourage you to verify whether your content is generated dynamically or statically on the frontend. OpenAI crawlers may struggle to access dynamically created content, which could result in your content being invisible to OpenAI.
With the help of OtterlyAI’s GEO Audit tool, you can check how much dynamic content you have on any particular URL or website:
The GEO Audit from OtterlyAI checks dynamic content ratio on any URLSummary
Here’s a summary on what to check:
- Make sure your CDN / hoster doesn’t blow AI bots
- Make sure your robots.txt file is correctly configured – allowing AI bots crawl your relevant content
- Make sure your content is not dynamically generated on the frontend
Top AI Crawlers: Which AI Search Bots and Crawlers should I check in the first place?
Below you can find a list of the most important crawlers and bots from AI Search Platforms.