Skip to content

AI Crawler Statistics: GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot

AI training now drives most crawler requests, and AI platforms crawl hundreds to tens of thousands of pages per visitor sent. Here is who crawls, why, and who blocks them.

By Jimmy PalFounder, GTM 8020

Published 11 min read View as Markdown

AI crawlers now make more requests for training than for anything else. In June 2026, 52% of the crawler requests Cloudflare saw were for AI training, up from 22% in spring 2025. The traffic they send back is tiny: AI platforms have made anywhere from 118 to nearly 50,000 crawls for every visitor referred. Below are 32 numbered statistics on GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, covering crawl volumes, growth, crawl-to-referral ratios and robots.txt blocking. Each one comes from Cloudflare, Vercel, Fastly, Ahrefs, OpenAI or another original source, and all were read on 2 November 2026.

Start a free trial, no card. See whether the AI engines crawling your site actually name your brand.

What are the key takeaways?

  • Training dominates AI crawling. 52% of crawler requests on Cloudflare were for AI training in June 2026, up from 22% in spring 2025.
  • GPTBot is the largest AI-only crawler. Its share of AI-only crawling rose from 5% to 30% between May 2024 and May 2025, and it made 569 million fetches across Vercel's network in one month.
  • User-triggered fetching is the fastest-growing kind. Cloudflare saw user-action crawling grow more than 15x in 2025, and daily requests from AI agents grew more than 1,700% in a year.
  • Crawls far outnumber clicks. Anthropic's crawl-to-referral ratio peaked at 500,000:1 in 2025, against 3:1 to 30:1 for Google.
  • Sites block training, not search. Fewer than 1% of Cloudflare sites block search crawlers, while 17% block training.
  • AI crawlers don't render JavaScript. None of the major AI crawlers in Vercel's data executed it, so content that loads client-side is invisible to them.

How much traffic do AI crawlers generate?

AI crawlers are a large and growing slice of web traffic, but still smaller than Googlebot. In late 2024, GPTBot, Claude, AppleBot and PerplexityBot together made about 1.3 billion fetches a month on Vercel, a little over 28% of Googlebot's 4.5 billion. Cloudflare says more than half of internet traffic is now non-human.

  1. GPTBot made 569 million fetches across Vercel's network in one month, Claude 370 million, AppleBot 314 million and PerplexityBot 24.4 million. Source: Vercel and MERJ, "The rise of the AI crawler", 17 December 2024.
  2. Googlebot made 4.5 billion fetches in the same month. The four AI crawlers combined came to a little over 28% of that. Source: Vercel, 17 December 2024.
  3. AI bots averaged 4.2% of HTML requests on Cloudflare in 2025, ranging from 2.4% to 6.4%. Googlebot alone averaged 4.5%. Source: Cloudflare Radar 2025 Year in Review, 15 December 2025.
  4. GPTBot was about 7.5% of all verified bot traffic on Cloudflare in 2025. Source: Cloudflare Radar 2025 Year in Review.
  5. More than half of internet traffic was non-human in 2026, the first year that happened. Cloudflare's average load rose from 63 million HTTP requests a second at the end of 2024 to 115 million. Source: Cloudflare, "The Internet has a second audience", 30 September 2026.
  6. Bots made 49% of requests on Fastly's network in January 2026. AI bots were about 8% of the bot traffic Fastly classed as wanted. Source: Fastly, "AI, Bots, and the Agentic Future of the Web" (January 2026 data).
Crawler Fetches in one month (Vercel network)
Googlebot 4.5 billion
GPTBot 569 million
Claude 370 million
AppleBot 314 million
PerplexityBot 24.4 million

Source: Vercel and MERJ, December 2024.

Which AI crawlers are growing fastest?

GPTBot grew fastest among training crawlers in 2024 and 2025, while ClaudeBot and Bytespider lost share. Search and user-triggered bots grew from smaller bases: ChatGPT-User requests rose 2,825% in a year, and OAI-SearchBot's late-2025 peak was about five times its January level, according to Cloudflare's data.

  1. GPTBot's share of AI-only crawling rose from 5% to 30% between May 2024 and May 2025. Meta-ExternalAgent reached 19%, and Bytespider fell from 42% to 7%. Source: Cloudflare, "From Googlebot to GPTBot: who's crawling your site in 2025", 1 July 2025.
  2. GPTBot requests grew 305%, lifting it from ninth to third among AI and search crawlers, with 7.7% of that traffic. Source: Cloudflare, 1 July 2025.
  3. ClaudeBot requests fell 46%, cutting its share from 11.7% to 5.4%. Source: Cloudflare, 1 July 2025.
  4. ChatGPT-User requests grew 2,825% and PerplexityBot requests 157,490%, though they still held only 1.3% and 0.2% of AI and search crawling. Source: Cloudflare, 1 July 2025.
  5. OAI-SearchBot's late-October 2025 peak was about 5x its level at the start of the year. PerplexityBot ended 2025 about 3.5x higher and ClaudeBot about 10% higher. Source: Cloudflare Radar 2025 Year in Review.
  6. Googlebot requests grew 96% over the same year, and its share of AI and search crawling rose from 30% to 50%. Source: Cloudflare, 1 July 2025.

What are AI crawlers collecting data for?

Most AI crawling feeds model training. Training was about 80% of AI crawling in mid-2025, and by June 2026 52% of all crawler requests on Cloudflare were for training. Crawling for search is small and shrinking. Fetching triggered by a user's question is growing fastest, and OpenAI's bots handle almost all of it.

  1. Training made up about 80% of AI crawling in the year to July 2025, up from 72%. Crawling for search fell from 26% to 17%, and user action rose from 2% to 3.2%. Source: Cloudflare, "The crawl-to-click gap", 29 August 2025.
  2. 52% of crawler requests were for AI training by June 2026, up from 22% in spring 2025. Mixed-use crawlers made up more than 36%. Source: Cloudflare, "Content Independence Day, one year on", 1 July 2026.
  3. User-action crawling grew more than 15x in 2025, mostly from ChatGPT-User. At its peak, training crawling was up to 7x to 8x search crawling. Source: Cloudflare Radar 2025 Year in Review.
  4. Daily requests from AI agents grew more than 1,700% in a year. Source: Cloudflare, 30 September 2026.
  5. OpenAI's ChatGPT-User and OAI-SearchBot made 97% of AI fetcher requests on Fastly in January 2026, and DuckAssistBot 2%. 57% of fetcher requests were for content that could not be or had not yet been cached. Source: Fastly, January 2026 data.

How many crawls does an AI platform make for each visitor it sends?

AI platforms crawl far more than they refer. In July 2025 Anthropic made 38,065 crawls per referral, OpenAI 1,091 and Perplexity 195. Google ranged from 3:1 to 30:1 through 2025. Cloudflare has also cited a range of 118:1 to nearly 50,000:1 across leading AI companies.

  1. Anthropic's ratio fell from 286,930:1 in January 2025 to 38,065:1 in July 2025. Source: Cloudflare, "The crawl-to-click gap", 29 August 2025.
  2. OpenAI's ratio moved from 1,217:1 to 1,091:1 over the same period, and Perplexity's rose from 54:1 to 195:1. Source: Cloudflare, 29 August 2025.
  3. Anthropic's ratio peaked at up to 500,000:1 in 2025, and OpenAI's at up to 3,700:1 in March. Perplexity stayed below 200:1 from September. Source: Cloudflare Radar 2025 Year in Review.
  4. Google's ratio ranged from 3:1 to 30:1 in 2025, and DuckDuckGo's was about 1.5:1. Source: Cloudflare Radar 2025 Year in Review.
  5. Leading AI companies showed ratios from 118:1 to nearly 50,000:1 around July 2025. Source: Cloudflare, "Attribution Business Insights", 1 July 2026.
  6. Google sends about 88% of referral traffic, and Cloudflare estimates it has about twice the content access of the leading AI companies. Source: Cloudflare, 1 July 2026.
Platform Crawls per referral, Jan 2025 Crawls per referral, Jul 2025
Anthropic 286,930 38,065
OpenAI 1,217 1,091
Perplexity 54 195
Microsoft 38.5 40.7

Source: Cloudflare, August 2025. For what that traffic converts to, see our AI referral traffic statistics.

How many websites block AI crawlers?

About 6% of websites block GPTBot in robots.txt, and other AI bots are blocked at similar rates. Blocking of training is far more common than blocking of search. On Cloudflare, 17% of sites block training while fewer than 1% block search crawlers. GPTBot is both the most blocked and the most explicitly allowed AI bot.

  1. GPTBot is blocked by 5.89% of about 140 million websites, the highest rate of any AI bot. Counting subdomains across about 461 million robots.txt files, the rate rises to 7.3%. Source: Ahrefs, "The AI Bots That ~140 Million Websites Block the Most", 21 May 2025.
  2. ClaudeBot is blocked by 5.74% of sites, PerplexityBot by 5.61% and OAI-SearchBot by 5.54%. Source: Ahrefs, 21 May 2025.
  3. ClaudeBot block rates grew fastest, up 32.67% in a year, against 15.37% for PerplexityBot, 13.38% for GPTBot and 2.17% for OAI-SearchBot. Source: Ahrefs, 21 May 2025.
  4. About 14% of the top 10,000 domains have AI-specific robots.txt rules. Of 3,816 domains with a robots.txt, 546 named AI bots. GPTBot was disallowed on 312 of them and explicitly allowed on 61, more than any other bot on both counts. Source: Cloudflare, 1 July 2025.
  5. Fewer than 1% of Cloudflare sites block search crawlers, while 17% block AI training. Source: Cloudflare, "Have it both ways", 15 September 2026.
  6. GPTBot, ClaudeBot and CCBot were the most fully disallowed user agents in robots.txt in 2025, and allow rules for Google-Extended tripled. Source: Cloudflare Radar 2025 Year in Review.
Bot Operator Share of sites blocking
GPTBot OpenAI 5.89%
ClaudeBot Anthropic 5.74%
PerplexityBot Perplexity 5.61%
OAI-SearchBot OpenAI 5.54%

Source: Ahrefs, May 2025.

How do AI crawlers differ from Googlebot?

AI crawlers are simpler than Googlebot. None of the major ones render JavaScript, and they hit missing pages about four times as often. OpenAI also runs separate bots for training, search and user requests, each with its own robots.txt setting, so one rule rarely covers every crawler from the same company.

  1. None of the major AI crawlers render JavaScript. ChatGPT's crawler fetched JavaScript files in 11.50% of requests and Claude's in 23.84%, but neither executed them. Source: Vercel, 17 December 2024.
  2. 34.82% of ChatGPT crawler fetches and 34.16% of Claude fetches hit 404 pages, against 8.22% for Googlebot. Source: Vercel, 17 December 2024.
  3. OpenAI's GPTBot and OAI-SearchBot settings are independent, ChatGPT-User visits are user-initiated and may not follow robots.txt, and changes take about 24 hours to apply. Source: OpenAI, "Overview of OpenAI crawlers".

What do these numbers mean for brand visibility?

Crawler access is the entry ticket to AI answers, but it doesn't earn the citation. Blocking OAI-SearchBot or PerplexityBot removes you from those search answers, while blocking training bots mostly affects future models. Allowing every bot still doesn't mean an engine will name you. That depends on what your pages say.

Three moves follow from the data:

  • Split your robots.txt by purpose. Allow the search and user bots (OAI-SearchBot, ChatGPT-User, PerplexityBot), then decide on training bots such as GPTBot and ClaudeBot separately. Our Claude statistics and Perplexity statistics cover how those engines cite.
  • Serve key facts in HTML. AI crawlers don't render JavaScript, so pricing, product and comparison content should not load client-side.
  • Measure the answer, not the crawl. Crawls rose while referrals stayed low, and our zero-click search statistics show the same trend in Google. Track whether engines name you for real buying questions.

See which engines name your brand and which sources they cite. Start free, no card.

Where does Mentionova fit, and where does it not?

Crawler logs tell you who fetched your pages. They don't tell you whether ChatGPT named you. Mentionova covers the answer side. It runs your buyers' real questions through six answer engines on a schedule and logs whether each named you, cited your domain, where you placed and which sources it used instead.

What it does well:

  • Answers "why does AI cite my competitors and not me?" You see the rivals each engine named and the URLs it cited.
  • Covers six engines: ChatGPT, Perplexity, Claude, Gemini, Google AI and Reddit. That lets you see whether a crawler you block lines up with a gap in one engine.
  • Keeps every run on a weekly, two-weekly or monthly schedule, with cited sources exportable as CSV.

Limits worth knowing: for crawl volumes and bot traffic, use your CDN or server logs. Starter is $99 a month billed annually ($125 monthly) with 25 tracked prompts and three engines (ChatGPT, Perplexity and Google). Scale covers all six. See pricing, or follow our guide to checking if ChatGPT mentions your brand.

Where does each number come from?

  • Cloudflare blog: "From Googlebot to GPTBot: who's crawling your site in 2025" (1 July 2025); "The crawl-to-click gap" (29 August 2025); Radar 2025 Year in Review (15 December 2025); "Content Independence Day, one year on" and "Attribution Business Insights" (1 July 2026); "Have it both ways" (15 September 2026); "The Internet has a second audience" (30 September 2026).
  • Vercel and MERJ, "The rise of the AI crawler" (17 December 2024).
  • Fastly, "AI, Bots, and the Agentic Future of the Web" (January 2026 data).
  • Ahrefs, AI bot block rates across about 140 million websites (21 May 2025).
  • OpenAI platform docs, "Overview of OpenAI crawlers".
  • Mentionova plan details: mentionova.com/pricing.

All sources were read on 2 November 2026.

What do people ask about AI crawlers?

What is GPTBot?

GPTBot is OpenAI's crawler for collecting training data. It is separate from OAI-SearchBot, which powers ChatGPT search, and from ChatGPT-User, which fetches pages when a user asks. GPTBot was about 7.5% of verified bot traffic on Cloudflare in 2025 and is blocked by 5.89% of sites.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot gathers content for training OpenAI's models. OAI-SearchBot indexes pages for ChatGPT search answers. OpenAI treats their robots.txt settings as independent, so you can block GPTBot and still be eligible for ChatGPT search, as long as OAI-SearchBot is allowed.

Should I block AI crawlers?

Block by purpose, not by brand. Blocking search bots such as OAI-SearchBot and PerplexityBot removes you from those engines' search answers. Blocking training bots such as GPTBot or ClaudeBot keeps your content out of future training. On Cloudflare, 17% of sites block training and fewer than 1% block search.

Do AI crawlers send traffic back to websites?

Very little. In July 2025 Anthropic made 38,065 crawls per referral, OpenAI 1,091 and Perplexity 195, according to Cloudflare. Google ranged from 3:1 to 30:1 in 2025. User-triggered fetches from ChatGPT are the AI traffic most likely to end in a visit.

Can AI crawlers read JavaScript?

No. Vercel found that none of the major AI crawlers render JavaScript. ChatGPT's and Claude's crawlers download some JavaScript files but don't execute them. Any content that only appears after JavaScript runs, such as prices or product tables, is invisible to them.

Which AI crawler crawls the most?

GPTBot. It made 569 million fetches across Vercel's network in one month, ahead of Claude at 370 million. On Cloudflare, its share of AI-only crawling rose from 5% to 30% between May 2024 and May 2025. Googlebot still out-crawls every AI-only bot.

Why isn't my brand in ChatGPT if GPTBot crawls my site?

GPTBot collects training data. It doesn't decide what ChatGPT search shows. Search answers depend on OAI-SearchBot access and on whether your pages answer the question better than competitors' pages. Track the buying questions your customers ask to see who ChatGPT names instead.

Find out whether AI engines name your brand. Start a free trial, no card.

Keep reading

More from the blog.

All articles
Research

Grok Statistics: Users, Subscribers, Models and Citations

32 sourced Grok stats: 117M monthly users, 1.9M SuperGrok subscribers, every model launch from Grok 3 to 4.7, X integration and the sites Grok cites most.

12 minRead
Research

AI Overviews Citation Statistics: Which Sources Google Cites

32 sourced AI Overviews citation stats: YouTube's 24.5% share, Reddit and Wikipedia, 17% to 43% top-10 overlap, citation odds by rank and industry splits.

13 minRead
Research

Perplexity Shopping Statistics: Product Cards, Sources and Checkout

36 sourced Perplexity shopping stats: product cards in 5% of answers, 10.2 cited sources per answer, no ads, PayPal checkout and the Amazon Comet ruling.

10 minRead

100+ agents on call · 24×7

Put the agentsto work.

100+ growth agents run your competitor research, watch six answer engines, write content briefs and build the reports — around the clock. You keep final approval.

Get startedBook demo