Block everything or block nothing are both wrong answers, and any dashboard that collapses AI crawler traffic into one number is hiding the decision that actually matters. Cloudflare’s homepage now leads with “build for the agent era,” and a vendor in that position has every reason to package which AI crawlers reach your origin as a business-attribution view rather than raw bot logs. The question such a view answers is old and sharp: which crawlers get to read which paths, and what do you get back.
Should you block AI crawlers or let them read your site?
Neither blanket policy survives contact with the economics. The defensible default is a per-path decision that weighs citation visibility against serving cost and content leakage, and the inputs to that decision have changed enough in the last two years that policies set in 2024 deserve a re-review.
The case for allowing comes from how search behavior has shifted. Searches increasingly end without a click to any website, with agency analyses of AEO, SEO, and GEO framing AI synthesis as a citation channel that can produce visibility even when it produces zero direct traffic. The same analyses argue that visitors who do arrive via AI search convert at higher rates than traditional organic visitors: fewer sessions, more qualified ones. Under that framing, “the bot is stealing bandwidth” and “the bot is the top of my funnel” are both true, depending on which path it touched.
The case for blocking is structural too. AI Overviews sit between the user and the blue links, which means a synthesized answer can substitute for the click whether or not you are cited in it, and the substitution gets worse if your competitors are cited while you are not. Blocking also remains the coherent choice for content whose entire value is exclusivity: paid tiers, licensed datasets, anything behind a meter.
One structural fact raises the stakes of how you decide. Cloudflare is used by approximately 21.3% of all websites on the Internet as of January 2026, per W3Techs figures reported on Wikipedia, and Cloudflare’s investor relations page states that 42% of the Fortune 500 were paying customers as of March 31, 2026, alongside more than 4,400 large customers in Q1 2026. Whatever framing Cloudflare ships in a crawler dashboard becomes the default mental model for a large fraction of the web’s operators, including most of the enterprises with legal departments. That makes it worth examining what the vendor view shows, and what it cannot.
What does a vendor attribution dashboard show out of the box?
It reframes AI crawler visits as a business signal: which AI company’s bots reached your origin, presented as a commercial metric rather than a security log. Treat any specific product name, launch date, or feature list in this category as unverified until you see it in the vendor’s own docs.
What is documented is the direction. Wikipedia’s Cloudflare entry notes the company has been integrating AI into its infrastructure, acquiring companies such as Replicate and launching tools to manage AI bots and scrapers. Read alongside the homepage’s “build for the agent era” tagline, that is not a security roadmap. It is an intermediary position: detect the crawlers, govern their access, and eventually broker the terms under which they read your content. An attribution layer is the natural reporting instrument on top of that stack, the thing that tells a publisher what their position is worth.
The limitation is structural, not a missing checkbox. A business-attribution rollup exists to answer an executive question: is AI crawler traffic going up, and whose is it. The operator’s question is different: which routes are being read, by which bot, and what happened to those requests. An aggregate count of GPTBot requests this week tells you nothing about whether those requests hit your public docs or your paid archive, and the correct policy response differs completely between those two cases. Rollups do not just omit that detail; they make it easy to forget the detail exists.
What does rolling your own log analysis look like?
The ground truth for crawler decisions is your own access log. On nginx, the $http_user_agent variable already records everything you need, and on Cloudflare’s own edge, a short Worker gives you the same per-route visibility without shipping logs anywhere.
The nginx version is a one-line config change. Make sure the user agent is in your log format (the default combined format already includes it):
log_format crawler '$remote_addr $time_local "$request" ' '$status $body_bytes_sent "$http_user_agent"';access_log /var/log/nginx/access.log crawler;Then the analysis is a pipeline you can run over SSH in seconds. A rough pass that buckets AI crawler hits by path:
grep -Ei 'gptbot|claudebot|bytespider|ccbot|google-extended|perplexitybot' \ /var/log/nginx/access.log \ | awk -F'"' '{split($2, r, " "); print r[2]}' \ | sort | uniq -c | sort -rn | head -30The output is the thing the attribution dashboard cannot give you: a ranked list of the exact paths each family of AI crawlers is pulling, with counts you can diff week over week. Add $status to the extraction and you can see whether your blocks are actually returning 403s or silently failing. Add $body_bytes_sent and you have the bandwidth cost per crawler per route, which is the real input to the content-cost side of the decision.
The Workers version keeps the data inside Cloudflare, which matters if your origin never sees the traffic because it is cached or blocked at the edge. A minimal collector using Analytics Engine:
const AI_BOTS = /gptbot|claudebot|bytespider|ccbot|google-extended|perplexitybot/i;
export default { async fetch(request, env, ctx) { const ua = request.headers.get('user-agent') || ''; if (AI_BOTS.test(ua)) { const path = new URL(request.url).pathname; ctx.waitUntil(env.CRAWLER_LOG.writeDataPoint({ blobs: [ua, path], doubles: [1], indexes: [ua.slice(0, 32)], })); } return fetch(request); },};Query that dataset grouped by pathname and you have the per-route crawler map for traffic that never touched your infrastructure at all.
The honest cost accounting: nginx log analysis is free but manual, and you own retention and rotation. Workers Analytics Engine is operationally light but samples and aggregates on Cloudflare’s terms. Neither gives you the polished “business insights” framing, which is precisely the point. The framing is the part you are supposed to interrogate, not consume.
Which paths should stay open, and which should close?
Open the paths where a citation pays you back, and close the paths where a read only costs you. That sounds obvious, and yet the dominant policy in the wild is still a single sitewide directive, because a sitewide directive is what robots.txt makes easy and what a rollup dashboard makes legible.
The concrete version, using the split most content businesses actually have: documentation, reference material, and marketing pages are citation-positive. If GPTBot reads your docs and ChatGPT starts citing them in answers, you gain qualified visibility at zero marginal cost beyond serving bytes you were serving anyway. Paid archives, licensed content, metered APIs, and anything whose business model is scarcity are citation-negative: a synthesized answer that absorbs the content is a substitution, not a referral. The policy that follows is asymmetric by design.
In robots.txt terms:
User-agent: GPTBotAllow: /docs/Allow: /blog/Disallow: /premium/
User-agent: BytespiderDisallow: /premium/Disallow: /api/Robots.txt is advisory, and everyone in this industry knows it. It is a request that well-behaved crawlers honor, not a control. Enforcement lives at the edge: Cloudflare’s bot management surface is this layer productized, and the same effect is a five-line nginx if block or a WAF rule matching user agents to path prefixes and returning a 403. A more aggressive option is to stop blocking the unauthorized crawler and feed it generated filler instead, which converts your content-cost problem into their training-data-quality problem. That trade is harder to attribute but more satisfying.
The decision matrix per route looks like this:
| Path type | GPTBot / ClaudeBot | Bytespider and opaque crawlers | Rationale |
|---|---|---|---|
| Docs, API reference | Allow | Allow | Citation-positive; synthesis drives qualified visits |
| Blog, marketing | Allow | Case-by-case | Visibility upside; watch bandwidth |
| Paid or metered content | Block | Block | Synthesis substitutes for the product |
| Authenticated areas | Block | Block | No citation channel exists anyway |
| Expensive dynamic endpoints | Block | Block | Pure cost, no attribution return |
No aggregated dashboard will hand you this table. The table is the job.
Do all AI crawlers deliver the same value?
No, and the difference is measurable, which is what makes the per-crawler decision more than superstition. The value of allowing a crawler is downstream of whether the product it feeds cites sources at all, and whether your audience uses that product.
The sourced version of that claim: the AEO/GEO comparisons define GEO success as citation inclusion and mention frequency across ChatGPT, Perplexity, Gemini, and Claude. Those four products are where citation return is currently measurable, so the crawlers attached to them (GPTBot, ClaudeBot, PerplexityBot, and Google’s AI-specific agents) are the ones with a defensible allow case on citation grounds. Crawlers whose downstream product you cannot identify, or whose answers never surface sources, offer no measurable citation channel. Allowing them is a donation.
This is where your own logs earn their keep a second time. The per-route data tells you behavior, not just identity: a crawler that reads only your docs index and leaves is behaving like a citation engine building a map, while one that crawls your paid archive at depth is behaving like a bulk harvester. Two bots can have equally recognizable names and opposite intents on your specific site. Vendor attribution tells you the name. Your logs tell you the behavior, and behavior is what the policy should key on.
One caution on identity: user-agent strings are self-declared and trivially spoofed, and a sizeable share of “GPTBot” traffic in any large access log is scrapers wearing the name because it is less likely to be blocked than an empty agent string. If a route matters enough to protect, enforce on more than the UA: known IP ranges where the operator publishes them, request-pattern heuristics, or a verified-bot program where one exists. Treat the UA as a hint and the log pattern as the evidence.
How do you measure citation instead of traffic?
AEO and GEO run on different scoreboards, and conflating them is the most common measurement error. Per the agency breakdown of the two disciplines, AEO success is measured by snippet and answer capture on Google AI Overviews and voice assistants, while GEO success is measured by citation inclusion and mention frequency across ChatGPT, Perplexity, Gemini, and Claude. Any attribution product, vendor-built or homemade, that rolls those into one “AI visibility” number is measuring something nobody can act on.
The deeper problem is that your existing analytics cannot validate the decision either way. The 2023 Princeton GEO paper by Aggarwal et al., which the agency breakdown cites as the formal introduction of GEO, argued that generative engines respond to different optimization signals than traditional search. If you allow GPTBot next month and your rankings dashboard does not move, that tells you nothing, because the return you are buying does not appear in rankings. It appears in whether ChatGPT names you when someone asks the question your docs answer, and measuring that means sampling prompts across the four products and counting mentions, not watching Search Console.
There are on-page levers that complement the infra decision. Structured data such as FAQ and entity markup is the consistently recommended input across agency analyses of the field, and a parallel comparison from Search Engine People draws the same three-way split between SEO, AEO, and GEO. Treat any specific multiplier on citation lift as unverified until you see the underlying study; the taxonomy is stable even where the numbers are not.
When is a vendor dashboard enough, and when do you need your own logs?
If the question is whether AI crawler traffic is trending up and roughly whose it is, a vendor dashboard is enough. If the question is which route to open to which crawler, you need your own logs, because the decision lives at path granularity and rollups structurally cannot hold it.
The comparison:
| Dimension | Vendor attribution dashboard | DIY $http_user_agent analysis |
|---|---|---|
| Setup effort | Toggle in dashboard | Log format plus a pipeline, or one Worker |
| Granularity | Aggregate by crawler, business framing | Per path, per crawler, per status, per byte |
| Covers edge-blocked traffic | Yes | Only if you log at the edge (Workers) |
| Answers “is AI traffic growing” | Directly | With a cron job and a diff |
| Answers “open /docs to GPTBot?” | No | Yes |
| Audit trail if access becomes paid | Vendor’s numbers | Your numbers |
| Consumer | Executives, partnerships | The operator writing the WAF rule |
The last rows deserve emphasis because of where Cloudflare’s positioning points. The “build for the agent era” tagline and the documented moves to manage AI bots and scrapers describe a direction in which crawler access is governed, and eventually sold, not just permitted. The moment access is billed, attribution data stops being analytics and becomes a settlement input. In that world, running your own logs is how you audit the invoice, the same reason you never trusted an ad network’s impression count over your own server logs. Operators who lived through ad-tech reconciliation will find the shape familiar.
The practical split: a brochure site or a small SaaS marketing site can take the dashboard and stop. A publisher with a paid tier, a documentation business whose traffic is collapsing into zero-click answers, or anyone who expects to sell crawl access within two years needs the log pipeline, because every decision that actually moves revenue is per-route.
What should you verify before acting on any of this?
Two things in this piece deserve a second check before you act on them.
First, any named vendor product in the crawler-attribution category. Cloudflare’s homepage markets bot management as part of a broader security bundle (DDoS protection, WAF, bot management, Zero Trust) rather than naming a dedicated attribution product, so confirm any specific capability, particularly per-path or per-crawler breakdowns, against the vendor’s own documentation before writing policy around it. Second, the crawler roster itself. Bot names, operators, and their robots.txt posture change quarterly; re-pull each vendor’s crawler documentation at the time you set policy, not from memory, including from this article.
The durable part needs no verification, because it is a method rather than a claim. Your access log is the ground truth. The allow/block decision is per-path and per-crawler. The return on allowing is measured in citations, not sessions, and citations are measured per engine. Vendor dashboards will keep arriving with friendlier framings. Use the dashboards for the trend line. Keep the logs for the decision.
Frequently Asked Questions
How does Cloudflare Attribution differ from the GoDaddy AI Crawl Control partnership?
Cloudflare Attribution is a reporting layer that reframes crawler visits as business signals, while the GoDaddy partnership, announced in April 2026, extends the underlying permission-based blocking policy to GoDaddy-hosted sites. Attribution tells you who is reading your content; the GoDaddy integration lets site owners on that specific hosting platform decide whether to allow or block those readers at the point of sale.
What happens if a crawler spoofs its user-agent string?
User-agent strings are self-declared and easily spoofed, meaning a scraper can impersonate GPTBot to bypass blocks. To mitigate this, enforce access controls on known IP ranges where operators publish them, or use request-pattern heuristics rather than relying solely on the UA string. This is critical because a sizeable share of ‘GPTBot’ traffic in large access logs is often scrapers wearing the name to avoid empty-agent blocks.
Can I use Cloudflare Attribution to audit invoices if crawler access becomes a paid product?
No. Vendor dashboards provide aggregate business insights, not the granular audit trail required for financial reconciliation. If access becomes billed, you need your own logs because every decision that moves revenue is per-route. Operators who lived through ad-tech reconciliation know that you never trust an ad network’s impression count over your own server logs, and the same principle applies to crawler billing.
How does GEO measurement differ from traditional SEO analytics?
Traditional SEO analytics track rankings and direct traffic, which the 2023 Princeton GEO paper notes have little to no effect on generative engines. GEO success is measured by citation inclusion and mention frequency across ChatGPT, Perplexity, Gemini, and Claude. If you allow GPTBot and your rankings dashboard does not move, that tells you nothing about GEO performance, which appears in whether ChatGPT names you when someone asks the question your docs answer.
