Which Pakistani Websites Block AI Crawlers? We Checked 93

On 20 October 2026 we read the robots.txt file of 93 well-known Pakistani business websites to see which AI crawlers each one lets onto its home page. The sites cover banking, telecom, retail, news, real estate, education, healthcare, software and industry. The short answer: almost every site lets every AI crawler in, very few have made a choice about it, and not one blocks the crawlers that decide whether a business shows up in ChatGPT, Claude or Perplexity answers. Below are the numbers by sector, the five sites that block something, why training and search are two separate decisions, and what to do about your own site this week. The full dataset is free to download and reuse.

Key findings

• 83 of 93 sites publish a robots.txt with rules in it. 9 have no usable file and 1 could not be reached.
• 78 of those 83 (94%) let all 12 crawlers we tested read the home page, including GPTBot, ClaudeBot and CCBot.
• Only 10 of 83 (12%) mention any AI crawler by name. Most Pakistani sites have no AI policy at all. They allow AI bots by default.
• 5 sites block at least one AI training crawler: Jazz, ARY News, Bookme, Foodpanda and TechJuice.
• 0 sites block an AI search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) or ChatGPT-User, the fetcher that reads a page when a person asks about it.
• 74 of 83 (89%) list a sitemap in robots.txt.

How we checked

We read the robots.txt file of each site and tested the home page against 12 user agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Googlebot, Bingbot, Applebot-Extended and meta-externalagent. We count a crawler as "named" when the file has a rule group written for it, and as "blocked" when the rules that apply to it disallow the home page. The matching follows the robots.txt standard, RFC 9309, the same way our free checker does.

We picked the 93 sites ourselves as large, well-known brands in each sector, so this is not a random sample. The results are one snapshot, of the home page only. And robots.txt only shows what a site asks crawlers to do. It cannot show what a CDN, firewall or bot-protection service does to the same crawler when it actually turns up.

Results by sector

Software and IT companies are the ones that have thought about it. 4 of the 10 (10Pearls, Arbisoft, Devsinc and Tkxel) name AI crawlers in their robots.txt, and all four choose to let them in. None of the 20 banks names an AI crawler, and 3 of them have no usable robots.txt at all.

Who blocks what

Every one of these blocks is aimed at a training crawler, the kind that collects pages to train a model. None of the five blocks a search crawler or an on-request fetcher. So these sites have opted out of model training and stayed visible in AI search answers, which is the policy most businesses would pick if someone asked them.

What this means for Pakistani businesses

Allowing by default is not the same as having a policy. 73 of the 83 sites with a robots.txt never mention an AI crawler. They are open to every AI bot only because nobody decided otherwise. That works until the day someone adds a blanket rule, or the host or CDN switches on AI-bot blocking, and the site quietly drops out of AI answers with nothing in the analytics to show why. Our guide to WordPress and AI crawlers goes through the three layers where that can happen.

Training and search are separate decisions. Blocking GPTBot does not remove a site from ChatGPT search. OAI-SearchBot and ChatGPT-User are separate agents with their own names. Anthropic splits the same way, ClaudeBot for training and Claude-SearchBot for search, and so does Google: Google-Extended controls Gemini training and grounding, not Google Search. On 30 October 2026 Google also said it plans to support a shared AI-training control in robots.txt from 2027, so the training choice will soon fit in one line that several AI companies honour.

robots.txt is only one layer. A site can allow every crawler in robots.txt and still turn them away at the CDN or firewall, which sends the bot a 403 or a challenge page instead of the content. If AI search visibility matters to you, test the crawler you care about against the real page, not only the file. That is part of what our generative engine optimization work checks first, before any content is touched, and it sits alongside the crawl and index work our SEO team does.

What to do this week

1. Decide your policy in one line. For most businesses: allow AI search crawlers and on-request fetchers, then decide separately whether training crawlers may read your pages.
2. Write it into robots.txt with named rule groups, so the choice is yours rather than inherited from the rules for every bot.
3. Check your CDN and security plugin for AI-bot blocking settings that override robots.txt without telling you.
4. Test it. Our free AI Crawler Access Checker reads any site's robots.txt for 22 AI crawlers and gives you a block to copy that allows search and blocks training.

Get the data

The full dataset, with all 93 sites, the result for each of the 12 crawlers, the sector, the robots.txt status and the sitemap line, is free to reuse under CC BY 4.0 on Hugging Face: 313Automations/pakistan-ai-crawler-access-2026. If you use it, please credit 313 Automations and link to this page.

Bottom line: Pakistani business websites are almost all open to AI crawlers, but very few chose to be, and the handful that block anything block training while staying in AI search. Make the choice on purpose, write it into robots.txt by crawler name, and check what your CDN does with the same request. To see what your own robots.txt says to 22 AI crawlers right now, run it through our free AI Crawler Access Checker.

Frequently asked questions