Can AI read India's and the UAE's biggest companies?
- 4%
- block OpenAI's GPTBot in robots.txt
- 7 of 171 readable files
- 0
- NIFTY 100 companies block an AI search crawler
- in robots.txt, of 89 readable
- 1 in 3
- turned away an automated visit or showed a near-blank page
- 68 of 200 home pages
- 62%
- open to AI on all four of our checks
- 124 of 200 companies
WHAT WE FOUND
01
Hardly anyone blocks AI in robots.txt
Of the 171 companies whose robots.txt we could read, 7 block OpenAI's GPTBot, the crawler that collects training data. In the NIFTY 100 that is 2 of 89, both Adani group companies. In the UAE it is 5 of 82.
Search crawlers fare even better. ChatGPT's search crawler and its on-demand fetcher are blocked at just 2 companies, International Holding Company and du, and both do it with a blanket rule that admits only a short list of classic search engines. No NIFTY 100 company blocks any AI search or user-triggered crawler, and none of the 171 blocks Googlebot or Bingbot.
02
Where it is blocked, it's training, not search
Five of the seven GPTBot blockers still let OAI-SearchBot in: Adani Enterprises, Adani Green Energy, Emaar Properties, Emaar Development and Space42. The message reads as "don't train on us, but do show us in ChatGPT search", which is a reasonable position and the one most AI policies now make possible.
03
The firewall decides more than the policy
19 of the 161 sites that loaded for our browser-identified request refused the same page to a request identifying itself as GPTBot, in two separate tries: 13 in India and 6 in the UAE. Only one of the 19 also asks GPTBot to stay out in robots.txt.
A caution we take seriously: our GPTBot request came from our own network, not OpenAI's. Firewalls rightly check whether a bot is who it says it is, so some of these refusals may be protection against impersonation rather than a decision about AI. What it does show is that the security layer, not the published policy, is where access is really decided.
04
Nearly one in five turned away any automated visit
37 of the 200 home pages blocked, challenged or dropped both of our requests, the browser-identified one included: 24 in the UAE and 13 in India. Half of the UAE aviation and travel names behaved this way. Verified crawlers from Google, Microsoft or OpenAI may well get through; we can't test that from outside. But any AI tool that fetches a page on a user's behalf faces the same wall we did.
05
About one in eleven home pages is close to blank without JavaScript
15 of the 161 home pages we could load showed fewer than 500 characters of text before scripts run. Most AI crawlers don't run JavaScript, so for them these pages say almost nothing about the company. Examples include Bharti Airtel, BSE, Cipla, DMart and Jumeirah.
06
Indian companies are ahead on llms.txt
28 of the 200 companies publish an llms.txt file, a newer and still unproven convention for pointing AI systems at a site's key pages: 19 in India and 9 in the UAE. Indian banks lead, including HDFC Bank, Axis Bank, Kotak Mahindra Bank, Bank of Baroda and Union Bank of India. We don't count it towards being "open to AI", because evidence that engines use it is thin.
INDIA AND THE UAE, SIDE BY SIDE
India invites AI in by name. The UAE locks the door more often.
Indian robots.txt files that mention AI crawlers mostly do it to welcome them (16 of 18). In the UAE, three of the four that name AI crawlers do it to keep them out, and far more sites turn away automated visits altogether.
Open to AI on all four checks
share of all companies
- India
- 65 of 100
- UAE
- 59 of 100
robots.txt names AI crawlers
share of readable files
- India
- 18 of 89
- UAE
- 4 of 82
Publish an llms.txt file
share of all companies
- India
- 19 of 100
- UAE
- 9 of 100
Blocked our automated visit outright
share of all companies
- India
- 13 of 100
- UAE
- 24 of 100
robots.txt blocks GPTBot
share of readable files
- India
- 2 of 89
- UAE
- 5 of 82
BY SECTOR: OPEN TO AI ON ALL FOUR CHECKS
Property is the most open. Healthcare and aviation, the least.
Both countries pooled. Groups are small, so treat these as a direction, not a league table.
- Real estate10 of 12
- Automobiles7 of 9
- Transport and logistics7 of 9
- Industrials and materials23 of 32
- Energy and utilities8 of 12
- Conglomerates and holdings7 of 11
- Banking and financial27 of 43
- Consumer and retail13 of 22
- Oil and gas8 of 14
- Telecom and tech8 of 15
- Healthcare3 of 10
- Aviation and travel3 of 11
WHO BLOCKS AI IN ROBOTS.TXT
These are published policies, so we name them. Firewall and JavaScript results are in the dataset by company, with the caveats above, rather than singled out here.
| Company | Country | What robots.txt blocks | How |
|---|---|---|---|
| Adani Enterprisesadanienterprises.com | India | GPTBot | Names GPTBot |
| Adani Green Energyadanigreenenergy.com | India | GPTBot | Names GPTBot |
| Emaar Properties, Emaar Developmentemaar.com | UAE | GPTBot, PerplexityBot and four training crawlers | Names each crawler |
| Space42space42.ai | UAE | Eight training crawlers | Cloudflare-managed robots.txt |
| International Holding Companyihcuae.com | UAE | All 14 AI crawlers we tested | Blocks everyone except named search engines |
| dudu.ae | UAE | All 14 AI crawlers we tested | Blocks everyone except named search engines |
IF YOU RUN A COMPANY WEBSITE
01Decide training and search separately
Blocking GPTBot keeps your pages out of training data; blocking OAI-SearchBot keeps you out of ChatGPT's search answers. Most companies want the first to be a choice and the second to be open.
02Check the firewall, not just robots.txt
Ask whoever runs your CDN or WAF how verified AI crawlers are treated. Bot protection that blocks everything automated also blocks the crawlers you want.
03Make sure the page says something without JavaScript
View your home page with scripts off. If the company name, what you do and where you work aren't in the HTML, most AI crawlers won't see them.
04Serve robots.txt to everyone
Nearly one in ten sites showed robots.txt to a browser but not to a declared crawler. A crawler that can't read the rules has to guess.
05Re-check after every platform change
CDN rule updates, redesigns and new bot settings change access silently. A quarterly check takes minutes.
Our free checker runs the robots.txt and no-JavaScript checks from this study on any site, in a few seconds.
Check your site →
HOW WE DID IT, AND WHAT IT CAN'T TELL YOU
METHOD
- Sample: all 100 NIFTY 100 constituents (NSE Indices file, 19 October 2026); the 70 largest UAE-headquartered companies on ADX and DFM by market cap; and 30 major unlisted UAE companies, an editorial pick.
- Collection: one pass from one network location on 21 October 2026, 17:48 to 17:54 UTC. For each company we fetched the home page as a desktop browser and as OpenAI's published GPTBot user agent, robots.txt with a declared research crawler, and /llms.txt. Every difference between the two home-page requests was re-tested; it counts only if both passes agree.
- robots.txt was parsed to RFC 9309, the standard Google and others follow, with the same parser as our open-source AI crawler checker, for 16 crawlers. A missing file (404) means everything is allowed; a refused or unreachable file is excluded from robots.txt percentages, which is why those use 171 companies.
- "Open to AI on all four checks" is our own composite: robots.txt readable; it allows OAI-SearchBot and ChatGPT-User; the home page was served to the GPTBot-identified request; and the page has at least 500 characters of text without JavaScript.
LIMITS
- One day, one place, about six minutes. Results vary by network, country and time, and sites change often.
- Our GPTBot request was an imitation from a non-OpenAI address. Real, verified crawlers may be treated differently.
- We tested home pages only. robots.txt states intent, not enforcement.
- The 30 unlisted UAE companies were chosen by us. Sector groups are small (9 to 43 companies), so read sector figures as indicative.
- Four websites are shared by two listed companies (for example Emaar Properties and Emaar Development), and counts are per company.
USE THIS DATA
Free to use, with credit.
The full company-level dataset is under a CC BY 4.0 licence. Quote it, chart it, or check our work. If you spot an error, tell us and we’ll correct it in the open.
SUGGESTED CITATION
Seamwire, “Can AI read India's and the UAE's biggest companies?”, 22 October 2026. www.seamwire.com/research/ai-access-india-uae-2026