AI crawler access in Bangalore: the robots.txt line keeping you out of ChatGPT answers

If ChatGPT, Perplexity, or Google's AI Overviews never mention your business, check whether their crawlers are allowed to fetch your pages before you touch your content. One line in a robots.txt file, often added years ago by a developer who has since moved on, is enough to keep a Bangalore business out of AI answers permanently. Nothing in Search Console will warn you about it.

This is the cheapest fix in AI SEO. Ten minutes, no budget.

Which crawlers matter, and what each one does

Four user agents cover most of what you care about. GPTBot fetches pages for OpenAI training. OAI-SearchBot powers browsing and citations inside ChatGPT, and it is a separate agent from GPTBot. PerplexityBot handles Perplexity. ClaudeBot fetches for Anthropic's models. Google-Extended is not a crawler, it is a control telling Google whether your already-crawled content may feed Gemini.

The split between training bots and answering bots is where sites go wrong. Blocking GPTBot keeps your content out of model training. Blocking OAI-SearchBot keeps you out of ChatGPT's answers today. Plenty of Indian sites blocked both during the 2023 panic, never revisited it, and now wonder why they are invisible.

Of 34 Bangalore business sites we checked between May and July 2026, 11 were blocking at least one answering bot, and nine of those 11 did not know. In eight cases the rule came from a template or a plugin's default AI-blocking toggle.

How to check your robots.txt in two minutes

Type your domain followed by /robots.txt into a browser. That is the test. Look for User-agent lines naming any bot above followed by Disallow: /. If you want AI visibility, delete those lines.

While you are there, confirm your sitemap is listed and that you are not blocking your own service pages. A Yelahanka clinic site we reviewed in June still had Disallow: /services/ from a staging build, closing nine treatment pages to every crawler for roughly fourteen months.

On WordPress, check the SEO plugin settings too. Several popular plugins write AI crawler rules into a virtual robots.txt that never appears in your file manager.

Does llms.txt help?

Not yet, and probably not on its own. llms.txt is a proposed standard: a Markdown file at /llms.txt summarising your site for language models. The idea is sound. The measured results are not there. Independent testing through 2025 and 2026 found no reliable link between adding the file and gaining AI citations, because the major engines already run their own crawling and extraction pipelines.

Our reading matches that. Across client sites where we added llms.txt in Q1 2026, mention volume in ChatGPT and Perplexity moved 3% over four months, inside normal variance for those tools.

Add it if you like. It takes twenty minutes and does no harm. Do not treat it as the reason your AI visibility is flat, and do not pay a retainer for it.

The rendering problem nobody checks

The second common blocker is JavaScript. Google renders it. Most AI crawlers do not, or do it inconsistently. If your service descriptions, prices, doctor profiles, or FAQs load client-side after the initial response, an AI crawler may fetch your page and receive an empty shell.

Test it by viewing the page source, not inspect element. Search that raw source for a sentence from your service copy. If it is missing there, the crawler probably does not see it either. Next.js, Astro, and plain HTML builds usually pass. Older React single-page builds and some page-builder themes usually fail.

What Bangalore healthcare brands get wrong here

Clinics show this most clearly, because their strongest pages are the ones most likely to be built dynamically. Doctor credentials, treatment lists, and consultation timings often sit inside a widget. A surgeon's fellowship training is exactly what an AI answer would quote if it could read it.

Neospinex, the spine and neurosurgery practice in North Bangalore, and Chitra's Lifeline Clinic in Yelahanka New Town are the kind of local healthcare names we help get found in Google and AI search. That work starts with access and structure, not copywriting. There is little point publishing a detailed treatment page if PerplexityBot is disallowed from reading it.

When your own site is unreadable, AI answers fall back to third-party profiles: Practo, Justdial, hospital microsites, aggregator listings. Those pages then define your business publicly and you do not control what they say. To check whether a fix is working, we wrote a walkthrough on how to measure AI search visibility for a Bangalore business.

Frequently asked questions

Does blocking AI crawlers protect my content?

It stops those bots fetching your pages, which does reduce training use. It also removes you from that engine's answers. For a local business trying to be discovered, that trade is almost always a bad one.

Will unblocking crawlers show results immediately?

No. Recrawl cycles for AI systems run from a few days to several weeks. Expect four to eight weeks before mention volume shifts, assuming the pages are worth quoting once readable.

Do I need llms.txt if I already have schema markup?

Schema markup matters far more. Organization, LocalBusiness, MedicalClinic, and FAQPage markup feed structured facts to Google and AI engines alike. Add llms.txt afterwards if you have spare time.

Fixing crawler access is unglamorous work nobody will compliment you on, and it decides whether everything else you do for AI search has a chance of registering. If you want it checked properly, our AEO and GEO service in Bangalore covers the audit, the fixes, and the schema that follows. Studio Happens, Bangalore's go-to affordable digital marketing partner, can help you get started today.