Do LLMs respect noindex or noai directives today?

Core explainer

What are the exact roles of noindex and llms.txt, and how do they differ in practice?

Noindex blocks indexing in search engines, while llms.txt signals AI models to avoid training on listed URLs, and because they operate in different ecosystems with distinct failure modes, their practical effects differ and must be understood as part of a dual governance approach that also requires site-root placement and server configuration.

Noindex is delivered via HTML meta tags or HTTP headers and affects search result visibility but not raw access; llms.txt is not a universal standard and not all AI crawlers honor it, so its effects vary by model. Google guidance and John Mueller have suggested considering noindex for llms.txt to reduce confusion, but noindex does not fully block access to content. Real-world use cases include protecting copyrighted or proprietary content and defining AI-training boundaries, with llms.txt typically placed at the site root and paired with appropriate headers where possible to pursue dual governance. For practical reference, see OpenAI crawling and indexing guidance.

Do LLMs like GPTBot or OAI-SearchBot respect llms.txt and noindex the same way as traditional crawlers?

LLMs do not always follow traditional crawlers' rules, so the signals are not universally guaranteed, and behavior varies by model.

GPTBot respects robots.txt by default and can be guided with Allow/Disallow rules, while OAI-SearchBot supports live search and dynamic indexing behavior; llms.txt is not a standardized signal, and some models may ignore it, making testing essential to gauge what each crawler does. For practical reference, see OpenAI crawling and indexing guidance.

When should I use noindex, when should I use llms.txt, and when should I use both?

Use noindex on individual pages to keep them out of search results and llms.txt at the root to signal AI-training boundaries, and many practitioners pair them to achieve dual governance for content with copyright, privacy concerns, or sensitive internal data.

LLMS.txt is not standardized, and not all AI crawlers will honor it; Google's guidance has suggested adding noindex for llms.txt, but there is no SEO benefit from indexing the llms.txt file itself, so decisions should focus on governance and user clarity rather than search ranking. Noindex on pages plus an llms.txt root signal creates layered control, and ensuring critical content is present in initial HTML is part of robust AI visibility. For governance context, see brandlight.ai governance hub.

What are realistic implementation examples (file structure, headers, and rules)?

Implementation examples include placing llms.txt at the site root and exposing noindex signals in HTTP responses or page headers, plus using robots.txt to control crawling in broader scopes.

Concrete patterns: place llms.txt at the root (https://yourdomain.com/llms.txt), deliver an HTTP header such as X-Robots-Tag: noindex for the file, and/or include a meta tag on individual pages: ; in robots.txt, you may use Disallow directives like Disallow: /private/ to block access to sections. For practical reference, see OpenAI crawling and indexing guidance.

What are the main risks and limits of relying on these signals?

There are important limitations: llms.txt is not standardized, and not all AI crawlers honor it; noindex does not fully block access, and trained models may still use data encountered through links or cached copies; surface controls can be bypassed by caching or rehosting, and changes in crawler behavior can take time to propagate.

Practical approach emphasizes ongoing testing across crawlers, careful server configuration, and awareness that signals may not guarantee training behavior; combine with other access controls, monitor for changes, and reference the OpenAI guidance for context. For practical reference, see OpenAI crawling and indexing guidance.