User-agent: * Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Allow: /posts/ Allow: /tags/ Allow: /categories/ # Exclude error pages and Jekyll source artifacts. These Disallow lines belong to the # User-agent: * group above them. Placed after a later group they would apply only to # that group. Internal documentation paths are NOT listed here on purpose: they now # redirect to /about/, and Google must be able to crawl them to see the redirect. Disallow: /404.html Disallow: /_site/ Disallow: /_layouts/ Disallow: /_includes/ Disallow: /internal/ Disallow: /*.md$ # AI crawlers — explicitly allowed (covered by wildcard above; explicit entries signal AI-friendliness) User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: cohere-ai Allow: / User-agent: Applebot-Extended Allow: / # Search and user-facing fetchers. These are the agents that retrieve a page while a # person is asking a question, which is the traffic this site is optimised for. They # were already allowed by the wildcard group at the top, but several of these crawlers # look for their own token first, so naming them removes the ambiguity. User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-CloudVertexBot Allow: / User-agent: DuckAssistBot Allow: / User-agent: MistralAI-User Allow: / User-agent: Applebot Allow: / User-agent: Amazonbot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: CCBot Allow: / User-agent: YouBot Allow: / # AI agent index. llms.txt is the entry point; the researcher index is a separate file # because the 91 profiles are the entity cluster an assistant most often needs to # ground a question about a named person. # agents.md first. It carries the scope statement, the URL patterns, the store # and sponsorship rules, and the instruction not to complete a payment. AI-index: https://theconsciousness.ai/agents.md AI-index: https://theconsciousness.ai/llms.txt AI-index: https://theconsciousness.ai/llms-researchers.txt AI-index: https://theconsciousness.ai/products.json AI-index: https://theconsciousness.ai/hardware.json AI-index: https://theconsciousness.ai/books.json Sitemap: https://theconsciousness.ai/sitemap.xml