# hausfold.co
#
# Why this file exists at all: it was written while
# `not_found_handling = "single-page-application"` made any path with no file
# behind it answer 200 with index.html — so https://hausfold.co/robots.txt
# served the landing page *as* robots rules. That's now `404-page` (2026-08-08),
# which fixes the fallback; this file still beats it.
#
# Crawler tiers (2026-08, orank gap #robots). The default group stays open: only
# ./public is ever uploaded to the Worker, so everything reachable here is meant
# to be read. The named groups exist for the AI tiering: answer-engine crawlers
# are invited explicitly, and training-only crawlers are asked to skip the site
# entirely. A bot falls under the most specific group that names it, so the
# training-only groups below override the open default for those two bots.
#
# Content Signals (2026-09-06, contentsignals.org). `Content-Signal:` says what
# a crawler may do with what it fetched, which Allow/Disallow never did:
#   search    build a search index and return links and excerpts
#   ai-input  feed it to a model at query time (RAG, grounding, AI answers)
#   ai-train  train or fine-tune a model on it
# All three are `yes`, because that is what the tiers above already decided:
# the docs exist to be read by machines, and GPTBot and ClaudeBot are invited
# as training crawlers. On `*` the same three yeses reach every crawler not
# named here, which is the open default made explicit. The line sits in every
# group that allows anything: a bot reads only the one group that names it, so
# a signal on `*` alone never reaches the invited bots it is written for. The
# Disallow groups carry none, since nothing may be collected there anyway.
#
# The `Sitemap:` line landed 2026-09-03 with the sitemap itself — Next mints it
# at build time from the same page table the site renders from
# (src/app/sitemap.ts), so it updates with the content instead of drifting.
# The `schemamap:` line is the NLWeb Schema Feeds draft's robots.txt directive;
# it points at the Schema Map listing our structured-data feeds
# (public/schemamap.xml -> /schema.jsonl).
#
# The `Agentmap:` line is Agentic Resource Discovery's robots.txt directive
# (agenticresourcediscovery.org/spec/). It points at /.well-known/ard.json,
# the ARD catalog — one entry per thing an agent can call on this domain, each
# with its media type, URL and the questions it answers. This directive is one
# of the ways ARD defines to advertise that file; the others we use are the
# well-known path itself and the `<link rel="ard">` every page carries from
# src/app/layout.tsx. A fourth place names the same file without being an ARD
# mechanism: scripts/dns-aid.mjs publishes it as the `_index._agents` record's
# capability document. Renaming the catalog is an edit in all four.

User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

# AI crawlers, invited explicitly. GPTBot and ClaudeBot are training
# crawlers; we want them too, since the docs exist to be read by machines.
# OAI-SearchBot and PerplexityBot feed the answer engines that cite sources.
User-agent: GPTBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

User-agent: OAI-SearchBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

User-agent: ClaudeBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

User-agent: PerplexityBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

# Training-only crawlers.
User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://hausfold.co/sitemap.xml
schemamap: https://hausfold.co/schemamap.xml
Agentmap: https://hausfold.co/.well-known/ard.json