# Ravenpost — https://ravenpo.st
#
# The dashboard, admin and auth routes are a client-rendered SPA behind Clerk:
# a crawler gets an empty shell there, so disallowing them keeps thin,
# identical pages out of the index rather than protecting anything.
#
# Everything else is open to everyone, on purpose — search crawlers, AI
# crawlers and agents fetching a page on a person's behalf alike. Being
# quotable in an assistant's answer is worth as much to us as a blue link.
#
# NOTE: there are deliberately NO per-crawler groups below. A robots.txt group
# is winner-take-all: a bot matched by a `User-agent: GPTBot` group ignores the
# `*` group ENTIRELY, so naming a crawler here to say "you are welcome" also
# silently hands it the /dashboard and /admin paths this file exists to keep
# out of indexes. `Allow: /` under `*` already welcomes them. If a specific
# crawler ever needs its own group, that group must repeat every Disallow.

# Content Signals (contentsignals.org). A machine-readable statement of what we
# consent to, which robots.txt on its own cannot express: `Allow: /` says a
# crawler may fetch a page, not what it may then do with it. All three are
# `yes` on purpose and it is the opposite of most sites' answer — being quoted
# in an assistant's answer, and being part of what the next model knows, are
# both worth more to us than the pageview we would be protecting. Read as
# permission, not as a technical control; the honest ones honour it.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /dashboard
Disallow: /admin
Disallow: /sign-in
Disallow: /sign-up
Disallow: /invite
Disallow: /connect

# Written for models rather than crawlers: llms.txt is the map (a line per
# page), llms-full.txt is every page's text in one fetch, and each page has a
# Markdown twin at <page>.md linked from its <head>.
# https://llmstxt.org

Sitemap: https://ravenpo.st/sitemap.xml

# The same idea one layer up, for an agent rather than a crawler: where the
# sitemap lists pages to read, the catalogue lists the interfaces this domain
# exposes to be *called* — the MCP server and the REST API, each with its own
# URL and a description of what it does. An assistant that can act on a
# person's behalf needs the second list, and until it existed the only way to
# find our MCP endpoint was to read a marketing page and hope.
Agentmap: https://ravenpo.st/.well-known/ai-catalog.json