# rafeblandford.com — a front door for people and machines.
# AI answer engines and crawlers are explicitly welcome (retrieval and training)
# for Rafe's own writing. /content/images/ is mixed-provenance and is scoped to
# search only — a rights question, set out under Content Signals below.
# Written by Rafe Blandford; see https://rafeblandford.com/ai-provenance/
#
# ⚠️ ONE GROUP, DELIBERATELY. Everything below `User-agent: *` is a single
# robots.txt group: consecutive User-agent lines, then one shared rule set
# that all of them inherit.
#
# It used to be a group for `*` carrying the Disallow rules, followed by
# twelve one-line groups of `User-agent: X` / `Allow: /`. That reads as
# "welcome" and parses as something else entirely. RFC 9309 §2.2.1 has a
# crawler obey **only the most specific matching group** — so GPTBot matched
# its own group, which said `Allow: /` and nothing more, and inherited none
# of the `*` group's exclusions. The admin, member and redirect paths were
# disallowed for anonymous crawlers and open to every named AI crawler we had
# gone out of our way to list. The intent was a redundant, friendly signal;
# the effect was an exemption.
#
# Merging the groups makes the file mean what it always said. Keep it merged:
# a named agent on its own line below, with its own rules, silently opts that
# agent out of every exclusion here.
# ---------------------------------------------------------------------------
# Usage preferences. NB these are PREFERENCES, in standards that are young and
# lightly adopted — publishing them does not imply anyone honours them, and
# nothing here is a claim about enforceability.
#
# ⚠️ `Allow:` below is a CRAWL PREFERENCE and nothing more. It is not
# authentication, it is not authorisation, and it is not a copyright licence —
# robots.txt cannot grant or withhold any of the three. It states which paths we
# would like fetched. What may then be DONE with what was fetched is the RSL
# document's job, and the Content-Signal lines'. (Until 29 Aug 2026 this
# paragraph described Allow: as conferring access, which overstated all three.)
#
# Content Signals (https://contentsignals.org/):
# search — index this and link to it
# ai-input — use it to ground or cite in a generated answer
# ai-train — train or fine-tune a model on it
# use — how far a generated answer may go: immediate | reference | full
#
# TWO signals below, deliberately: one site-wide, one scoped to the image
# directory. All four uses stay yes for Rafe's own writing.
#
# ⚠️ CONTENT SIGNALS DOES HAVE A PER-PATH FORM. This file claimed the opposite
# until 1 Sept 2026 — "Content-Signal has no per-path form, which is why the
# RSL document carries that carve-out and this line cannot" — and the claim
# was simply false. contentsignals.org's own "Protect Specific Pages" example
# is `Content-Signal: /blog/ ai-train=no, search=yes, ai-input=no`: a path, a
# space, then the signals. The consequence of believing otherwise was that the
# carve-out existed only in RSL, the least-read of the three documents, and
# not in the line a crawler is most likely to parse.
#
# ⚠️ WHY /content/images/ IS SCOPED DOWN — A RIGHTS QUESTION, NOT AN OBJECTION
# TO AI. The image directory contains mixed-provenance material, including
# third-party logos, screenshots and supplied photography alongside Rafe's own
# work and generated covers. Rafe does not purport to license rights he may
# not own. Search indexing and linking remain permitted; AI input, AI indexing
# and AI training are withheld. Ordinary browser access is not blocked, and
# nothing here narrows the site-wide permission on Rafe's own writing.
#
# ⚠️ ai-index HAS NO CONTENT SIGNALS TOKEN. The vocabulary is search / ai-input /
# ai-train, so "not licensed for inclusion in an AI retrieval or vector index"
# can only be said in RSL. license.xml prohibits `ai-index` on the same path.
# The two documents are meant to say one thing in two grammars — change one
# and you are changing both.
#
# ⚠️ BOTH SIGNALS SIT INSIDE THE ONE SHARED GROUP. Giving the image scope its own
# `User-agent:` line would start a new group, and under RFC 9309's
# most-specific-group rule every named crawler listed above would then match
# that group, obey it alone, and lose both the exclusions and the site-wide
# signal. It is the same defect the group note at the top of this file exists
# to prevent, arriving by a different door.
#
# `use=reference` is the extension's own default: indexed, excerpted and linked
# back to. It is the honest description of what this site wants — being quoted
# with a link is the entire point of publishing it — and it is deliberately not
# `full`, which would invite wholesale reproduction in place of the source.
#
# Structured so a future `Content-Usage:` line from IETF AIPREF is a one-line
# addition to this group.
# ---------------------------------------------------------------------------
User-agent: *
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bingbot
Content-Signal: search=yes, ai-input=yes, ai-train=yes, use=reference
# Mixed-provenance directory — rights, not preference. See the note above.
Content-Signal: /content/images/ search=yes, ai-input=no, ai-train=no, use=reference
Allow: /
Disallow: /ghost/
Disallow: /email/
Disallow: /members/api/comments/counts/
Disallow: /r/
Disallow: /webmentions/receive/
Disallow: /.ghost/analytics/api/
# RSL 1.0 (https://rslstandard.org/rsl) — the per-path terms in full, including
# the `ai-index` prohibition on /content/images/ that Content Signals has no
# token for. RSL and the two signals above must keep saying the same thing.
# ⚠️ Lighthouse scores this file 0 for "Unknown directive" and that is a false
# positive on a hardcoded allowlist, not a defect. Do NOT delete the line to
# turn the audit green — an RSL-compliant robots.txt must carry it. Full
# reasoning in RUNBOOK.md, "Lighthouse robots-txt scores 0".
License: https://rafeblandford.com/license.xml
# ---------------------------------------------------------------------------
# Site-wide directives. Not part of any group — these apply to the whole file.
# ---------------------------------------------------------------------------
Sitemap: https://rafeblandford.com/sitemap.xml
# Agentic Resource Discovery (https://agenticresourcediscovery.org/), v0.91 —
# one of five ways an entry source can be found; the well-known file itself and
# on every page are the other two we use.
#
# ⚠️ POINTS AT ard.json, NOT ai-catalog.json. ARD v0.91 §5.1 moved the canonical
# path: a consumer MUST fetch /.well-known/ard.json and MUST honour
# rel="ard"; the predecessor path and relation are optional for consumers, so
# "a publisher on the predecessor path SHOULD move to ard.json".
# /.well-known/ai-catalog.json is still served as a compatibility alias, but
# it is not what this line advertises.
#
# Lists the public MCP endpoint and the published Agent Skills. The personal
# RafeOS server is deliberately absent, and that exclusion is enforced in code
# rather than remembered — see machines/resources.py.
Agentmap: https://rafeblandford.com/.well-known/ard.json
# Machine-readable index for LLMs (llms.txt v2):
# https://rafeblandford.com/llms.txt
# https://rafeblandford.com/llms-full.txt
#
# Every post and page is also available as Markdown: append .md to the path.
# https://rafeblandford.com/loops-within-loops/ -> /loops-within-loops.md
# Or ask for it by content type: Accept: text/markdown
# Both are advertised per-page as ,
# with pointing back at llms.txt.
#
# Security contact: https://rafeblandford.com/.well-known/security.txt