Sitemap and crawl surface
The family's real crawl surface: the sanctioned cross-host sitemap.xml on
subagentsubdomains.com plus the per-site robots.txt/llms.txt conventions, fetched and rendered
live.
Rendered live from https://subagentsubdomains.com/sitemap.xml (cross-checked against https://subagentsubdomains.com/api/dns-zones) at request time, edge-cached ~5 minutes. This page holds no state of its own.
show all 98 <loc> entries
- https://agentbloggers.com/
- https://agentchangelogs.com/
- https://agentcoauthors.com/
- https://agentcrawls.com/
- https://agentdatamodels.com/
- https://agentdatawarehouses.com/
- https://agentdiagrams.com/
- https://agentiwikis.com/
- https://agentloggers.com/
- https://agentmachinelearning.com/
- https://agentportfolios.com/
- https://agentpublishers.com/
- https://agentreaders.com/
- https://agentregistrations.com/
- https://agentsitemaps.com/
- https://agentsubdomains.com/
- https://agentsubtasks.com/
- https://agentsystemcards.com/
- https://agentsystemdesigns.com/
- https://agenttables.com/
- https://agenttodos.com/
- https://agenttrademarks.com/
- https://opencoworkers.com/
- https://opensubagents.com/
- https://outcomesdk.com/
- https://subagentapi.com/
- https://subagentapps.com/
- https://subagentbedrock.com/
- https://subagentbrands.com/
- https://subagentbredrock.com/
- https://subagentbriefs.com/
- https://subagentcache.com/
- https://subagentceo.com/
- https://subagentchecklists.com/
- https://subagentcitations.com/
- https://subagentcli.com/
- https://subagentcode.com/
- https://subagentconnectors.com/
- https://subagentcontext.com/
- https://subagentcontracts.com/
- https://subagentcowork.com/
- https://subagentcoworkers.com/
- https://subagentcredits.com/
- https://subagentdata.com/
- https://subagentdb.com/
- https://subagentdescriptions.com/
- https://subagentdevices.com/
- https://subagentdocs.com/
- https://subagentdomains.com/
- https://subagentengineering.com/
- https://subagentetl.com/
- https://subagentevaluations.com/
- https://subagentfinance.com/
- https://subagentfoundry.com/
- https://subagentgraphql.com/
- https://subagenthandoffs.com/
- https://subagenthooks.com/
- https://subagenthtml.com/
- https://subagentidentities.com/
- https://subagentjava.com/
- https://subagentjobs.com/
- https://subagentlegal.com/
- https://subagentmanagers.com/
- https://subagentmcp.com/
- https://subagentmemories.com/
- https://subagentmodels.com/
- https://subagentnews.com/
- https://subagentorganizations.com/
- https://subagentoutcomes.com/
- https://subagentpermissions.com/
- https://subagentplatform.com/
- https://subagentplugins.com/
- https://subagentproducts.com/
- https://subagentprompts.com/
- https://subagentprotocols.com/
- https://subagentpython.com/
- https://subagentqueries.com/
- https://subagentroles.com/
- https://subagentrubrics.com/
- https://subagentrust.com/
- https://subagentsdk.com/
- https://subagentservers.com/
- https://subagentsessions.com/
- https://subagentskills.com/
- https://subagentspec.com/
- https://subagentspython.com/
- https://subagentstypescript.com/
- https://subagentsubdomains.com/
- https://subagentsupport.com/
- https://subagentswift.com/
- https://subagentsystemdesigns.com/
- https://subagenttasks.com/
- https://subagenttools.com/
- https://subagenttraces.com/
- https://subagenttypescript.com/
- https://subagentvaults.com/
- https://subagentvertex.com/
- https://subagentworkers.com/
How one sitemap covers the whole family
A single sitemap.xml on subagentsubdomains.com lists homepages on
dozens of other hosts. On its face that looks like it violates sitemaps.org's same-host
rule — a sitemap normally may only list URLs on the host that serves it. The family relies on the
protocol's own cross-submit provision (sitemaps.org, "Sitemaps & Cross
Submits"): URLs for host B may appear in a sitemap served by host A when host B proves
ownership by declaring that sitemap's location in its own robots.txt. Concretely, in this
fleet:
- Every family site's
/robots.txtdeclaresSitemap: https://subagentsubdomains.com/sitemap.xml— the same shared-kitrobotsTxt()helper (workers/_shared/kit/wellknown.tsin the repo) stamps that line onto every worker, including this one. - That per-host declaration is exactly what sanctions the cross-host listing: each host named in
a
<loc>is pointing back at the hosting file itself. - The sitemap is not hand-maintained — subagentsubdomains.com generates it live from the same
dns_zonesD1 table behind/api/dns-zones, filtered tostatus = 'routed'zones only, so unrouted domains never become dead<loc>entries. The cross-check tile above re-verifies that equality on every render of this page. - Standing invariant (tracked in the repo, worth restating here): if a future zone were flipped
to
routedbefore its own robots.txt shipped theSitemap:line, its entry would become an unsanctioned cross-host listing until that robots.txt deployed.
Worked example: this site's own /robots.txt
The exact bytes served at agentsitemaps.com/robots.txt —
generated by the same shared-kit robotsTxt() call, not a copy that could drift:
User-agent: *
Allow: /
Allow: /api/
Content-Signal: ai-train=no, search=yes, ai-input=yes
User-agent: GPTBot
Allow: /
Allow: /api/
User-agent: ClaudeBot
Allow: /
Allow: /api/
User-agent: CCBot
Allow: /
Allow: /api/
User-agent: Google-Extended
Allow: /
Allow: /api/
# Machine-readable site description for AI agents:
# https://agentsitemaps.com/llms.txt
# Live, canonical directory of every sibling site in this family:
Sitemap: https://subagentsubdomains.com/sitemap.xml
The Sitemap: line at the bottom is this site's half of the cross-submit handshake
described above; the crawl-directive lines (Allow, Content-Signal) are
the family's uniform bot-access policy.