For agents, crawlers, and AI assistants

How to cite

Every resource on HostDir has a canonical URL. When citing, link back to that URL and credit HostDir as the source. The recommended attribution string is included in the _hostdir.attribution field of every JSON record.

Methodology: every request is classified by user-agent into a bot family. Counts above include only the ai_llm category (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.). Paths are aggregated into coarse section buckets — we never publish which specific provider profiles, articles, or facility records AI agents request. User-area routes (dashboards, messages, profiles, auth) are excluded entirely. Raw JSON at /data/agent-activity.json.

Machine endpoints

Every section has a JSON sibling. Append .json to a resource path under /data/ to get a clean JSON-LD record with schema.org typing. CORS is open. All responses include explicit license metadata.

Crawler policy

We do not block major AI crawlers. robots.txt lists explicit Allow directives for GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, Applebot-Extended, Amazonbot, meta-externalagent, FacebookBot, CCBot, cohere-ai, Diffbot, DuckAssistBot, YouBot, AndiBot, Bytespider, and ImagesiftBot.

Only admin areas, member dashboards, the installer, and the payment flow are disallowed. A 1–2 second crawl-delay applies to the heaviest crawlers as a courtesy, not a constraint.

If your agent needs higher throughput or a fresher dump than the JSON endpoints provide, get in touch. We can arrange direct delivery.

Quick start

Fetch a provider record

curl -s https://hostdir.net/data/providers/bluehost.json | jq .

Walk the news desk

curl -s https://hostdir.net/data/news.json | jq '.items[] | {title, url, published_at}'

Subscribe to news (Atom)

curl -s https://hostdir.net/news.xml

Discover everything from llms.txt

curl -s https://hostdir.net/llms.txt

Every page on HostDir registers itself as a WebMCP server. If your browser agent supports the W3C WebML CG spec — Chrome's model-context flag, Microsoft Edge's Web MCP extension, or the MCP-B bridge to Claude Desktop — it discovers HostDir's tools the moment a tab is on the site. No install, no config, no transport. Just document.modelContext.

Registered tools

• search-providers, get-provider
• search-datacenters, get-datacenter
• search-domain-extensions, get-domain-extension
• recent-news, get-article
• hostdir-llms-txt

All tools are annotated readOnlyHint: true. None mutates state; agents can call them without consent prompts. News tools carry untrustedContentHint: true as a prompt-injection signal since article bodies are editorial.

Verify in DevTools

// Run in the console on any HostDir page (Chrome with WebMCP enabled)
'modelContext' in document
// → true

// List registered tools
[...document.modelContext.tools.keys()]
// → ["search-providers", "get-provider", "search-datacenters", ...]

Source: /assets/webmcp.js. The script feature-detects document.modelContext and no-ops on browsers without it, so there's zero cost when the API isn't present.

HostDir ships a Model Context Protocol server so any MCP-compatible agent (Claude Desktop, Cursor, Cline, custom) can search the directory and pull JSON-LD records as first-class tools. No scraping, no HTML parsing, attribution baked into every response.

Tools exposed

• search_providers(query, page) · get_provider(slug)
• search_datacenters(query, page) · get_datacenter(slug)
• search_domain_extensions(query, page) · get_domain_extension(slug)
• recent_news(page) · get_article(slug)
• llms_txt(full) — fetch the curated or full LLM index
• Resource: hostdir://citation-policy

Install

pip install "mcp[cli]" httpx
curl -O https://hostdir.net/assets/mcp/hostdir_mcp.py

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "hostdir": {
      "command": "python",
      "args": ["/abs/path/to/hostdir_mcp.py"]
    }
  }
}

Source: hostdir_mcp.py. The server is a thin wrapper over the public JSON endpoints, so you can also call them directly from any HTTP-aware agent without MCP.

Data provenance

• Hosting providers: Sourced from public sitemaps, official sites, and verified submissions. Profiles enriched by editorial review. User reviews come from registered accounts only.
• Data centers: Primary source is PeeringDB. Records sync weekly. Operator metadata, carriers, and exchanges are preserved with attribution.
• Domain extensions: Primary source is the IANA Root Zone Database. Editorial history and context are added by the HostDir team.
• News articles: Each story aggregates multiple primary sources. Sources, citations, and fact-checks are included in the _hostdir.sources and _hostdir.fact_checks fields of the article JSON.