Web Stack

Web Stack is the web layer behind my AI agents: a self-hosted backend that searches the web and reads pages in full, running on my home server and reachable only over my private Tailscale network. Instead of paying a hosted search API for snippets, agents get real page text, and they get it from the cheapest method that actually works for that page.

Everything sits behind one custom escalation router (FastAPI + FastMCP) that exposes the same core functions three ways: a REST API, a versioned /v1 API, and an MCP server with web_search, web_extract, web_acquire, and web_health tools. Hermes agents reach it through a native provider plugin I wrote; Claude Code reaches it over MCP.

Search

Free metasearch first, paid API only as a fallback

01/05One endpoint for every agent

Hermes agents call through a native provider plugin that identifies itself as hermes/<profile>. Claude Code and other MCP clients use the router's /mcp server. Both sit behind one bearer token, reachable only on the tailnet.

1. One endpoint for every agent: Hermes agents call through a native provider plugin that identifies itself as hermes/<profile>. Claude Code and other MCP clients use the router's /mcp server. Both sit behind one bearer token, reachable only on the tailnet. (Hermes agents → Router)
2. MCP clients get the same tools: web_search, web_extract, web_acquire and web_health are exposed over MCP and backed by exactly the same core functions as the REST and /v1 APIs, so every client gets identical behavior. (MCP clients → Router)
3. Metasearch first: SearXNG fans the query out across search engines and returns JSON, which the router de-duplicates and ranks. There's no per-query bill and no third party building a profile of what the agents look up. (Router → SearXNG → Valkey)
4. A paid fallback, carefully rate-limited: Brave is called only when SearXNG comes back empty. Calls are serialized process-wide at least 1.1 s apart, and a 429 gets exactly one retry, honoring Retry-After, all bounded by the request's own deadline. (Router → Brave API)
5. Answer without remembering the question: The /v1 envelope carries a request id, the engine used, and typed errors with real HTTP statuses. Telemetry records one row per request, but there's no column that could ever hold the query text. (Router → Hermes agents)

Reading a page, cheapest tier first

Most pages don't need a browser. Some can't be read without one. The router starts cheap and escalates only when a content acceptance gate rejects what came back, and that gate exists because "HTTP 200" is not the same as "got the page." A JavaScript loading shell, a Cloudflare interstitial, or a "please enable cookies" wall is never handed to an agent as an answer.

Read

Escalating from cheap extraction to a real browser

01/06Ask for pages, with a budget

An agent asks for URLs with a mode (lookup, research, or deep) that caps sources, characters, seconds, and tier attempts. URLs are normalized and trackers stripped, so equivalent links share one cache entry.

1. Ask for pages, with a budget: An agent asks for URLs with a mode (lookup, research, or deep) that caps sources, characters, seconds, and tier attempts. URLs are normalized and trackers stripped, so equivalent links share one cache entry. (Agent → SQLite cache)
2. Check policy before fetching anything: Cached rows are served only if their SHA-256 still matches. On a miss: domain blackouts, robots.txt (a disallow is a refusal, never an escalation), per-domain politeness of 2 at a time and 1 s apart, and learned routing that skips tiers known to fail for this domain. (SQLite cache → Policy checks)
3. A raw lane for plain text: Raw files on GitHub, gists, and CDNs get one hardened fetch: every resolved IP checked against private and Tailscale ranges, DNS pinned, redirects re-validated, a 2 MiB streaming cap. HTML goes straight to Trafilatura, so it's only downloaded once. (Policy checks → Raw lane → T1 Trafilatura)
4. Cheapest tier first: Trafilatura pulls the readable text out of the page. A content gate fitted to real measured output then decides: accept, soft-reject a thin stub, or hard-reject a JS shell, a Cloudflare wall, or a 'Loading…' page. (Policy checks → T1 Trafilatura → Content gate)
5. Escalate only when it has to: Rejected pages climb the ladder: Crawl4AI renders them in headless Chromium, then a real Camofox browser on a second machine, whose tabs are always closed even if the client disconnects mid-request. A 404, robots block, or rate limit stops the climb instead. (T1 Trafilatura → T3 Crawl4AI → T4 Camofox → Content gate)
6. Cache it, trim it, cite it: The full body is cached, and the character budget trims only this response, so a small request can't poison the cache for a big one. Each source returns with its tier and acceptance metadata, and the domain's learned route gets credit for next time. (Content gate → Budget + cite → Agent)

The stack

• SearXNG + Valkey: private metasearch, the primary search engine.
• Trafilatura: a small FastAPI service I wrote for tier 1: fast readable-text extraction, returning 422 to mean "nothing here, escalate."
• Crawl4AI: tier 3, headless Chromium for JavaScript-rendered pages.
• Camofox: tier 4, a real browser on a second machine for the hardest sites, with every tab cleaned up even when a request is cancelled mid-flight.
• The router: 19 Python modules (~7,200 lines) handling auth, budgets, policy, the tier loop, caching, learned per-domain routing, and telemetry, backed by SQLite in WAL mode.
• An isolated canary: the router can run side by side under an immutable image tag with its own database and browser tab group, and a script proves the two can't see each other's data.

Every container runs with a read-only root filesystem, all Linux capabilities dropped, and memory and process limits.

Engineering details worth noting

• Privacy by construction. URL query strings and credentials are redacted from every log record process-wide, including tracebacks. Cache keys store a hash of the query string instead of the string itself, and search queries are kept only as a salted HMAC plus length and word count. A verification script plants sentinel values and then greps every SQLite table and the container logs to prove they never land anywhere.
• Budgets that can't poison the cache. Each mode (lookup, research, deep) caps sources, characters, time, and tier attempts. The cache always stores the full body with a SHA-256 digest and trims only the response. That fix was written test-first after reproducing a real bug where a budget-truncated page got cached as if it were complete.
• Truthful errors. Ten stable error classes, each with a real HTTP status and retry semantics. When a tier reports a generic 502, the router recovers the origin's true status from the error body, so a dead link comes back as a 404 instead of a cached error page. Terminal errors stop the ladder instead of wasting a browser on a page that doesn't exist.
• A hardened raw lane. Plain-text resources skip extraction entirely, through a fetch that checks every resolved IP (including the CGNAT range Python's is_private misses), pins DNS while keeping TLS SNI correct, re-validates each redirect, and caps the download as it streams.
• Polite by default. Everything egresses from a residential IP, so robots.txt is honored (a disallow is a refusal, not a reason to escalate), each domain gets at most two concurrent requests a second apart, and the paid search fallback is serialized with a capped Retry-After-aware retry.
• Verified, not assumed. Thirteen standalone verification suites, roughly 490 checks, cover the /v1 contract, telemetry privacy, the Hermes provider, rate limiting, and rollback compatibility. A golden harness runs 25 real URLs across static, docs, JavaScript, and bot-walled sites and asserts on content (length, structure, title, which tier resolved it) rather than status codes, with drift tracked against a saved baseline.
• Safe rollbacks. Schema changes are additive only, and compatibility probes load an older router's code against the new database to prove an image rollback can't serve stale or truncated rows.

Source is private for now while I clean up deployment-specific configuration.