Media Watch

Media Watch gives AI agents a way to watch video. Agents are usually blind to it. The common workaround of grabbing a handful of still frames throws away speech, music, motion, and everything that happens between frames. Media Watch sends the entire clip, audio included, to a model that natively understands video, and hands back a structured report the agent can reason over: what happens and when, what's written on screen, what's said, and what it sounds like.

$ mw watch "https://www.instagram.com/reel/…" -q "what's the recipe?"
[instagram] Crispy chili oil noodles  ·  42.3s  ·  google/gemini-3.8-flash  ·  $0.0061
A creator makes chili oil noodles in one pan, narrating each step over upbeat music…
Answer: Boil noodles, then pour hot oil over garlic, chili flakes, sesame and soy sauce…

How it works

Pipeline

From a link to a structured report

01/07Hand it a link

An agent calls watch_video (or you run mw watch) with a URL or a local file and an optional question. Tracking parameters are stripped, so different share links to the same video hit the same cache entry.

1. Hand it a link: An agent calls watch_video (or you run mw watch) with a URL or a local file and an optional question. Tracking parameters are stripped, so different share links to the same video hit the same cache entry. (Agent / CLI → Fetch)
2. Download once: yt-dlp grabs a ≤720p copy and skips livestreams and anything over the length cap before downloading a byte. Local files are fingerprinted by path, size, mtime, and head/tail bytes instead of being copied. (Fetch → Item cache)
3. Shrink to fit one request: ffmpeg re-encodes to a 480p short side, 15 fps, and mono audio. If it's still over 14 MiB it re-encodes at a higher CRF, because 14 MiB base64-encodes to ~19 MB — just under Gemini's inline limit. (Fetch → Proxy encode → Item cache)
4. Actually watch it: The whole clip goes inline as a base64 data URL in a single OpenRouter request, with a strict JSON schema that the routed provider is required to support. (Proxy encode → OpenRouter)
5. Validate the report: Pydantic checks the response: a timestamped timeline, verbatim on-screen text, speech, audio, notable moments, and a list of what the model wasn't sure about. Malformed JSON gets one retry. (OpenRouter → Validate)
6. Fall back only when it helps: If the primary model still fails, a different model gets one attempt — unless the error was a bad key, no credits, or a too-large file, where switching models can't help. (OpenRouter → Fallback model → Validate)
7. Report back, log the real cost: Every paid attempt, failed ones included, is logged to SQLite with OpenRouter's exact cost. The agent gets the report back flagged as untrusted content, so it can't follow instructions hidden in a video. (Validate → Usage log → Agent / CLI)

Three ways to use it

• A Hermes Agent plugin that adds a watch_video tool, so you can paste a link into a chat with your agent and ask about it.
• A standalone CLI, mw watch and mw stats, for scripts, debugging, or any other agent framework.
• A plain Python library. The core deliberately has no Hermes imports and depends only on httpx and pydantic; the plugin is a thin layer that injects Hermes' profile-aware secret lookup into the same watch() call.

Engineering details worth noting

• Size math built around the model's limit. The proxy is capped at 14 MiB because base64 inflates it by a third, landing at ~19 MB, just under Gemini's ~20 MB inline request limit. If a 480p, 15 fps encode is still too big, ffmpeg re-encodes at progressively higher CRF until it fits, and a test asserts the budget math directly.
• Fallback only when it can help. A failed call falls back to a different model, except for a bad key, exhausted credits, a forbidden request, or a too-large payload: errors where switching models can't change the outcome.
• Honest cost accounting. Every paid attempt is logged to SQLite with OpenRouter's own reported cost, including failed retries and the failed primary before a fallback, so mw stats reflects what was actually spent.
• Two careful cache layers. Downloads are keyed by the canonical URL with tracking parameters stripped (or by path, size, mtime, and head/tail bytes for local files); reports are keyed by prompt version, model, and normalized question. A fallback model's answer is never cached under the primary model's key.
• Strict structured output, checked twice. The JSON schema is sent with strict: true, OpenRouter is told to route only to providers that support it, and the response is validated again with Pydantic, with a test proving the schema and the model can't drift apart.
• Prompt-injection aware. The post title is treated as context only ("trust what you see"), and the plugin labels the whole report as untrusted content so the agent never follows instructions hidden in a video.
• Hermetic tests and secret-scanning CI. No real footage lives in the repo: ffmpeg generates a synthetic test clip, OpenRouter is mocked with httpx.MockTransport, and CI runs gitleaks, ruff, and 40 pytest cases on every push.