Dense week. The throughline was efficiency at every layer — smarter caching, smarter routing, smarter model orchestration. Plus a genuinely surprising piece of AI science from Anthropic.

Deep Dives

1. Rethinking KV Caching for Production Inference

Source: Daily Dose of DS — “Rethinking KV caching for production inference” (Jul 7)

Stanford researchers found that ~62% of what an agent sends to the LLM on every call is just repeated content — system prompts, tool definitions, documents it already processed last turn. Token prices dropped 80% between 2023 and 2026, but agentic workflows consume 5–30x more tokens per task than a chatbot query. The volume outran the price cuts.

The standard fix is prefix caching: if two requests share the same opening tokens, the provider stores the KV tensors from the first and skips recomputing them on the second. Anthropic gives a 90% discount on cached input tokens. For stable system prompts and tool definitions, this is the highest-leverage optimization available.

But prefix caching has a hard ceiling. The match has to be byte-for-byte from position zero. Three common failure modes:

  • Multi-document RAG: Cache doc A and doc B separately. A query needing both causes a full miss because doc B’s cached state was computed without doc A’s context.
  • Document order changes: Same three docs in different order = different hash = full miss.
  • Growing conversation history: Each turn changes the context after the stable prefix, making earlier cached states useless.

Alibaba Cloud’s production data validates this: 10% of cache blocks serve 77% of hits. Most cached content never gets reused.

LMCache takes a different approach. Instead of running cache management inside the inference engine, it runs as a completely separate process alongside it. The engine tells LMCache “here are the block IDs I need” (tiny messages) and all the heavy tensor I/O happens inside LMCache’s own process. Benefits:

  • No resource contention between cache I/O and inference compute
  • Zero-copy sharing across GPUs (both read/write the same memory region directly)
  • Multi-tier parallel loading: checks GPU memory, CPU RAM, SSD, and remote storage simultaneously

On H200 GPUs with Qwen3-235B and 50 concurrent users: 14x faster time-to-first-token, 4x faster decoding, startup time from 3+ minutes to ~30 seconds.

The paper CacheBlend (EuroSys 2025 Best Paper) solves the multi-document problem. The key observation: most tokens in a transformer attend to their own local context. Only a small fraction have strong cross-document connections. CacheBlend identifies and selectively recomputes only those tokens, reusing everything else from the independent per-document caches. Result: 2–4x faster processing for multi-doc RAG queries with no quality loss.

LMCache integrates with vLLM, SGLang, and TensorRT-LLM. If the inference engine crashes, LMCache preserves cache on CPU and storage. If LMCache crashes, inference continues in downgrade mode and reconnects automatically.

My take: The architectural insight is right. Cache management and inference are fundamentally different workloads — one is I/O-heavy, the other is compute-heavy. Conflating them creates contention. If you’re running inference at any scale and haven’t looked at disaggregated caching, this is the paper to read.

2. Production LLM Routing: The Model Affinity Problem

Source: Daily Dose of DS — “How LLM routing actually works in production” (Jul 12)

A good interview puzzle: you add a routing layer to your agent that sends easy calls to a 15x cheaper model. The router works. But the bill doesn’t change. Where did the savings go?

The answer is in how agents consume tokens. A typical agent task is never one LLM call — it’s planning, tool use, analyzing results, each sending the accumulated context back through the model. Providers handle this with prompt caching: input already seen costs ~90% less than fresh input. But that cache is model-specific. Every time the router switches models mid-task, the accumulated warm cache gets thrown away and the full context is re-billed at cold rates. A cache-hot expensive model can cost less than a cache-cold cheap model.

Production routers solve this with model affinity: make the routing decision once per task, not once per call. The first call routes normally. The winning model gets pinned to a session ID (Redis-backed for multi-replica deployments). Every call after that goes to the same model, so the cache stays warm and the context stays coherent. When the task changes, the pin resets.

The 4-stage pipeline:

A) Guardrail filter — jailbreaks and unsafe inputs get caught before reaching the router; every call is logged from this point.

B) Router model — a small model (Arch-Router is 1.5B, trained on human preference data) reads the prompt, infers domain and action, and matches against candidate models. Has to be tiny — a routing decision that costs as much as the call it’s routing saves nothing.

C) Selection policy — picks the winner by cost or latency preference from a live pricing catalog. Adapts when providers reprice; runner-up stays on standby as fallback.

D) Model affinity — pins the winner to the session ID so the cache never resets and context never splits across providers.

Plano implements this pipeline as an open-source proxy. Config lives in one YAML file, swapping any layer never touches agent code, and it ships an observability console with per-request USD cost. The Daily Dose team put it in front of their Hermes agent and usage dropped 2x without changing a line of agent code.

My take: Most teams I’ve seen are doing per-call routing, which is exactly the wrong level. The cache hit rate story is the real reason to centralize routing — not just cost splitting, but keeping the context coherent and the cache warm across an entire task chain.

3. Fable as Advisor, Fable as Orchestrator

Source: Alpha Signal — “Anthropic’s Claude agent trick hits 96% top model performance at half the cost” (Jul 8–9)

Anthropic published two patterns for getting near-Fable quality without routing everything through Fable. The idea: stop using the frontier model as a workhorse. Use it only where it actually matters.

Advisor pattern: Sonnet 5 does all the heavy lifting. It only calls Fable 5 once per task when it needs guidance — roughly once per task, not once per call. On SWE-bench Pro: 92% of Fable’s score at 63% of the cost. Most tokens bill at Sonnet’s cheaper rate.

Orchestrator pattern: Fable plans the task and hands off execution to Sonnet 5 workers running in parallel. On BrowseComp: 96% of Fable’s performance at 46% of the price. A concrete example from the cookbook: verifying 20 facts cost $1.61 with the split team versus $4.00 solo. Caching bonus — each sub-agent maintains its own cache independently, so repeated context doesn’t get billed twice.

Both patterns run through Claude Managed Agents. Working code for both setups is in the claude-cookbooks repo.

My take: The orchestrator pattern is more interesting architecturally. It essentially makes the frontier model a high-level planner and the cheaper model a parallel executor pool. The cost advantage compounds with task complexity — the more steps, the more you save on execution while the planning overhead stays fixed. The hard part is getting the task decomposition right: if Fable breaks the task poorly, Sonnet workers execute the wrong things efficiently.

Week in AI News

  • Anthropic discovers Claude’s “J-space” — Researchers found a hidden internal workspace inside Claude that nobody designed on purpose. J-space acts like a mental scratchpad where Claude silently thinks before responding, holding active concepts the model uses while processing. It’s different from chain-of-thought text — these internal representations stay off-screen even when they shape the final answer. In tests, editing J-space directly changed Claude’s answers: swapping an internal “spider” pattern for “ant” during a question about legs flipped the response from 8 legs to 6. Deleting J-space let Claude recall facts and chat normally, but multi-step problem completion collapsed. Anthropic notes this doesn’t reveal whether Claude is conscious or “feels anything at all” — but finding an undesigned, brain-like workspace that emerged spontaneously during training is striking regardless of what you think about AI consciousness. (Alpha Signal, The Rundown AI, Jul 7)

  • OpenAI releases GPT-5.6 Sol + ChatGPT Work — The GPT-5.6 family launches with Sol (flagship), Terra, and Luna tiers. Sol lands slightly below Fable 5 on the Intelligence Index but tops it on agentic coding, with improvements to computer use and design. Pricing matches GPT-5.5 ($5/$30 per M tokens for Sol down to $1/$6 for Luna). Sol notably “autonomously post-trained Luna” during development. ChatGPT Work is OpenAI’s answer to Claude Cowork — Codex’s engine behind a more accessible work agent. The Codex app is merging into a revamped ChatGPT desktop app with a built-in browser and computer control. (The Rundown AI, Jul 10)

  • Meta Muse Spark 1.1 hits the API — Meta released Muse Spark 1.1 for agent-style tasks, computer use, and long sessions via paid API. Priced at $1.25/$4.25 per million input/output tokens — about a quarter of top rivals. Zuckerberg called it “state-of-the-art or very close to it” on agent reasoning and tool use, claiming it leads Opus 4.8 and GPT-5.5 on several related benchmarks. 1.1 can split jobs across parallel subagents and uses a 1M context window. API access starts with $20 in credits; the heavier ‘Watermelon’ model is still in training. (The Rundown AI, Jul 10)

  • xAI rebrands to SpaceXAI — Following the February merger (valued at $250B), xAI has officially taken the SpaceXAI name. (The Rundown AI, Jul 9)

  • SWE-Bench Pro: 30% of tasks are broken — OpenAI researchers found that 30% of SWE-Bench Pro tasks are broken or incorrectly specified. OpenAI pulled its recommendation of the benchmark. This is a significant problem — SWE-Bench has been the de facto agentic coding leaderboard, and if nearly a third of the eval is invalid, scores across the board need re-reading. (Alpha Signal, Jul 9)

  • Tencent open-sources Hy3 — Small MoE model that runs on under half the hardware needed for GLM-5.2, with a permissive Apache 2.0 license (avoiding the regional restrictions of earlier Chinese models). Benchmarks show it trailing GLM-5.2 on SWE-bench but topping open rivals on web research and tool use. Dario Amodei put Chinese frontier models at 6–12 months behind; Hy3 doesn’t appear to challenge that timeline but the efficiency gains and open licensing make it compelling for deployment. (The Rundown AI, Jul 7)

  • MIRA: neural world model plays Rocket League — Kyutai and General Intuition released MIRA, an open-source world model that runs live 2v2 Rocket League for four players — generated entirely inside a neural network with no game engine underneath. Trained on 10K hours of AI bot gameplay, it renders boost meters and crashes while keeping four screens in sync on a single GPU at 20 FPS. Its memory spans ~4 seconds; when a goal replay rolls, MIRA hallucinates convincing footage of a goal that never actually happened. The point isn’t to replace games — it’s to simulate the physical world for robot training data. (The Rundown AI, Jul 7)

Tools & Reads

  • LMCache — Disaggregated KV cache for LLM inference. Runs as a separate process alongside vLLM, SGLang, or TensorRT-LLM. Multi-tier parallel loading (GPU → CPU → SSD → remote storage), zero-copy GPU sharing, fault-tolerant failover. If you’re running inference at volume and haven’t revisited your caching stack, this is where to start. GitHub (Daily Dose of DS, Jul 7)

  • Plano — Open-source LLM router with 4-stage pipeline: guardrail filter, Arch-Router 1.5B for intent classification, cost-aware selection policy, and Redis-backed model affinity. YAML config, OpenAI-compatible proxy, built-in observability console with per-request cost. GitHub (Daily Dose of DS, Jul 12)

  • Transformer Lab — Open-source ML platform that orchestrates GPUs across any cloud. Supports LoRA, QLoRA, DPO, ORPO, SIMPO; works with MLX, vLLM, Ollama, and HF Transformers; one-click conversion between HF/GGUF/MLX; LLM-as-judge evals plus EleutherAI harness; submits jobs to Slurm and SkyPilot from the same UI. Solo researcher on a MacBook or 64-GPU cluster, same interface. GitHub (Daily Dose of DS, Jul 12)

  • Rowboat — Local AI second brain (15K+ GitHub stars). Background agents index your emails, meetings, and notes into a living knowledge graph; work surfaces include email client, meeting note taker, browser, and code mode. Bring your Obsidian vault, connect Slack and Fireflies, schedule agents on cron. Inspired by Karpathy’s LLM wiki pattern. GitHub (Daily Dose of DS, Jul 8)

  • Claude Managed Agents cookbook — Anthropic’s working code for the Advisor and Orchestrator patterns with benchmarks and cost breakdowns. Worth keeping nearby if you’re building multi-model pipelines. GitHub (Alpha Signal, Jul 8–9)

These are synthesized notes from newsletters I read each week. Content credits: Daily Dose of DS (Avi Chawla & Akshay Pachaar), Alpha Signal, The Rundown AI. Views are my own.