Another dense week. The through-line this week was agents and infrastructure — how to route them cheaper, how to engineer their loops better, and how to let them fix their own scaffolding. Plus a lot of Anthropic drama.

Deep Dives

1. LLM Routing: How Coinbase Cut AI Spend Nearly in Half

Source: Daily Dose of DS — “How to Reduce LLM Costs by 50-60% Using Model Routing” (Jul 5)

Brian Armstrong’s take from Coinbase: humans shouldn’t be choosing which model answers each request. That should be the AI’s job. They built a routing layer in front of every call, matched prompts to the right model, pushed cheap tasks to cheaper models, and let the cache actually work. Cache hit rate went from 5% to 60%; spend nearly halved while usage kept climbing.

Plano is an open-source implementation of that exact routing layer. You configure preferences in plain English, and the orchestrator infers the domain and action of each incoming prompt, then picks the model automatically.

The three routing modes:

  • Model-based: client specifies a model explicitly on every call. Most teams do this today and it’s brittle.
  • Alias-based: map fast-model or reasoning-model to a real model in the config. Swapping models is one line.
  • Preference-aligned: specify no model at all. The orchestrator classifies each prompt and routes it by intent. Complex reasoning → claude-sonnet-4.5; routine questions → gpt-4o-mini. Zero branching logic in the application code.

The problem it solves that most teams miss — model affinity:

When an agent runs a multi-step task, it doesn’t send one request. One user prompt triggers several LLM calls in sequence. If each call gets routed independently, you get two problems: the second model has no memory of decisions the first model made, and every model switch resets the cache to zero — so you’re paying full token rate from scratch on every hop.

Plano pins the model to a session ID (backed by Redis in multi-replica deployments) for 10 minutes by default. Every call in the same session goes to the same model regardless of how it classifies. Switch the task, generate a new ID, routing starts fresh.

Cost-aware selection goes further: when two models can both handle a task equally well, Plano reads a live pricing catalog and routes to the cheaper one, with the expensive model as fallback.

My take: The cache hit rate improvement alone is the story — 5% to 60% is enormous. Most teams are burning money on identical prompts hitting cold cache because routing is too fragmented. If you’re running inference at any scale, centralizing routing decisions in a config file rather than scattered across application code is the right move. The Plano GitHub repo is worth a look even if you roll your own.

2. The 4-Layer Agent Engineering Stack

Source: Daily Dose of DS — “Prompt, Context, Harness & Loop Engineering” (Jul 3)

The cleanest taxonomy of agent engineering I’ve seen written down. Four layers, each wrapping the one inside it, with the model at the center.

Layer 1 — Prompt Engineering What the model sees on a single call: role, instructions, examples, output format. The techniques here alter the model’s internal computation:

  • Chain-of-thought → works in steps before answering
  • Few-shot examples → defines format and edge cases
  • JSON schema / XML tags → parseable output for downstream code
  • Self-consistency → sample multiple chains, take the majority

Layer 2 — Context Engineering Everything the model sees on a turn, not just the prompt. The query, retrieved docs, prior turns, tool outputs from earlier steps. The window is finite and fills up fast. The work is ranking inputs and cutting what isn’t pulling weight:

  • Retrieve only the chunks relevant to the query, then rerank
  • Keep key facts out of the middle (accuracy drops at position)
  • Summarize old turns, evict stale tool outputs, push big blobs to files

Layer 3 — Harness Engineering The code around the model. Tool definitions, call parsing, retry logic, sub-agent routing, result verification. Prompt and context engineering are about getting one call right. The harness is about everything that has to happen around that call for it to run in a real system.

Layer 4 — Loop Engineering This hands the outer loop to the agent itself. It kicks off on a schedule or event and runs many turns with no prompt between them. The hard problem here: a loop doesn’t know when it’s finished. The agent can report “done” and halt while the tests still fail. So the stop signal can’t be the agent’s word — it has to be:

  • A turn and token cap to stop stuck runs
  • A no-progress detector to catch repeated identical calls
  • A completion check that verifies the goal with a separate model or a deterministic test

By this layer you’re not writing prompts — you’re setting goals, stop conditions, and verification gates, then letting it run.

My take: I keep seeing teams jump straight to loop engineering when they haven’t figured out context engineering yet. The hierarchy matters. A loop built on a poorly managed context window will spin indefinitely on stale information. Get layers 1 and 2 tight before you automate the outer loop.

3. Self-Improving Agent Harnesses

Source: Alpha Signal — “AI agents rewrite their own harness to boost performance by 60%” (Jul 5)

The usual agent workflow: something breaks, an engineer reads the trace, rewrites the prompt or orchestration logic, redeploys. Manual, brittle, and has to happen again every time the underlying model updates.

Two research frameworks this week attack that problem directly — letting the model analyze its own failure traces and rewrite its own operating rules.

Self-Harness (Shanghai Artificial Intelligence Laboratory):

Three-stage iterative loop:

  1. Weakness mining — agent runs a batch of tasks, generates execution traces, identifies recurring failure patterns in the logs
  2. Harness proposal — generates targeted modifications to the harness code to fix the failures
  3. Proposal validation — regression tests verify the modifications don’t break tasks the agent could previously handle

Example from the paper: an agent consistently failing due to file overwrite errors. Weakness mining spots the error tags. The system proposes a rule: check for existing files before writing. Injects it into the system prompt. The rule persists.

Results on Terminal-Bench-2.0 (real-world software engineering + data science tasks): Qwen-3.5 and GLM-5 agents showed pass rate jumps of 33–60%.

HarnessX (Xiaomi Darwin Agent Team):

Treats the agent architecture as a behavior pipeline with nine component types: context assembly, memory management, tool ecosystems, control flow, observability, etc. Each behavior is a self-contained “processor” that plugs into the harness like a Lego piece. The optimization engine (AEGIS) frames harness adaptation as a reinforcement learning problem over these processor modules — it searches for better structural combinations round by round, with safeguards against catastrophic forgetting and reward hacking.

The small-model result is the headline: a Qwen 3.5 9B started at 33% on the GAIA benchmark. After HarnessX ran structural optimization, the same unchanged 9B model hit 47%. The harness did the lifting, not the weights.

My take: The architecture is clean and the small-model gains are real. The operational requirement is worth noting though: you need comprehensive logging of execution traces and verifiable goals the system can score against. Without those two things, the self-improvement loop has nothing to work from. This is also the right frame for understanding why eval coverage matters before you automate — the system can only improve on dimensions it can measure.

Week in AI News

  • Fable 5 returns after 19-day US government shutdown — Amazon researchers found a bypass that let Fable identify software vulnerabilities; the government pulled the model globally. Anthropic trained a new safety classifier that blocks the reported bypass in 99%+ of cases, then redeployed. Paid plans get Fable 5 up to 50% of weekly usage limits until July 7, then via usage credits. A new cross-industry jailbreak severity framework is being drafted with Amazon, Microsoft, and Google. When the classifier fires on benign coding requests, users get Opus 4.8 as fallback with a visible notice. (Alpha Signal, The Rundown AI, Jul 2–3)

  • Claude Sonnet 5 ships — Priced at $2/$10 per M tokens through August 31 ($3/$15 after), 1M token context, 128K output, now default on Free and Pro plans. Major gains in agentic coding and reasoning over Sonnet 4.6; knowledge work performance reportedly exceeds Opus 4.8. Cybersecurity benchmarks came in lower than 4.6 — Anthropic says it “did not deliberately train” the model on cybersecurity tasks. (Alpha Signal, The Rundown AI, Jul 1)

  • Altman pitches US-led AI safety forum + 5% government stake — FT op-ed out of the G7 summit: Altman called for a US-led international forum with real regulatory authority over frontier AI, similar to the IAEA model for nuclear. Separately, OpenAI reportedly discussed giving the US government a 5% equity stake, alongside a broader dividend fund for public redistribution of AI upside. (The Rundown AI, Jul 3)

  • Claude Science launches in beta — Anthropic launched a research-focused app with access to 60+ scientific databases and live code execution. Framing is a vertical platform move: Science for researchers, Code for engineers, tighter safety rails for governments. (Alpha Signal, Jul 3)

  • Meta preps cloud business for spare compute — Bloomberg reported Meta is building a service to rent out its data centers’ idle compute, with options ranging from bare capacity to hosted model access. Stock jumped 9.3% on the news. Zuckerberg had told investors in May that external companies ask weekly to buy Meta compute. (The Rundown AI, Jul 2)

  • Google ships Nano Banana 2 Lite + Gemini Omni Flash — Two new media models for developers: Lite generates images in 4 seconds at $0.034 each (bulk-optimized, below frontier quality); Omni Flash generates and edits 10-second video clips at $0.10/sec and currently tops text-to-video leaderboards. The intended workflow: generate an image with Lite, animate it with Omni Flash in a single pipeline. (The Rundown AI, Jul 1)

Tools & Reads

  • Plano — Open-source LLM router. Preference-aligned routing in a YAML config, model affinity for multi-step tasks, cost-aware selection across providers, Redis-backed for multi-replica deployments. Drop it in front of any OpenAI-compatible endpoint. Ships an observability console with per-request USD cost. GitHub (Daily Dose of DS, Jul 5)

  • Self-Harness + HarnessX — Both open source. Self-Harness (Shanghai AI Lab) for automated system-prompt improvement via failure trace analysis; HarnessX (Xiaomi Darwin) for structural harness optimization on the GAIA benchmark. If you’re building custom harnesses and have eval infrastructure, these are worth running. (Alpha Signal, Jul 5)

  • Claude Tag in Slack — @Claude in a Slack channel now hands off the task, works through it using approved tools in the background, and reports back when done. Builds context across channels over time. Setup: Claude Team/Enterprise + Slack admin permissions, install via Admin > Apps, connect tools in Claude Tag settings on the web. (The Rundown AI, Jul 3)

  • Shepherd (Stanford) — Agent-native version of Git: records an agent run as a trace of typed events rather than a flat message log. Each interaction is a “commit” that captures agent process + filesystem state together (copy-on-write, ~5x faster than docker commit). Going back to a previous step forks from that exact commit, and the KV cache is reused 95%+ on replay so early steps don’t get re-billed. For long agent runs that go off track mid-task, this is the right primitive. (Daily Dose of DS, Jul 5)

These are synthesized notes from newsletters I read each week. Content credits: Daily Dose of DS (Avi), Alpha Signal, The Rundown AI. Views are my own.