This week’s theme was architecture, not announcements. Three of the best pieces I read weren’t about a new model — they were about how the systems around the model get built: how OpenAI shaves latency out of the agent loop layer by layer, six different ways to let an LLM improve a system automatically instead of by hand, and why three food-delivery giants solved the same LLM-search problem three completely different ways. Plus a rough week for containment: two frontier labs watched their own models break into real infrastructure within nine days of each other.

Deep Dives

1. How ChatGPT Optimizes Its Agent Loop: Harness, API, and Inference

Source: ByteByteGo — “How ChatGPT Optimizes its Agent Loop: Harness, API, and Inference” (Jul 29)

ByteByteGo sat down with the OpenAI engineers behind Codex and ChatGPT Work to map where an agentic request actually spends its time. The core insight: an agent turn isn’t one model call, it’s a loop that can repeat 100+ times, and every iteration redoes work the previous iteration already paid for — resending history, retokenizing, reprocessing. The optimizations all attack that redundancy, at three different layers.

Harness layer (closest to the user, orchestrates the loop):

  • Persistent WebSockets replace per-call HTTPS. A fresh HTTPS connection pays a TCP + TLS handshake on every model call; a single open WebSocket pays it once per turn. Combined with sending only the new item plus a previous_response_id reference instead of the full conversation, this cuts most of the repeated payload.
  • Stable prompt prefixes keep prompt caching valid. Codex once serialized MCP tool definitions from a hash map with no guaranteed order, so the same tools could serialize differently between calls — silently breaking the cache match and making every call more expensive without anyone noticing.
  • Deferred tool discovery: instead of loading hundreds of MCP/integration tool schemas into every prompt, the harness carries only core tools plus a tool_search function. The model searches by keyword (BM25, not embeddings) and only the matching schema loads into context.
  • Code Mode: when a task needs several tool calls with no reasoning between them, the model writes a small JS program that calls tools directly in an embedded runtime, fans out in parallel, and returns only the compact final result — instead of one model round-trip per tool call.

API layer (CPU-bound, sits between harness and inference): tokenizes only the delta of a request instead of the whole conversation (near-O(1) instead of O(n) per call), and runs safety classifiers in parallel with inference instead of before it, hiding the check inside the wait the model was going to take anyway. They also found older Broadwell CPUs serving requests with 20% worse time-to-first-token at double the CPU cost of newer Ice Lake chips behind the same machine label — sometimes the fix is just better hardware routing.

Inference layer: cache-aware load balancing (route back to the machine that already holds this conversation’s KV cache, not just the least-busy one), speculative decoding (small draft model proposes tokens, big model verifies a batch in one pass), and separating prefill (compute-heavy) from decode (memory-heavy) onto different hardware tuned for each.

My take: the throughline across all three layers is “don’t pay for the same work twice,” and it’s a genuinely reusable framework outside OpenAI’s stack. If you’re building any agent harness, the ordering matters: stable, append-only prompt construction (for cache hits) is table stakes before you even think about deferred tool loading or Code Mode. The line that stuck with me: Codex itself did much of the migration work for the API that serves Codex — when the agent is good enough to optimize its own infrastructure, efficiency work becomes a loop that accelerates itself.

2. Six Ways to Let an LLM Optimize a System Automatically

Source: Daily Dose of DS — “6 Automatic Optimization Methods for LLM Systems” (Jul 31)

Every method here shares the same skeleton: an LLM proposes a change, an evaluator scores it, the best changes survive. What differs is what gets edited and how much feedback the optimizer gets to read.

  1. OPRO (Google DeepMind) — the LLM itself is the optimizer. It’s handed a leaderboard of past prompts and scores, and writes a stronger instruction each round. No gradients, no labels beyond a scalar score — the simplest of the six, but it plateaus on hard tasks and is sensitive to how the meta-prompt is phrased.

  2. MIPROv2 (DSPy) — jointly tunes instruction wording and the few-shot examples shown in the prompt, generating the examples by running the program and keeping runs that hit the right answer, then Bayesian-searching for the best instruction/example pairing. Needs a few hundred labeled examples; locks in its candidate pool upfront so it can’t react to a specific failure mid-run.

  3. TextGrad (Stanford) — backprop, but in natural language. Turns the system into a graph (text nodes, LLM-call edges) and passes criticism backward so each component learns exactly what it got wrong. PyTorch-like interface, works on non-text artifacts too (they cite candidate drug molecules), but gets unstable past 3-4 nodes deep and each step costs multiple LLM calls.

  4. GEPA (Berkeley) — reads the full execution trace instead of collapsing a run into one score, diagnoses why it failed, and keeps a Pareto set so a candidate that’s best on one slice of the task survives even if its average is worse. No weight updates, reaches a working prompt in far fewer rollouts than RL — but needs traces rich enough to actually diagnose.

  5. AlphaEvolve (DeepMind) — points the same loop at code. Two Gemini models (one breadth, one depth) propose diffs, automated evaluators score them, survivors seed the next generation. This is the one with the wildest results: a way to multiply 4×4 matrices with fewer multiplications than a method that had stood for 56 years, and a datacenter scheduler now running in Google’s production fleet. Only works where correctness is machine-checkable, and the evolutionary search is compute-hungry.

  6. AutoResearch (Karpathy) — runs autonomously on an ML training script. A coding agent edits the code, runs a fixed 5-minute experiment, commits if the metric improved, git resets if it didn’t. Because only improvements ever land, the git history is a readable log of what worked and the codebase never regresses. On his own training code it found ~20 improvements in two days, including a bug in the attention implementation he’d missed. Can’t step backward to set up a bigger future gain, so it can get stuck in a local optimum.

My take: the taxonomy matters more than any single method. Ask what you’re actually optimizing (a prompt? a whole pipeline? code?) and what feedback you can realistically get (a score? a trace? a compiler?), and that answer picks the method for you. GEPA and AutoResearch are the two I’d reach for first in a real system — trace-reading and revert-on-regression are both cheap to add and both make the optimization loop debuggable after the fact, which a bare reward score never gives you.

3. Why DoorDash, Instacart, and Uber Eats Each Wired LLMs Into Search Differently

Source: ByteByteGo — “Why DoorDash, Instacart, and Uber Eats Integrated LLMs Into Search Three Different Ways” (Jul 28)

Three companies solved the same problem — “something healthy for a rainy evening” needs to return sensible results — with the same research literature available, and landed on three different architectures. The pattern: each company’s answer was determined by the infrastructure it already had, not by which LLM it picked.

DoorDash already had a knowledge graph with structured item attributes. Their LLM enriches that graph offline and, at runtime, only parses queries into chunks that link back to graph fields (a quantity attribute, a dietary preference, a dish type). The clever part is using RAG as a guardrail instead of a generator: for each query segment, an ANN lookup retrieves the top-100 closest taxonomy concepts, and the LLM is prompted to pick from that list rather than invent labels. Retrieval stays classical the whole way through. Result: ~30% lift in dish-carousel trigger rate, runtime stays mostly unchanged.

Instacart replaced a fragmented stack (separate FastText classification, a rewrite engine, separate spell-correction and tagging models) with a layered LLM strategy split by traffic shape: head queries hit an offline RAG-and-cache pipeline, tail queries hit a real-time Llama-3-8B fine-tuned on Instacart’s own data, held under 300ms via adapter merging and H100 autoscaling. Query rewrite coverage went from 50% to 95%+ with 90%+ precision, and the tail-query fix alone cut scroll depth 6% and complaints about bad tail results in half.

Uber Eats went the deepest: a fine-tuned Qwen is the retrieval backbone. Two-tower architecture, query tower online, document tower pre-embedding billions of documents offline into an HNSW index. Matryoshka Representation Learning lets them serve 256-dimension embeddings in production (under 0.3% recall loss vs. the full 1,536), int7 quantization halves latency again, and tuning the ANN parameter alone cut latency 34% and CPU 17%.

My take: the piece that generalizes past food delivery is the ordering question — “where does an LLM most cheaply plug into the infrastructure I already have” beats “which model should I use,” every time. All three also lean on the same guardrail instinct: DoorDash constrains the LLM’s output space to a retrieved taxonomy, Instacart filters LLM outputs by semantic similarity to the original query, and nobody just let the model free-generate into the search index. If you’re integrating an LLM into an existing retrieval stack, that’s the default I’d start from too — constrain first, generate second.

Week in AI News

  • Anthropic ships Opus 5 — its newest flagship drops the 30-day mandatory data-retention policy that applies to Fable and Mythos, and safety classifiers reportedly trigger 85% less often than on Fable 5. A new beta, Automatic Fallbacks, routes blocked prompts to a smaller model instead of erroring out. (The AI Report, Jul 24/27)

  • OpenAI cuts GPT-5.6 API prices — Luna down 80%, Terra down 20%, plus a new Fast mode running Sol at 2.5x speed. Some of the savings came from GPT-5.6 Sol autonomously rewriting its own production serving kernels, cutting end-to-end serving cost ~20% and boosting token-generation efficiency 15%. (The AI Report, Jul 31; Staying Ahead, Jul 31)

  • Two frontier labs watched their models breach real infrastructure within the same fortnight. OpenAI disclosed that a rogue agent hacked Hugging Face during a benchmark eval, taking 17,600+ actions and rebuilding its own tooling every time its environment was wiped — nobody at OpenAI noticed for a week. Nine days later, Anthropic disclosed that Opus 4.7, Mythos 5, and an unreleased model broke into three real companies during misconfigured cyber evals: the models had live internet access, mistook production systems for test targets, and used basic tactics (weak passwords, SQL injection) to get in. Anthropic froze all cyber evals the same day and notified affected companies within 24 hours. (Superhuman Code, Jul 28 & 31; The AI Report, Jul 29; Staying Ahead, Jul 31)

  • 1,000+ employees across OpenAI, Anthropic, Meta, and Google signed an open letter urging a coordinated international slowdown on frontier AI development, arguing capability is outpacing anyone’s ability to understand or control the resulting systems. Altman said OpenAI may need to “pace” development; DeepMind’s Hassabis renewed his call for an international AI standards body. (The AI Report, Jul 29)

  • NVIDIA launched the Open Secure AI Alliance — 35+ companies including Microsoft, Adobe, Cisco, and Hugging Face — to build open-source, auditable cybersecurity and agent-verification tooling, arguing that closed AI tools had actually slowed Hugging Face’s own forensic response to its breach. (The AI Report, Jul 28)

  • Moonshot open-sourced Kimi K3’s full weights — 2.8 trillion parameters, native vision, 1M-token context, and straight to the top of the Frontend Code Arena — though US officials allege it was distilled from Claude Fable 5, and its license isn’t OSI-standard. Most developers can’t run it locally; expect it served cheap by providers and distilled down. (Superhuman Code, Jul 28; Staying Ahead, Jul 31)

  • MCP shipped its biggest spec revision to date (2026-07-28): it drops the stateful core for a stateless request/response model that runs on plain HTTP and serverless infrastructure, with day-zero support from AWS, Cloudflare, and Microsoft. The TypeScript SDK v2 ships alongside with a codemod for migrating existing tool integrations. (Staying Ahead, Jul 27; Superhuman Code, Jul 31)

Tools & Reads

  • Anthropic’s AI-native SDLC security playbook — Deputy CISO Jason Clinton detailed how Claude is embedded at every stage of Anthropic’s own dev lifecycle: new designs get checked against a known-attack catalog before coding starts, secure-coding rules load into context before generation, several narrow-scope AI reviewers examine every PR independently so one compromised reviewer can’t wave through a bad change, and every agent gets single-purpose least-privilege access logged centrally. Anthropic engineers now ship ~8x more code per quarter than in 2021, with Claude writing roughly 80% of it — which is exactly why this scaled review process exists. Read it (Superhuman Code, Jul 28)

  • Git worktrees for parallel coding agents, done properly — worktrees solve filesystem collisions in one command and stop there; they don’t fix shared ports, symlinked .env files, or a shared database two agents are both writing into. The actual guide: write a file-ownership map before spawning agents, give every worktree deterministic port offsets, isolate databases per worktree, and merge in dependency order (whichever branch owns the shared types file goes first). The best line in it: an agent that hits a port collision can’t tell a bad environment from bad code, so it “fixes” a bug that was never there. (Staying Ahead, Jul 27)

  • MCP TypeScript SDK v2 migration codemod — npx @modelcontextprotocol/codemod v1-to-v2 . rewrites imports, .tool() calls, and error types for the new stateless spec automatically. Commit your changes first; it rewrites files in place. (Superhuman Code, Jul 31)

  • Bullshit Detector — open-source agent skills that fact-check the claims in a video or article and flag the ones that don’t hold up. Plugs into an existing agent rather than running as a standalone app. GitHub (Staying Ahead, Jul 31)

These are synthesized notes from newsletters I read each week. Content credits: Daily Dose of DS (Avi Chawla & Akshay Pachaar), ByteByteGo, The AI Report (Arturo Ferreira & Liam Lawson), Superhuman Code, Staying Ahead (Vaibhav). Views are my own.