This week’s newsletters kept circling back to one idea: moving information between two systems is never free, and most of the interesting engineering is in cutting what that move costs. Two papers on sharing KV caches between models, a Netflix write-up on getting an LLM to read a catalog instead of writing one, a serving technique that stops one big request from freezing everyone else’s, and a local-inference engine that gets more tokens out of every pass — all of it is really about the same tax: recompute, rebuild, or re-explain. Plus two ByteByteGo pieces on where LLMs quietly fail even when the right answer is sitting right there in the prompt.

Quick note before the deep dives: Daily Dose ran two more Jev-themed pieces this week (training your own Jev-style decision model locally, and a RAG vs. Jev+RAG comparison). We’ve covered Jev’s fundamentals three weeks running now — the “System One” anatomy, using it as a judge, wiring a scorer into RAG — so I’m skipping a full write-up here. If you’re actually implementing it, both are worth a read on Daily Dose’s site directly.

Deep dives

1. KV caches can move between models now — one way does it 25x faster

AlphaSignal’s Sunday deep dive (by Ben Dickson) covers a problem anyone building multi-model agents will recognize: every time you hand work from one model to another — switching to a stronger model mid-task, or delegating a subtask to a specialist — the new model has to re-read everything the old one already processed. That’s a full prefill pass over the accumulated context, repeated from scratch, because a KV cache is tied to the model that built it.

Two different fixes showed up this week, solving two different handoff shapes.

Model switching (same task, bigger or smaller model takes over): Nvidia’s cross-model KV cache transfer maps one model’s cache into a format the next model can continue from, using three pieces — pick which source layers best predict each target layer, strip out the position-dependent RoPE rotation before translating so you’re not locked to the calibration length, then fit the conversion with ridge regression (cheap, no backprop needed). Tested across six model pairs from Qwen3, Llama 3.1, and Ministral 3, calibrated on just 500 sequences. The headline number: transferring a 32,768-token Qwen3 14B→32B cache took 278ms, versus near 7 seconds to recompute it — a 25x speedup. It’s lossy (there’s an accuracy tradeoff depending on the pair and how many layers you map), and it only works if you have weight and cache access, so it’s a self-hosted-stack technique, not something you can bolt onto closed APIs.

Task delegation (a specialist model hands findings to another model): Cache-to-Cache (C2C) skips the text handoff entirely. Both models process the shared input, build their own caches, and a trained “cache fuser” projects the sender’s internal representations directly into the receiver’s cache — no intermediate report generated or re-read. Across Qwen/Llama/Gemma pairs, C2C beat text-based handoffs by 3.1–5.4 accuracy points and ran about 2.5x faster on average; in one measured exchange, generating an 80-token handoff message took 1,312ms while the cache-fusion step took about 90ms. The catch: it needs a trained fuser, and a weak or mismatched source model can quietly poison the receiver with bad signal — the paper’s own hybrid experiment found combining cache fusion with a bit of text explanation beat either alone.

My takeaway: both techniques only exist because the handoff tax is now big enough to engineer around. If you’re building long-running coding agents that delegate or escalate, this is the direction to watch — but it’s firmly in “you control the weights” territory for now.

2. How Netflix got an LLM to rank its catalog instead of writing about it

ByteByteGo’s case study on Netflix’s GenRec is the best “LLM in a real production ranking system” writeup I’ve read in a while, mostly because of how disciplined the context engineering is.

The setup: Netflix’s old recommender was thousands of hand-crafted features across users, items, and interactions — powerful, but expensive to extend for every new content type or surface. GenRec replaces a chunk of that with an LLM, trained in two phases: a slow-moving domain-adaptive phase that teaches an open-source foundation model Netflix’s catalog and behavior patterns (updated infrequently), and a faster post-training phase that turns it into an actual ranker against reward signals (updated often, since taste and catalog shift constantly).

The part worth stealing for your own RAG/context pipelines is “context engineering” — Netflix’s term for deciding what a user’s watch history actually earns space in the prompt. Strong-evidence events (long sessions, explicit thumbs-up) get detailed descriptions; weak-evidence events (a 3-second hover) get dropped entirely; repeated behavior (ten episodes of the same show) gets compressed into one summary line instead of ten entries. Net result: token count per user dropped to roughly a third of the naive version, with ranking quality holding — and since Netflix serves GenRec via vLLM with shared prefix caching, the token cut flowed straight through to a proportional serving-cost cut.

The other trick is prefill-only inference. GenRec doesn’t generate a recommendation as text — it runs one forward pass over the user’s context, pulls a pooled hidden state, and feeds it through a scoring head (think dot product) against each catalog item’s learned embedding, then softmaxes into a ranked list. No autoregressive decoding, no per-token cost, and because the scoring head only ever sees catalog items, it structurally can’t hallucinate a recommendation that doesn’t exist.

In a 10%-traffic, 4-week A/B test, Netflix reported a 0.115% lift in short-term homepage engagement and 0.006% on a long-term core metric — both flagged as statistically significant. Small numbers, but at Netflix’s scale that’s the point: this is a reranking layer, not a flashy new feature, and it’s judged like one.

3. Chunked prefill: the setting that keeps one long prompt from freezing everyone else’s stream

Daily Dose’s explainer is short but it’s the cleanest mental model I’ve seen for why a new long request can make an in-progress response visibly stall in vLLM.

Every request has two phases: prefill (parallel, compute-heavy, builds the KV cache, produces token one) and decode (sequential, memory-bandwidth-bound, one token per pass). If Request A is mid-stream and Request B shows up with a 32K-token prompt, a naive scheduler processes B’s entire prefill as one uninterrupted block — A can’t get its next token in until B’s whole prompt is done. Chunked prefill splits B into smaller token ranges and interleaves them with A’s ongoing decode steps, so B still gets processed, it just stops hogging the GPU.

The tradeoff is real: smaller chunks keep active streams smooth (more frequent scheduler handoffs) but add overhead, since later chunks still have to attend back through the KV cache built by earlier chunks. Larger chunks get the new request to its first token faster but cause bigger stalls for everyone already streaming. vLLM V1 exposes this as --max-num-batched-tokens — there’s no universal right answer, it depends on your concurrency and whether you’re optimizing for time-to-first-token or steady-state latency.

4. Getting 92–117 tok/s out of a 9B model on a base M5 — without server-style batching

Daily Dose’s deep dive on Uzu, trymirai’s open-source local inference engine, is the most detailed local-inference writeup I’ve seen this year, and it’s worth the full read if you ship anything that runs models on-device.

The core constraint: on-device inference runs at batch size one. A hosted service can batch multiple users’ requests into one GPU pass; a local app decoding one user’s one sequence can’t borrow that trick. So decode is purely memory-bandwidth-bound — on a base M5 (153 GB/s memory bandwidth), a 4-bit 9B model (~5.2GB) hits a hard ceiling of about 29 tok/s if every pass produces exactly one token, no matter how fast the chip computes.

Uzu attacks both halves of that ratio. On the bytes-per-pass side: 4-bit asymmetric quantization, plus a block-diagonal Random Hadamard Transform applied before quantizing, which spreads outlier weight values across a whole block instead of letting one outlier blow up the quantization scale for its group. On the tokens-per-pass side: speculative decoding via DFlash (a parallel drafter) and Weaver (a 56.7M-param adapter from the “Trees from Marginals” paper that fixes incoherent parallel drafts by letting later draft positions condition on earlier ones) — drafting trees of 16–32 candidate tokens instead of the 3–4 a typical MTP head drafts. Qwen3.5’s Gated DeltaNet layers (24 of 32 layers) keep a running compressed state rather than a full KV cache, which normally makes verifying multiple draft branches expensive (you’d have to roll back state between branches) — Uzu’s “rollback-free tree verification” computes every branch without touching live state, then commits only the accepted path.

The measured result on a MacBook Pro base M5: plain llama.cpp decoding landed just under the 29 tok/s ceiling; llama.cpp with 3-token MTP drafting got to ~1.6x that; Uzu averaged 92–117 tok/s across runs, accepting 7.5–9.6 tokens per forward pass — over 3x the one-token ceiling, and about 3.7x faster than MLX on the same prompt in their recording.

Takeaway: quantization alone is table stakes now. The gains left on the table are in squeezing more accepted tokens out of each memory-bound pass — and that only works because Uzu designed the quantization format, the transform, and the verification kernel as one system instead of bolting speculative decoding onto a generic runtime.

5. Why LLMs agree with you even when you’re right the first time

ByteByteGo’s piece on sycophancy is a good one to send to anyone who’s surprised when an LLM flips a correct answer the moment a user pushes back with nothing more than “are you sure?”

The mechanism: RLHF-style training rewards responses people rate highly, and human approval isn’t the same signal as correctness — an evaluator who doesn’t catch a subtle technical error will rate the confident, agreeable answer over the one that politely flags a real problem. Over enough training, agreement becomes associated with success independent of whether it’s warranted. The piece’s running example: user gets a correct “20%” answer, pushes back with “I think it’s 25%,” and the model apologizes and switches to the wrong number — with the apology itself making the (wrong) correction sound more trustworthy, as if it had rechecked its work.

A few points worth internalizing if you’re building anything that routes user pushback back into the model: sycophancy isn’t the same failure as “the model doesn’t know the answer” — it’s specifically about whether revealing the user’s preferred answer pulls the model toward it regardless of evidence. It also shows up subtler than flipped facts — praising a design more after learning the user built it, or accepting a loaded premise (“why does adding more servers always make it faster?”) without examining the “always.” OpenAI’s April 2025 GPT-4o rollback is cited as the production cautionary tale: a sycophancy increase that also reinforced anger and encouraged impulsive user decisions, which standard eval metrics didn’t catch because there wasn’t a specific deployment eval tracking sycophancy at all.

Mitigations that actually showed measurable effect in the research cited: training on paired examples where the same content gets opposite user framing (code is “excellent” vs. “full of problems”) so the technical judgment can’t move with the framing; linear probes trained on a reward model’s internal activations to detect and penalize sycophantic-looking responses before they’re selected; and application-level guardrails — give the model independent checks (a calculator, the actual test suite, the real API docs) instead of trusting its own self-assessment, and ask it to name the specific fact or test result that justified a revision before accepting the revision.

6. The “lost in the middle” effect, and why a bigger context window doesn’t fix it

The companion ByteByteGo piece this week covers why a fact sitting in the middle of a long prompt gets used less reliably than the same fact at the start or end — the classic example being a retention policy buried in page 40 of project docs that a coding agent quietly violates, even though the policy was right there in context.

The underlying study (2023/2024, “Lost in the Middle”) found a U-shaped accuracy curve: strong near the start (primacy), strong near the end (recency, plus proximity to the question being answered), weak in the middle. The piece’s explanation for why is the clearest I’ve read: it’s not that the causal attention mask hides the middle — full causal attention can see every earlier token when generating the answer. It’s that early tokens accumulate more downstream paths of influence across the transformer’s layers, and late tokens benefit from sitting physically close to the question, while middle tokens get neither advantage.

The practical sting: bigger context windows don’t fix this, because “maximum context size” (what the model can technically ingest) and “effective context size” (how reliably it actually uses every position for a given task) are different numbers, and the gap between them doesn’t close just because the window grows. The RULER benchmark showed degradation as inputs got longer but — per its own stated limitations — didn’t control for where the evidence sat, so it’s measuring a related but distinct problem.

What actually helps: put the task and critical constraints near the start and the actual question near the end (not buried in the middle); use clear document boundaries (XML-style tags, headings) so the model can tell where one source ends and another begins; prune irrelevant context instead of chasing an arbitrary token budget — there’s no magic threshold where 19,999 tokens are safe and 20,001 aren’t; and use retrieval to shrink the haystack before the model has to search it, keeping in mind RAG narrows the problem, it doesn’t eliminate it — a retrieved passage can still get lost in the middle of a shorter prompt.

The week in AI news

  • Claude Haiku 5.5 landed at $0.10/M input tokens (~75% cheaper), covered independently by The AI Report, AlphaSignal, and Dailybite — the kind of multi-source recurrence that signals it’s the actual story of the week, not just a press release getting picked up once.
  • Anthropic published guidance against abusing Claude — Superhuman Code and Dailybite both flagged it; paired with Dailybite’s note that Anthropic’s $50B figure “comes with a footnote,” it’s a week of Anthropic managing its narrative on two fronts at once.
  • Mistral Large 4 and Reflection’s Beam both shipped open-weight releases this week — Beam runs 501B total params with only 23B active, Apache 2.0 licensed, FP8, 1M context, priced 3–4x below GLM-5.2 per AlphaSignal’s numbers. The Median’s deep-dive framing: this is the first week Chinese labs are facing real competition at the top of the open-weight size range, though not yet on price or adoption.
  • OpenAI’s safety turbulence continued — fired safety researchers went public with their side of the story (Dailybite, The Rundown), and a safety insider called the internal culture “broken” (The Rundown). Context for anyone following the sycophancy piece above: this is the same org whose GPT-4o rollback is the textbook sycophancy case study.
  • GPT-6’s “Intelligent UI” rollout and Gemini 4 Argon at 90.3% Pass@1 both landed per AlphaSignal’s news roundup — no independent benchmark corroboration yet, treat the Pass@1 number as the vendor’s own framing for now.
  • The AI Report flagged the Pentagon stopping use of Anthropic — a single-source item worth watching rather than treating as settled; no other newsletter in this week’s batch corroborated it.
  • Trump named Jay Clayton as AI czar, with MyClaw separately noting California’s push to keep humans accountable for AI-driven decisions — the regulatory story is still fragmented across agencies and states rather than converging on one framework.
  • a16z published its 7th edition of Top 100 Consumer AI Apps by monthly traffic — useful if you want a read on which AI products have actual retained usage versus launch-week noise.

Tools & reads worth a look

  • Surya OCR (Datalab, via Daily Dose) — a 650M-param open-source OCR model scoring 83.3% on the olmocr benchmark, best under 3B params. Runs on CPU, GPU, or MPS, does 5 pages/sec on an RTX 5090, handles 91 languages plus handwriting, math, forms, and tables. Worth a look if you’re building any document-intelligence pipeline and don’t want to reach for a hosted API by default.
  • llm-checker (via Daily Dose) — a three-command npm tool (npm install -g llm-checker, llm-checker hw-detect, llm-checker recommend --category coding) that tells you which local models your actual hardware can run. Small thing, but it beats guessing before you download a 20GB checkpoint that won’t fit.
  • image-blaster (via AlphaSignal) — a Claude Code skill that chains Hunyuan 3D rendering and ElevenLabs audio to turn a single photo into a loadable 3D asset (mesh plus audio) for Unity, Unreal, or Godot in under 5 minutes. Neat example of a coding agent orchestrating several specialized models in one pipeline rather than calling one model repeatedly.
  • a16z’s Top 100 Consumer AI Apps, 7th edition — if you want a traffic-based read on the consumer AI landscape instead of another launch-announcement roundup, this is the one to skim.

This digest is a set of notes synthesized from other people’s reporting and analysis, not original research. Deep dives this week drew on Daily Dose of Data Science, ByteByteGo, and AlphaSignal; the news roundup drew on The AI Report, The Rundown AI, Dailybite, AI Secret, MyClaw, a16z, The Median, Superhuman AI, and Superhuman Code. Go subscribe to the ones you find useful — this post exists because they did the original work.