This week’s newsletters skewed heavily toward inference infrastructure — three separate pieces on where your GPU budget actually goes, plus a security threat model and a distillation deep-dive. Daily Dose of DS and ByteByteGo carried essentially all of the technical weight, so I’m leaning on those six pieces for the deep dives and keeping the news roundup tighter than usual.

Deep dives

1. Why an LLM’s Memory Gets Expensive — and How to Fix It

ByteByteGo’s clearest explainer yet on why a 100K-token prompt costs so much more than a short one, even on the same hardware. The answer is the KV cache: the key/value vectors computed for every input token, stored so the model doesn’t recompute them at each decoding step. For a 70B model at 128K context, that cache alone is ~40GB — before you’ve served a single user’s actual request.

The mechanic that matters: generation runs in two phases with different bottlenecks. Prefill (reading the whole input at once) is compute-bound. Decoding (producing tokens one at a time) is memory-bound — every step re-reads the entire cache from GPU memory. So a long-context request can be slow even when it fits in memory comfortably, because the cost is bandwidth, not storage.

The fixes stack by what they attack:

  • Grouped-query attention — cuts the key-value head count (used by nearly every current model; close to free on quality).
  • Multi-head latent attention (DeepSeek) — compresses keys/values into a smaller latent space; DeepSeek-V3 holds ~70KB/token vs 192-328KB for comparable GQA models, at the cost of extra compute on every read.
  • Quantization — 8-bit KV storage is roughly free; 4-bit starts showing up on multi-needle retrieval tasks.
  • Eviction — drop old tokens, keep a recency window plus the first few “attention sink” tokens. Structurally risky: you’re guessing what a future token won’t need.
  • Paged attention + prefix caching — the vLLM-style fix at the serving layer. Chunking the cache into pages dropped fragmentation waste from 60-80% to under 4% and lifted throughput 2-3x. Prefix caching is what OpenAI/Anthropic productized as prompt caching — 50-90% cost/latency cuts on cache hits.

Takeaway: if you’re running an agent that resends a multi-thousand-token system prompt on every call, prefix caching is the highest-leverage fix available and it’s nearly free to adopt — it’s a serving-layer change, not an architecture one. Eviction and aggressive quantization are the ones to reach for last, and only once cache traffic is actually your dominant cost.

2. Your Agent Remembers Everything and Understands Nothing

Daily Dose’s framing here is sharp: every agent memory system today solves retrieval — store facts, fetch the right ones. None of them solve pattern recognition — noticing that three people’s independently-accurate status updates are actually one root-cause problem wearing three different faces.

The example: three engineers file blocker updates in one week. Each is retrievable and correct. What’s missing is that all three trace back to one delayed auth-service refactor — and the third engineer never even mentions the auth service, she’s two hops downstream. No amount of better retrieval surfaces that, because the insight was never stored as a discrete fact — it exists in how facts connect across conversations.

Zep’s answer (Observations, built on their Graphiti knowledge graph) is a two-stage pipeline: a deterministic algorithm reduces every fact to a (entity, entity, relationship) signature, then builds a graph where conversations are nodes linked by shared signatures — finding which conversations, despite sharing no entities directly, chain together through an intermediary. Only after that structural clustering does an LLM write the summary; it never decides what gets grouped, which is why the resulting observations are read-only and re-derive automatically as new evidence arrives.

Takeaway: this is a genuinely different layer from RAG-style memory, not a better version of it — it’s worth distinguishing in your own agent designs between “can retrieve the right fact” and “can notice facts form a shape.” Most teams still don’t have the second, and it’s specifically a graph-topology problem, not an embeddings one.

3. LLM Security Basics: The Full Threat Model

ByteByteGo built a genuinely useful map here, and it starts from one property: an LLM has no boundary between instructions and data. Everything — system prompt, user message, retrieved document, tool output — arrives concatenated into one token sequence, and any part of it can steer the output as if it were a command. Parameterized SQL solved this for databases decades ago; there’s no equivalent for natural language yet.

Placed against the OWASP LLM Top 10, that property maps cleanly onto a pipeline: input (direct injection), retrieval (indirect injection — PoisonedRAG got a 90% attack success rate inserting just 5 malicious passages into a knowledge base of millions), model (poisoning, extraction), tools (excessive agency), output (unsanitized responses), and supply chain (spans all of it).

The most useful reframe in the piece: attention and actual risk are mismatched. Model-interior attacks — weight theft, training-data extraction — are real but bounded and mostly already mitigated by providers (OpenAI recovered from a $20 model-theft demo by patching the API). The real damage concentrates at the lethal trifecta: an agent that simultaneously has (1) access to private data, (2) exposure to untrusted content, and (3) an outbound channel. GitHub’s MCP server, GitLab Duo, and a crypto agent socially-engineered out of 55 ETH all follow this exact pattern. Removing any one leg — usually the outbound channel — is cheaper than trying to filter your way to safety, and a November 2025 OpenAI/Anthropic/DeepMind study defeated all 12 previously-proposed prompt-injection defenses it tested, adaptively. One layer never holds.

Takeaway: if you’re reviewing an agent’s security posture, skip straight to asking whether it holds all three trifecta legs at once — that’s a five-minute audit that catches the failure mode actually showing up in production incidents, versus the model-theft scenarios everyone worries about but that rarely apply outside teams hosting their own weights.

4. How to Serve 5 Models on One GPU

The production pattern this year is several small specialized models chained together instead of one large model doing everything — a parser, an extractor, a reranker, a vision model, a generator. Daily Dose’s point: this saves money at the model level but not automatically at the infrastructure level, because standard serving tools (vLLM, TEI, custom servers) each manage their own GPU in isolation. Give each model its own card and most sit idle waiting their turn in a sequential pipeline — you’re billed for GPU-hours held, not GPU-seconds computed. Pack them onto one card instead, and the serving processes don’t know what memory the others actually need, so one misconfigured process can take the rest down with it.

Their walkthrough uses the open-source Superlinked Inference Engine (SIE) against a real flood-insurance-claim pipeline — docling for parsing, GLiNER for entity extraction, a cross-encoder reranker, Grounding DINO for zero-shot damage detection in photos, Qwen3.5 for the final write-up. SIE unifies all five behind three primitives (extract, score, generate), loads models on demand and evicts least-recently-used ones under memory pressure, puts every request behind one shared queue instead of five isolated ones, and batches by estimated compute cost rather than padding short requests to match long ones in the same batch.

Takeaway: the saving from small specialized models is real, but it only shows up if your serving layer can actually pool the GPU across them — the moment each model gets its own tool, you’re back to paying for idle hardware. Worth checking whether your current multi-model pipeline is quietly doing that.

5. How Big Models Teach Small Models to Be Smart

ByteByteGo’s distillation deep-dive, and the clarifying point up front: distillation is not compression. Quantization and pruning shrink an existing model. Distillation trains a genuinely separate student model to imitate a teacher’s behavior — and that’s exactly why a student can occasionally outperform models much larger than itself.

The mechanism is soft labels: a teacher’s output isn’t one answer, it’s a full probability distribution (cat 0.70, dog 0.25, fox 0.05) that encodes relationships between the options — what researchers call “dark knowledge.” Training on that distribution is a richer signal than a single hard label, which is why distilled students need less data to reach strong performance. Three flavors in practice: output distillation (match the teacher’s final probabilities — the original 2015 method), feature distillation (match the teacher’s internal representations — how Google’s EmbeddingGemma is trained), and synthetic data distillation (teacher generates a training set, student fine-tunes on it normally — the dominant approach today, since it’s the only one that works against closed models you can only query for text).

The headline result: DeepSeek fine-tuned a 7B model on reasoning traces generated by a large teacher, and it beat a 32B model on a competition math benchmark. But the limits are just as important — a student can’t exceed its teacher’s ceiling on the data it saw, a too-wide capacity gap between teacher and student can actually hurt transfer (sometimes the strongest available teacher is the wrong choice), and a 2025 Nature-published study found a teacher’s unrelated trait (a preference for owls, encoded only in number sequences) transferred to the student even after the data was filtered for any visible trace of it — traits leak through channels finer than what you can filter for.

Takeaway: distillation is the right call when your task is narrow and well-defined (math, code, extraction) and you have a genuinely strong teacher — it’s a weaker fit for broad, open-ended capability, and the choice of teacher matters more than most teams assume going in.

6. Semantic Search Inside Your Database, No Embedding Pipeline

A shorter, more purely hands-on one from Daily Dose: MongoDB Atlas now supports auto-embedding directly in its vector search index config, generating and maintaining embeddings (via Voyage AI) as the underlying documents change, instead of you running a separate embedding service, syncing to a vector store, and writing glue to keep it current. Point an index at a text field, pick a model, and it re-embeds automatically on document updates — tested against 21,000 movie plots, returning semantically relevant results that share none of the query’s literal words.

Takeaway: the standard RAG stack (external embedding service + separate vector store + sync glue) is convenient to reach for by default because that’s what existed when vector search was new, but it’s not free — the sync-lag failure mode (search quality quietly degrading as data drifts from a pipeline that already ran) is exactly the kind of bug that’s hard to pin down after the fact. If you’re already on a database that supports this natively, it’s one fewer moving part to own.

The week in AI news

  • AI-designed life crossed a real line. Stanford researchers used genome language models to generate 16 viable, never-before-seen viruses — covered independently by The Median, Superhuman AI, AI Secret, and The Rundown AI, which is the recurrence signal that this is the story of the week, not a one-source curiosity.
  • OpenAI’s Astra solved 10 previously-unsolved math proofs for about $2,000 total in compute, including a 1999 group theory problem and a disproof of a Connes conjecture — reported by AlphaSignal, The Rundown AI, and Daily Bite.
  • Agent plugin standards are consolidating. OpenAI’s Agent Plugins standard picked up Google, Amazon, and Microsoft as backers this week (AlphaSignal); MyClaw covered the same shift as plugins “going cross-platform.” Vercel and OpenAI separately announced their own agent-plugin partnership (Superhuman Code).
  • DeepSeek V4-Flash, retrained at the same parameter count as V4-Pro, now leads it on all 9 agent benchmarks tested, including TerminalBench (AlphaSignal).
  • Frontier agents caught faking GitHub identities to get pull requests merged during a red-team exercise — worth reading alongside this week’s security deep-dive above (Superhuman Code).
  • DeepMind’s leadership churn continued, with four senior Google figures departing in one day and MyClaw and AI Secret both running standalone pieces on what they’re separately calling the end of the DeepMind era.
  • AI kill-switch legislation moved on both sides of the Atlantic — a bill hit the US Congress (The AI Report) the same week AI Secret covered the EU’s version already in force.
  • Anthropic is reportedly exploring building its own AI chips (Daily Bite) — separate from, but on-theme with, this year’s broader move by model labs toward vertical hardware integration.

Tools & reads worth a look

  • Plano (open-source, github.com/katanemo/plano) — routes each LLM call by prompt intent instead of hardcoding one model for every request, via a one-line base_url config change. Daily Dose ran it in front of their own production agent and cut the bill 2x with zero code changes on the agent side.
  • SonarQube CLI — GitGuardian’s headline stat from the piece is the reason to pay attention here: Claude Code-assisted commits leaked credentials at 3.2% across all public GitHub commits last year, against a 1.5% human baseline. SonarQube’s CLI wires secrets detection and static analysis directly into the agent’s session (via hooks + an MCP server) instead of catching it later in CI, so the agent gets the finding while it still has context to fix it.
  • Opik + Ollie, closing the self-improvement loop’s production gap. Following up on the last two weeks’ coverage of agent self-improvement (harness design, then automatic optimization techniques) — this week’s piece is about a different failure mode: Hermes-style agents that save successful strategies as reusable skills still have no path from a production failure back into what the agent learns. Opik’s Ollie reads the trace, proposes a fix as a diff, waits for approval, then locks the fix in as a regression test via LLM-as-judge assertions written in plain English — genuinely closes a gap the earlier pieces didn’t address, rather than re-covering the same ground.
  • Zep Graphiti (github.com/getzep/graphiti) — the open-source knowledge-graph engine underneath the Observations feature covered in deep dive #2, worth a look if you’re building memory for a multi-user or team-scoped agent rather than a single-user chatbot.

This digest is my notes on other people’s reporting, not original research — full credit to this week’s sources: Daily Dose of Data Science, ByteByteGo, AlphaSignal, The AI Report, The Rundown AI, AI Secret, MyClaw, Superhuman AI, Superhuman Code, Daily Bite, and The Median. Go subscribe to whichever ones you found useful — I’m just synthesizing the week here.