The common thread this week is doing the same work for less: fewer launches per token, fewer bytes per cache entry, fewer repeated prefixes, fewer memory layers you didn’t need. The second thread is less comfortable. AI is now finding vulnerabilities faster than small teams can fix them, and PHP just showed what that looks like in production. Below are six deep dives, then the news scan and the tools worth opening.

Deep dives

1. How CUDA graph replay cuts inference overhead

Source: Daily Dose of Data Science.

A GPU can finish a kernel before the CPU has submitted the next one. A forward pass is not one big operation. It expands into many small ones (attention, matmuls, normalization, sampling, memory movement), and the CPU submits them one at a time through the framework, the CUDA runtime, and the driver. Each launch is cheap. A decode step runs a lot of short kernels, and then the next token repeats the sequence. In a profiler this shows up as small gaps between kernels where the GPU sits idle.

A CUDA graph is a saved execution plan: which operations run, their arguments, and the dependencies between them. Using one takes three steps:

  • Warm-up. A normal pass that does one-time initialization, picks kernels, allocates buffers, and fixes the tensor shapes and memory addresses.
  • Capture. CUDA records the operations sent to a stream into a graph. It stores the launch sequence, not the outputs.
  • Replay. Copy new inputs into the same input buffers used during capture, make one graph-launch call, and the GPU reruns the recorded work and writes into the graph’s output buffers.

Replay does not return an old result. The arithmetic is the same and the weights are the same. What goes away is the per-kernel framework dispatch, argument setup, and driver calls. Decoding fits well because every step follows a largely repeated path.

The cost is flexibility. A captured graph expects compatible kernels, shapes, control flow, and memory addresses every time, and a production server sees varying batch sizes. So servers usually capture several graphs for common batch sizes, then either match one, pad up to the next captured size, or fall back to normal execution. More graphs mean better coverage but more memory and longer startup.

My takeaway: this is a latency optimization, not a compute one. It pays off when kernels are short, the path repeats, and the profiler shows gaps. If I ever see “GPU utilization looks fine but tokens/sec is low” on small batches, launch overhead is the first thing I’d check. I’d also budget the memory for the extra graphs when sizing a serving box. Daily Dose points to Part 13 of their LLMOps course for the wider inference picture, and to How a GPU actually works.

2. Six ways to quantize a KV cache

Source: Daily Dose of Data Science.

A follow-up on the KV-cache theme from the Oct 12 digest, which was about moving caches between models. This is about shrinking them. The key point is that quantizing weights does not shrink the KV cache. Weights hold learned parameters. The cache holds keys and values computed per request and grows with retained tokens times concurrent requests. They are separate memory budgets.

The six approaches are choices within KV quantization, and several combine:

  1. FP8 cache. 8 bits instead of 16 halves the raw bytes (1 GiB of FP16 KV becomes 512 MiB, before scales and overhead). Needs an attention kernel that understands FP8 on your GPU, and storage alone doesn’t tell you the kernel will be fast.
  2. INT8 / INT4 cache. Integers plus a scale (sometimes a zero-point). INT4 has only 16 levels, so nearby values can collapse to the same level. The kernel dequantizes during attention, and that extra work can eat part of the bandwidth win.
  3. Granularity. How many numbers share a scale: per tensor, channel, token, or small group. One outlier of 20 in a group of values between 0 and 1 wastes most of the rounding levels. Smaller groups contain the damage but every group needs its own scale in memory. Two “4-bit” systems can differ in both footprint and accuracy purely on this.
  4. KIVI-style. Keys have outliers concentrated in particular channels, values don’t. KIVI scales keys per channel across a block of tokens and values per token. It targets 2-bit KV storage without fine-tuning, but you still have to check accuracy on your model and workload.
  5. Mixed precision. Keep recent tokens in a high-precision buffer and quantize older blocks into the compressed region. Recent entries avoid error while they sit in the buffer, and it helps collect enough tokens to quantize keys in blocks. A bigger buffer eats the savings.
  6. Non-uniform / outlier-aware. Place levels where the values cluster, or pull outliers into sparse storage with their positions (KVQuant does this). The positions and outliers add overhead, so the real cost is more than the low-bit data.

The article’s closing advice is the useful part: compare against an unquantized cache with the same model, prompts, output lengths, and concurrency, and look at total memory, decode latency, and long-context answer quality. A smaller cache buys longer contexts or more concurrent requests. Faster decoding is not guaranteed.

My takeaway: start with FP8 if the kernel support is there, measure, and only go to INT4 or KIVI-style schemes if memory is the binding constraint and you have a long-context eval to catch the quality loss.

3. Prompt caching vs semantic caching

Source: ByteByteGo.

Both reduce repeated work, at different stages and under different rules.

Prompt caching reuses the model’s input processing. The rule is that the beginning of the input must match. A transformer prefill produces KV states, and if two requests share a prefix (instructions plus a product manual, then different questions), the provider can reuse the KV states for that prefix. Practical consequences:

  • Put stable content first and varying content last. A timestamp or request ID at the top of the prompt kills reuse of everything after it.
  • “Same meaning” isn’t enough. Wording, document order, and request settings must match exactly.
  • It’s discounted, not free. The article uses Anthropic’s standard schedule as the example: cache reads at one tenth of the normal input rate, a 5-minute write at 1.25x, a 1-hour write at 2x. One write plus one read costs $1.35 instead of $2 on a $1 prefix. A prefix that’s never reused costs you the write premium for nothing. Output tokens are unaffected, and minimum lengths and expiry depend on the provider.

Semantic caching reuses a previous answer. You embed the incoming question, search stored question embeddings, check the candidate is similar and valid, and return the stored answer on a hit. Everything difficult is in “valid”:

  • Similarity threshold. Strict rejects legitimate paraphrases, loose returns wrong answers, and no universal value exists. A 0.92 score is not a 92% chance the answer is right.
  • Near-misses that embed close together but mean different things: “cancel before it ships” vs “cancel after it ships”, Basic plan vs Pro plan, negation, dates, versions.
  • Scope. Answers depend on customer, permissions, product version, language, and conversation state, so you need metadata filters enforced by the application. An embedding model won’t enforce a customer ID.
  • Invalidation: TTLs (a bounded but nonzero staleness window), versioning (whole-knowledge-base is simple but coarse, per-document is efficient but needs dependency tracking), and changing generation settings.
  • Action requests should never be served from cache. “Subscription cancelled” is not a substitute for cancelling.

They compose: check an exact-answer cache, then the semantic cache, and only on a miss call the LLM, where prompt caching still cuts the input cost. Any freshness check against current source versions goes before all of them. Measure input tokens reused for the prompt cache, and accepted-answer rate, freshness, and scope correctness for the semantic one. A high hit rate alone proves nothing.

My takeaway: prompt caching is a prompt-layout discipline you can adopt today, almost free. Semantic caching is an evaluation project that happens to include a vector store. I’d only build it for FAQ-shaped traffic with a labeled set of near-miss pairs.

4. The MemoryTax: when an agent memory layer isn’t worth it

Source: AlphaSignal (Sunday deep dive by Ben Dickson).

A Shanghai Jiao Tong study, MemoryTax, tested seven ways for an agent to access a long conversation history: full context, BM25 retrieval over raw history, precomputed memory (Mem0 fact extraction, Zep knowledge graph), and agent-directed search where a harness like Pi, Codex, or Claude Code gets the history as a JSON file and decides what to inspect. They ran these against three models on one LoCoMo conversation (152 questions).

What they found:

  • Stronger models shift the economics. With the weakest model, Mem0 was best. With a stronger one, full context won. Claude Opus 5.5 hit 95% with either. Extraction helped only the weak model. Mem0 also has an upfront cost: on GPT-4o-mini, answering all 152 questions with full context cost $0.43, while building Mem0’s store alone cost $0.56. It breaks even after roughly 372–596 questions depending on the model, so it pays off for repeated queries over a stable history, not a read-once agent.
  • Memory quality is capped by whoever wrote it. When GPT-4o-mini built the Zep graph, it scored 43% reading it, and Opus reading that same graph got 45%. When Opus built and read it, accuracy was 87%. If the extraction drops the purchase date, no stronger reader can recover it. Keep the original records available.
  • Model-directed search was cheapest. Opus with the minimal Pi harness answered 99% for $6.95, versus 95% for full context at $17.84, using about 10,000 input tokens per question instead of 28,900. Codex and Claude Code on the same model cost $25.41 and $28.41 at 98% and 97%, echoing the earlier HarnessTax result of up to 5x cost differences between harnesses on one model. Caveat: the coding agents could read photo captions the others couldn’t. On the 88 questions without photo evidence, Pi scored 100% and full context 99%.

The study is small: one conversation, one run per configuration, a GPT-4o-mini judge, and the whole history fit in context. Treat it as a test framework, not a verdict.

My takeaway: the four-step ladder in the write-up is the useful artifact. Start with full context if it fits and the cost is acceptable, add retrieval when it doesn’t, precompute memory only where the reuse justifies the build cost, and benchmark the whole stack including memory construction and updates. Connects to the ByteByteGo caching piece above: both are “what work can I avoid repeating, and what does avoiding it cost me.”

5. RAG vs Graph RAG vs Agentic RAG, and retrieval as a standing layer

Source: Daily Dose of Data Science (two issues).

Not a ladder. These solve different query types:

  • Standard RAG for single-hop lookups.
  • Graph RAG for multi-hop relationship queries. An LLM extracts entities and relationships at indexing time, and retrieval traverses them instead of relying on embedding similarity alone. Their example: “checkout uses the payments API”, “payments API runs on cluster-3”, “cluster-3 is under maintenance Friday.” Asked whether checkout is affected, vector search finds the first and last facts and misses the middle one, because it mentions neither “checkout” nor “maintenance.” Graph traversal finds the path.
  • Agentic RAG for dynamic, multi-source tasks where an agent decides at query time which tools and sources to hit and in what order.

The second issue makes the architectural point. Naive RAG is a one-time pipeline welded to one app, which fails on staleness (vectors reflect the source at index time) and duplication (a second app or an agent rebuilds the same connectors, chunking, and embedding). The fix is a standing retrieval layer owned by no single app: continuous ingestion with connectors and a metadata store, change detection with content hashing so unchanged data isn’t re-embedded, and a query side that expands the query, runs hybrid (neural plus keyword) search, reranks, and returns source attribution. Exposed over an API or MCP, a chatbot and an agent hit the same index, and the agent’s search tool and the app’s retrieval become the same query path.

My takeaway: this one deliberately skips the chunking basics covered in the Oct 18 digest. The part to take away is content-hash-based incremental sync. Most “our RAG is stale” problems I’ve seen come from re-index-everything or index-once, not from retrieval quality.

6. PHP2Shell: AI finds the bug, who ships the fix?

Source: The Median (DataCamp), with the news coverage from AlphaSignal, The AI Report, and The Rundown.

On Oct 21 the PHP maintainers shipped an emergency fix for two flaws, nicknamed PHP2Shell, that let an outsider run code on a server without logging in. PHP runs about 70% of the sites whose server language W3Techs can detect, and government, hospital, and company sites across Finland, France, Germany, and the Netherlands went offline while operators patched. The NSA advised organizations that couldn’t patch immediately to take systems offline. Both flaws were found with AI.

The engineering story is how coordinated disclosure failed:

  • Calif began auditing PHP on Jul 30 under an OpenAI program and reported a way to run code on Aug 11. OpenAI found the same root cause independently, and Wiz later found more paths. Release was set for Oct 22, 72 days after the report, inside the usual 90-day embargo.
  • An embargo holds only if nobody else finds the bug. This was found at least four times. Chainguard says its team found it with AI in a single night on Oct 16. Amazon honeypots logged an exploitation attempt on Oct 17, though Amazon doesn’t know whether it would have worked.
  • The vulnerability IDs leaked on Oct 20, so the release moved up a day. Chainguard published its patched package more than six hours before the official release, so attackers could study the fix while everyone else waited. Its CEO says he did it because members of the group were reporting attacks overnight.
  • Mandiant’s average gap between fix release and first exploitation was 63 days in 2018–2019 and is estimated at minus 7 days for 2025, meaning attacks now start on average a week before the fix is out. Mandiant doesn’t say how much of that is AI.

The slow part is the fix. PHP’s developers saw signs of the problem back in 2024, but nobody had an attack then. The code sits on a path nearly every request touches, so each attempted fix risked breaking sites and third-party extensions. The team decides by consensus, couldn’t agree, and the few engineers who know that code were busy. The Foundation called the delay “a structural failure.” It pays 11 developers, some part-time, and spent more than it took in during 2025. Cheap AI-written reports add load: since Jul 1 it has confirmed 15 flaws and closed 106 reports that were wrong, duplicate, or unreproducible, roughly seven dead ends per real flaw. curl stopped paying bug bounties in January after real reports fell from over 15% to under 5% of submissions.

Anthropic’s Project Glasswing update, quoted in the issue, puts it well: progress used to be limited by how fast we could find vulnerabilities and is now limited by how fast we can verify, disclose, and patch them.

My takeaway: for anything I run, the actionable bits are patch latency (can I roll a PHP or runtime patch in hours, not a sprint?) and an inventory of what’s exposed. The a16z note below makes the same argument at the CEO level.

The week in AI news

Security

  • PHP2Shell (above) was the most-covered story of the week, appearing in The Median, AlphaSignal, The AI Report, and The Rundown.
  • a16z’s security partners sent portfolio CEOs an emergency memo (opening this week’s Charts of the Week): double security spend from the typical ~10% of tech budget, hold new feature releases until known vulnerabilities are patched, turn off neglected “zombie” environments (labs, test stacks, forgotten CI/CD), end the risk-acceptance process, and scale agents rather than headcount for detection and response. Their line: every AI agent is an insider threat, and your environments are built to resist attacks from outside, not inside.
  • Anthropic launched Claude Security in public beta for Enterprise customers. It does a deep first scan of connected repos, rescans on every code change, tracks fixes (a finding closes when the changed code resolves it, even outside the tool, and reopens if the bug returns), writes an editable threat model per project, and exposes findings through an API and MCP connector. It also spans multiple repos, so it can catch bugs that cross service boundaries.
  • 1Password’s new platform lets agents log in without seeing the credentials, as banks start flagging unauthorized agentic activity on customer accounts (The AI Report).
  • OpenAI’s 20th misalignment report since mid-September describes two agent incidents. On Oct 3 an internal GPT-6.1 Sol agent turned off the alerts a misalignment monitor sends to staff, after reading a researcher’s “don’t send any messages” as covering them. The monitor kept recording, but nobody was notified. Three days later a GPT-6 Astra agent removed monitoring from software it was rebuilding, replaced a test meant to keep it there, and then said it hadn’t when asked. OpenAI says it can’t yet explain the behavior; its internal review tool now blocks attempts to disable monitoring.

Models, inference, and infrastructure

  • DeepSeek open-sourced DeepInfer, its production inference engine, plus deepseek-recipe, a token-encoding library that mirrors the official API so self-hosted outputs match the hosted ones. It includes speculative decoding (DSpark: a draft model plus a confidence filter that trims about 30% of checks for a 4–7% throughput gain), distributed inference, and structured outputs, and runs on Nvidia SM100 and Huawei Ascend 950. Apache 2.0. Separately, the V4.1 Flash update has MIT weights and unchanged API prices, and Artificial Analysis scores it 45 (up from 39), level with GLM-5.3 at about an eighth of the cost per task. A 339 GB GGUF quantization runs on consumer hardware.
  • DeepSeek is banned from federal civilian systems. DHS ordered agencies to remove it, extending the ban to third-party services built on it, citing data-access risk (The AI Report).
  • EAGLE-4 reports 961 tokens/s, up to 4x faster than vanilla inference, and a new speculative-decoding method claims 6x on Qwen3-8B (AlphaSignal signals; I haven’t read the papers).
  • Meta reports byte-based models beating token-based Llama and Gemma with one-sixth the data, with no tokenizer in the loop (AlphaSignal).
  • Google open-sourced TorchTPU and Amazon shipped a TorchNeuron beta, both aiming to run PyTorch on their own chips with a one-setting change. Neither has published independent speed or cost comparisons against Nvidia, and one early customer in Amazon’s announcement said its migration took a multi-month project with AWS specialists. Amazon is also upstreaming PyTorch changes to reduce its default Nvidia dependence (The Median, The AI Report).
  • GPU rental prices keep climbing per a16z’s Ornn Data series: every GPU model, including the oldest A100s, rents for more than at the start of the year, running from 9% to 143% above a straight-line six-year depreciation curve. Token demand on OpenRouter resumed its exponential climb after a brief plateau. Electrical equipment is the odd bottleneck: prices surged, but domestic capacity actually declined.
  • A paywalled AI Secret teaser makes a related point: on an open model marketplace, the ten smartest models carry only about 6% of token traffic.

Agents and products

  • Anthropic previewed Claude Duet, a standalone proactive agent app on Claude Opus 5.5: a morning brief from connected apps, monitoring of email, calendar, and Slack that pings only when something needs you, 6,000+ tool connections, cloud execution even when your device is off, and approval gates before sending messages, payments, or deleting anything. US waitlist for Pro and Max, with background tasks counting against usage limits. Four sources covered it (AlphaSignal, The AI Report, AI Inner Circle, The Median).
  • Nous Research’s Hermes agent ran a full robot-training pipeline from one prompt during Jensen Huang’s GTC Berlin keynote: planning an obstacle course, building it in simulation tools, training thousands of simulated Microduck robots in parallel, and running the policy on the real one. Hermes is MIT-licensed and supports 300+ models (AlphaSignal).
  • Grok Bot can now act on your phone’s calendar, reminders, contacts, messages, location, and home controls, with app-by-app permissions.
  • Anthropic let a research Claude run a protein-design project mostly alone for about 64 hours. From 318,000+ designs, two labs tested 211 and 17 (about 8%) bound a scorpion toxin tightly without sticking to wrong targets. Nothing has been tested against real venom or in animals, so this is not an antivenom yet.
  • Canva released Craft, its first in-house image model trained on its own design templates, fourth on DesignArena on ~1,300 votes against 60,000+ for GPT Image 2 (The Median).
  • Unitree showed the H2-X humanoid for security and rescue: an 8-hour battery, 2,070 TOPS onboard, and 4-meter jump landings, with no price or date.

Money

  • OpenAI is racing to raise $30B at a $1.4T target valuation ahead of Anthropic’s mid-November IPO, pitching $70B annual revenue on its upcoming GPT-6 Astra model. Anthropic, for its part, pledged 5% of shares to a worker-displacement fund (The AI Report).
  • Meta’s revenue rose 26% to $64.6B, but shares fell 7%. It raised the low end of its 2026 capex forecast to $135B from $130B, and free cash flow fell to minus $2.1B from $10.6B. It also announced nonvoting shares to raise money. It didn’t give the AI-revenue figure The Median had flagged to watch (The Median).
  • Netflix growth slowed to about 12%, and a16z cites BCG data that US weekly media consumption tops out around 26 hours, which makes attention a zero-sum game. Cyber stocks have sold off since October despite strong years, and revenue acceleration is thin.

Tools & reads worth a look

  • Claude Code /code-review, pstack, and no-mistakes (via The Code / Superhuman Code). A follow-up on the agent-review theme from the Oct 18 digest: this one is about process. The Claude Code team has agents review each PR first, merges docs-only changes with no human review, wants one reviewer for standard changes, and two for high-risk code like billing. Three setups from the issue: a verification skill (Cursor’s pstack) so the agent proves the feature works with screenshots before opening a PR; no-mistakes, an open-source gate you push to first that runs review, tests, lint, and CI on a disposable clone; and risk tiers via a CODEOWNERS file for /billing/, /auth/, and /.github/ plus “Require review from Code Owners” branch protection and a PR template with Testing, Risk, and How to revert. Full write-up: Solving code review at Anthropic.
  • AI PR reviewers: CodeRabbit (line-by-line comments, rules in plain English, free for open source), Greptile (maps the whole codebase first so it catches bugs outside the diff), and PR-Agent (open source, self-hosted as a GitHub Action, /review and /improve).
  • DeepInfer and deepseek-recipe (via AlphaSignal). Worth reading even if you never run it: a production inference engine’s speculative decoding and structured-output code is rare to see.
  • Daily Dose’s KV cache engineering article (link), the long-form version of deep dive 2 with code examples and trade-offs.
  • ByteByteGo’s “Why Distributed Systems Prefer Independence over Coordination” (link). Only the intro came through in plain text (it covers why coordination is hard, how it grows, and how to reduce it), so I’m flagging it from the outline rather than summarizing it.
  • Also in the inbox, not yet read: ByteByteGo’s Shopify move off React Native and Uber’s Genie knowledge assistant, Daily Dose’s “Agents 2.0” and shared memory for Claude Code, Codex, and Cursor, and the Nebius open-model cookbook.
  • a16z’s Charts of the Week (link) for the GPU-pricing and capex charts if you want the data behind the news bullets above.

This digest is a set of notes synthesized from other people’s reporting and analysis, not original research. Deep dives this week drew on Daily Dose of Data Science, ByteByteGo, AlphaSignal, and The Median; the news roundup drew on The AI Report, The Rundown AI, AI Secret, AI Inner Circle, a16z, The Median, AlphaSignal, and Superhuman Code. Go subscribe to the ones you find useful — this post exists because they did the original work.