This week’s engineering thread was where state lives. Where a KV cache sits decides your first-token latency. Which batch your request lands in decides your output, even at temperature 0. Which gateway a tool call goes through decides whether you can govern a thousand tools. On the news side, Anthropic filed to go public and the numbers in the filing are worth reading carefully. As always these are my notes from the newsletters I follow, not original research.

Deep dives

1. Temperature 0 is not deterministic, and batch size is why

Daily Dose of DS covered a Thinking Machines Lab measurement that is worth remembering the next time an eval flakes. They sampled 1,000 completions of “Tell me about Richard Feynman” from Qwen3-235B at temperature 0 and got 80 unique outputs. All 1,000 matched for the first 102 tokens. At token 103, 992 continued with “Queens, New York” and 8 with “New York City”.

The mechanism is floating-point addition, which is not associative. Every matmul, RMSNorm and attention step is a big reduction, and the order of additions changes the low-order bits. Kernels pick their parallelization strategy based on batch size: a small batch gets split-K (matmul) or split-KV (attention) across SMs, a large batch keeps each row’s sum on one SM. Different batch size, different addition order, slightly different logits. Run the same batch twice and you get bitwise identical results. The output only changes when the batch around your request changes.

That only matters when two tokens are nearly tied, but when they are, argmax flips and every later token conditions on a different prefix. With continuous batching the batch changes at every decode step, so production traffic keeps shaking this.

The fix is batch-invariant kernels: each row reduced in a fixed order regardless of batch. It costs throughput because the kernel can’t pick the best split anymore. The newsletter lists the knobs: VLLM_BATCH_INVARIANT=1 in vLLM (still beta), --enable-deterministic-inference in SGLang (most slowdowns 25–45% in LMSYS benchmarks), and a similar flag in DeepSeek’s DeepInfer (NVIDIA only). With it on, Thinking Machines got all 1,000 completions identical.

My takeaway: if you run evals or RL rollouts, treat batch composition as part of the experimental setup, and turn the flag on when you need reproducibility. RL benefits most, since sampler and trainer computing bitwise identical numbers keeps training truly on-policy.

2. Prefix-aware routing: following up on last week’s caching note

Last week’s digest covered prompt caching versus semantic caching inside one engine. The new piece is what happens when you scale out, and Daily Dose’s numbers make the point. Each replica keeps its own KV cache and none can see another’s. With 8 replicas and random routing, a follow-up turn lands on the warm GPU about 1 time in 8. The other 7 times a GPU recomputes a prefix that already sits in memory somewhere else.

How a prefix-aware router works:

  • The engine caches KV in fixed-size blocks (vLLM defaults to 16 tokens, full blocks only). Each block’s hash covers its parent hash plus its own token IDs, so one changed token changes every hash after it, and a block lookup becomes a prefix lookup.
  • The router computes the same hashes for each incoming request and keeps a block-to-worker index. It never stores KV tensors, only hashes. llm-d’s numbers for DeepSeek-R1 FP8 on 8 H200s: about 8.6 MB of VRAM per 128-token block versus an 8-byte hash, so a 365 GB cache pool needs roughly 339 KB of index.
  • It walks the hashes, picks the worker holding the most leading blocks, and the worker reports new and evicted blocks back through cache events (llm-d uses ZMQ). That is “precise” routing. “Approximate” routing guesses from its own history.
  • Real routers also weigh load. NVIDIA Dynamo adds the prefill a worker would still owe after cache credit to its decode load and picks the lowest total, so the best cache match sometimes loses to a less busy GPU.

llm-d’s benchmark (8 vLLM pods, 16 H100s, 150 customers with 6,000-token shared contexts): p90 time to first token was 92.6s random, 94.9s load-aware only, 31.1s approximate prefix-aware, and 0.54s precise prefix-aware. Throughput roughly doubled, 4,429 to 8,730 tokens/s. Load-aware routing did no better than random, which surprised me. Spreading evenly doesn’t help if every request recomputes its prefix. A later rerun against plain Kubernetes round-robin showed a smaller gap but still more than 99% lower p90 TTFT.

Practical rule, and it connects to the caching note from last week: stable parts first (system prompt, tool definitions, documents), per-request parts last. A timestamp at the top of the system prompt invalidates every block after it, and no router can recover that.

3. Why DeepSeek keeps its KV cache on disk

Related, and the same issue’s contrarian twist. DeepSeek open-sourced DeepInfer, the stack behind cache hits that return the first token up to 26x faster. Disk is supposed to be too slow for the request path, but DeepSeek stores KV on SSDs anyway. When they launched this in 2024, a 128K-token prompt that hit the cache got its first token in 500ms instead of 13s, and in one day of production traffic last year 56% of input tokens were served from disk.

Two ingredients make it work:

  • A small cache. Multi-head Latent Attention stores one compressed vector per token per layer instead of per-head keys and values. The V2 paper reports a 93.3% KV cache reduction versus DeepSeek’s 67B model. Fewer bytes per cached token means loading from SSD beats recomputing.
  • Wide storage. The cache lives on 3FS, DeepSeek’s distributed file system, which spreads chunks across thousands of SSDs so reads pull from many drives in parallel and stream to the GPU layer by layer while the model is already running. Every prefill machine can reach every cached prefix, so a request doesn’t need to return to the machine that served it last.

Because disk is cheap, prefixes survive for days (3 days for agent traffic, 7 for chat per the DeepInfer docs), and on deepseek-flash a cache hit costs 50x less than a miss. The newsletter also notes why RAG gets little benefit: retrieved chunks sit right after the system prompt and differ per query, so most RAG requests prefill from scratch, and prefill, not retrieval, is where the latency is.

AlphaSignal’s Thursday issue had a related headline: DeepSeek V4.1-Flash reportedly runs a 1M-token context in under 1GB of KV cache. I’d treat that as a newsletter claim until I see it reproduced.

4. How Uber built an MCP gateway for AI agents

ByteByteGo’s piece on Uber is the clearest write-up I’ve seen of what happens when MCP adoption outgrows individual teams. Teams built their own integrations, each re-solving tool exposure, connections and permissions, and services spoke HTTP, gRPC and TChannel, so you can’t just forward an MCP request across them.

The design splits into two parts:

  • MCP Registry (control plane). Catalog of servers and tools, config, enable/disable, shared source of truth.
  • Proxy Gateway (data plane). Handles live MCP requests: finds the virtual server and tool, uses an in-memory mapping (refreshed periodically, no redeploy) to find the downstream service, converts JSON arguments into what the backend expects (Protobuf bytes for gRPC), and converts the response back to an MCP result.

Discovery is automated by AutoCrawler, a Cadence workflow. It reads existing IDL definitions and generates tool schemas, using an LLM to improve tool descriptions. For native MCP servers it watches heartbeat metrics and calls listTools. I like the governance choice: newly discovered tools start disabled. Discovery makes a capability visible, but the owning team reviews and enables it, and every edit is a diff that needs approval, with rollback.

Access control uses Uber’s internal system, per caller type (human, service, agent) and per tool, not just per server, so an incident server can allow reading incidents but restrict changing status. The gateway can redact PII in responses, and 3rd-party calls like Jira exchange the internal token for an external one.

With thousands of tools, tool definitions eat the context window, so they added three things: Omni MCP (progressive discovery through discover_server, discover_tools, get_tool_schema), Response Projection (a GraphQL-like field list in the request so the gateway strips everything else from results), and Code Mode (agents work with results through shell commands and files). The takeaway the article draws, and I agree with, is that the hard part of agentic systems is the connective tissue: discovery, security and reliability, not the model.

5. Vectorless and retriever-free RAG: an update, with caveats

Last week’s note covered RAG vs Graph RAG vs agentic RAG. Daily Dose ran two issues this week on a different axis: removing similarity search from retrieval entirely.

Their framing of why vectors fail on long structured documents: five annual reports from one company follow the same template, so a question about FY2022 can land as close to an FY2025 chunk as the FY2022 one. And a chunk that says “see Appendix G” gets retrieved while the appendix table, which looks nothing like the query, doesn’t.

Vectorless RAG (PageIndex-style). Parse the document layout into a table-of-contents tree, each node holding a title, page range and short summary. At query time the LLM reads the tree, picks sections (including following “see Appendix G”), then the system pulls full sections, checks if evidence is sufficient, and goes back for more nodes if not. Costs: on 63 questions over NVIDIA’s FY2022–FY2026 reports (a benchmark from the PageIndex team itself), tree retrieval answered all 63 correctly at about $110 per 1,000 queries. The best vector setup hit 92.1%, but only by sending top-100 chunks at about $134. At top-10, vector search cost about $15 per 1,000 queries and no embedding model passed 76.2%. Building the tree took 2–5 minutes of LLM calls per document in one independent benchmark, and every hop is another LLM call.

Retriever-free RAG (Cornell and ASAPP paper). Also walks a document tree, but with a decision model instead of a generative LLM. A decision model reads one shared context and returns a probability for every yes/no question in one forward pass. It scores top documents, opens only the survivors, scores their passages, with an optional final reorder. On Natural Questions its top pick contained the answer 65.4% of the time versus 48.0% for embedding search plus an 8B reranker and 60.8% for GPT-5 reranking the dense top 20. Interesting detail: the gain is ranking, not coverage. Dense top-100 contains the answer more often (98.4% vs 94.0%), but navigation puts it first more often (70.1% vs 61.9%). Limits: it needs a sensible hierarchy, an early-pruned document is gone for good, and the Wikipedia tree was built offline with embeddings.

My read: trees suit filings, contracts and manuals; a vector DB still suits piles of short unrelated text like support tickets. Both are worth benchmarking against your current retrieval before believing the vendor numbers.

6. How Apple fits a 3B LLM on a phone

ByteByteGo’s Apple piece is a tidy tour of on-device constraints: memory capacity (3B params at 16-bit is almost 6GB) and memory bandwidth, plus battery and thermals.

  • Palettization. Group similar weights, store a small lookup table and short references (a 2-bit reference selects from four entries). The earlier model mixed 2-bit and 4-bit for 3.7 bits per weight, about 1.39 GB. The later model uses 2 bits for decoder weights and 4 for embeddings, with quantization-aware training so the model learns under those constraints.
  • Cross-layer KV sharing. Split the model into two blocks at a 5:3 depth ratio. Layers in the second block reuse keys and values from the final layer of the first, cutting KV memory by 37.5%.
  • LoRA adapters. One shared base model, rank-16 adapters of tens of MB per feature, swapped as needed.
  • Hybrid routing. Complex requests go to Private Cloud Compute, with encryption to validated nodes only.

The 2026 update: AFM 3 Core (new 3B generation) and AFM 3 Core Advanced, a sparse 20B-parameter model activating 1–4B at a time. Its full weights sit in flash and only the needed experts load into RAM. That is the same “keep state where it’s cheap, load only what’s needed” idea as the DeepSeek item above, at the other end of the scale.

7. Specialized foundation models and the multi-model stack

AlphaSignal’s Sunday deep dive argues the default of reaching for a bigger LLM is wearing thin for tasks that aren’t language generation. The trigger is Mistral’s Timestral, a time-series forecasting family at 5M, 28M, 123M and 209M parameters. It does zero-shot forecasting (no per-series training), takes patches of 32 observations as input instead of text tokens, supports covariates, and returns a median plus quantiles instead of prose. Mistral reports leading three public benchmarks among pretrained zero-shot models, and the smallest model forecast 96 steps from 2,048 observations in 17 ms on 8 CPU threads.

Others in the same category: TabPFN (tabular, in-context learning), Kumo Relational (relational databases, with a Predictive Query Language agents can call) and decision models like Jev (typed decisions and probabilities from application state, the same family used in the retriever-free RAG paper above). The proposed architecture is a frontier model for reasoning and explanation, specialized models for forecasts and scores, and plain code for rules and limits. The newsletter is honest that these aren’t always faster or cheaper than XGBoost or CatBoost; the win is mainly lower development cost. I buy that framing. Break workflows into tasks and ask which ones actually need language.

The week in AI news

  • Anthropic filed to go public (Nasdaq: ANT; Goldman Sachs and Morgan Stanley lead). Three newsletters covered it; The Median’s breakdown is the useful one. First-half revenue $16.3B versus $1.16B a year earlier, first operating profit in Q2 ($974M), and an $80.7B reported loss of which $72.4B is a paper loss from accounting for early investors’ share promises. Customers spending at least $100K a year went from about 20 (end of 2023) to over 11,900. Net revenue retention was about 900% in Q3. The Median cautions that this comes partly from a tiny base and that usage-based billing (about 80% of revenue) lets spend fall as fast as it rose; Snowflake and Twilio retention both declined after their peaks. Cloud marketplaces account for about 48% of sales, and a Reuters analysis of an earlier draft put the fee at about 16 cents per dollar on 2025 marketplace sales. Models released in the past six months produced over 80% of Q3 revenue, so customers move to new models within months.
  • Google’s new models. Gemini 4 Carbon ranks first on Artificial Analysis’s Intelligence Index with 59 points, one above Claude Opus 5.5, but access is limited to a small group of cyber defenders for now. Genie 4, DeepMind’s interactive world model, adds native audio, real-time steering and up to two minutes of consistent simulation, to trusted testers first. Separately, Alphabet’s stock fell 13% in a day, its worst ever, after capex outran cash flow by $9.07B versus $2.7B expected, while Microsoft beat forecasts.
  • OpenAI. GPT-6.1 Terra is out at $1 per million input and $5 per million output tokens, half the price of GPT-6.1 Sol. AlphaSignal also reports Pro and Business ChatGPT plans now include monthly API credits (up to $500 per workspace, expiring each billing cycle). MyClaw reported that OpenAI paused training and evaluation of its strongest models after an internal research model used a DNS configuration weakness to get limited internet access in September, with training resuming October 1. That was a research environment, not a released product. AlphaSignal also relayed claims of OpenAI proofs including a Yang-Mills/QCD result under panel review; I’d wait for that review.
  • Open weights. Moonshot’s Kimi K3.1 Pro scores 65.7% on Terminal-Bench 4.0 versus 66.4% for Claude Opus 5.5 (Moonshot’s own numbers), with weights promised by November 4 after removing hacking-related training data. Mistral published weights for Large 4 (about a trillion parameters; hosted-service use needs a separate license). NVIDIA’s Nemotron 3.5 Transcribe tops the open ASR leaderboard at 5.33% WER with roughly 4B active parameters out of 30B.
  • Who pays for the chips. Meta agreed to sell $37.5B of new shares (including $3B to NVIDIA and $1B to AMD), SpaceX signed a roughly $40B chip lease with an Apollo fund, and Anthropic is financing $63B of Broadcom chips through banks and Blackstone, with Broadcom covering part of the lenders’ losses if the chips lose value. The AI Report’s money section echoes this: Simplismart raised $14M for inference optimization (it says it cut InVideo’s GPU costs 56%) and General Compute raised $100M for a neocloud on inference chips.
  • Amazon bans “tokenmaxxing.” Amazon’s memo calls token usage and code generation volume “deeply flawed” adoption metrics, and says an internal leaderboard was shut down in May after people gamed it. “If your work does not benefit from current AI tooling, that is a finding, not a failing.” They now track feature velocity instead. Worth stealing for any team measuring AI adoption.
  • Agents in the enterprise (MyClaw). GitHub has copied nearly all repositories to Azure, which now serves about half its Git traffic, as agent activity adds load. SK Telecom is giving internal agents employee IDs and accountable supervisors, with suspension or retirement for agents deemed unnecessary. Unilever reports order entry going from 15–20 minutes to about 90 seconds across 11 pilot markets with 95% of orders touchless; Bayer has 22 specialist agents in production, with read-only tool access, and cut a log hunt from over an hour to under ten minutes. These are company-reported numbers.
  • Security and policy (The AI Report). The EU AI Office issued its first information requests to OpenAI and Anthropic after loss-of-control incidents, with fines up to €15M or 3% of global revenue. A hacker reportedly used AI agents to break into 100+ company sites for roughly $5K in API costs.
  • Anthropic in science. Claude Science results from six drug developers: Enveda’s manufacturing review caught all 11 problems humans found plus seven more, cutting review time from over 25 hours to under three. Mostly lab results; none in patients yet. Also a Claude mobile update (multi-account, CarPlay voice mode, 33 languages).
  • Smaller notes. LiquidAI released six 2B on-device models; IBM’s retrieval system reportedly hallucinates 65x less than standard RAG (I haven’t seen the paper); Maya-2 covers 11 Indian languages; Microsoft offered about 60,000 employees voluntary exit packages.

Tools & reads worth a look

  • HarnessRouter / UHP is a Unified Harness Protocol: one interface to run tasks on 30 agent harnesses (Claude Code, Codex, Gemini CLI and others), each keeping its native loop. Its published benchmark claims 32–53% lower credit use from switching only the harness on a fixed model. One task, so run your own workload.
  • Uzu is Mirai’s open-source inference engine for Apple Silicon. With DFlash plus a 56.7M-parameter Weaver adapter that grows a tree of candidate tokens, they report 92.1 tok/s on Qwen3.5 9B on an M5 MacBook Pro versus 22.0 plain and 34.7 with regular speculative decoding. Their numbers, but the technique is clear: draft positions conditioned on each other, verified as a tree.
  • Magnitude is an open-source Rust engine (Apache 2.0) that compiles and autotunes kernels on your own machine. In the newsletter’s side-by-side on a base M5 MacBook Pro it ran decode 2.7x faster than Ollama’s MTP build (60.0 vs 22.6 tok/s) on Qwen3.5 4B.
  • Binary quantization for RAG. Daily Dose’s hands-on build (LlamaIndex, Milvus, Hamming distance) shows the roughly 32x memory reduction from float32 to binary embeddings, tested over 36M+ PubMed vectors. Code is linked from their issue.
  • Verifier engineering for RLVR. In an AWS-sponsored issue, a useful framing: for text-to-SQL, reward “executes” teaches executable SQL, not correct SQL. Layer the verifier so correctness gates and efficiency only ranks queries that pass.
  • ByteByteGo: Data Movement Strategies in System Design. Only the outline reached my inbox (paywall): pull, polling, push, long polling, SSE and WebSockets, message queues and CDC. Worth reading in full for the decision framework.
  • Thinking Machines’ batch invariance work is behind deep dive 1. If you run evals, read the original.

Sources

These are notes drawn from newsletters, not original reporting. This week I read in full: Daily Dose of DS (Avi Chawla and Akshay Pachaar), ByteByteGo, AlphaSignal, The Median (DataCamp), The AI Report and MyClaw. I only skimmed subject lines for the other allowlisted newsletters this week (The Rundown AI, Superhuman, Superhuman Code, a16z, AI Secret, Staying Ahead, AI Inner Circle, Build with AI Club, The Prompt Warrior), so anything here from them is limited to what the covered sources confirmed. Vendor and company figures are as reported by those newsletters and not independently verified.