This week’s engineering thread was about spreading work evenly. MoE routers have to spread tokens across experts, a new attention scheme spreads a query across clusters of keys instead of every key, and Netflix spreads performance review across a fleet by giving a stateless model a shared catalog. On the news side, two stories read together: OpenAI shipped a vulnerability-hunting model, and METR measured how much AI is now speeding up AI research. As always, these are my notes from the newsletters I follow, not original research.

Deep dives

1. Seven MoE routing strategies, and why balanced routing isn’t the same as fast serving

Source: Daily Dose of DS.

The problem: if every token goes to its highest-scoring experts, many tokens pick the same ones. Those experts overload, others sit idle, and with a fixed capacity and a dropping rule some assignments get dropped. Across GPUs, uneven load plus communication overhead eats the compute savings. Routing has to pick useful experts and distribute work, and each strategy trades those off differently.

The seven:

  1. Token-choice. Each token picks its top-k experts. The expert count per token is fixed; the load per expert is not. If four tokens all pick the same two experts in a four-expert layer, two experts get four assignments each and two get none. Training-time balancing losses help but don’t guarantee a balanced serving batch.
  2. Expert-choice. Reverse it: each expert picks its top tokens up to a capacity. Load is equal, but compute per token isn’t. One token may reach both experts, another neither. The original paper also flags trouble for autoregressive generation and tiny serving batches, since choosing across a token group needs care when future tokens don’t exist yet.
  3. Dynamic routing. The number of experts varies per token, for example picking experts in score order until cumulative router probability crosses a threshold. Caveat: router confidence is not a direct measure of token difficulty, so check quality and real runtime rather than assuming more experts means better answers.
  4. Balanced assignment. A global matching step assigns tokens under load constraints (BASE Layers is the research example). Some tokens lose their first choice, and the matching itself costs time. It’s a research idea, not something every production decoder runs per step.
  5. Hash-based. A deterministic hash maps tokens to experts with no learned gate. Simple, but less adaptive, and a fixed mapping doesn’t balance real traffic by itself because frequent tokens concentrate.
  6. Soft MoE. Experts process weighted mixtures of token representations through slots. It changes what the expert sees. The original work targets vision and names autoregressive decoding as a limitation, because mixing tokens has to preserve causality.
  7. Group-limited. Restrict which expert groups are eligible before picking experts. Tie the groups to hardware placement and you cut cross-device traffic. DeepSeek-V3’s node-limited routing sends each token to at most four nodes. Arbitrary logical groups don’t guarantee less traffic, and bolting a restriction onto an already-trained model can change what it computes.

My takeaway: these are routing designs, not seven toggles in a serving engine. And even perfectly even token distribution doesn’t guarantee low latency, since expert batches can still be tiny and moving tokens between devices takes time. The advice I’d steal is to profile the whole MoE path as separate stages (routing, token movement, expert compute, combination) and go after the slowest one rather than staring at router scores.

2. Latent Sparse Attention: stop the indexer from scanning every key

Source: AlphaSignal (Sunday deep dive, by Ben Dickson).

Following up on the recent weeks of inference-cost notes, this one attacks a different layer: attention itself at very long context. Full attention gets quadratic as context grows, which hurts prefill (processing the whole input) and decode (each new token consults the whole context). Sparse attention picks a limited set of keys per query, but methods like DeepSeek Sparse Attention use an indexer that still scores every previous key or block, so prefill selection cost keeps growing quadratically. In a 10M-token run on GLM-5.3-Flash, which uses a DSA variant, selecting tokens took 98% of an attention layer’s prefill time.

Latent Sparse Attention (Prefrontal AI and ETH Zurich) organizes keys like a library: look at shelves first, then books. Four steps:

  • Represent keys compactly in a learned latent space.
  • Organize them into clusters, each with a centroid.
  • Search by comparing the query to centroids and choosing promising groups. Flat search compares to every centroid; tree search follows a hierarchy of clusters.
  • Attend conventionally over the original keys from those groups, plus a recent-token window and a few initial tokens.

Two ways to adopt it: Distilled LSA adds a trained indexer to a frozen existing model; Native LSA bakes clustering into training with ordinary next-token loss.

Reported numbers, 10M tokens on an H200: one GLM attention layer’s prefill 22.7x faster and decode 4.6x faster than its sparse baseline; Qwen prefill 72.4x faster than full attention. End to end on eight H200s, GLM’s time to first token improved 2.2x and per-output-token time 2.1x; Qwen’s time to first token improved 9.2x. The honest caveats are in the same issue: on RULER, Qwen improved (63.9 vs 59.9 for full attention at 10M) but GLM lost up to eight points at two to five million tokens versus its sparse baseline, and the gains shrink a lot at shorter contexts. Native LSA was only tried on a 500M-parameter hybrid at up to 128K tokens, where it matched full-attention validation loss and trained 1.24x faster end to end.

My takeaway: this only matters if your agents really live in the million-token regime. The code is Apache 2.0, with indexer weights for Qwen and GLM and an SGLang integration, so the sensible move is to benchmark on your own input lengths and watch both time to first token and retrieval accuracy.

3. How Netflix uses agents for performance engineering

Source: ByteByteGo.

The setup: AI-assisted coding ships code faster than anyone can profile it, and LLMs have no idea that a helper runs ten million times a second on a hot path. Traditional performance work (trigger a profiler, pull a flame graph, hunt the method in the code, patch, redeploy) depends on a few experts, so only big incidents get investigated and small steady drains go unnoticed.

Netflix’s finding is that LLMs read profile data fine, because profilers across runtimes (async-profiler/JFR, py-spy, pprof) all reduce to call stacks with inclusive and self time, and models already know the classic anti-patterns. Two practical lessons:

  • Scope the input. A raw stack trace distracts the model with GC pauses, kernel syscalls and third-party framework overhead. Strict package-prefix rules keep it on code the team actually controls.
  • Verify without AI judging AI. Existing tests must pass, then a canary runs next to an identical baseline, then hard telemetry (relative CPU, median and P99 latency, allocation rate) decides, and only then a PR opens for human review with the flame graph and canary report attached.

The production wins are concrete: an O(N²) immutable-map accumulation inside a stream reduction (8.8% of CPU), a per-request replaceAll regex (46% of a service’s CPU, replaced by a lookup table that ran 22x faster), a heavy HashSet built on every call (47% of heap collections), and a metric-counter lookup on every call (66% of CPU versus 0.3% for the real work, and it turned up in three independent services).

The part I liked most is the shift-left design. Models are stateless, so Netflix paired one with a stateful Pattern Catalog: a plain Git repo of Markdown files, no vector DB, in the “LLM wiki” style. It’s diffable, reviewable, works across offline profiling, CI and local tools, and scales to hundreds of rules. In the reactive path, findings from production get written back to the catalog; in the proactive path, CI matches a PR diff against the catalog index and an LLM leaves an inline comment before merge. Netflix describes four maturity levels, from manual to assist to orchestration (where they are now) to fully autonomous.

My takeaway: a Markdown repo as shared agent memory is a very cheap thing to copy, and “the canary decides, not the model” is the verification rule I’d want on any agent that edits code.

4. The OpenCode bug class: local servers, and the triage bottleneck

Sources: Daily Dose of DS (sponsored by CodeRabbit, so read it with that in mind), with a related AlphaSignal item.

The incident worth understanding: OpenCode’s opencode serve starts an unauthenticated local API on 127.0.0.1:4096, and one endpoint accepts a version and installs it, with package setup code running on install. Any web page you open can make your browser send a request to localhost. A plain old form submission can do it, and it worked because the server read a plain-text body as JSON without checking the content type. An earlier version did reject non-JSON, but the check was lost when that part was rebuilt on a new framework. The flawed code shipped April 30 and those versions were downloaded over 647,000 times before a fix. No single file looked wrong; the bug only existed along the path from request to install. The fix (1.18.22) requires the target to be a real version number.

The newsletter reproduced the pattern in a Claude Code-generated dashboard (about 2,150 lines, 90 passing tests) with one hand-added module, then ran a repository-wide scan. It returned six findings: the injection, a CSRF issue on the same endpoint, a missing origin check, an SSRF that checks one DNS lookup but lets the client resolve again, a DoS in the same fetcher, and an unauthorized settings endpoint. The useful property wasn’t the count; it was that each finding came with the entry point, the path across files, whether the route was reachable, and exploit conditions.

Separately, the same issue cites Anthropic’s Project Glasswing numbers: 29,439 candidate vulnerabilities flagged in open-source projects, and as of October 2 only 516 of 6,157 reported findings patched. Anthropic names human triage as the bottleneck.

My takeaway: if you run model runners, MCP servers or coding agents on localhost, treat “only listens locally” as no defense, and check content types and origins. And when evaluating any scanner, judge it on the evidence it hands you per finding, because verification, not discovery, is where the time goes.

The week in AI news

Models and capability

  • OpenAI GPT-6 Cyber (AlphaSignal): a security-focused model with 81 high-severity Firefox vulnerabilities found and patched by Mozilla, 1,724 Linux kernel bugs reproduced with working exploits for 948, 48.5% on ExploitGym (up from 32.1%), and out-of-scope behavior under 1% (down from 5%). Access is through OpenAI’s vetted Daybreak program, with a red tier for offensive testing and a blue tier for validating and fixing. Another AlphaSignal item put the 948 kernel patches at $1.2M of cost.
  • METR on Anthropic (AlphaSignal, and covered across several others): after 11 weeks inside Anthropic, METR estimates models are improving at roughly 2x the historical rate versus March 2024 to October 2025 (plausible range 1.3x to 3x), with about 80% of the speedup attributed to Anthropic using its own AI tools for research, and no clear bottleneck found. Staff report big gains but the most critical research steps aren’t fully automated. The 100-page report was heavily redacted, so this leans on trusting METR’s judgment.
  • Kimi K3.1 Pro weights (AlphaSignal): 2.8T-parameter open-weight MoE with a 1M-token context, about 1.56 TB of weights, ranked first among open-weight models on Agent Arena, already hosted on Ollama, Nebius and Databricks.
  • OpenAI Images 2.6 (AlphaSignal, The Code, AI Secret): animated GIF output from text or photo, reusable reference images, and two API models, Flare (as fast as 6 seconds) and Sunburst (higher quality edits), both with SVG export, and 50% faster overall.
  • Smaller items from AlphaSignal’s headlines: Hydrogen-1.0 with 753B open weights, an OpenAI bug bounty of $172K, and “GPT-6 Astra weight edits”. I only saw the teaser lines for these, so no detail from me.

Tooling

  • Claude Code v2.1.316 cuts background subagent cost (AlphaSignal): each background subagent refreshed its status line every 30 seconds by re-reading its full conversation. Now Haiku 5.5 writes the update from short notes, making each update 99% cheaper, about 5% less usage for sessions with background subagents and roughly 6% more prompts before the limit.
  • Claude gets personal-finance features (Superhuman, The AI Report, AI Inner Circle, Staying Ahead): the recurring headline mid-week was Claude seeing your spending, summarized in one newsletter as able to see it but not touch it. I only saw teasers, so I’m not going to guess the details.
  • Claude limits: AlphaSignal’s Nov 2 issue said verified Ultra accounts lose the 5-hour session cap, and “Claude limit reached. again” was the subject of a Staying Ahead issue, so limits are clearly on people’s minds.

Security

  • Cursor flagged an SSH backdoor in a fake VS Code extension hitting 5,000 machines (AlphaSignal, Nov 2 teaser).
  • a16z: “The Exploit Was the Bottleneck” and “Defense in Depth, Rebuilt for Machine Speed” ran the same week as GPT-6 Cyber, subtitled “What 23 Years of Data Say About AI and the Patch Gap”. Worth reading alongside deep dive 4.

Infrastructure and money

  • Google Cloud A5X instances (AlphaSignal): 80,000 Vera Rubin GPUs in one data center, now available.
  • AI chip stocks fell (The Median): investors bet on fewer new data centers and chipmakers paid for it.
  • Apple cuts AI budgets and flags agents as a business risk (MyClaw, The AI Report headlines), while the Rundown and The Code had Apple opening Siri to use third-party apps and AI.
  • Goldman cuts traders’ AI access (MyClaw headline), and a16z’s charts of the week led with “Apple skips capex”.
  • Anthropic keeps showing up: an IPO-direction story (The AI Report), a residency paying $350K to $500K prorated with no prior ML experience required (AlphaSignal), and a note that Claude mapped a poorly understood protein structure.

Research and science

  • Claude and vault proteins (AlphaSignal): Claude searched 15 billion predicted proteins and found vault relatives across 29 bacterial groups, cutting structure mapping from weeks or months to days. One generated protocol included an unsuitable additive and cost two weeks until a human caught it. AlphaSignal’s framing, “fast, imperfect lab partner”, is the right one.
  • LeCun on convergence (AlphaSignal): models trained on different data converge to the same world view.
  • MIT: AI improving AI with no training required (The Rundown headline).

Agents

  • AI agents making phone calls (The Rundown), publishers preparing a feed for agents (MyClaw), and one-ticket-becomes-a-coding-team stories (MyClaw) kept the agent theme going, though I only saw the headlines.

Tools and reads worth a look

  • LSA code and weights from Prefrontal AI: Apache 2.0, distilled indexers for Qwen and GLM, SGLang integration. A good excuse to benchmark long-context serving on your own lengths.
  • Daily Dose of DS’s inference-serving roadmap: a 10-week, 50-session GitHub plan that builds one service end to end (deploy, instrument, load test, tune, cost-aware router, public benchmark). The checklist itself is good: learn the roofline model, read the vLLM and SGLang schedulers, track time to first token and inter-token latency, report p50/p95/p99, and use queue depth as an autoscaling signal. They are careful to say the 30-minute session timing is a study plan, not a guarantee.
  • Onyx (Daily Dose of DS): self-hostable, MIT-licensed AI chat that works with any LLM, with agents, MCP support, a code interpreter, and RAG that indexes 40+ connectors instead of querying tools at runtime.
  • InsForge (Daily Dose of DS): Apache 2.0 open-source backend (Postgres, auth, storage, edge functions) with a CLI that lets an agent inspect backend state, used in a four-agent PM/backend/frontend/tester setup where only the PM updates the board and handoff files carry the interface contract.
  • ByteByteGo, “The Hidden Work Behind Every Database Request”: read, write and space amplification, with B-trees versus LSM trees, indexes and filters, compression and caching. I only saw the intro, but the framing of logical versus physical work is the right lens for storage tuning.
  • ByteByteGo EP232, six orchestration frameworks in an AI system, covering task management across agents, failure handling, shared memory and context. I saw only the teaser.
  • ByteByteGo on Nvidia Nemotron and on Anthropic’s Scion source control (Nov 2 and Nov 4 issues): the topics are on my reading list; I haven’t read them yet.
  • Daily Dose of DS on agent framework vs agent harness and 12 optimizers for training neural networks (Nov 3 and Nov 2): saved for later, not summarized here.

Sources

These are my notes, not original work. This week’s digest draws on Daily Dose of DS, ByteByteGo, AlphaSignal, The AI Report, The Rundown AI, AI Secret, MyClaw, Superhuman AI, Superhuman Code, Staying Ahead, AI Inner Circle, The Prompt Warrior, Build with AI Club, The Daily Bite, a16z, The Median, and True Positive Weekly. Items I only saw as headlines or teasers are marked as such. Go subscribe to the originals.