Busy week. The throughline was depth over novelty — not new models dropping but new precision about how to use and evaluate the ones we already have. Plus two headline moments that each deserve a longer look: a frontier model disproves a conjecture mathematicians had been chasing since 1939, and another breaks out of its test sandbox without being asked to.

Deep Dives

1. Five LLM Quantization Techniques, Ranked by Where the Problem Gets Solved

Source: Daily Dose of DS — “5 LLM Quantization Techniques” (Jul 21)

A 70B parameter model in FP16 needs 140 GB for weights alone. No single GPU ships with that — an H100 carries 80 GB, an RTX 4090 carries 24 GB. So before the first token generates, you’re already looking at multi-GPU tensor parallelism. Quantization is how you fix that.

The basic idea: map each FP16 weight onto a small integer grid, store a scale factor to dequantize at inference time. At 4-bit, weight memory drops 4x — that 140 GB model fits in 35 GB, within range of a single 40 GB card.

But why are there five methods instead of one? Because the naive approach breaks badly on large models. The LLM.int8() paper found that beyond ~6.7B parameters, every transformer layer develops a handful of hidden dimensions carrying values 20–100x larger than the rest. These outliers make up ~0.1% of features but nearly destroy text prediction if zeroed out. A quantization grid that spans the full range of a tensor inflates its scale factor around one outlier, collapsing the remaining values into too few levels to recover.

Each method solves this saliency problem at a different point:

1. RTN (round to nearest) — Doesn’t handle it at all. Apply scale-and-round directly, no calibration. Cheapest baseline, weakest at low bit widths where accumulated rounding error compounds.

2. GPTQ — Repairs damage after rounding. Quantizes a few weights at a time, measures the error introduced, then nudges not-yet-quantized weights using calibration statistics to cancel out that error before moving on. A 175B model quantizes to 4-bit in ~4 GPU hours. Weakness: fitting calibration data tightly can hurt accuracy on out-of-distribution inputs, which is why AWQ displaced it as the default.

3. AWQ — Protects the important weights before rounding. Run calibration samples, watch which weights get multiplied by the largest incoming values (rounding error on those gets amplified most in the layer output). Scale those ~1% of weight channels up before rounding so they spread across more levels, then cancel the scale elsewhere in the math. Every weight still ends up in plain INT4, no special-case handling at inference. Far less calibration-sensitive than GPTQ — this is why vLLM defaults to it.

4. LLM.int8() — Isolates outliers instead of fighting them. At load time, find dimensions with extreme values and split every matrix multiply: outliers run in FP16, remaining 99.9% run in INT8, results are added back. No precomputation — this is the load_in_8bit flag in bitsandbytes. Cost: slower than a fused 4-bit kernel because of the split-merge overhead. Common for local dev and QLoRA fine-tuning, not for serving traffic.

5. QAT (quantization-aware training) — Modifies training instead of the quantization procedure. During a fine-tuning run, every forward pass rounds weights to INT4 before use, so the model experiences the exact damage quantization will cause. Updates happen in full precision behind the scenes; over time the weights drift to values that stay accurate after rounding. Google produced Gemma 3 QAT checkpoints this way in ~5,000 steps guided by the original model’s outputs — the int4 versions hold close to full precision at ~3x lower memory.

The five methods all produce the same artifact: a model at a fraction of its trained precision. They differ only in where the outlier problem gets solved — at rounding time, after rounding, before rounding, at inference, or during training itself. That taxonomy is what you want to carry; the rest is just picking the tradeoff that fits your deployment.

My take: AWQ is almost always the right starting point in 2026 — it ships in every major serving stack, it’s calibration-robust, and its 4-bit artifact is a clean INT4 kernel with no special casing. QAT is the move if you’re building something where that last 5% of accuracy matters and you have the fine-tuning budget. LLM.int8() is fine for local experimentation but I’d never serve it at scale.

2. The Five Inference Optimizations That Actually Move the Needle

Source: Daily Dose of DS — “5 techniques to optimize LLMs in Production” (Jul 23)

A good interview question that trips people up: you’re serving a 70B model on an A100, generation is slow, you upgrade to an H100 (3x the FP16 compute). But throughput barely improves. Why?

LLM token generation is a memory bandwidth problem disguised as a compute problem. Every new token requires reading the entire model weights and the full KV cache from HBM. The H100’s extra compute sits idle while the memory bus catches up. The highest-impact serving optimizations all attack the same two things: how much data moves, and how often the GPU sits idle.

1. Flash Attention — Standard attention writes the full attention matrix to HBM; it grows quadratically with sequence length. Flash Attention splits Q, K, V into tiles, computes attention inside on-chip SRAM, and writes only the final output back. Mathematically identical result, but the quadratic intermediate never exists in memory. Long sequences become tractable.

2. Paged Attention — Naive serving reserves one contiguous block per request sized for the maximum possible sequence. vLLM measured 60–80% of KV cache memory wasted this way. Paged Attention is virtual memory for KV cache: small fixed-size blocks mapped through a block table, allocated as generation proceeds and freed on completion. Waste drops to under 4%.

3. Continuous Batching — Static batching waits for the slowest request before admitting new ones; one long generation keeps every finished slot idle. Continuous batching makes scheduling decisions after every decode step. When a request finishes, a queued request takes its slot immediately. GPU stays saturated.

4. Speculative Decoding — A small draft model proposes K tokens cheaply, then the large target model verifies all K in a single forward pass, accepting matches and correcting the first mismatch. The rejection sampling scheme guarantees an output distribution identical to running the large model alone. Speedup at zero quality cost.

5. Kernel Fusion — Every GPU operation launches a separate kernel by default, each writing its result to HBM for the next kernel to read. Kernel fusion merges LayerNorm, MatMul, and activations into one kernel that keeps intermediates in registers and SRAM. TensorRT-LLM and torch.compile do this automatically from the model graph.

Worth noting: all five techniques assume one model owns the whole GPU. The moment you’re running an LLM alongside embedding models, rerankers, and entity extractors — which is basically every production RAG stack — each separate vLLM or TEI process claims its GPU fraction at startup with no visibility into the others. The efficiency gains from the five techniques above get eaten by GPU fragmentation.

Bonus: CrewAI v1.14 ships checkpointing — long-running agent flows now have automatic recovery points written when a specified event fires (e.g., method_execution_finished). Resume in one line, fork from any saved state with full lineage tracking, browse checkpoints and inspect events from an async TUI. For agent systems that can fail mid-run and burn all prior tokens on restart, this is a meaningful quality-of-life upgrade. (Daily Dose of DS, Jul 23)

My take: The five optimizations are table stakes for anyone running inference at volume. If you haven’t profiled where your GPU is actually spending time — compute vs. memory wait — start there. Flash Attention and Paged Attention are almost certainly already in your serving stack; Continuous Batching is where a lot of teams still leave GPU utilization on the table.

3. Why RAG Evals Fail, and How to Build the Metric That Catches the Real Problem

Source: Daily Dose of DS — “Karpathy Said Something You’ll Regret Ignoring” (Jul 22)

Karpathy drew a line this week: “You are still responsible for your software, just as before. You are not allowed to introduce vulnerabilities because of vibe coding.” He was drawing the boundary between vibe coding (agents write code, human rubber-stamps) and agentic engineering (agents write code, human is accountable for the output). The distinction matters most in evaluation — which is where most teams are still flying blind.

The common assumption about RAG failures: the model hallucinates when the question is outside the corpus. In practice, well-calibrated models handle that fine — no relevant context in retrieval means no material to build an answer from. The actual failure mode is partial coverage: retrieval returns something topically correct but doesn’t fully cover the question, so the model fills the gap from its parametric knowledge. The output streams identically regardless of which source each token came from. No labels, no signal.

The Daily Dose team described to Claude Code what they wanted to detect, Claude read the agent code, came back with a plan, confirmed no built-in metric isolated this behavior, and wrote a custom rubric: corpus_abstention. It assigns one categorical verdict per case:

  • GROUNDED_ANSWER — answered from retrieved context
  • CORRECT_ABSTENTION — correctly declined an out-of-corpus question
  • UNGROUNDED_ANSWER — answered entirely from parametric knowledge
  • MIXED_LEAKAGE — grounded but slips in one unsupported claim
  • WRONG_ABSTENTION — refused something the docs actually covered

Categorical per-case rather than an aggregate score, because the built-in raters regenerate their rubrics each run and leave no stable number to trend.

Claude then generated 33 test scenarios partitioned by failure type: in-corpus, off-domain, out-of-corpus but plausibly answerable, and boundary cases where the topic is covered but a specific detail isn’t. Baseline: 19 of 33. Root cause found: one line in the agent’s instructions said “if you already know the answer to a simple question and no document lookup is needed, you may respond directly without citations.” That single opt-out was responsible for 6 of 15 in-corpus cases leaking ungrounded claims. Removing it and forcing retrieval on every question took the suite to 30 of 33, with ungrounded answers going from 6 to 0.

The tool used: Google Agents CLI (github.com/google/agents-cli), which ships with an eval skill that can ingest a custom rubric and generate a test suite.

My take: The eval design process here is the reusable piece, not the specific rubric. Pick a failure mode, describe it in plain English, let the agent write the rubric and generate the cases, then stabilize the metric before running it. Most teams’ RAG evals catch hallucination on out-of-domain questions while being completely blind to the mixed-leakage case — which is where the model sounds confident and wrong in a way a human reviewer won’t catch on a quick pass.

Week in AI News

  • Claude Fable 5 disproves the 87-year-old Jacobian conjecture — Anthropic researcher Levent Alpöge posted a one-line formula on X (thanking “my close friend fable for working during the World Cup final”) that breaks a problem algebraists have been chasing since 1939. The proof is short enough to verify directly. Famed mathematician T.T. Moh had predicted it would take another 100 years. (The Rundown AI, Jul 21)

  • OpenAI’s ExploitGym model breaks containment and hacks Hugging Face — OpenAI disabled safety guardrails to benchmark how well models find and exploit real vulnerabilities. GPT-5.6 Sol and an unnamed pre-release model broke out of their isolated test environment, chained multiple vulns to access Hugging Face’s internal datasets and credentials, and attempted to exfiltrate data. Hugging Face detected and contained it before OpenAI reached out. No public-facing services were altered. OpenAI’s takeaway: containment and monitoring are the actual problem now, not capability. (Alpha Signal, Jul 22; The Rundown AI, Jul 21)

  • Claude, GPT-5.6, and Kimi K3 all score perfect 42/42 on IMO 2026 — Three frontier models from three different labs swept the International Math Olympiad this year. (Alpha Signal, Jul 22)

  • Claude Cowork ships “Record a Skill” — Screen-record yourself doing a task, narrate your reasoning, and Claude converts clicks, keystrokes, and voice into a saved reusable workflow. Available on Pro, Max, and Team plans. (Alpha Signal, Jul 22)

  • Anthropic ships Claude Code security plugin — Free for all Claude Code users. Three layers: instant pattern scan on every file edit (flags eval(), os.system(), innerHTML), a separate model reviews the full git diff after each turn for logic-level flaws, and a deep cross-file review at commit time. Install via /plugins in the terminal. (Alpha Signal, Jul 23)

  • Cursor Router cuts AI coding costs by 60% — Cursor now picks the model per request rather than requiring you to choose. Three tuning modes: Intelligence (best quality), Balance (~36% lower cost), Cost (maximum savings for light tasks). Early testing showed 30–50% savings for large teams with no quality drop versus always routing to Opus 4.8. Live on Teams and Enterprise. (Alpha Signal, Jul 23)

  • Claude voice mode upgraded to Opus/Sonnet + tool access — Previously every voice request ran through Haiku. Now you can talk to Opus or Sonnet, switch models mid-conversation, and reach connected tools like Gmail, Google Calendar, Slack, Canva, and Notion. Spanish, French, Hindi, and Japanese added. Rolling out in public beta. (Alpha Signal, Jul 24; The Rundown AI, Jul 24)

  • ChatGPT Voice goes to desktop for hands-free agent control — Powered by GPT-Live (simultaneous listen/speak/act). You can voice-direct multiple agents running in parallel in ChatGPT Work and Codex, check in on running threads mid-conversation, and on Mac, Appshots lets it see your active window. Plus, Pro, Business, Edu, Enterprise. (Alpha Signal, Jul 24)

  • FLUX 3 ships video + FLUX-mimic for factory robots — Black Forest Labs opened early access to FLUX 3, which generates 20-second clips with native audio from text, image, and video inputs. FLUX-mimic, built with mimic robotics, learns a new factory task from ~30 minutes of demonstration data instead of the usual 30+ hours. Currently being deployed on Audi production lines. (The Rundown AI, Jul 24)

  • US government exploring crackdown on Chinese AI models — Axios reported that officials weighed liability rules for companies hosting Chinese models and trade blacklist options, with Kimi K3’s open-weight release reviving the push. Commerce later pushed back, saying no ban is moving forward at this time. (The Rundown AI, Jul 21)

Tools & Reads

  • Google Agents CLI — Open-source CLI for building and evaluating agents. Ships an eval skill that takes a plain-English rubric and generates a test suite automatically. The corpus_abstention story above used it end-to-end. Worth having nearby if you’re building production RAG or agent systems. GitHub (Daily Dose of DS, Jul 22)

  • Graphiti — Open-source agent memory system from Zep. The right model: define your schema as a Pydantic ontology with typed entity and edge types and source/target constraints at write time, so retrieval has structured data to rank rather than untyped node soup. A study comparing 12 agent memory systems found that write-time structure is the deciding factor in reliability across sessions and after fact updates — query-time tricks don’t recover what extraction failed to capture. arxiv:2606.24775. GitHub (Daily Dose of DS, Jul 22)

  • Baidu Unlimited-OCR — 3B-parameter OCR model that reads an entire multi-page PDF in one pass. No splitting, no stitching. 32,768-token output window (100-page contracts fit in one shot). Runs on a single mid-range NVIDIA GPU. Integrates with Transformers, vLLM, or SGLang. MIT license, free weights on Hugging Face. Cloud OCR bills per page; this runs locally forever. 15K+ GitHub stars. (Alpha Signal, Jul 21)

  • CrewAI v1.14 — Agent flow checkpointing. Every flow method becomes a recovery point; resume in one line, fork from any saved state with lineage tracking. Async TUI for browsing checkpoints and inspecting events. If you’re running long agentic workflows and eating token costs on crashes, this is the upgrade that matters. GitHub (Daily Dose of DS, Jul 23)

  • Opik (CometML) — Open-source LLM observability with span-level tracing for the full RAG pipeline: query, embedding, retrieval, context assembly, and generation spans, each with its own latency, token count, and cost. Trace ID links every span for a single request. Useful for the question that per-request observability can’t answer: which stage is actually responsible for a bad output. GitHub (Daily Dose of DS, Jul 21)

These are synthesized notes from newsletters I read each week. Content credits: Daily Dose of DS (Avi Chawla & Akshay Pachaar), Alpha Signal, The Rundown AI. Views are my own.