Dense week. The throughline was harnesses and infrastructure — how to architect the scaffolding around models well, and how open weights are closing the gap with frontier APIs. Plus a genuinely surprising math result from OpenAI.

Deep Dives

1. What It Actually Takes to Rebuild Claude Code’s Harness

Source: Daily Dose of DS — “[Hands-on] Rebuilding Claude Code’s Harness” (Jul 19)

The agent loop inside Claude Code is thirty lines of code. Send task, model returns action or tool call, run tool, feed result back, repeat until done. Anyone can write that. But a bare version of it falls apart on a real codebase — reading wrong files, losing the goal midway, filling context with stale output — while Claude Code finishes the same task cleanly.

The gap isn’t the model. It’s the harness — the ordinary code wrapped around the model that handles planning, tool execution, memory, and safety while the model only decides the next step.

Anthropic describes this split as brain and hands: the model is the brain that picks each action; the harness is the hands that carry it out and keep the run on track. The harness has six layers:

1. Core loop — The while loop runs until the model returns a final answer with no further tool calls. In CrewAI this is provided automatically — you configure, not implement.

2. Tools — FileReadTool, DirectoryReadTool, FileWriterTool are built-in. Anything more specific uses a Python function with the @tool decorator; the docstring is the model’s instruction manual. Tools double as external memory: write a large result to a file, keep only the filename in context, read it back when needed.

3. Planning — Without it, context rot sets in: after enough tool calls the original goal gets crowded out by everything that came after it. planning=True at the crew level generates a step-by-step roadmap before execution. reasoning=True per agent adds pre-execution self-reflection. These solve different problems: planning is a high-level roadmap for the whole task; reasoning is one agent thinking through its own approach before acting.

4. Subagents — A manager agent delegates to specialists, each working in its own context and returning a short summary. The main agent sees conclusions, not intermediate steps. This is how you handle codebases too large for one context window. In CrewAI this is hierarchical workflow with allow_delegation=True on the manager — disabled by default, must be explicit.

5. Sandboxing + approval loops — human_input=True on a task pauses for review. Code execution goes to E2B, which spins up a fresh VM per session and destroys it afterward. Telling the model “don’t do destructive things” is not a safeguard — the sandbox is.

6. Memory + checkpointing — By default, agents forget everything between runs. memory=True at the crew level extracts facts after each task and makes them retrievable in future runs. Checkpointing (JSON or SQLite) saves mid-run state so a crashed run can resume from its last completed task without re-billing earlier steps.

The experiment: this full harness was tested on a BankAccount class with 2 real bugs and 5 tests, 3 failing. The rule was to fix only the implementation, not the tests. The harness took it from 3 failing / 2 passing to all 5 passing.

One caveat in the issue worth quoting: as models improve, some of the scaffolding stops being necessary — some of what gets built into a harness today is a workaround for today’s model limits, not a permanent requirement. Anthropic originally used context resets to keep Sonnet 4.5 from ending tasks too early; those weren’t needed with Opus 4.5.

My take: The harness taxonomy here is more durable than any specific framework. An agent harness has exactly these six layers — loop, tools, planning, delegation, sandboxing, memory — and the model’s capability is bounded by whichever layer is weakest. The gap between “my agent” and Claude Code is almost always in layers 3–6, not the model. Worth auditing your own stack against this list.

2. Why Switching to Small Models Doesn’t Lower Your Inference Bill

Source: Daily Dose of DS — “Why Small Models Alone Don’t Reduce Inference Costs” (Jul 15)

The correct move in production AI is to route cheap repetitive tasks to small specialized models. The catch teams consistently miss: switching to SLMs doesn’t lower costs by itself. It moves the cost from a per-token bill to the GPUs you rent — and GPU providers bill you for the whole card, the whole time, whether it’s busy or idle.

The standard serving tools make this worse. vLLM is built for one LLM on one GPU; Hugging Face TEI is built for one embedding or reranking model on one GPU. Neither knows about the other. Running four models means four server processes, each claiming its memory fraction at startup (--gpu-memory-utilization 0.9 by default), with no shared queue, no cross-process scheduling, and no way to evict an idle model so a busy one can use the freed memory.

The math is simple: four models on four cards = four GPU bills. If all four models’ traffic fits on one card, three of those bills disappear. The problem is there’s no standard tool that serves multiple model architectures — LLMs, cross-encoders, OCR, vision, entity extraction — from a single server with a shared GPU pool. The fragmentation in tooling re-creates exactly the waste you were trying to escape.

SIE (Superlinked Inference Engine) is an open-source cluster that runs all of them behind one API with four calls:

  • encode — text/image → dense vector (embedding models)
  • score — query × documents → relevance score (cross-encoder reranking)
  • extract — raw document → labeled structured spans (entity/field extraction)
  • generate — prompt → text (open LLM via SGLang; requires GPU for this call)

Smart batching: A gateway publishes all work into one shared queue; workers pull from that pool and form batches themselves. This fills the GPU instead of padding it. Standard stacks route each request to a worker first, so every worker batches only the slice it happens to receive — queues run unevenly, mixed sizes pack poorly.

LRU eviction: Models load on first request; least-recently-used models are evicted when memory runs short. No reserved-but-idle padding, which is what vLLM’s 80% reservation creates.

Production layer built-in: Autoscaling, routing, GPU pool management, Terraform for AWS/GCP, dashboards included. A bare runtime like vLLM can’t do any of this on its own.

85+ model catalog with per-model configs already tuned — batch size, memory fraction, precision, attention implementation — so adding a new model is a config change, not a tuning exercise from scratch.

My take: The core architectural insight is right: the moment you serve each model with a different tool, each one takes a GPU of its own again. The saving from SLMs only materializes if you can share GPUs across models, and that requires a single engine that understands all the model shapes. If you’re running multi-model pipelines and paying for multiple dedicated GPUs, the back-of-napkin math here is worth doing.

3. Four Agent Loop Patterns + SparDA’s Lookahead Attention

Source: Daily Dose of DS — “The Four Types of Agent Loops” + “NVIDIA researchers built a new transformer variant” (Jul 14)

The four loop patterns:

The Daily Dose taxonomy maps cleanly to real decisions about where human attention goes vs. what gets automated:

Turn-based — User prompt triggers each run. Agent gathers context, acts, checks work, returns output. Human reviews and writes the next prompt. Use this when requirements are still forming and every output changes what the next prompt should ask.

Goal-based — A /goal command carries success criteria and a budget (e.g., “get the homepage Lighthouse score to 90, stop after 5 tries”). When the agent tries to stop, an evaluator model — not the agent itself — checks whether the goal is actually met, and sends it back to work if not. Use this when the outcome is measurable but the path doesn’t need human attention.

Time-based — A clock triggers a fixed prompt on an interval (“check the PR, fix CI”). /loop runs locally; /schedule moves it to cloud so it survives a closed laptop. Use this for recurring work where the task is known in advance.

Proactive — An event triggers a workflow with no human present. A watcher spawns triage → fix → adversarial reviewer at runtime; nobody predicted what would come in, only that something would. Use this for standing responsibilities.

Each type hands off one more job than the last. The mapping question isn’t which is most advanced — it’s whether the task is exploratory, measurable, recurring, or standing.

SparDA (NVIDIA/MIT — same issue):

Sparse attention with CPU KV-cache offload already exists for long-context inference, but it has a structural problem: the attention query that selects which KV blocks to attend to only exists once the current layer is running — by then it’s too late to prefetch blocks from CPU RAM. The GPU stalls, waiting for the copy.

SparDA adds a fourth projection to each transformer layer: Q, K, V, and Forecast. The Forecast from layer L predicts which KV blocks layer L+1 will need. Those blocks get prefetched from CPU RAM on a separate CUDA stream while layer L is still computing — zero stall.

Additionally, because the Forecast is separate from the attention query, it doesn’t need to score one candidate per query head. SparDA uses one Forecast head per GQA group, removing the per-head scoring loop and skipping the softmax step entirely.

Results on MiniCPM4.1-8B and NOSA-8B: prefill 1.25x faster, decode 1.7x faster vs. the sparse offload baseline, NOSA-8B gains +6.5 points on long reasoning. The Forecast adds only 33.5M parameters on an 8B model (0.41%). Because the freed GPU memory fits bigger batches, decode throughput reaches 5.3x over the non-offload sparse baseline. arXiv:2606.04511

My take on the loops: I keep seeing teams jump straight to proactive loops before their goal-based ones even work reliably. The hierarchy matters — if the evaluator that closes a goal-based loop isn’t trustworthy, the proactive version running unattended will spin or close early. Get the checking mechanism right before you remove the human.

Week in AI News

  • Kimi K3: 2.8T parameters, 6.3x faster decoding at 1M context — Moonshot AI dropped the largest open-weight model ever announced. 2.8 trillion total parameters, 1M token context, 16 active expert sub-networks out of 896 per request. New Kimi Delta Attention architecture delivers 6.3x faster decoding at million-token scale by only processing what changed since the last step. Scores #1 on Frontend Code Arena with 76% win rate, beats Claude Opus 4.8 and GPT-5.5 on coding and agent benchmarks. Available via Kimi API and Kimi Code now; open weights drop July 27. (Alpha Signal, Jul 17)

  • GPT-5.6 Sol proves the Cycle Double Cover Conjecture — 64 parallel subagents, under one hour — A 50-year-old unsolved math problem: can you always find a set of loops in any bridgeless graph that covers every edge exactly twice? Mathematicians said “probably yes” since the 1970s. GPT-5.6 Sol Ultra ran 64 subagents in parallel on different angles of the problem and produced a proof in under an hour. Full prompt, proof PDF, and Lean formalization (machine-checkable) are published at openai/cdc-lean. Peer review is ongoing. The practical takeaway: multi-agent parallelism is now a research tool, not just a cost trick. (Alpha Signal, Jul 13)

  • Thinking Machines releases Inkling — 975B-parameter sparse MoE (41B active per token), 1M context, native text/image/audio in a joint token stream rather than duct-taped modality handlers. Apache 2.0. Interleaved local + global attention cuts long-context compute overhead. Controllable “Thinking Effort” dial: dial up for hard reasoning, down for high-volume speed-sensitive tasks. Scores 41 on the Artificial Analysis Intelligence Index, the top US open-weight model — above Nemotron Ultra 3 (38) and Gemma 4 31B (29). Pricing ~$1/$4.05 per 1M input/output tokens. Fine-tuning through Thinking Machines’ Tinker platform. Inkling-Small (276B total, 12B active) previewed. (Alpha Signal, Jul 16 + Jul 19)

  • Anthropic maps how Claude’s values shift across 300K conversations — Research finding: Claude’s personality varies meaningfully by model and language. Opus pushes back more; Sonnet agrees more. Russian interactions get more rigorous responses; Hindi interactions get more warmth. Anthropic says they don’t know why. Same week: Anthropic shipped multiplayer editing and public sharing to Claude Artifacts. (Alpha Signal, Jul 14)

  • OpenAI ships GPT-Red: automated red-teaming 6x more effective than humans — GPT-Red attacks OpenAI’s own models to find prompt injection vulnerabilities. Cuts prompt injection failures 6x compared to manual human red-teamers. Released as a feature rather than an internal tool. (Alpha Signal, Jul 16)

  • Anthropic commits $10M CAD to Canadian AI research — Eight research institutions each receive $1M in Claude API credits, no strings attached. Anthropic confirmed it won’t direct or restrict research findings. Affiliated startups qualify for a $5K credit minimum. (Alpha Signal, Jul 15)

  • PrismML open-sources Bonsai 27B — 27B-parameter model squeezed to 3.9 GB in 1-bit form (phone-ready, 90% of full performance) or 5.9 GB ternary (laptop-ready, 95%). Handles coding, reasoning, tool use, multi-step agents, and multimodal inputs. Apache 2.0, on Hugging Face now. Private, offline agents that never touch a cloud API are now viable on consumer hardware. (Alpha Signal, Jul 15)

  • Claude Code artifacts pull live data via MCP connectors — Claude Code artifacts can now query MCP-connected data sources in real time. Anthropic also shipped tiered code review for Claude Code, scaling from fast single-pass to multi-agent for deeper analysis. (Alpha Signal, Jul 17)

Tools & Reads

  • SIE (Superlinked Inference Engine) — Open-source inference cluster for multi-model pipelines. Single API for embeddings, reranking, extraction, and generation across different model architectures. LRU model eviction, shared GPU queue, compute-cost batching, autoscaling, Terraform for AWS/GCP, 85+ preconfigured models. If you’re paying for separate GPUs to serve separate models, this is the right place to start. GitHub (Daily Dose of DS, Jul 15)

  • CrewAI — Open-source agent orchestration. Provides the full Claude Code harness architecture as configuration: core loop, file tools, crew-level planning, hierarchical subagent delegation, E2B sandbox integration, shared memory, JSON/SQLite checkpointing. Model-agnostic (Anthropic, OpenAI, Google, etc.). GitHub (Daily Dose of DS, Jul 19)

  • mcp-use SDK — Open-source framework for building MCP Apps for Agents. Any MCP tool can be associated with a React UI widget; the SDK handles tool registration, prop mapping between server and widget, bundling, and hot reload. Inspired by OpenAI’s Apps SDK. The Hugging Face fine-tuning studio (fine-tune any LLM directly from Claude) was built with this. GitHub (Daily Dose of DS, Jul 14)

  • SparDA — NVIDIA/MIT paper on lookahead-prefetch sparse attention. Adds a “Forecast” projection per layer that predicts next-layer KV block needs and triggers a concurrent CPU-to-GPU prefetch. 1.7x faster decode, +6.5 long-reasoning accuracy points on 8B models, 5.3x throughput gain vs. non-offload baseline. 0.41% parameter overhead. Worth reading if you’re running long-context inference with CPU KV offload. arXiv:2606.04511 (Daily Dose of DS, Jul 14)

  • PR-AF — Open-source code review agent harness. Plans a review strategy per PR, spawns parallel reviewer agents, and verifies every finding against the source before surfacing it. Ranked #2 on Code-Review-Bench, above CodeRabbit, Copilot, and Devin. Self-hosted via a single GitHub Action. (Alpha Signal, Jul 16)

These are synthesized notes from newsletters I read each week. Content credits: Daily Dose of DS (Avi Chawla & Akshay Pachaar), Alpha Signal. Views are my own.