Writing

Why things break, and what to do about it.

Plain-English essays on AI agents, running language models and distributed systems. Most include an interactive demo you can poke at. I post when I have something worth saying, so there’s no schedule.

  1. 48 min read

    An Agent Is a Model Plus a Harness. The Harness Now Rewrites Itself.

    Same model, same 89 terminal tasks. 58% inside one harness, 76% inside another that no person wrote. What a harness is, what it’s worth once somebody reruns the numbers, the five things a machine is now allowed to rewrite in one, and the results that should slow everyone down. One loop issued 43 warnings about what it might break. Five came true.Strong opinionAI agents, Evaluation, LLMs
  2. 18 min read

    AI That Improves Itself Is Real. Does It Compound?

    The phrase “recursive self-improvement” was on four arXiv papers in 2024 and a hundred and five in the first nine months of 2026. I read the ones that measured something. The improving is real. The one experiment that tested the recursion came back 0.780 against 0.782.Strong opinionAI agents, Evaluation, LLMs
  3. 108 min read

    Building Effective Autoresearch Systems: 2

    A system that runs its own experiments has about seventeen moving parts, and each one has a way of quietly lying to you. Here is every mechanism I found that stops it, in the order the loop runs them.Strong opinionAI agents, Evaluation, LLMs
  4. 39 min read

    Building Effective Autoresearch Systems: 1

    Systems that run their own experiments, read their own results and keep the changes that worked. The ones that have discovered anything real share one thing, and it isn’t the search algorithm. It’s who marks the homework.Strong opinionAI agents, Evaluation, LLMs
  5. 10 min read

    Your Agent Isn’t Confused. It’s Timing Out.

    Almost every failed agent run I’ve debugged came down to a timeout, a retry that did something twice, or context that got cut off without warning. The model was fine. The system around it wasn’t.Strong opinionAI agents, Reliability, LLMs
  6. 22 min read

    Where the Money Goes When You Run an LLM

    These days, much of an LLM bill goes on words you never read: the model thinking out loud, charged by the word. Here’s where the money goes, why more thinking isn’t always better, and how to stop paying genius rates for easy questions.Strong opinionLLMs, AI agents, Cost
  7. 14 min read

    22,580 GPT-2s Fit Inside Kimi K3. Scale Is the Boring Part.

    Seven years took language models from 124 million parameters to 2.8 trillion, and the block diagram barely moved. The interesting part is quieter: what a model keeps while it reads, what it throws away when it runs out of room, and who gets to decide.Strong opinionLLMs, Attention, Architecture
  8. 15 min read

    A Thousand Sandboxes That Agree on What Random Means

    An agent that writes and runs code is executing untrusted input, so each tool call wants a microVM. But microVM boot times don’t fit the budget, so everyone snapshots, and a snapshot is a copy of the guest’s idea of randomness.Strong opinionAI agents, Security, Infrastructure
  9. 15 min read

    The Cheapest GPU Is the One That Already Knows You

    Load balancing an LLM fleet isn’t a load balancing problem, it’s a cache affinity problem. The cheapest machine isn’t the idle one, it’s the one already holding your KV cache, and the textbook answer picks wrong almost every time.Strong opinionAI agents, LLMs, Distributed systems
  10. 15 min read

    You Can’t Replay a Model Call

    Every durable workflow engine recovers from a crash by re-running your code and trusting it to reach the same decisions. A model call breaks that trust in one line. So durable execution for agents isn’t replay, it’s recording.Strong opinionAI agents, Reliability, Distributed systems
  11. 40 min read

    How Subscription Billing Systems Actually Work

    A subscription isn’t a row with a boolean on it. It’s a long-running state machine that outlives every process and most of the services that touch it. Here’s how the real ones are built, and three problems that only show up once there are hundreds of millions of them.Battle-testedDistributed systems, Payments, Reliability
  12. 14 min read

    Which of Your Invariants Actually Needs a Lock?

    Whether an operation needs coordination isn’t a property of your database, your isolation level, or your budget. It’s a property of the rule you’re protecting, and there’s a test that tells you which is which.Strong opinionDistributed systems, Consistency, Databases
  13. 15 min read

    Write the Simulator First

    The bug that takes you down lives at one exact interleaving of messages, clocks, crashes and slow disks, and production will never show it to you twice. FoundationDB’s answer was to build the simulator before the database, so that an entire cluster’s history becomes a pure function of one number.Strong opinionDistributed systems, Testing, Reliability
  14. 12 min read

    Slow Database Writes? Blame the Map.

    Every consensus protocol has the same speed limit, a round trip to most of your servers. So the real engineering happens in where you put those servers and what you’re willing to call a failure.Battle-testedDistributed systems, Consensus, Latency
  15. 10 min read

    Too Much Traffic? Decide Who Waits.

    Every queue in your system is a decision about who waits when things get busy. Most outages I’ve read about happened because nobody made that decision on purpose.Battle-testedDistributed systems, Reliability
  16. 15 min read

    Removing the Cause Doesn’t Fix It

    A distributed system has two stable states, not one. The load level that breaks you and the load level you have to return to before you recover are different numbers, and the second one is far lower than almost anybody expects.Strong opinionDistributed systems, Reliability