Performance Reviews for Agents

More shipped, less struggle

The Performance page counts shipped tickets by the day they reached Done, and smooths each rate over three days. Here's Every King Dies Mobile from October 6 to October 27.

Output is up: code lines and tickets shipped per day both trend upward. Meanwhile the median ticket shrank from about 800 lines to about 300, and the time from an agent's first job to my Final Review fell from about four hours to about two.

Quality is moving the right way. Testing used to send defects back to Implementation more than once per ticket. Now it happens on about one ticket in five. Failed jobs, like a merge conflict or a stopped recovery, are down by about three quarters from their peak.

Tokens per code line fell by more than half. And the times the Factory needed me for anything other than design and final review fell from more than five per ticket to under one in two.

Some of that last drop is bookkeeping. Early on, the Factory counted routine notices and machine alerts as needing me even when nothing asked me to act. Fixing that count was part of the work too: a metric that cries wolf is friction of its own.

Review the work, not just the tickets

Engineering managers have done this for years. You look for where someone is struggling and you ask why. The answer is rarely "they're bad at their job." It's a poor process, a weak upstream deliverable, missing tools or access, unclear goals, or a scope that's too big. You agree on a change, and you check back to see whether it worked.

The best teams I've been on ran retros that were constructive and effective. They could dream big and still be pragmatic about what to improve first and how much to take on. What made those retros work best was data: real numbers to ground the discussion, and to judge later whether the change we agreed on actually worked.

Plenty of retros ran on feel instead, because feel was the best anyone had. Most human teams were too busy to gather data. The product gets priority, and improving how the team works slips down the list. That excuse is gone. "We should gather data on that" is now a ticket an agent can knock out over a lunch break.

My Performance page is an example. It started as an ad hoc conversation with Claude, digging into the numbers Software Factory already records. The report it put together was useful enough that I wanted it one click away, so it became a page. The easier it is to see performance, the easier it is to do something about it when it turns the wrong way.

"Measuring how your team works has always mattered. It's never been easier to do, or harder to excuse skipping."

Me, on measuring the work

Agents also make the review itself easier. Every job has a log, a timing, a token count and an outcome. Nobody has to fill in a timesheet. What's left is the habit of reading it every week, if not every day, and acting on it.

None of us enjoyed a manager peering over our shoulder all day, or tools tracking every keystroke, pull request and hour of focus. Our agents work just fine under those conditions, even if we feel a little dirty about it. Maybe just a little.

Here are the changes that I think moved my lines the most. Each started as friction I could see in the data or in a ticket thread.

A supervisor that answers first

The biggest drain on my time was agents waiting on me. In one week the Factory needed me 245 times in a single day for things other than design and final review. One agent's question sat in my queue for about eight hours. When the supervisor agent finally looked at it, it solved it in minutes.

So the Factory's supervisor agent now answers every agent question before it reaches me. It screens bugs that agents file, steps in when a ticket is thrashing, clears stale blockers, carries out recoveries I've already approved, and moves work to another machine when one machine is the problem. I wrote down the rule for it when one game ticket burned through more than 200 million tokens:

"If Factory sees a ticket is struggling then the right thing to do is pause, assess, regroup and get the train tracks rebuilt and the train moving."

Me, setting the supervisor's job

Then the supervisor became the friction. In 48 hours it was called in 172 times, mostly to confirm that a healthy agent was healthy, at about a sixth of one provider's usage. Now simple rules and a cheap monitoring agent screen the alarms before the supervisor gets involved.

Between those changes and fixing what counts as needing me, the amber bars fell from 245 on the worst day to 15.

Why not just make the workers better?

There's a reasonable argument against a supervisor. It's another layer, and layers cost money; my own numbers showed that. It can become a crutch: if it keeps answering the same question, the worker never learns. And one capable agent is simpler than two.

I think that's half right. A worker sees one ticket in one phase, and its job is to finish. The supervisor sees across tickets, machines and my past decisions, and it holds the calls I've handed it. A thrashing ticket can't see that it's thrashing, any more than an engineer three hours into a bug can. And I'd rather pay for strong judgment rarely, in one place, than brief every worker for every exception.

The bigger value for me is that the supervisor drags friction into the open. When it steps in, the decision and its reasons land in the ticket thread. I can read it, understand it, and agree or push back. Those threads are exactly what I bring to a retro. By making the inefficiency a squeakier wheel, it makes me more likely to learn what's really going on, and to decide how I'd like it fixed.

Where the critic is right is that every intervention should teach the workers something. A question answered twice should become a note or a change to the worker's brief. An alarm that's usually false should be screened more cheaply. The supervisor's workload should fall over time, and if it doesn't, the workers are leaning on it.

That's what I'm building into Software Factory next: the design and tooling to turn supervisor interventions into worker improvements, and to track whether they fall. It's the same loop as the rest of this story. Look at what's good and what's bad, make a change, and find out with data a few weeks later whether it was right.

Memory that gets graded

My agents keep notes for each other: workarounds, conventions, lessons from deploys. Early on, an agent could see only the eight most recently updated notes in each area, so useful notes were missed. Now agents see the whole index, and every time they use a note they rate it as helpful, unhelpful or possibly stale, and say why.

Those ratings let me groom the index on real usefulness instead of guesswork. Periodically an agent audits the notes against them. Notes that get pulled often and marked helpful stay. Notes that overlap get consolidated. Notes nobody finds useful get rewritten or cut.

In one audit, my game had collected 19 separate deploy lessons in six days. One had been pulled 87 times and marked helpful only twice. The lessons were folded into a shorter set that agents actually use.

Memory without grades just grows. Memory with grades can be pruned like any other code.

Sealed designs

One large game feature went through twelve design versions before I approved it. Implementation then built a scrolling page where the approved design was a menu, and the test plan still included difficulty tiers I had cut. The friction was immediate and painfully visible: it was right there in Final Review, with the whole history behind it in the ticket thread.

So I had a side conversation with each agent about why this one was such a struggle. Their answers pointed to something I'd seen many times over my career. Large tickets are hard. And a long history of debate and decisions is a good paper trail, but it also confuses, and it leaves room for misreading. The agents were reading drafts and arguments that had since been settled, and picking the wrong parts.

"A paper trail is great for remembering how you got there. It's a poor map for where you're going."

Me, on why designs get sealed

That's how the sealed design was born. Once I approve a design, the Design agent seals a clean version that contains only what was approved. Implementation and Testing build from that version and nothing else. It's been a real help ever since, and the fix was upstream, not in the agents doing the building.

Cheaper rework

Rework used to be where the tokens went. One evening, three game tickets bouncing between Implementation and Testing used about 213 million of roughly 400 million tokens. On the feature with twelve design versions, the rounds after the first build cost nearly twice as much as the first build itself.

Five changes went after it:

Smaller tickets

The ticket size chart is mostly my doing, not the agents'. Agents struggle with oversized tickets the same way people do: more to hold in their head, more ways to drift, and more to retest when something changes. So I write smaller tickets now, and I interview the big ones before they start to find where they split.

The harder question is what to do when a ticket is already in flight and Testing keeps finding problems. I've split one off by hand: a ticket that had run for hours, which I called a "perfect example of when/why splitting off a lightweight follow up fix vs continuing to thrash." But I haven't made "ship what works, file the rest as follow-ups" a standing rule. When Testing finds a real defect, my agents still fix it on the ticket that found it.

The small-fix send-back is the half step. Final Review can now return a short, focused change instead of reopening the whole ticket. Whether to go further is something I'm watching closely in the numbers before I decide.

The next conversation

Agent minutes per 100 code lines is roughly flat. It moves around between 20 and 46 minutes and trends down only slightly. Tickets are cheaper in tokens and reach me faster, but each line takes about as much agent time as it did a month ago.

I don't know why yet, and that's the point of having the chart. It's this week's conversation: is it slow machines, slow test runs, agents waiting on tools, or something I haven't thought of? I'll make a change, give it a week, and look at the line again.