TL;DR
- I published a prescription for making a repo AI-ready, then realized I had no way to check whether my own repo followed it — so I built a scorer, pointed it at my agent harness, and got the lowest possible grade.
- The overall level is the minimum across pillars rather than an average, because a missing test entry point isn’t compensated by great docs; deterministic checks run in a script and only the two judgment items go to an isolated read-only agent, merged with
minagain.- Pushing three pillars to the ceiling moved the total one notch and a single linter config moved it three — the score reliably predicts its own next step, but whether it predicts agent performance is the part I haven’t measured, and I lay out how I plan to test that.
What My Repo Scored
The first time I ran the scorer, I pointed it at the agent harness I build and use every day — a repo I had spent months shaping so agents could work in it. Here is what came back.
| Pillar | Level |
|---|---|
| agent-config | L1 |
| test-discoverability | L0 |
| ci-automation | L0 |
| safety-hygiene | L1 |
| Overall | L0 |
A repo that bills itself as an engine for agents scored the lowest possible AI-Ready grade.
None of it was unfair. Tests existed but nothing standard could find them, there was no CI config, and .gitignore was missing the .env pattern. All true. I just didn’t know it.
Why I Started Measuring
In an earlier post I argued that firmware AI is blind and needs prosthetics. That was a prescription — here’s what your repo should have.
Something nagged at me afterward. I had written a prescription with no way to check whether it was being followed. I didn’t actually know whether my own repo took my own advice; I only had a feeling. Writing “do this” for other people while not knowing whether I do it myself kept bothering me.
It took me a while to name what was missing. My harness already had plenty of tools that find problems — validators that catch a broken contract or an implementation drifting from spec. But every one of them answers “what went wrong,” and none of them answers “is this repo ready in the first place.” Detection and measurement sit at different layers. I had instruments for finding a fault and nothing resembling a checkup.
Other people were circling the same gap. Through 2026 a handful of projects appeared that score a codebase for agent readiness, and several were already running implementations. So this wasn’t a new itch, and it wasn’t mine alone.
So I built a scorer. It came back L0.
Your repo is probably in a similar state. A carefully written CLAUDE.md, but nothing anywhere that says how to run the tests. A linter everyone runs locally and nobody runs in CI. These are exactly the things you’re most wrong about when you’re sure you’re right. And checking costs almost nothing — that’s the first reason to bother. The scan reads files and changes none of them.
This post is about the design decisions I made building that scorer, and what I watched happen as the score climbed from L0 to L4. It ends with how far I think this number can be trusted, and how I plan to test the part I haven’t proven.
Average Says L0.5, Lowest-Pillar Says L0
The first thing to settle wasn’t the point values. It was how to aggregate — how four pillar scores collapse into one.
That question already had answers. Several projects had shipped something by 2026, so I read those first.
| Project | Structure | Aggregation |
|---|---|---|
| Factory.ai Agent Readiness | 8 pillars × 5 maturity levels | Not published |
| Kodus agent-readiness | 39 checks / 7 pillars, only 4 use AI judgment | Pass 80% of a level’s criteria to unlock the next |
| Workstream | 5 categories, 120 points normalized to 100 | Weighted sum |
| Haolin Dai, agent-readiness | 5 pillars. Safety is treated as a cap, not a weight | Not published |
| Kenogami codebase-readiness | 9 dimensions, 3 of them blocking | Lowest score dominates |
Two things stood out.
First, everyone publishes their structure and half of them don’t publish their aggregation. You get the pillar names and the check lists in detail, and nothing about how those pillars become a single number. That’s the part a reader of the score most needs. Whether an L3 means “everything is mediocre” or “one thing is broken” is decided entirely there.
Second, the ones that do publish disagree. Weighted sum, level gate, lowest-score-dominates. There’s no settled answer, which meant this wasn’t a decision I could copy. I had to pick.
The candidates were exactly those three.
| Candidate | Aggregation | Who actually ships this | Verdict |
|---|---|---|---|
| ① Weighted sum | Assign points per pillar, add, normalize | Workstream (Agent Config 30 · Documentation 25 · CI/CD 25 · Code Structure 20 · Security 20) | Rejected |
| ② Level gate | Clear 80% of a level’s criteria to unlock the next | Kodus | Rejected |
| ③ Lowest pillar dominates | The lowest pillar’s level is the overall level | Kenogami | Adopted |
I rejected ① for a simple reason. An average hides your weakest axis. Excellent docs and perfect secret hygiene don’t help if there’s no way to run the tests — the agent can’t verify its own work. That repo isn’t “middling,” it’s unusable. A weighted average scores it as middling anyway. The whole point of the tool is to surface the weakest axis, so an aggregation that conceals it works against the tool.
② was harder to let go. Clearing one level to unlock the next is a good learning path. What killed it was character. A gate speaks in pass/fail, and I had already decided this score would be advisory only. Wrapping unlock mechanics around a number that blocks nothing makes the name and the behavior disagree.
What pushed me to ③ was Kenogami’s justification for blocking dimensions. Their scorecards explicitly refuse to reduce themselves to an average — “the ceiling (lowest score) sets the readiness level” — and three of their nine dimensions are marked blocking because a low score there can’t be compensated by high scores elsewhere. Averaging treats non-compensable things as compensable. That sentence articulated the discomfort I’d had with ① and couldn’t name.
The choice carries a bonus. Secret hygiene needs no separate cap rule. The safety pillar is already inside the min, so a committed secret drags the whole repo to the floor no matter how perfect everything else is. The cap is implicit in the formula — which also means one fewer branch in the implementation.
Dai’s rubric pinning Safety as a cap rather than a weight reads like the same instinct. That design keeps the cap as its own rule; lowest-pillar-dominates gets there without one. Two roads to the same conclusion, one of them shorter.
Here is what it actually looks like — the per-pillar level, and the overall.
def pillar_level(checks):
"""A pillar's level comes only from the deterministic pass ratio."""
ratio = sum(1 for c in checks if c["pass"]) / len(checks)
if ratio == 1:
return 5 # all checks pass
if ratio > 0.5:
return 3 # more than half
return 1 if ratio > 0 else 0 # at least one / none
overall = min(p["level"] for p in pillars) # overall = lowest pillarThat last line is the entire design. No cap handling, no weight table.
The trade-off is real. The score barely moves. You can fix a lot and see nothing change unless you happen to touch the lowest pillar. Effort doesn’t come back as a number, and for a measurement tool that’s a genuine drawback.
The drawback showed up on day one.
Three Pillars Hit the Ceiling and the Total Moved One Notch
After the L0 I took all four suggestions. I exposed the test runner through a test: target in the Makefile, registered a CI workflow, wrote AGENTS.md, and added the .env pattern to .gitignore. Twenty-nine minutes. That night I added a linter config too.
| Pillar | First scan | 29 min later | After the linter |
|---|---|---|---|
| agent-config | L1 | L5 | L4 |
| test-discoverability | L0 | L5 | L5 |
| ci-automation | L0 | L1 | L5 |
| safety-hygiene | L1 | L5 | L5 |
| Overall | L0 | L1 | L4 |
The middle column is the point of this post. Three pillars jumped from the floor to the ceiling and the overall level moved one notch. No linter config meant ci-automation stayed pinned at L1.
Pushing three pillars to the maximum doesn’t move the overall level; the lowest pillar sets it
On the numbers alone it’s deflating. Four fixes, one notch. But this was exactly the designed behavior, and watching it behave that way is what made me trust it. The score isn’t measuring how hard I worked. It’s an arrow pointing at where the agent gets stuck, and the arrow hadn’t moved.
Then I fixed that one thing and got the last column. The overall level rose three steps. The property I called a drawback earlier flips here: fix the lowest pillar and the whole thing jumps in a staircase. Never having to wonder where to start is the practical payoff of this aggregation.
One honest caveat. What raised the score was the existence of a config file, full stop. The same commit also cleaned up what the linter flagged, and none of that registered. The deterministic check asks “is a linter configured,” not “is the code clean.” An empty config file would have scored identically — a limitation I come back to when I get to gates.
I also turned two linter rules off. The existing scripts used that style, and there was no reason to overturn the codebase’s conventions for a score. Adopting a linter and rewriting your code to satisfy one are different projects.
What a Machine Measures and What a Model Measures
Once you start listing checks, two kinds get mixed together.
Does .gitignore contain the .env pattern? True or false, unambiguous. grep settles it.
Does CLAUDE.md actually help an agent? You have to read it.
Keeping those two apart was the second design decision. Anything with an unambiguous answer goes to a Python script; only the parts requiring judgment go to an isolated read-only agent.
| Pillar | What the script measures | What the agent measures |
|---|---|---|
| agent-config | CLAUDE.md exists · AGENTS.md exists | doc quality |
| test-discoverability | standard test command discoverable | modularity |
| ci-automation | CI config exists · linter config exists | — |
| safety-hygiene | .gitignore hygiene · no secret files | — |
Six deterministic, two judgment. That ratio is deliberate — the more judgment items you add, the more the score wobbles, which is the next section.
Split what a machine can decide from what has to be read, then merge the two with min again
I didn’t invent this split either. Kodus from the table above draws the line in exactly the same place — only four of its 39 checks use AI analysis, and it names them: naming consistency, whether tests are meaningful, whether the README is useful, whether the docs are friendly to an agent. All things you have to read. The other thirty-five look for files and configs.
By ratio, Kodus is nine-in-ten deterministic and I’m six-in-eight. Very different scale, same place to draw the line. Multiple implementations converging there reads to me as a constraint rather than a preference.
The agent that handles judgment operates under two restrictions.
First, read-only at the tool level. It gets file reading and search, and no write tool at all. I didn’t ask it not to modify anything; I didn’t give it the means.
Second, it can’t grow its own role. Its definition spells out four prohibitions: don’t redo the deterministic checks, don’t aggregate scores, don’t write files, don’t invent judgment items you weren’t handed. It judges, and it can’t act on its judgment.
The command does the merge — with min, again.
pillar_level = min(deterministic_level, judgment_level)Run the script by itself and the judgment items stay marked status: "pending". The scorer doesn’t imitate judgment. What it doesn’t know, it leaves marked unknown.
This split is worth something even if you never build a scoring system. Any time you automate a check on a repo — a pre-release checklist, a PR template — this is the first line to draw. Unambiguous items go to a script, and only the read-it-to-know items go to a model. Hand the whole list to a model and you’ll get wobbling answers on the items that were unambiguous to begin with. The next section is that failure in miniature.
This Scorer Gets Firmware Repos Wrong
Firmware is the clearest illustration of what deterministic checks can’t do.
The test-command-discoverable check looks for exactly three things: scripts.test in package.json, a test: target in a Makefile, or a pytest/tox config. Those are web and Python conventions, transplanted.
Zephyr’s standard test runner is Twister. You invoke it as west twister -T tests/, and what runs is decided by testcase.yaml files scattered through the tree plus platform filters. It isn’t that Zephyr lacks a discovery mechanism — it has its own, and that mechanism matches none of the three.
A Zephyr repo with excellent test coverage scores L0 on this check. And because the lowest pillar dominates, the whole repo goes to L0.
The fix is the same one I applied on day one. Wrap Twister in a Makefile line.
test:
west twister -T tests/From an agent’s point of view this isn’t cosmetic. There’s a real difference between an answer to “how do I run tests here” scattered somewhere in the tree and a make test standing at the front door. I ran into this once before while building a HIL CI pipeline, except that entry point was for humans. Now agents are the ones looking for it.
Same Code, Different Score
Adding judgment items meant accepting one thing up front: model verdicts drift.
The problem isn’t the drift itself. It’s that drift looks like a regression. When the code hasn’t changed and the score drops, that reads as a signal that something broke.
I observed it twice, in both directions.
| Interval | Judgment score change | Code changed in between |
|---|---|---|
| 2nd → 3rd | doc quality 5→4, modularity 5→4 | none |
| 3rd → 4th | modularity 4→5 | the linter commit only; module structure untouched |
Once down, once up. Same files, same criteria, read again. On the third scan the judge introduced a new deduction — the constitution is verbose — for a document that hadn’t changed by a single character since the second scan.
I had three options: drop the judgment items, run them repeatedly and average, or admit the drift and constrain what the results are used for.
I took the third. Regression detection compares deterministic levels only. Judgment levels still land in the scorecard, but they’re excluded from the regression verdict. So the third scorecard records a judgment drop with all four deterministic levels unchanged and concludes not a regression, and the fourth notes the opposite direction as “judge variance, not subject to machine comparison.”
Writing it down instead of hiding it paid off later. Reading back through the history, I could tell at a glance that a rise wasn’t something I’d earned. The very first entry, which had no baseline to compare against, is marked “no baseline” rather than quietly passing — I didn’t want “couldn’t measure” to look like “measured fine.”
I Didn’t Turn the Score Into a Gate
Build a measurement tool and the urge to make it a pass condition arrives immediately. Block deploys below L3. Reject a review when the score drops.
I didn’t. And I made it hard to do accidentally.
- The score JSON carries
advisory: trueas a required field. The schema itself declares what this is. - Everything is off by default. Scoring runs only when a command gets
--score; without it, behavior is byte-for-byte identical to before. - Even printing the score alongside a review happens only when a scorecard file exists. Never scored the repo? Then it’s off, with no flag or config file to introduce.
- The spec says it in one line: never use the score or its exit code as a gate or a blocking condition.
I made the exit codes deliberately inconsistent with the rest of the repo. The validators return exit 2 when they find a problem. The scorer returns exit 0 no matter what the score is. The spec explains why: a score is a measurement, not a finding. Findings demand a response; measurements don’t.
The most important safeguard is a different one. It never creates a missing file.
If AGENTS.md is absent, the report says to consider adding one. The scorer doesn’t add it. That’s not an unfinished feature; it’s an explicit prohibition.
The reason connects back to the limitation from earlier. The check looks at whether a file exists, so an empty AGENTS.md raises the score. Had I let the scorer fill its own gaps, I would have built a tool that improves its own grade. Put the thing being measured under the control of the measurer and the measurement stops meaning anything.
This isn’t a hypothetical I invented to justify a decision. Workstream, from the table earlier, went the other way — per the paper, its generator creates AGENTS.md for you when the repository lacks one. Gap-filling is a feature.
I won’t claim one of us is right. That design gets your repo improved quickly; mine keeps the measurer’s hands off the thing being measured. What’s clear is that we were protecting different things. That a real implementation took the opposite answer is what makes this a fork in the design, not a matter of taste.
There’s a familiar line about a measure ceasing to be a good measure once it becomes a target. It’s usually attributed to Goodhart, but that phrasing is Marilyn Strathern’s, from 1997; Goodhart’s own 1975 formulation is drier — statistical regularities collapse once you apply control pressure to them.
Refusing to give the score teeth costs something too. Nobody has to care about it. A metric that blocks nothing is easy to ignore, and I ignored one of its suggestions across two consecutive scans.
So what did a number that blocks nothing actually buy?
What I Got Was the Next Single Step
Across four scans, the useful part was never the number. It was the sentence underneath it.
The scorecard writes improvement suggestions for the lowest pillar only. It says nothing about what the other pillars could still do. That’s not laziness — it falls out of the min. If the lowest pillar is the overall level, the next thing to do is always exactly one thing.
| Scan | What it suggested |
|---|---|
| 1st | Expose the test runner on a standard path · register CI · AGENTS.md · .env in .gitignore |
| 2nd | Add a linter config |
| 3rd | Add a linter config (same as the 2nd, marked still unadopted) |
| 4th | Trim the CLAUDE.md body |
Not a to-do list — a next step. A vague sense that the repo is a bit messy turns into one thing to do this week.
The Prediction Held
What convinced me wasn’t the moment the score went up. It was the moment before.
The third scan’s suggestion stated the outcome in advance: add a linter config, ci-automation’s pass ratio goes to 1.0 and the pillar hits L5, and the overall rises to L4 behind the next-lowest pillar.
I added the linter and rescored a minute later. Exactly those numbers. ci-automation L5, overall L4.
This is possible only because the deterministic layer is six checks. A judgment-laden item can’t be predicted — as the drift section showed, rereading the same file is enough to change the answer. A measurement tool earns its keep not by assigning accurate scores but by telling you what your next action will produce. That requires the layer to be deterministic.
What Survives Deleting the Score
Delete every scorecard and four things stay in the repo.
One line of make test. An agent no longer has to guess how tests run here.
A CI workflow. Changes an agent makes get verified without a human looking. It’s the minimum version of what I called a log feedback channel in the blind-AI post.
AGENTS.md. This one isn’t just for my scorer. OpenAI published it in August 2025 and donated it that December to the Linux Foundation’s Agentic AI Foundation alongside MCP and goose, with more than 60,000 open source projects already using it as of that announcement. Codex, Cursor, Copilot, Gemini CLI, Devin, and Factory all read it. A file I created because my own tool asked for it turned out to line up with where the industry landed.
One .env line in .gitignore. One path for secrets to reach an agent’s context or a commit, closed.
Scoring was just what prompted those four. The value is in the four. The score is the finger pointing at them.
If You Want to Try This
You don’t need my tool. All six deterministic checks amount to asking whether a file exists, which is a shell script you can write in half an hour. The work is deciding what to check.
- Is there an instruction file an agent will read (
CLAUDE.md,AGENTS.md)? - Can a test command be found the standard way (
make test,npm test, or a pytest config)? - Are a CI config and a linter config committed to the repo?
- Does
.gitignorecover.env, and is the tree free of secret files?
Grade each pillar on pass ratio alone — all checks pass is 5, more than half is 3, at least one is 1, none is 0. The coarser the scale, the less room judgment has to creep in.
And don’t average when you combine them. Take the lowest item as the overall score and the output stops being a grade and becomes one thing to do next. If there’s one thing worth taking from this post, I think that’s it.
There’s an equally clear list of what not to do. Don’t score someone else’s repo with it, and don’t make it a pass condition. The reason is the next section.
How Far This Number Can Be Trusted
Interrogating a measurement you designed yourself feels odd, but skipping it turns this post into a tool advertisement. Four grounds, as I see them.
One, the prediction held. Covered above: the tool named the resulting level before the change and the change produced it. That’s the most practical demand you can make of a measurement — does it make falsifiable predictions, and are they right?
Two, the predictable layer is separated from the unpredictable one. Six deterministic checks give the same output for the same input. The two judgment items don’t, and once I confirmed that empirically I removed them from the regression verdict entirely. The structure only asks you to trust the part that’s trustworthy.
Three, an independent implementation reached the same conclusion. Kenogami, from earlier. Two designs that never referenced each other converging on “lowest, not average” suggests the property belongs to the problem rather than to my taste. The same goes for the dual rubric and where Kodus drew its line.
Four, it publishes where it’s wrong. It gets Zephyr repos wrong. A config file alone raises the score even if the code is a mess. The judge gives different scores for the same file on a rereading. A measuring instrument that knows where it fails is more trustworthy than one that claims it doesn’t.
And one part you shouldn’t trust.
I never measured whether agents actually work better. I didn’t compare task success or rework rates before and after scoring, and I don’t have the data to. So this post does not claim that raising the score improves your agent. What’s been verified is that the scoring system behaves consistently by its own rules — not that the score correlates with real outcomes. Those are different claims.
That’s also why it stayed advisory instead of becoming a gate. You don’t get to block someone’s work with an unproven metric.
How I Plan to Check That
Ending on “I’ll measure it someday” felt cheap, so here’s the design. Fortunately no new instrumentation is needed — the harness already emits signals as work happens.
Five events get recorded: rework, circuit-breaker trips from repeated failure, automatic dispatches, fact-check failures, and phase blocks. On top of those, a few values are aggregated per unit of work: rework rate, how far actual work drifted from the plan, and how many items shipped without tests.
The design is this. For each unit of work, collect a pair — the repo’s AI-Ready level at that moment, and that work’s quality signals. Then check whether rework rates and circuit-breaker trips fall for work done after the level rose. If they fall, that’s one piece of circumstantial support. If they don’t, that’s evidence this score is a hygiene checklist and not a performance indicator. Either result leaves me knowing more than I do now.
Nailing down the falsification condition in advance matters here. “The level went up and the rework rate held steady or climbed” — if that’s what comes out, I have to shrink what this score is for.
The obstacles, stated up front. The sample is a few dozen units of work, which won’t support strong statistical claims. The model itself changes over that span and task difficulty varies wildly, so confounders abound. Most of all, the observer and the subject are the same person. So this is closer to hunting for a counterexample than proving causation — showing that no link exists is comparatively easy, and I think there’s a real chance that’s what surfaces first.
I’ll write up the result separately, whichever way the numbers go.
The Remaining Deduction Is a File I Wrote
After the climb to L4, the lowest pillar was agent-config. Both deterministic checks passed, so on the deterministic layer alone it’s an L5. What pulled it to L4 was a single judgment item: doc quality.
The deduction was the same two scans running.
Third scan: CLAUDE.md is imperative and progressively disclosed, and AGENTS.md delegates rather than duplicating rules. But the constitution is verbose.
Fourth scan: progressive disclosure and delegation are strong. But the constitution body is bloated.
Told twice, and I didn’t fix it either time. The suggestion field reads “a cleanup skill already exists, awaiting user decision.” I am the user.
This one stings for a specific reason. In the earlier post I named CLAUDE.md over-injection as a failure mode — put in too much and you create two sources of truth, and when they drift the AI trusts the wrong one. The CLAUDE.md of the person who wrote that is losing points for exactly that reason.
Build a measurement tool and the first thing you measure is yourself, and then you find out you’re breaking your own advice. I think that’s the most valuable thing the scorer produced. Sentences that persuade other people are cheap to write. Your own repo’s numbers aren’t.
The next task is settled: trim the constitution. The tooling for it already exists. I just haven’t done it.
And a larger one is waiting — the validation described above, testing whether this score connects to actual work outcomes. Split the rework rates and circuit-breaker records around the level change and the answer falls out. It may well come back as no connection, in which case I narrow what this tool is for. The next post will be about those numbers.
References
Earlier in this series
- Giving a Blind Firmware AI Eyes — Repo Setup for Claude Code — what a repo should have; the direct predecessor to this post
- The Build Passed, So Why Doesn’t It Run — Automating Firmware Tests on Real Hardware — building HIL CI on Twister
- How to Actually Use AI Coding Agents — 6 Skill-Specific Tips — where the machine-vs-model split originated
Readiness rubrics I read
- Factory.ai — Agent Readiness — 8 pillars × 5 maturity levels
- kodustech/agent-readiness — 39 checks / 7 pillars, 4 using AI judgment
- Kenogami-AI/codebase-readiness — 9 dimensions, lowest score dominates
- harrydaihaolin/agent-readiness — Safety treated as a cap
- Happy Bhati, Workstream: A Local-First Developer Command Center — 5 categories over 120 normalized points, with a gap-filling generator
Standards and terms
- Linux Foundation — Agentic AI Foundation launch (AGENTS.md donation)
- Zephyr Project — Twister test runner
- Goodhart’s law — the commonly quoted phrasing is Marilyn Strathern’s (1997)