Maintenance was always the bottleneck

Every product team I know is shipping faster than it ever has. Features that used to take three weeks take three days. That part is great.

What nobody talks about is what happened on the other side of it. We ship more, so there is more to verify, and the way we verify has not changed at all. On the team I am on now, our fix is a band-aid. We point an agent at the codebase, hand it very detailed instructions, and iterate until it writes the suite we wanted. The tests still sit in the codebase. They still run in CI/CD. And a week later the product has moved and half of them are lying to us.

I keep coming back to this, so I am writing it down.

The claim

Writing tests was never the bottleneck. Maintenance is.

This was true before AI and it is worse now. When a team shipped once a fortnight, a stale suite was an annoyance. When a team ships twice a day, a stale suite is a suite nobody trusts, and a suite nobody trusts is one people start skipping. Every team I have been on had a set of tests they quietly ignored.

There is a second problem underneath it. The tests never captured the nuance in the first place. The engineer writing them is the person furthest from the actual user flow. They know the code path. They do not know that a user coming from the email link lands in a different state, or that the empty state is the one people actually see on day one. Even with an agent writing the code, someone still has to spell all of that out by hand, and the person who knows it is the PM. The PM does not have time to spell it out for every feature, because of how much is shipping. So it does not get spelled out.

What everyone tried

This maze has been walked a lot. Selenium, then Cypress and Playwright, then a wave of record-and-replay tools that promised testing without writing code.

Every one of them held two assumptions fixed. A test is code, and it lives in CI/CD.

The record-and-replay tools are the interesting casualty, because they tried to break the first assumption. Let a non-engineer click through a flow, record it, replay it on every build. It sounds right. It failed on maintenance. The recording is bound to selectors, the UI moves, the selectors break, and fixing them requires understanding the DOM. So maintenance fell back to engineering and the tools became another thing engineers owned.

The second assumption never got questioned at all. If tests live in CI/CD, they belong to the people with commit access. Everything downstream follows from that.

What changed

Two things, both recent.

Models can reason about intent. A test can be a goal rather than a script. “A new user can sign up, connect their calendar, and see their first meeting” is a sentence, not a set of selectors, and a model can now work out what that means in a specific product.

Computer-use agents are reliable enough to drive a real interface. Not perfect, but past the line where they can log in, navigate, and notice when something is wrong.

Together those two things break the first assumption properly, not the way record-and-replay tried to break it. If the test is a goal, then when the UI moves, the goal has not moved. There is nothing to repair.

What I would build

You point an agent at a URL and give it login access. It researches the company, works out who the users are, derives the flows itself, and keeps them current as the product changes. It checks whether the flow works, not whether a specific input box still exists.

No repo access. No CI/CD.

The interesting consequence is who ends up owning it. Not because I think PMs should own testing as a matter of principle, but because if maintenance stops being an engineering cost, there is no longer a reason for it to sit with engineering. The PM wrote the flow. The PM approves the release. The PM can now change what gets checked without asking anyone, without GitHub access, and without touching code they are right to be nervous about touching.

Ownership moves as a side effect of fixing maintenance. That is the part I find compelling.

What I do not know yet

A few things I am actively working through.

• How does the agent tell an intentional change from a regression? If the signup flow gains a step, that is a product decision, not a bug. Getting this wrong in either direction kills the product. This is the hardest problem in the whole thing.
• What does it cost? Tens of thousands of runs a day at even a handful of customers. Cost per run is not a detail, it decides whether this is a business.
• How much flake is survivable? Trust is the actual product here. A suite that is wrong five percent of the time gets ignored the same way a stale suite does.
• Where does it break? Anything with state that is expensive to set up, anything that depends on external systems, anything where the correct behaviour is a judgement call.

I am building a prototype. If you have run into this, or if you think I am wrong somewhere, do reach out. I would like to hear it.