· 6 min read

Spec-Driven AI Coding: Make the Agent Prove It Is Done

AI coding agents say a task is done when their tests go green. Here is the verification checklist I put in every spec so done means the feature works.

The weakest point of spec-driven AI coding is not the code. It is the word "done". An AI coding agent finishes a task, runs its own tests, sees green and reports success. Often that is true. Sometimes the tests only prove that functions were called, not that the feature works. This post shows the verification checklist I add to every spec, so the agent has to prove the outcome, not the mechanics.

How green tests lie

I write specs before I let an AI assistant touch code. A spec says what the change is, why it exists and how we will know it works. The agent then works task by task from that document. I covered the testing side of mobile UI in waiting for state in Maestro; this post is about the step before that: what the agent must show before it says "done".

The failure mode is always the same. The agent writes the code and the tests in one go. Both come from the same understanding of the task. If that understanding is a little off, the tests are off in the same way, and they pass.

Some typical shapes:

None of these is a bad agent. A person in a hurry writes the same tests. The difference is that an agent is fast and confident, so the gap reaches review more often. The fix is to write down, in the spec, what kind of proof counts.

The verification checklist in every spec

Every spec folder in my projects has a verification-checklist.md next to the spec itself:

docs/specs/SPEC-002-example-feature/
  epic.md
  tasks.md
  plan.md
  verification-checklist.md

The checklist has two parts. The first is a fixed floor that is the same in every project. The second is a short list for this spec only, written together with the spec, before any task starts. The agent reads it as part of the task, and it must walk through it before it marks the task complete.

The floor is a set of plain rules. These are the ones that matter most in mobile work.

Assert the stored result, not the helper call

When a task says "the trip is saved", the test reads the trip back from storage and checks its fields. In a Flutter app with Drift, that means a query against a real in-memory database, not a mock of the repository:

test('saving a trip writes a row', () async {
  final db = AppDatabase(NativeDatabase.memory());
  final repo = TripRepository(db);

  await repo.saveTrip(Trip(countryCode: 'PT', status: Status.visited));

  final rows = await db.select(db.trips).get();
  expect(rows, hasLength(1));
  expect(rows.single.countryCode, 'PT');
  expect(rows.single.status, Status.visited);
});

The most common false positive is a helper that returned without error and never wrote anything. A read-back catches it. The same goes for status changes: read the row after the change, do not trust the return value.

Every guard gets a negative test

A validator that is only tested with bad input may reject everything, and the test still passes. So each guard gets two cases: it fires on bad input, and it stays quiet on clean input. The clean-input case is the one that catches a guard that blocks normal users.

Check content, not only shape

"The list has three items and each has an id" is a shape check. It is needed, but it is not enough. Add at least one content check: the title is not empty, there is no TODO or Lorem ipsum, there are no raw Markdown fences inside a JSON string. Generated code is good at producing the right shape with junk inside.

Read each step's result

For scripts and CI jobs, success is read from each step, not from the last command in a pipe. A background job that printed "exit 0" is not proof. The agent reads the artifact or the log file the run produced. This matters a lot in mobile CI, where a build step can fail quietly and a later step still exits cleanly.

Green tests are not a UI check

For any screen the task touches, someone has to open it. If the agent can run a simulator, it walks the screen and attaches a screenshot. If it cannot, it must say so in plain words: "could not verify the UI here, please check the trip details screen". It may not mark the task complete in silence.

The hand-off summary

The checklist only works if the agent shows its work. So every task ends with a fixed summary, in this order:

  1. What changed, in one paragraph.
  2. Files touched.
  3. Tests added, and what each one asserts.
  4. UI check: verified on which screens, or "not verified, please check X".
  5. Storage checks: the actual query and its result, pasted in.
  6. The checklist items, each ticked or waived with a one-line reason.
  7. Open questions and places where the work differs from the spec.

Point 5 is the one that changes behaviour. "The row was written" is a claim. A pasted query result is evidence. Once the agent knows it has to paste the result, it runs the query, and it gets a chance to catch the bug itself.

Point 6 keeps the checklist honest. A rule can be skipped, but only out loud. "Waived: no UI in this task" is fine. A missing line is not.

Keep it light enough to use

A checklist that nobody reads is worse than none, because it gives a false sense of safety. Three habits keep mine useful:

This is not about trusting the agent less. It is about giving it the same definition of done a careful teammate would use, written down where it cannot be missed.

Takeaways