Your AI Coding Agent Needs a Test Harness, Not Just Better Prompts

Everyone Wants Test-Driven Development. Few Have the Infrastructure.

AI coding agents can generate software faster than many engineering teams can reliably verify what they generate.

You can instruct an AI agent to implement a feature, follow architectural guidelines, and write tests. But how do you establish that the resulting software actually behaves as your customers expect? The answer should be straightforward: execute tests derived from product requirements before implementation begins. Unfortunately, many engineering teams lack the infrastructure needed to make this practical.

The challenge is not simply writing End-to-End (E2E) tests. It is creating an environment where those tests exercise the real application, produce reliable results, and run repeatedly without requiring manual setup. Tests built around mocked APIs may validate assumptions that the real backend violates. Shared test environments introduce dependencies between tests, while fragile startup scripts, fixed ports, and inconsistent database states make failures difficult to reproduce.

What is needed is a test harness, which is defined as an environment comprising a collection of scripts, stubs, drivers, and execution engines needed to test a component or system in isolation. Think of it as the control layer that allows humans and AI agents to interact with the software under test.

Consider a multi-tenant application in which the first customer signup initializes the entire installation. How do you test that behavior repeatedly when initialization can happen only once per database? How do you give every test a clean starting state without slowing development to a crawl? These are infrastructure problems, and solving them requires deliberate engineering investment.

Without a reliable test harness, test-first AI development remains an aspiration rather than an operating practice.

Fragile Test Environments Turn AI Speed Into Rework

When E2E testing is difficult to execute, AI-generated code increasingly becomes the authority for its own correctness.

The failure usually starts with an innocent shortcut. A coding agent implements a feature and then generates tests against what it has already built. The tests pass because they reflect the implementation's assumptions, not necessarily the customer's requirements. The organization gains a growing collection of green tests without establishing that it is building the right behavior.

The consequences extend beyond just defects. Without an independent behavioral reference, agents can improvise different solutions to similar problems across modules and services. Architectural decisions drift, integration defects accumulate, and developers spend increasing amounts of time understanding and repairing previously generated work. More code gets produced, but even more code may need to be revised or discarded.

The missing test harness also creates operational waste. Developers manually start services, reset databases, repair broken environments, and investigate failures caused by test infrastructure rather than product defects. QA cannot consistently reproduce the same conditions, and automated tests become unreliable delivery signals. The apparent speed of AI coding turns into rework, cognitive load, and unpredictable delivery.

A test suite that validates what AI has already decided is not an independent quality control mechanism.

Inside Our Test Harness: Real Backends, Isolated Databases, Repeatable Tests

Build an executable test harness that lets AI coding agents verify implementation against product intent, using real application behavior.

You have several implementation choices:

Managed test environments: Provide centrally managed environments with controlled resets and isolated test data.

• Benefit: Teams reuse existing infrastructure and standardize execution.
• Risk: Environment contention, state leakage, and installation-wide behavior remain difficult to manage.

Fully disposable environments: Create a complete containerized application stack for every test.

• Benefit: Strong isolation and greater fidelity to production deployment.
• Risk: Startup cost and resource consumption can make rapid development feedback impractical.

Hybrid isolated test harness: Combine fast local execution with production-like container testing and controlled database seeding.

• Benefit: Reliable feedback without requiring full production infrastructure for every change.
• Risk: Requires investment in lifecycle management and explicit coverage of differences from production.

In my practice, we chose the third approach.

We built a browser E2E harness using Playwright, a real Python/FastAPI backend, SQLite databases, and Docker. Each test accesses a real application stack through its browser-facing URL. There are no mocked APIs. The harness creates isolated environments with dynamic ports and known starting states, then manages their entire lifecycle.

Three named database seeds address different testing needs: a populated installation, a completely empty installation, and a minimal tenant without application data. Ordinary tests reuse a populated stack within a worker where safe; scenarios requiring fresh state receive their own backend and database. A backend-owned launcher creates the environment, issues valid authentication tokens, and emits a structured readiness event only when the server is ready.

The same design handles failures outside normal test execution. Owner records, process lifelines, and cleanup procedures recover resources left behind by interrupted runs. A manual testing CLI uses the same seeded stacks, allowing testers to reproduce starting states and capture traces, logs, and screenshots. Local mode provides fast feedback, while Docker mode verifies the real Nginx configuration, containers, and persistence behavior.

Blueprint: Browser E2E Tests Against a Real, Isolated, Seeded Backend

1. Context

The system under test has two parts that are developed and released separately:

• A web front end: static HTML/JS pages served by Nginx in production. The pages call the API on the same origin under /api.
• An API backend: a Python FastAPI app run by Uvicorn. Production uses PostgreSQL and runs in Docker, with persistent state on a named volume.

The product is multi-tenant. A brand-new installation has a one-time first-start path: the first tenant signup creates the tenant and a superuser, runs the database migrations, and marks the installation as initialized. After that, the same API call refuses to run again.

2. Problem

We wanted browser tests that exercise the real front end against the real API, with no API mocks, and we wanted them to be practical to run on every developer machine. The obvious options all failed somewhere:

• Mocked APIs test the front end against assumptions instead of real behaviour.
• One shared test server gives tests hidden dependencies on one another. It also cannot test first-start behaviour more than once, because that path works only once per database.
• Ad-hoc launcher scripts, as found in our sibling repositories, had grown fragile: fixed ports, sleep(6.5) instead of a readiness signal, hard-coded machine paths, a JWT checked in that had already expired, and a schema step that silently did nothing.
• Interrupted runs (Ctrl+C, a SIGKILL, a cancelled CI job) left orphaned server processes, temp directories and Docker containers behind.
• Manual testers had no supported way to get the same clean, known starting state that the automated tests used.

3. Forces

• Fidelity vs. speed. A real browser, proxy, backend and database catch more defects. Each real component also adds start-up time.
• Isolation vs. cost. A fresh database per test is the cleanest option but costs a backend start each time. A shared database is cheap but lets state leak between tests.
• Different starting states. Some scenarios need a populated installation, some need a never-used one, and some need a single tenant with no data.
• Local vs. production topology. The local loop must start in seconds without Docker. The production-like topology (container, reverse proxy, volume) still needs coverage.
• Credentials. Tests need valid tokens, and tokens checked into the repository expire and leak.
• Abnormal termination. Teardown code does not run when the runner is killed, so cleanup cannot depend on it.
• Concurrency. Two runs on the same machine, such as a developer's suite and a manual session, must not interfere or clean up each other's resources.

4. Solution in one paragraph

Every test gets a stack: a real backend process on its own copy of a named seed database, plus a web tier (a proxy locally, the real Nginx in Docker), on dynamic loopback ports. A launcher owned by the backend repo seeds the database, starts the server, mints a token and prints one JSON ready event when it is accepting connections. A test-runner fixture starts stacks lazily, sharing one populated stack per worker and giving each test that needs a fresh installation its own stack. Each stack records its owner (host and PID), and its child processes hold a stdin lifeline, so resources from crashed runs are swept safely by the next run. The same stack managers also drive a manual session CLI, so humans explore exactly the same starting states that the automated tests use.

5. Reference architecture

5.1 Building blocks

5.2 Contracts

These contracts between the parts carry most of the design's value.

Launcher command line (in the backend repo, so every consumer uses the same one):

run_temp_db_server.py --seed {populated|empty|minimal-tenant}
                      --host 127.0.0.1 --port <free port>
                      --runtime-dir <private temp dir>
                      --exit-on-stdin-eof

Ready event. The launcher prints this line on stdout only after Uvicorn accepts connections, so the event is also the readiness signal. No sleeps and no polling loops on the backend.

{"event": "e2e-token", "seed": "populated", "token": "eyJ..."}
{"event": "e2e-token", "seed": "empty", "token": null}

Readers scan stdout for a valid event rather than reading a fixed line, so diagnostic output at import time cannot break setup. A null token is accepted only for the empty seed, and a seed mismatch is a setup error.

Stack fixture. Specs never see processes, ports or Docker:

stack is { seed, webUrl, backendUrl, runtimeDir, credentialDir, signInToken }, and baseURL is overridden to stack.webUrl, so relative URLs reach the right stack.

Owner record. Every runtime directory contains owner.json (host and PID). Every Docker container and volume carries io.<app>.e2e.owner-host, owner-pid and keep labels.

5.3 Seeds

Seeding rules:

• Seed once per runtime directory. The launcher seeds only when the database file is absent. A replaced container on the same volume therefore keeps its state, which is what the persistence tests check.
• Copy the fixture; never change it in place.
• Fail loudly. A seeding step that fails must exit non-zero. One old harness hid a failed insert in a bare except:.
• Tokens come from the backend. They are minted at start-up with the production signing code, never checked in.

5.4 Stack lifetime: shared where safe, fresh where needed

Two details matter:

• Laziness. stack must not depend on a worker-scoped populated fixture. Playwright sets up every fixture a fixture depends on, so that would start a populated backend even for empty tests. The pool's lazy method avoids this.
• Fresh database means fresh backend. The backend reads its database URL at import and keeps a module-level engine, so swapping or resetting the database under a live server is not viable. A fresh state is always a new process.

A failed test is retried in a new worker, so the retry gets a new populated stack and a clean database.

5.5 Two deployment modes, one model

The Docker mode builds the image once in global setup, so a build never happens inside a test. Each stack starts with docker compose up --no-build --wait under its own project name and a dynamic host port. Helpers let specs do what an operator would: read a file from the volume through a throwaway read-only alpine container, recreate the container on the same volume, or inspect mounts.

5.6 Manual sessions use the same stacks

npm run test:manual -- --mode local|docker --seed <seed> [--no-browser] [--video] [--keep] starts one stack and opens a managed, headed Chromium at the seed's entry URL: signup for empty, or a sign-in link for the others. A manual> prompt offers status, reset [seed], restart (Docker), logs, credentials, mint, helper, exec, screenshot and quit.

• Evidence goes to manual-results/session-<timestamp>/: a Playwright trace per browser context, JSONL console and network errors, screenshots, optional video, and stack-log snapshots. It is kept even when the stack is deleted.
• Pass/fail. The CLI gives no verdict. The tester records findings.
• Reset replaces the whole stack and the browser context. It never resets a live database.

5.7 Cleanup that survives a killed runner

Live owners are never touched, so concurrent runs are safe. --all removes everything, including kept stacks. In CI, run the sweep as an always-run post-step.

6. Results

Positive

• Real defects, not mock drift. Every spec crosses the real browser, proxy, routing, auth, ORM and database code paths.
• First-start behaviour is testable at all, and as often as needed. Each empty test gets its own never-used installation.
• Tests are isolated by design. Fresh seeds are fully isolated, and the shared populated stack is per worker, not per run. Retries start clean.
• Deterministic start-up. The ready event replaced sleeps, and dynamic ports replaced fixed ones. Flaky "server not up yet" failures went away.
• No secrets in the repository. Tokens are minted by the backend's own signing code at start-up, so they never expire in a fixture file.
• One launcher, many consumers. The web repo, a CLI client repo and the manual CLI all start the backend the same way. Fixing the launcher fixed every one of them, and three old ad-hoc launchers were retired.
• Specs stay simple. They ask for { page, stack } and pick a seed. All process and Docker details sit behind the fixture.
• Production topology is covered when it matters, without making the everyday loop slow: local mode runs in seconds, Docker mode runs nightly.
• No leftovers. Killed runs, cancelled CI jobs and concurrent sessions leave nothing behind that the next run cannot safely remove.
• Manual and automated testing share starting states. A tester can reproduce a spec's exact initial state with one command, and hands back a trace and logs instead of a description.
• Building it surfaced real defects. The harness exposed a migration that could not run after the ORM had created the schema. Designing the seeds showed that the older harnesses had silently stopped creating a schema at all.

Costs and limits

• Stand-in database. Both modes use SQLite built from the ORM, not production PostgreSQL. SQL specific to PostgreSQL and the real migration chain are not exercised. A separate migration test suite remains the authority.
• The local proxy can drift from Nginx. Header forwarding and the SPA fallback are matched, but timeouts and other Nginx behaviour are covered only in Docker mode.
• Wall time grows with fresh-seed tests, by a few seconds locally and about 15 s in Docker for each one. Use fresh seeds only where the scenario needs them.
• The shared populated stack can still leak state between tests in one worker. Specs that change data must undo their changes or pick a fresh seed.
• The launcher is a cross-repo contract. A change can break every consumer at once. The event schema and seed names are versioned by documentation and a contract spec.
• Execution is serial today. Stacks are already per worker, so raising the worker count is a separate decision, not a redesign.

The harness does not replace product specifications or engineering judgment. Instead, it makes approved behavioral expectations executable, repeatable, and available to developers, QA, and AI agents.

The test harness turns testing from an infrastructure obstacle into a reusable engineering capability.

AI Coding Agents Become Accountable to Executable Requirements

With a dependable test harness, Test-Driven Development becomes a practical control mechanism for AI-assisted engineering.

The workflow changes fundamentally. Product managers define expected behavior through specifications, business rules, and acceptance criteria. QAs, assisted by AI, turn those expectations into E2E tests before implementation and obtain approval of the tests. Only then does the coding agent implement the feature, repeatedly running the tests and correcting its code until the required checks pass.

Crucially, the agent must not silently weaken or rewrite those tests to achieve a green result.

This operating model has costs. Fast local execution may not reproduce every production characteristic: our SQLite test environments, for example, do not validate PostgreSQL-specific SQL or the complete production migration chain. Shared worker stacks also require discipline to prevent tests from contaminating one another. These limitations must be covered through complementary verification, not ignored because the E2E suite passes.

The choice is between investing in dependable feedback now and continuing to pay for uncertainty during delivery. A reliable harness does not guarantee correct requirements, complete test coverage, or defect-free software. It does, however, give engineering teams a practical way to detect observable mismatches earlier and prevent AI-generated implementations from defining their own acceptance criteria.

In AI-assisted development, the test harness is no longer just a testing tool. It is part of the system that governs what AI is allowed to deliver.

Next Step

Before scaling AI coding agents across your organization, make one engineering team demonstrate a complete test-first workflow with approved E2E tests, isolated real application environments, automated implementation, and independent verification and use that as the foundation for wider adoption.