How to Build Quality Into Your Software Factory

Your board wants "more AI." Your CEO wants the team to build, "one of those software factories".

So you add coding agents, automate the backlog, and push more pull requests into your merge queue.

It works, at first. Your PR count has gone up. The roadmap is moving faster. Your reports show that your factory is producing more than ever.

Now, your quality bill is arriving. Your reviewers are skimming changes they cannot fully understand. Your engineers are manually clicking through flows before merge because they do not trust the checks. Customers find bugs that passed CI.

Your factory is producing more code, but more code is not necessarily better software.

Testing is a big part of why.

Whether it's due to a lack of testing or blind spots in the tests you have, it's often the biggest reason your software factory isn't delivering the impact you want (customer wins) instead of simply vanity metrics (increased velocity and a board off your back).

Your factory needs speed and stability

A software factory is a delivery system built for throughput. Tickets come in. Agents write code. PRs open. CI gates the merge.

Done right, it's a machine.

DORA's four key metrics measure speed and stability together: deployment frequency, change lead time, change failure rate, and recovery time. Shipping ten PRs a day only helps if those PRs do not create ten more emergencies for customers and on-call engineers.

A factory that optimizes only for throughput is going to have major quality problems.

Merge volume can rise while change failure rate climbs right alongside it. As AI coding tools push PR volumes higher, the pressure on the stability side gets worse, not better.

Faros' 2026 AI Engineering Report tracked 22,000 developers across 4,000 teams and found the strain building downstream: bugs per developer rose 54%, median review time grew fivefold, and incidents per PR more than tripled as AI adoption increased.

You cannot ask reviewers to manually click every affected path every single time. There aren't enough hours in the day, nor patience from your team.

You also cannot make every pull request wait for the perfect regression suite to run a million tests. A full regression suite on every change turns the merge queue into a traffic jam and trains your people to bypass it.

The answer is a testing stack that runs the right tests in the right order: find and fix the cheapest failures first, then run the heavier tests as your confidence in the PR grows.

What automakers can teach software teams

Toyota learned this same lesson decades ago. The Toyota Production System was developed to ensure consistent, quality cars at a high rate.

Despite the lofty goal of building millions of cars per year, it has just two pillars: Just-in-Time, which keeps production moving without waste, and jidoka, which builds quality controls into the line itself.

When a machine detects an abnormality, it stops. When a worker sees one, they can stop the line and signal the problem with an andon board. The defect is fixed at its source before more work gets added on top of it.

That is what makes it a production system, not simply a faster assembly line. Toyota did not treat quality as a final inspection after the car was built. Quality controls are part of every stage of making the car.

Software factories need the same design. More output only helps when the factory also detects failures early enough to fix them before they reach customers.

Build your testing stack from cheapest to most complete

The testing stack is the software equivalent of Toyota's quality controls. You want the cheapest checks to find the simple failures early, before they absorb more engineering time to fix or review. Then you add slower, more complete tests as the cost of a missed failure rises.

Here is a practical order. Times are planning ranges, not service-level commitments. Your codebase, test environment, and change risk will change the numbers.

The first two layers should usually happen before a pull request is ready for a teammate to review. They give the author immediate feedback and avoid spending CI time or reviewer attention on a missing import or failed branch of known logic.

Static review, integration tests, and targeted checks can then start on a draft PR, while runtime analysis and human review make the most sense once the change has the basics covered and fixed.

Human review belongs here, but it has a different job from testing. A reviewer should assess whether the team is building the right thing and whether the design is safe to maintain. They should not be asked to manually repeat every automated test. It's not a good use of their time, and it's one of the most common complaints we hear from engineers; they got into their profession to build, not be testers all day, every day.

These runtime and human tests are more expensive because they are testing more of the real system. Integration tests need services and data. Runtime analysis needs a running application, user state, and sometimes a browser. Full regression and performance testing can take even longer and require even more data and configuration.

The key is to recognize that each layer has blind spots. Lint cannot prove logic. Unit tests cannot prove that services work together. Static review cannot prove what happens when the app runs. Runtime analysis cannot cover every path. The stack works because the various layers of tests combine to cover each other's gaps, preventing bugs from reaching production.

Order matters. You should run the cheap tests broadly and first. Then, you run the expensive tests where the consequences justify them.

A passing test can still check the wrong thing

Earlier this year, DoltHub was running Ito on Doltgres, its open-source PostgreSQL-compatible database. One PR touched a shared query processor. The diff was clean. Lint passed. Static tests were green.

The problem: DoltHub already had test coverage for the exact SQL syntax Ito was about to test. The test expected the wrong answer. So it had been passing.

Ito opened a PostgreSQL connection, ran a query using a named SQL window, and compared the output against the expected running sum. The query returned the full partition sum for every row. The test said that was correct. It wasn't.

Nothing in the diff would have told you this. Static review could not flag it. A unit test that expects the wrong answer passes every time.

The failure appeared only when someone ran the product and compared its result with the behavior PostgreSQL requires.

If you'd like to go deeper on this example, DoltHub's full writeup includes the SQL queries, the wrong output, and the evidence Ito posted in the PR thread.

This is why a factory needs more than one green check. The unit test did its job as written. The runtime test exposed the job it failed to cover.

A green test suite is only as good as the behavior it actually checks. That's why having each layer matters.

Each one plays a part in giving you full coverage before you ship a change to your customers that should improve their experience, not hinder it.

Where runtime analysis belongs in your stack

Runtime analysis should run after your cheap checks pass and before your change merges.

That order matters. There is no reason to spend browser time on a PR with a type error. There is also no reason to wait until a nightly suite or a customer report to discover that a valid-looking change broke checkout.

The goal is not full-product testing on every push. The goal is to run the user paths your current change puts at risk, while you can still fix what the test finds.

Isolate each PR's environment

Shared staging breaks PR-level testing. When two PRs share an environment, their data bleeds together, feature flags conflict, and failures are often just collisions between your branches, not real bugs. That means each of your PRs needs its own environment so the signal is actually about that PR.

This is why our open source project, Moo, ties agent runtime isolation directly to git worktrees. A Playwright suite in shared staging can be useful, but it is a snapshot of staging. It is not reliable evidence about one specific PR. It's also why Ito spins up a separate environment for every PR we test.

Scope to the risk in the diff

Running the full product on every PR turns this layer into a bottleneck. Pricing logic changes warrants checkout flow tests. A copy change does not.

Code-aware scoping means you reserve the slower set of checks for when a change is large or risky enough to deserve a deeper set of testing. Simple, low-risk changes get released faster by skipping some of these.

Assert user outcomes, not just DOM state

"Button visible" tells you the button rendered. "Order persisted and confirmation shown" tells you the user finished your flow. Proper testing should care about the job the user came to do, not just that a screen loads at all.

Testing that an element exists is testing the UI layer. Testing that the behavior is correct is testing the product. To ship with confidence, especially at AI speeds of development, you need both.

Make your PR thread the source of truth

A runtime check that posts a red X and nothing else trains your engineers to ignore it. Useful evidence includes the flow name, the steps taken, what the app returned, a reproduction path, and screenshots or video.

Whatever evidence you have, it needs to be clear, easy to understand, actionable so you fix it, and located in a place that's easy to reference and share.

This matters even after you merge. When an agent loop runs out of control, or a customer escalates a bug three weeks later, the PR thread is where you reconstruct what happened.

Screenshots, logs, and reproduction steps in the PR thread create an audit trail the diff alone can't give you. GitHub Actions, checks, and status APIs all have hooks to automate this. Use them.

A well-instrumented PR thread becomes a debugging tool, a quality record, and an agent feedback loop all at once.

Make your test layers work together

The testing stack works best when layers have clear boundaries and clear owners.

A reasonable pull request sequence looks like this:

• First: lint, types, and unit tests stop cheap failures within seconds or minutes.
• Next: static review examines the diff for risks those checks do not model.
• Then: integration and selective runtime code review test the services and user paths the change can break.
• Finally: scheduled or pre-release suites look for the broad failures that are too expensive to run on every PR, but are essential to prevent bugs reaching production.

You can run some of these in parallel. The important decision is your budget. Do not make a browser check block every typo. Do not let a sensitive billing change skip a browser check because the unit suite happened to pass.

Start new runtime checks as advisory findings, not blocking. Engineers need to trust a signal before it has veto power over their merges.

Run it for a few weeks, fix flakey tests, improve your coverage of critical areas, then promote the reliable checks into your merge gates.

Treat these flakey tests like infrastructure debt. If the runtime station goes noisy, engineers route around it. Someone has to own flake burn-down the way a platform team owns flaky CI.

A testing layer that engineers cannot trust will not protect the factory.

Where Ito can help you

Ito supplies the selective runtime layer for teams that do not want to build and maintain that scaffolding themselves.

On each PR we review, Ito spins up an isolated environment, runs the user flows most likely affected by the diff, and posts the evidence in the GitHub review thread: what it tested, what it found, screenshots, reproduction steps, and logs. It works beside your linting, unit tests, integration tests, and static review tools.

The DoltHub result is the most detailed public picture of what this looks like on a real codebase over time.

Conclusion

Your software factory runs on throughput. But throughput without a working product is debt accumulation at scale.

What your board / CEO is asking for is proof that all the AI investment is leading to real gains. That comes from not just shipping more faster, but doing so with quality in mind.

Unfortunately, manual testing tires and frustrates your engineers while slowing down your factory. Meanwhile, waiting and fixing bugs in production hurts your customers and burns out your team who is on-call.

The happy medium is building a complete testing stack: catch the broad, cheap failures first, then spend more effort on the remaining risks that survive.

Lint, unit tests, static review, integration tests, runtime analysis, and deep regression suites each serve a purpose. The right order and knowing when to use each test keeps costs down. The complete stack keeps bugs from escaping through the blind spots between layers.

That is how you build a factory that ships fast and keeps working as you scale.