What Happened When DoltHub Ran Ito on 43 Pull Requests

Finding one bug can be luck, but a code review tool that keeps finding real problems is a lifesaver.

That is what DoltHub saw after six weeks using Ito on Doltgres, one of their open source repos.

In the review threads where its engineers judged Ito's findings, they confirmed more than two valid bugs worth fixing for every one they chose to ignore.

They also estimated that Ito's findings led to them fixing issues before production for roughly one out of every three pull requests.

That is the kind of proof you want to grow your confidence both in using an AI code review tool, and safely merging to production.

DoltHub did the work. Go read their full breakdown.

Jason Fulghum and the DoltHub team published the full story themselves: the SQL queries Ito ran, the bugs it found, the evidence in the pull request comments, the false starts, and the numbers behind their conclusion.

We are grateful they took the time to write it. This short post gives you the headline. Their post gives you the full, unfiltered proof.

Read DoltHub's complete technical review of Ito.

What six weeks of real Ito use looked like

Ito commented on 92 Doltgres pull requests. Many were automated dependency bumps or release metadata changes, so DoltHub set those aside. That left 43 pull requests where there was real logic for Ito to run and test.

Of the 38 of those pull requests that had merged or closed, Ito reported at least one issue on 18. Eight of those showed a fix and re-check cycle directly in the pull request comments before merge.

DoltHub also counted fixes that happened later or were moved into its backlog. With those included, it estimated that Ito's findings directly led to improvements made on roughly 33% of the pull requests it could meaningfully test.

That is the outcome you want from a review tool. It should keep giving your team things worth fixing as you build new features, fix bugs, and scale your product without being too noisy.

The takeaway: Ito gave DoltHub a repeatable way to catch and fix problems before merge.

Ito found more than regressions in the changed code

In 26 of those 43 pull requests, Ito surfaced at least one issue DoltHub labeled as pre-existing and unrelated to the code in that pull request.

That does not mean every one of those issues was a confirmed bug or needed a fix. DoltHub was clear about that. It does show what happens when Ito runs your product and explore the edges around a change instead of staring only at the changed lines.

Some problems live nearby. A pull request is often the reason you finally look closely enough to find them.

The takeaway: running the code can uncover problems that a diff-only review never gets near.

The two SQL bugs show why this matters

DoltHub documented two separate bugs in SQL window functions that Ito found while reviewing one pull request. Both were deep in a shared query processor used across its database products.

The first bug returned the full sum for every row when a query used a named SQL window. It should have returned a running sum. The bug was especially nasty because DoltHub already had test coverage for the syntax. The test expected the wrong answer.

The second bug involved inheriting or overriding a named window's ORDER BY clause. It created the same wrong full-partition result through a different path.

Neither problem was obvious from the pull request diff. Ito ran Doltgres, opened a PostgreSQL connection, issued the queries, and showed the wrong output next to the correct baseline.

You can see the exact SQL and results in DoltHub's first bug write-up and second bug write-up.

The takeaway: a green test suite is only as good as the behavior it actually checks. Dolthub shows how Ito can fill in the gaps in your test suite to catch important issues.

DoltHub's results came with clear limits

DoltHub found Ito especially useful when it was testing well-documented SQL behavior and edge cases. It also said Ito was less helpful on internal performance work that did not directly change SQL behavior, and sometimes added comments that did not warrant action.

That is the honest version of this story. Ito is not a replacement for human review. DoltHub kept human reviewers in the loop, but Ito made their lives easier by providing greater test coverage than they can get from static analysis and their own alone.

Ito also got better over the 6 weeks.

DoltHub found a problem where Ito fell back to an older binary after a branch build failed, which led to false alarms. We fixed it so failed builds now fail clearly instead of testing the wrong artifact. DoltHub said it had not seen the issue again, and noted how responsive we are:

“When we’ve hit an occasional issue using the tool, the Ito team has been responsive and quick to roll out a solution to improve our experience, including making their product work well for non-web-UI products, like Doltgres.”

Jason Fulghum, Software Engineer at DoltHub

Good tooling earns trust by being useful, by showing its work, and by getting better with every bit of feedback from customers.

Try Ito on your own pull request

The fastest way to see whether Ito helps your team like we helped DoltHub is to put it on a pull request and see the evidence yourself.

Ito runs your code, tests the behavior around the change, and reports what it found right in the comments of your PR in Github. And you get 10 free reviews when you sign up below:

Try Ito free on your next pull request.