# Ito — Full Content > Generated: 2026-10-26T01:53:39.642Z --- # Home ## Catch all runtime bugs before merge. Ito is the only AI code review tool that builds your app on every PR, runs the impacted flows, and attaches the failing test to anything that breaks. NEW The validation layer for AI-speed development Get Started no credit card required Watch 2 min demo *Trusted by engineering teams at* SOC 2 Type II compliant ## Ship faster than ever without sacrificing quality ### Catch runtime errors no one else can Service authorization, concurrency, failure handling, data migration, authentication, and endless more. ### Detailed proof on every finding Thorough runtime evidence on every pull request: video, logs, reproduction steps, re-run on fix, prompt to fix ### Removes the worst 3 hours of your week Automate the time each developer spends manually verifying AI code for runtime issues, with evidence ## Ito tests everything your app actually does when it runs Ito builds and runs your app from source and tests the frontend and backend as one system. It catches the bugs and broken flows that only show up at runtime, when both sides run together and real requests move through the app. ### Backend testing The UI can look right while the logic behind it is broken. Ito runs the real code behind your change, your business logic and database writes, and checks that what actually happens matches what the change intended. ### Frontend testing Ito navigates your app like a real user, catching UI regressions before they reach a reviewer. When something breaks, you get a video replay and full evidence trail with everything you need to fix it. how ito works Ship fast, test automatically ## Targeted test plans - When a PR opens, Ito reads the diff and the PR description, then focuses its testing on the user flows your change affects - It builds the plan on its own, no test cases to write and no suite to maintain - Add plain-English test instructions to prioritize the scenarios that matter and skip the ones that don't [See how it works →](/features/targeted-test-plans) ## Containerized test execution - For every PR, Ito builds and runs a real, single-use copy of your app from source - Ito navigates your app like a real user, and every action runs your real backend code, surfacing the logic and data bugs code review can't catch by reading the diff - Beyond the happy path, it runs the edge cases and adversarial scenarios that real-world use exposes [See how it works →](/features/containerized-test-execution) ## Full findings, right in your PR - Every PR gets a complete test summary: which flows passed, which failed, and exactly what Ito did to find out - Every failure includes everything you need to fix it fast: a video replay, the exact lines responsible, and steps to reproduce. - Severity ratings tell you what to tackle first, so you're not guessing what matters most - Push a fix and Ito automatically re-runs to confirm it's resolved [See how it works →](/features/evidence-rich-results) Agents are writing (and using) all the code. We need verification that changes behave as expected, and that there are no loopholes for agents to exploit. Every other tool reads the code and reasons about the diff. We built Ito to run it first. The problem: review without execution: PR volume from coding agents has outpaced what any team can verify by hand. Merge on faith or become the bottleneck. AI reviewers post confident comments, but a comment is an assertion. Nobody knows which are real until you reproduce them. Bugs that reach production are ones a diff can't show. They only exist when the code runs. The solution: execution as infrastructure: On every PR, Ito builds your app from source and exercises the entire flows your changes touch. Any issues come with evidence: video, logs, and reproduction steps on the exact lines of code. Push a fix and Ito re-runs until it passes. Merge with confidence. > "Nothing in the code diff in the PR that Ito reviewed had any direct sign of this bug. If Ito had only been looking at the diff lines from the PR, then it wouldn't have caught this." > > Jason Fulghum, CEO at DoltHub > "Ito helps my team ship faster by testing all edge cases right on their PRs. I refuse to go back to manual testing." > > John Gazzini, CTO at Truemed > "Ito caught 3-4 real things that got fixed in the first week. Impressed that backend mocking works autonomously." > > Dan Kokotov, CTO at CNaught > "I was very impressed with the zero configuration and all the different pathways it was testing. The computer use capability definitely sets it apart from other tools." > > Andrew, Founding Engineer at Inkeep > "Automating runtime checks lets my agents automerge without adding a code review bottleneck. We're shipping faster than ever. " > > Josh Dong, Founder at Temi Security-First Design Built for enterprises across financial services, healthcare, and defense. Architected for Security: SSO, audit logs, dedicated support. Isolated Test Runs: End-to-end encryption with zero data retention. SOC 2 Type II: Independent audits, reports available on request. ## Catch the runtime errors your suite can’t see. Every PR comes with a runtime report. Review cycles compress. Reviewers focus on quality, not verification. Merges happen the same day. ### 81% bugs found are unique to runtime ### PRs verified in hours, not days. ### 50% fewer regressions. ### 10x more coverage. No extra headcount. ## Questions teams ask before using Ito. Common questions from engineering managers, senior engineers, and QA leads. **Q: What is Ito?** A: Ito is the only code review tool that runs execution-based behavioral tests on every pull request to provide scriptless end-to-end validation and automated QA reports. **Q: How does Ito test pull requests?** A: Ito deploys your application in an isolated, single-use sandbox and uses computer-use agents and deep browser inference to execute user flows and detect behavioral regressions before code is merged. **Q: Does Ito access our source code?** A: Ito never stores your code. The AI Code Review runs secure code execution in an isolated, single-use sandbox environment, and the company is SOC 2 Type II compliant. **Q: How long does setup take?** A: Setup is autonomous and scriptless; you connect your GitHub repository and Ito maps your application logic and begins testing within minutes. **Q: How does Ito handle authenticated user flows?** A: Ito supports test environment credentials, including multi-factor authentication, and the QA Agent manages sessions automatically to validate behavior like a real user. **Q: Which tech stacks does Ito support?** A: Ito is framework agnostic and works with any stack, including React, Vue, Next.js, Rails, and Django. **Q: What does the testing report include?** A: Each pull request includes a full testing report with video, screenshots, and failure details delivered directly in the PR for faster review and triage. **Q: What outcomes can teams expect using Ito?** A: Customers report PRs verified in hours, faster shipping velocity with around 30% more features per sprint, 70% fewer production regressions, and dramatically increased QA coverage. **Q: How is Ito different from code review tools like Cursor's bugbot, Coderabbit, or Greptile?** A: Ito is an execution-based code review tool. While static analysis catches syntax or code-level issues, Ito catches behavioral regressions by actually running your app. It finds the runtime bugs, like broken UI logic or failed API integrations, that static tools cannot catch. **Q: How is Ito different from testing frameworks like Playwright or Cypress?** A: Playwright and Cypress are automated QA frameworks that require engineers to write and maintain brittle E2E test scripts. When your UI changes, those scripts break. Ito is the evolution of automated testing; it generates test cases from your app’s actual behavior. It provides scriptless end-to-end validation that adapts automatically as your UI evolves, eliminating the "maintenance tax." ## Ship with total behavioral certainty. Connect your repo and Ito starts testing pull requests right away. Each PR includes a full runtime report with video, screenshots, and failure details directly in the PR. GET STARTED no credit card required --- # For QA teams ## Code review that actually gives you useful artifacts. Ito catches runtime bugs with evidence so your team can focus on strategy. Reduce manual QA and shift from running tests to designing them with autonomous QA on every PR. FOR QA teams Get Started no credit card required Watch 2 min demo ## Zero-config onboarding. Test immediately. ### 60% QA time spent on repetitive regression. ### Under 40% Post-merge bugs caught manually. ### 3x More PRs than QA can manually cover. ## You hired experts. Not clickers ### The regression treadmill Every release, the same 50 flows: login, checkout, forms. Your team could run them blindfolded. That’s the problem. ### Coverage they can’t reach They know what should be tested. There just aren’t enough hours. So they triage, prioritize, and hope what gets skipped doesn’t break. ### Undervalued expertise They’re treated like test runners instead of test designers. Strategy gets squeezed out by urgency. ## QA automation software that keeps you in control. Automated QA built for QA teams. You direct the agent—it handles the grunt work of behavioral testing. ### PR opens, Ito runs Every pull request triggers automated QA. No manual kickoff required. ### Visual results, posted Screenshots, video test reports, and logs land in the PR so you can see exactly what happened. ### You review and decide Ito flags issues. You judge impact, spot false positives, and choose what to dig into. ### Focus on what matters With regression covered, spend time on edge cases, exploratory testing, and new feature validation. ## The power of behavioral testing. ### Every PR tested No more “we’ll test it in staging.” Every change is validated with automated pre-merge testing. ### Parallel execution Tests run across browsers and viewports simultaneously. What used to take hours with manual testing now takes minutes. ### Consistent execution No skipped steps. No “I thought someone tested that.” Our QA agent runs the same, thorough tests every time. > "Ito helps my team ship faster by testing all edge cases right on their PRs. I refuse to go back to manual testing." > > JOHN GAZZINI, CTO at TRUEMED > "I was very impressed with the zero configuration and all the different pathways it was testing. The computer use capability definitely sets it apart from other tools." > > ANDREW, FOUNDING ENGINEER AT INKEEP > "Ito caught 3-4 real things that got fixed in the first week. Impressed that backend mocking works autonomously." > > DAN KOKOTOV, CTO AT CNAUGHT > "Power of manual QA on every PR lets my team ship faster without a quality tradeoff." > > JOSH DONG, FOUNDER AT TEMI ## What teams ask before using Ito. Common questions from engineering managers, senior engineers, and QA leads. **Q: How do I see exactly what Ito tested and verify it covered what mattered?** A: Every behavioral testing session produces a full audit log: video test reports, screenshots, and test failure reproduction steps. This is accessible in the Ito dashboard per PR and also linked directly from the PR comment. **Q: How does Ito handle authenticated flows, test data dependencies, and multi-step state?** A: Ito’s QA automation tool uses what you give it. With zero config, it mocks external dependencies (payment processors, email, analytics, etc.) that would otherwise cause intermittent failures. We don't want them to get in the way of testing the code changes. You have the ability to add credentials to any services, environment secrets and variables, and any other seed data used in regular testing. **Q: Can I add my own test scenarios?** A: Yes. While Ito provides scriptless end-to-end validation, you can direct the QA copilot using natural language. Just describe a flow — like "apply promo code and checkout" — and the agentic QA platform executes it. It gives you the control of regression testing automation tools without the manual coding. **Q: How is Ito different from my E2E test suite?** A: Traditional end to end testing tools like Playwright or Cypress are "instruction-based"—if a CSS class changes, your scripts break. Ito provides scriptless end-to-end validation that adapts as your UI evolves. That’s what makes Ito a Playwright alternative for the "high-churn" surface area of your PRs, while you can keep your existing suite for "frozen" core flows if desired. There is no forced migration, but most teams do choose to retire parts of their maintained suites as Ito's coverage matures. ## Ready to ship faster? Join other QA teams reclaiming their time with Ito’s QA agent. Get started for free no credit card required --- # Pricing ## Full Coverage on Every PR Ito catches regressions and runtime bugs before merge by testing all of your PRs in a real environment before release. FOR Engineering teams Get Started no credit card required Watch 2 min demo ### Pro For startups and small teams $40/mo - Unlimited Repositories - 50 Code Reviews included per seat - $1 per additional Code Review - Unlimited Read-Only users - Create Custom Rules - Team Insights & Analytics ### Enterprise For teams of 25 + Engineers Custom - All Pro Plan Features - Security and compliance - Technical enablement and dedicated support - Custom invoicing and payment terms - Custom DPA and terms of service - Higher limits across the product ### Open Source Free for qualified non-commercial projects with MIT or Apache licenses Free - Runtime analysis on every Pull Request - Catch regressions from community contributions - Video & screenshots on each run - Unlimited public repositories - 1-click GitHub install - Dedicated long-term support What teams do today Hire more QA engineers: $120K+/year per hire. Slow to scale. Bottleneck before your releases. Write & maintain Playwright: Engineers spend >10% of their time writing and maintaining tests instead of shipping features for your customers. Manual review on every PR: Eats 1–2 hrs/day per engineer. Dreaded by your engineers. Skipped before deadlines. Ship and pray: On-call fires. Customer-reported bugs. Trust erodes. SLAs at risk. Coverage on every PR Autonomous scale: The equivalent of a dedicated QA hire for every dev. Scales instantly without adding to your payroll. Zero maintenance testing: No selectors to manage or scripts to fix. Adapts to UI changes so your tests stay current automatically. Instant behavioral feedback: Get PR results within 60 minutes of install. Automated video, logs, and screenshots posted inline. Pre-merge certainty: Catch regressions before they ship. Validates every unique user role and flow in a real browser to protect your users. ## FAQs Everything you need to know about billing and plans **Q: How does Ito save my team time?** A: Ito replaces the manual regression layer of QA. This is the part where someone, usually an engineer reviewing another engineer's PR or a test engineer, clicks through flows on every PR to make sure nothing broke. This is roughly 1-2hrs per day of manual work that Ito automates. Strategic QA, exploratory testing, and release planning still belong with humans. **Q: Does Ito merge code? ** A: No, but many customers use Ito as a critical signal that leads into merging code. We provide a critical signal of runtime verification that is hard to replicate and ensures software factories don't introduce bugs or regressions. **Q: What is "runtime analysis"?** A: Ito runs your actual product in a real browser and exercises it the way a user would: log in, navigate, fill forms, submit, verify state. We're testing behavior end-to-end as the code runs, not just looking at source code. **Q: What happens when Ito finds a bug?** A: Ito posts a GitHub comment on the PR sharing the exact lines of code with the issue, the severity of the issue, evidence of why it's an issue with a description, reproduction steps, and a simple prompt to fix it. Your team triages it like any other review comment. **Q: How long does a PR test take?** A: Reviews typically take 30 minutes to 2 hours, depending on the complexity of the change and the codebase. It is thorough testing on top of your actual running application: exploring flows, retrying when things are flaky, recording every step. We optimize for thoroughness. Many customers use Ito as an input to automerge and complete their software factories. **Q: What kinds of apps can it test?** A: Any tech stack for web apps and APIs. Native mobile is on the roadmap. **Q: How is it different from Playwright or Cypress that we'd write ourselves?** A: You don't write or maintain anything. Ito figures out the flows from your product and your PR. No selectors that break when you rename a button. No suite that goes stale. **Q: Do we have to write or maintain tests?** A: No. You can tell Ito what you want it to test, but it's not necessary. **Q: How long until it's running on our repo?** A: About 60 minutes from install to first PR tested. Any admin on your GitHub organization can set up Ito to run on their PRs. Activate other team members to try it on more of your PRs. ## Get started today Free for your first 20 reviews. Try Ito Free Get your first results in less than 60 minutes --- # Why Ito ## Code review that actually catches everything. The world handed engineering teams AI that writes code faster than ever, but left us with a legacy review process. More code, but the tools to validate that code haven't evolved fast enough. Ito bridges the gap. ITO Why we built Ito. Get Started no credit card required Watch 2 min demo ### The problem: the code review tax. Your engineers generate PRs at AI speed, but before they can merge, the flow stops. Someone has to manually verify the changes, fill out forms, and hunt for edge cases. In practice, this looks like: - **Wasted time -** Engineers spend an average of 3 hours per week on manual verification. For a 20-person team, that’s 60 hours weekly—1.5 FTEs doing nothing but clicking buttons. - **Sacrificing the details -** Because manual ROI is so low, teams only test the "important stuff" and ship the rest, hoping for the best. Existing code review tools just look at syntax, not real regressions. - **Constant maintenance -** Traditional QA with Playwright or Selenium doesn't solve this; it just shifts the burden from manual clicking to constant test maintenance. **Manual testing is technical debt pretending to be a process. To fix this, your team doesn’t need more discipline, they need the right infrastructure.** ### The solution: agentic infrastructure. We believe repetitive runtime testing should be invisible infrastructure, not a team activity. Ito introduces a new category: behavioral code review powered by agents. This looks like: - **Real browser execution -** Unlike static tools, Ito acts as a review copilot, running your app in an isolated environment to validate exactly how a user interacts with it. - **Pre-merge testing -** Testing shouldn't be a "gate" you reach at the end. It should be a continuous pre-merge activity that happens the moment a PR is opened. - **Scriptless end-to-end validation -** Our autonomous code review agents understand your application logic, so Ito doesn't break when a CSS class changes. - **Visual proof -** Every test generates visual bug reports with video, logs, and reproduction steps, eliminating the "it works on my machine" excuse. ### Our core principles. - **Testing should just work.** - No standup discussions or assigned test tasks. No checking if someone remembered to manually verify changes before merge. - **Speed and quality aren't opposites.** - They're just incompatible with current team processes. Engineers should be able to move quickly and innovate without worrying about missed bugs. - **Testing expertise is underutilized, not unnecessary.** - The best engineers shouldn't be clicking through login flows. They should be thinking about failure modes nobody else considered. - **Every commit should be production-ready by default.** - Not aspirationally. Not “if we had infinite time.” Actually ready. Right now. ### What Ito handles (so you don’t have to). Our AI code review is built for the "hard stuff" that other code review tools fail to catch: - **Observability with runtime evidence -** Video, logs, and more to ensure code behaves as expected. - **Environment parity -** Using ephemeral test environments to ensure clean, isolated runs. - **Data seeding -** Reaching specific application states automatically. - **Mocking -** Seamlessly handling external services and APIs. - **Complex auth flows -** Multi-factor and stateful login sequences. ### Your future with Ito. Ito reduces bugs in your codebase and helps your team to ship faster. We want to remove your manual testing bottleneck for good - **Reviewers focus on quality**, not verification. - **PMs stop clicking**, and start shipping. - **Bugs are caught pre-merge**, not in production. - **You can ship on your time**, not when someone has time to test. ## Your first PR tested within 60 minutes. Connect your repo and Ito starts testing pull requests right away. Each PR includes a full QA report with video, screenshots, and failure details directly in the PR. --- # Security ## Security at Ito Security is foundational to how we build and operate Ito. We are currently certified for SOC 2 Type II and undergo regular independent audits. This page describes our security program and how to report issues to us. Get Started no credit card required Watch 2 min demo ## Reporting a vulnerability If you believe you've found a security vulnerability in Ito, please email [**security@ito.ai**](mailto:security@ito.ai). Include: - A description of the issue and its potential impact - Steps to reproduce - Any proof-of-concept code or screenshots We'll acknowledge your report, investigate, and keep you updated on remediation. We ask that you give us a reasonable window to address the issue before public disclosure, and that you avoid privacy violations, data destruction, or service disruption while testing. ## Compliance Ito is SOC 2 Type II compliant. The report is available upon request: [security@ito.ai](mailto:security@ito.ai). ## Data protection - **Encryption in transit.** All traffic between users and Ito is encrypted with TLS 1.2 or higher. - **Encryption at rest.** Customer data is encrypted at rest using AES-256. - **Secrets management.** Credentials, API keys, and tokens are stored in a managed secrets service and never committed to source control. - **Data isolation.** Customer data is logically segregated, and access is scoped per installation. ## Infrastructure Ito runs on AWS in the United States. We rely on AWS for physical security, environmental controls, and hardware lifecycle management. Our production environment uses: - Private networking with restricted ingress - Managed, encrypted database and cache services - Container-based deployment - Infrastructure defined as code and peer-reviewed before changes ship ## Access control - Access to production systems requires SSO with mandatory multi-factor authentication. - Access is granted on a least-privilege basis and reviewed periodically. - Production access is logged and auditable. - Employee accounts are de-provisioned promptly upon offboarding. ## Application security - All code changes are peer-reviewed before merging. - Static analysis, dependency scanning, and secret scanning run in CI. - Dependencies are kept up to date, and known vulnerabilities are triaged on a defined SLA. - Authentication uses industry-standard OAuth flows; sessions are stored server-side. ## Monitoring and incident response - Production systems emit structured logs and metrics to centralized observability tooling. - We have a documented incident response plan with defined severity levels, roles, and communication procedures. - Customers affected by a security incident will be notified without undue delay. ## Employee security - All employees complete security and privacy training at onboarding and annually thereafter. - Background checks are performed where legally permitted. - Employees are bound by confidentiality obligations. ## Contact For security questions, reports, or to request our subprocessor list or SOC 2 Type II audit, please email [**security@ito.ai**](mailto:security@ito.ai) ## Your first PR tested within 60 minutes. Connect your repo and Ito starts testing pull requests right away. Each PR includes a full QA report with video, screenshots, and failure details directly in the PR. --- # Contact ## Talk to our sales team. We'd love to hear from you. Send us a message and we'll respond as soon as possible. - ZERO-CONFIG ONBOARDING - THOROUGH TESTING FOR EACH PR - VISUAL PROOF FOR ALL TESTS AND EDGE CASES ## Trusted by the best engineering teams. > "Ito helps my team ship faster by testing all edge cases right on their PRs. I refuse to go back to manual testing." > > JOHN GAZZINI, CTO AT TRUEMED > "I was very impressed with the zero configuration and all the different pathways it was testing. The computer use capability definitely sets it apart from other tools." > > ANDREW, FOUNDING ENGINEER AT INKEEP > "Ito caught 3-4 real things that got fixed in the first week. Impressed that backend mocking works autonomously." > > DAN KOKOTOV, CTO AT CNAUGHT > "Power of manual QA on every PR lets my team ship faster without a quality tradeoff." > > JOSH DONG, FOUNDER AT TEMI ## Your first PR tested within 60 minutes. Connect your repo and Ito starts testing pull requests right away. Each PR includes a full QA report with video, screenshots, and failure details directly in the PR. --- # Front End Validation ## See your front-end actually work before you merge. Ito opens every PR in a real browser, clicks through your app like a user, and hands you back a video of the feature working (or failing). No scripts to write. No suite to maintain. Just proof. NEW The validation layer for AI-speed development Get Started no credit card required Watch 2 min demo *Trusted by engineering teams at* ## Ship faster than ever without sacrificing quality ### Kill the manual click-through Reclaim the 3+ hours a week you spend clicking through your own flows to make sure the AI-written code actually renders and works. ### Video proof on every PR Every front-end change comes back with a screen recording of the feature in use, so you can see it work instead of guessing from the diff ### Eliminate fire drills in production Catch UI regressions, broken modals, dead buttons, blank states, before they reach a reviewer or production. ## Front-end bugs only show up when the app actually runs Ito drives your app in a real browser — filling inputs, clicking through flows, hitting edge cases — and records it. When something breaks, you get a video replay, the exact lines responsible, and steps to reproduce. When it passes, you get a video of it working. ### Frontend testing Ito navigates your app like a real user, catching UI regressions before they reach a reviewer. When something breaks, you get a video replay and full evidence trail with everything you need to fix it. ### Backend testing The UI can look right while the logic behind it is broken. Ito runs the real code behind your change, your business logic and database writes, and checks that what actually happens matches what the change intended. how ito works Ship fast, test automatically ## Targeted test plans Opens a PR, reads your diff, and figures out which screens and flows your change touches. No test cases to write, no suite to babysit. [See how it works →](/features/targeted-test-plans) ## Containerized test execution Spins up a real, single-use copy of your app and uses it like a person would. Every click runs your real code, so it catches the broken UI logic a code reviewer can't see by reading the diff. [See how it works →](/features/containerized-test-execution) ## Full findings, right in your PR A video replay of each flow, severity ratings so you know what to fix first, and a re-run that confirms your fix the moment you push it. [See how it works →](/features/evidence-rich-results) Why we built agentic infrastructure AI writes your front-end at terminal velocity. You're still clicking through it by hand. The problem: the manual testing tax: Every AI-generated UI change still lands on you to manually verify in the browser. The "automated" alternative — Playwright, Cypress — breaks every time the UI changes, so the scripts rot and you go back to clicking. The solution: A closed dev loop: Ito is scriptless. It learns your app from how it actually behaves and adapts as your UI evolves, so you automatically get a real browser walkthrough on every PR. > "Ito helps my team ship faster by testing all edge cases right on their PRs. I refuse to go back to manual testing." > > John Gazzini, CTO at Truemed > "It actually uses the UI and tests this stuff. When it passes, I see it in the video. When it fails, I see exactly what didn't work." > > Dan Kokotov, CTO at CNaught > "I didn't even know we could do this. It actually runs the code and makes changes on the fly. Way above what I was expecting." > > Josh Dong, Founder at Temi > "I was very impressed with the zero configuration and all the different pathways it was testing. The computer use capability definitely sets it apart from other tools." > > Andrew, Founding Engineer at Inkeep ## See every front-end change work before it ships. Every PR comes with a report. Review cycles compress. Reviewers focus on quality, not verification. Merges happen the same day. ### A video of every flow, on every PR ### PRs verified in hours, not days ### 10x more UI coverage. No extra headcount. ### 70% fewer production regressions. ## Questions teams ask before using Ito. Common questions from engineering managers, senior engineers, and QA leads. **Q: How is Ito different from testing frameworks like Playwright or Cypress?** A: Playwright and Cypress are automated QA frameworks that require engineers to write and maintain brittle E2E test scripts. When your UI changes, those scripts break. Ito is the evolution of automated testing; it generates test cases from your app’s actual behavior. It provides scriptless end-to-end validation that adapts automatically as your UI evolves, eliminating the "maintenance tax." **Q: How is Ito different from code review tools like Cursor's bugbot, Greptile, or Coderabbit?** A: Those tools rely on static analysis to study your code. Ito is an execution-based QA Agent. While static analysis catches syntax or code-level issues, Ito catches behavioral regressions by actually running your app. It finds the "runtime" bugs—like broken UI logic or failed API integrations—that static tools simply cannot see. **Q: How does Ito handle authenticated user flows?** A: Ito is designed for complex behavioral user flows. It supports test environment credentials for flows that require multi-factor authentication or secure logins. You provide the credentials, and the QA Agent handles session management automatically, validating your application’s behavior just like a real user. **Q: How long does setup take?** A: We offer an autonomous, scriptless setup. Connect your GitHub repository and Ito’s Agentic QA platform maps your application logic instantly. It creates code-aware testing plans with no manual setup required, though you can always add custom variables, secrets, and test data for deeper validation. **Q: Does Ito access our source code?** A: Ito never stores your code. Our QA Agent deploys your application in an isolated, single-use sandbox environment to perform secure code execution. We test rigorously using computer-use agents and are currently pursuing SOC 2 compliance to ensure enterprise-grade security for every pull request test. **Q: Do I get a video of every test?** A: Yes, a replay on pass and fail for front-end flows. You get runtime evidence/logs where a video doesn't apply. ## Watch your next PR test itself. Connect your repo and Ito starts running your front-end on every pull request. Video, screenshots, and failure details right in the PR. GET STARTED no credit card required --- # For engineering teams ## Only AI code review that catches runtime issues. Ito provides runtime evidence for teams that move too fast for manual bottlenecks. Comprehensive testing on every PR with no test writing or infrastructure maintenance. FOR Engineering teams Get Started no credit card required Watch 2 min demo ## Zero-config onboarding. Test immediately. ### 3 hrs Spent on manual QA every week, per engineer. ### Under 40% Of post-merge bugs are caught manually. ### 10% to 20% Of engineering time goes to end to end testing maintenance. ## You’re having to make the impossible choice between quality and speed. ### Hiring QA engineers Adds headcount, not velocity. Manual testing doesn't scale. You've traded one coordination problem for another. ### Building E2E infrastructure Playwright, Selenium, CI pipelines. Senior engineers pulled off product work. Ongoing maintenance. Flaky tests. Slow erosion of trust. ### Accepting slower releases Every extra day in the release cycle is a day your competitors are shipping. Thoroughness is good, but not if it’s costing you the market. ## Reclaim your team’s time and velocity. Ito doesn’t just add another layer to your stack it removes the work that’s halting your delivery cycle. ### Instant PR verification Every PR is tested the moment a branch is opened. Every change is verified before review begins. ### Empowered PMs PMs can focus on vision and strategy, instead of spending afternoons clicking through flows before every release. ### High-leverage engineering Senior talent is freed up to build core features and innovate, instead of wasting time verifying their own changes. ### Total behavioral certainty Your team can shift from "firefighting" post-merge incidents to preventing production regressions by default. ## What clients are saying about Ito’s QA automation for fast shipping teams. > "Ito helps my team ship faster by testing all edge cases right on their PRs. I refuse to go back to manual testing." > > JOHN GAZZINI, CTO at Truemed > "I was very impressed with the zero configuration and all the different pathways it was testing. The computer use capability definitely sets it apart from other tools." > > ANDREW, FOUNDING ENGINEER AT INKEEP > "Ito caught 3-4 real things that got fixed in the first week. Impressed that backend mocking works autonomously." > > DAN KOKOTOV, CTO AT CNAUGHT > "Power of manual QA on every PR lets my team ship faster without a quality tradeoff." > > JOSH DONG, FOUNDER AT TEMI ## What teams ask before using Ito. Common questions from engineering managers, senior engineers, and QA leads. **Q: How long does setup take?** A: Get started in minutes. Zero-config required. Connect your repository, open a PR, and Ito immediately starts analyzing changes and tests every pull request. **Q: How does Ito know which flows to test?** A: Ito creates a code aware test plan based on your codebase, PR changes, and PR description. Unlike static analysis tools, our QA agent knows what surface areas were affected by changes and likely failure patterns based on our QA experience. **Q: Does Ito need a staging environment?** A: No, Ito creates its own secure, single-use ephemeral test environments environments for each PR. **Q: How does Ito avoid becoming flaky noise in every PR?** A: This is where most E2E tools fail. Ito runs in a managed, deterministic browser environment with automatic service mocking for third-party dependencies (payment processors, email, analytics, etc.) that would otherwise cause intermittent failures. **Q: How does Ito work with third party services?** A: Ito uses what you give it. With zero config, it mocks external dependencies (payment processors, email, analytics, etc.) that would otherwise cause intermittent failures. We don't want them to get in the way of testing the code changes. You have the ability to add credentials to these services. ## Ready to ship faster? Join 100+ engineers reclaiming their time with Ito’s QA agent. --- # The Test Tax ## Trust your code is ready. Without spending your week proving it. Every PR gets the scrutiny of a full manual QA pass, without asking your engineers to be the ones doing it. Get Started no credit card required Watch 2 min demo *Trusted by engineering teams at* ## The afternoon it broke anyway Three weeks ago, an engineer shipped a small change to one API endpoint. He mapped out every place in the app the change could touch, tested each one himself, and merged it. Two days later, it broke, on a path he had not thought to check. He was not careless. He was fast, and fast is exactly the problem. The more your team ships, the less time anyone has to click through every path a change might touch, and the tax comes due exactly where nobody was looking. **Ito looks there before you merge**, so you get your week back, and you get to trust the PR is actually ready, not just quickly reviewed. ## Why this keeps happening to good teams You know how this goes: change a button used in fifty different places, and it looks right in the five you happened to check. The sixth is where the button's label runs long and the text cuts off. Ito catches these blind spots for you: a video file that played back perfectly on every MacBook in the office broke the moment a teammate opened it on a Windows machine, because nobody tested the file outside macOS. Neither bug came from careless engineering. Both came from the same limit every team hits: you can only click through as many paths as you have hours in the day, and the one you skip is too often the one that breaks. Yet, it's not about how the code got written. Even for teams where most of their codebase today is still hand-written, not AI-generated, the test tax still shows up there just the same. **The problem was never who wrote the code.** It is balancing how much you have to test to truly merge with confidence, and what's worth your time to test by hand. Ito vs. a bug bot on the same PR A 10-person Series A engineering team already runs a code review bot on every pull request, and it does catch real problems. But when its CTO compared it against Ito, the difference was not close. Reading code and running it are different disciplines. One tells you the code looks right. The other tells you it works. Reads the diff - Reviews your change as text - Flags problems it can infer from the code - Never actually runs the app - Tells you the code looks right Runs the app - Clicks through the same flow a QA engineer would - Surfaces issues a review bot never flags - Runs your change before a customer can - Tells you the code actually works ## Your judgment, encoded once Two engineering leads asked us for almost the same thing. One wanted the equivalent of IDE warning settings for their PRs: tell Ito which classes of issues are a blocking problem and which ones are fine to ignore, the same way an IDE lets you silence a warning you've already decided doesn't apply to you. Another asked for the same control, unprompted, on a separate call: guide the feedback once, tell it what you don't care about, and have it remember. So that's what Ito does. **Tell it what actually matters on your codebase**: security, access control, whatever keeps you up at night, **and what to ignore**, once. Ito keeps that in mind on every PR after that, without you re-explaining it every time. > "Ito helps my team ship faster by testing all edge cases right on their PRs. I refuse to go back to manual testing." > > John Gazzini, CTO, Truemed ## Questions, answered. What the test tax actually is, and how Ito fits alongside the process you already run. **Q: Does this replace our QA process?** A: No. It clears the manual verification work your engineers are already doing before QA ever sees the PR, and can extend into your release-candidate or regression testing once you trust it there too. **Q: Do we need to be shipping a lot of AI-generated code for this to matter?** A: No. If your engineers hand-write every line and still spend an afternoon clicking through flows before merging, this is for you, too. **Q: How is this different from a code review bot we already use?** A: A review bot reads the diff. Ito runs the code the same way a QA engineer would, and catches what static review tools can't see. ## Stop paying the test tax. Get your afternoons back. Get Started no credit card required, cancel anytime --- # Sandboxes for Coding & Security Agents ## Fully built, managed sandboxes for your coding and security agents. Running and connected to all of your services. No maintenance required. Easily add your coding agents to build and test. Get Started Watch 2 min demo ## Access sandboxes that are always ready. Stop wasting time setting up your environment. Access sandboxes that are consistent and high fidelity every time. ## How it works. Set up once: we maintain your builds, integrations, and services. ### Step 1: Your sandbox, built from scratch. On demand, Ito deploys your application in a clean, single-use container built specifically for your agent. Isolated from every other run, no state carried over. The container runs your actual code, not a mock or a snapshot. ### Step 2: Your agent gets to work. Your agents can plan, develop, or test by using your real product. Every action hits your real application code. ### Step 3: Results recorded and shared. Every interaction is recorded during the run. You can replicate a sandbox or just review the session artifacts. Easily spin the session back up if needed. ## What only appears when you run it Production sandboxes catch issues that lightweight copycats won’t run into. No wasted time setting up services and ensuring they work. These are the cases Ito is built to service: **Coding Agents.** Test and code changes quickly with high fidelity. **Runtime Security.** Rapidly stress application security. **On-demand Pentest.** Easily spin up agentic testing against real services. **Third-party integration paths.** Test real integrations instead of mocking. ## Questions teams ask before using Ito. Common questions from engineering managers and devops **Q: How are Ito's sandboxes different?** A: Unlike off-the-shelf sandbox providers, Ito ensures that your environment is always running perfectly compared to your real application. There's no work done to get it running and maintained, so you can focus on building. **Q: Does Ito store my source code?** A: No, Ito deploys your application in a single-use sandbox and destroys the container after you're done. Your code does not persist after execution completes. **Q: What access does Ito need to my codebase?** A: Ito needs read access to your repositories to build the environment. For higher-fidelity sandboxes, you can optionally provide sandbox credentials for external services your app depends on. We scope access during onboarding to what is needed for the environment, not more. **Q: How does Ito handle authenticated flows?** A: During onboarding, we surface theory, where sandbox credentials would be helpful. If you don't provide those, Ito will let you know where it isn't as authentic until credentials are provided. ## Your sandbox created automatically Connect your repo and Ito starts building your sandbox right away, including scoping out all dependencies or temporary mocking to ensure high fidelity. --- # Containerized Test Execution ## To know what the code does, you have to run it. For every PR, Ito builds a fresh isolated copy of your application, runs a swarm of AI agents through it, and posts the evidence before anyone merges. The diff shows you what changed. The run shows you what it actually does. Start testing PRs → Watch 2 min demo ## Reading code tells you what it says. Running it tells you what it does. A diff can look completely reasonable and still break the moment real data moves through the system. The logic error that causes a token refresh to run after a redirect instead of before it is not visible in any single file. An off-by-one in date handling that only surfaces in a specific timezone needs a real session and real routing to appear. A drag-and-drop reorder that writes to local state and never calls the API looks correct in every static view of the code. None of these is a careless diff. Each one depends on timing, state, or an integration path that reading cannot observe. Static analysis and code review are useful for what they do: surface likely problems in the code as written. The failure class they cannot cover is the one that only appears when the application runs. Code that passes review and breaks in production is not a code quality problem. It is a visibility problem. That is the gap Ito is built for: running your application on every PR so the behavior is observable before the merge. ## How it works. Set up once: connect your repo, configure how deeply you want Ito to test, and add any plain-English instructions about what matters most to your team. From there, every PR runs on its own. ### Step 1: Your app, built from scratch. For every PR, Ito deploys your application in a clean, single-use container built specifically for that change. Isolated from every other run, no state carried over, nothing shared across customers. The container runs your actual code, not a mock or a snapshot. ### Step 2: Ito navigates your app like a real user. AI agents open your product in a real browser and work through the flows the PR touches. They sign in, navigate between pages, submit forms, and trigger backend calls. The agents work from the test plan Ito built from the PR diff and description, your organizations stated preferences, and from learning your organizations behavior. Every action hits your real application code. ### Step 3: Ito tests what only appears at runtime. Beyond the happy path, Ito runs the scenarios that break things in production: double submits, expired sessions, rapid repeated actions, inputs at the boundary of what the code expects, and flows that cross permission levels. These are the cases that look safe in the diff and fail when a real user reaches them. ### Step 4: Results posted. Container destroyed. Every interaction is recorded during the run. Once execution completes, the container is destroyed and your code does not persist beyond the run. What remains is the evidence package: video of what the agent did, screenshots at the moment of failure, reproduction steps, and the exact lines of code responsible, posted to the PR comment and the Ito dashboard before anyone merges. ## What only appears when you run it The bugs that reach production share a shape. They read cleanly in the diff, pass static checks, and survive code review. They break the moment a real user reaches them because the failure lives in behavior, not in code structure. These are the cases Ito is built to catch: **Auth and session flows.** A returning user opens the app to finish a purchase. An expired session drops them into a login loop because the token refresh check runs after the redirect instead of before it. The test against the refresh helper is green. The loop only appears with a stale token and real routing. **Data persistence.** A drag-and-drop reorder looks correct in the UI, then reverts after the user navigates away and returns, because the reorder writes to local state and never calls the API. Every static view of the code looks fine. The failure lives in the round trip. **Third-party integration paths.** A mocked dependency passes. The real one returns something the code did not expect, and the payment never clears or the confirmation never goes out. These flows only surface when the live path runs. **Race conditions.** Two actions hit the same shared resource at the same moment. The outcome depends on timing that no diff review can reproduce. **Permission boundaries.** A change that looks UI-only turns out to affect what a lower-privilege user can reach. The access control holds in isolation and breaks in the flow. ## Questions teams ask before using Ito. Common questions from engineering managers, senior engineers, and QA leads. **Q: Does Ito store my source code?** A: No. Ito deploys your application in a single-use container for each PR, runs the tests, captures the results, and destroys the container. Your code does not persist after execution completes. **Q: What access does Ito need to my codebase?** A: Ito needs read access to your repositories to build the environment. For higher-fidelity testing, you can optionally provide sandbox credentials for external services your app depends on. We scope access during onboarding to what is needed for the test environment, not more. **Q: How does Ito handle authenticated flows?** A: The agent navigates login and session flows as part of test execution. It can test the full surface of your product, including dashboards, settings, and role-gated features, without exposing real credentials. **Q: Why can't static analysis catch the same bugs?** A: Static analysis reads code and can flag known patterns. It cannot run your application, which means bugs that only appear when data moves between services stay invisible until they reach production. Ito runs the code. ## Your first PR tested within 60 minutes. Connect your repo and Ito starts testing pull requests right away. Each PR includes a full QA report with video, screenshots, and failure details directly in the PR. --- # Self-healing Codebase ## Don't just find the bug. Fix it. Ito already runs your app, builds the test plan, and captures exactly how each bug reproduces. Self-healing closes the loop: for the small, well-understood issues it surfaces, Ito proposes the fix as a pull request. Bugs get resolved, not just reported. Alpha Now in private alpha Talk to us about the alpha → available to paying customers ## How it works. Self-healing builds on the testing Ito already does. When Ito finds a bug it understands well enough to fix, it closes the loop for you. ### Step 1: Ito finds a real bug On every release, Ito runs your app end-to-end and catches behavioral bugs that code review and linters miss, complete with video, reproduction steps, and the exact lines of code responsible. ### Step 2: Ito attempts a fix Because Ito already has the isolated environment, the test plan, and the runtime evidence, it's uniquely positioned to fix the small, well-scoped issues it surfaces. ### Step 3: You get a fix as a PR to review Ito opens a pull request with the proposed fix, re-runs the same test to confirm the bug is gone, and hands it to you to review and merge. You stay in control; Ito does the legwork. ## What engineering leaders say. > ""Ito doesn't just report bugs, it fixes them and proves the bug is gone. For the small stuff, that's hours back every week."" > > CTO > ""We ship too fast to manually review everything. A self-healing loop that finds the small bugs and proposes the fix is exactly what we need. Especially as we move to auto-merge"" > > Head of Engineering, Series B > ""Because Ito already has the environment and the repro, its fixes actually land. It's the first tool that closes the loop instead of adding another dashboard. We trust Ito to auto-merge."" > > Founder / CTO ## Questions teams ask about self-healing. Common questions from engineering leaders evaluating autonomous fixes. **Q: Does Ito merge code on its own?** A: No, Ito proposes a fix as a pull request and re-runs its test to confirm the bug is resolved. A human always reviews and merges. Self-healing is about doing the legwork on well-scoped bugs, not removing you from the loop. **Q: What kinds of bugs can Ito fix?** A: The small, well-understood issues it surfaces during testing. The bugs where the reproduction is clear and the fix is contained. Larger or ambiguous changes are still reported for your team to handle. **Q: How does Ito prevent an endless loop?** A: Ito already runs your app, builds the test plan, and captures runtime evidence for every issue. That context of the environment, the failing flow, and the responsible code is exactly what's needed to attempt a reliable fix and verify it worked. Unlike other codegen tools, Ito knows how to scope the change appropriately to the PR itself and doesn't allow scopecreep to refactor your whole codebase. **Q: How do I get access?** A: Self-healing is in private alpha with paying customers. If you want early access, talk to us about how your team ships and we’ll see if you’re a fit. ## From finding bugs to fixing them. Self-healing is in private alpha. Tell us how your team ships and we'll see if it's a fit. Talk to us about the alpha --- # Targeted Test Plans ## When a PR opens, Ito already knows what to test. Ito reads your diff and PR description, maps the change to the user journeys it could put at risk, and builds a targeted test plan on its own. The more your team uses it, the sharper the plans get. No test cases to write. No suite to maintain. Targeted test plans Start testing PRs → Watch 2 min demo Before any test runs, someone has to figure out what is worth testing.: On most teams, that work falls on a senior engineer who ends up acting as the QA for every PR: reading the diff, mapping it to the journeys it could break, and turning that judgment into test cases that go stale the next time the markup moves. More time goes into maintaining that coverage than into catching real bugs, and the bottleneck hits hardest when velocity should be highest. AI made that bottleneck worse. Six months ago your team was merging ten PRs a week. Now it is thirty. The code output tripled, and the process for deciding what to test stayed the same. Someone still has to connect the diff to the risk surface, and when the volume outpaces that person, things slip through without anyone noticing. Reading the diff shows what changed, not what breaks when it runs.: Reading the diff, whether a code review tool or person, tells you which files changed. It does not tell you which user journeys the change puts at risk, which edge cases are now live, or what the rest of the codebase assumed about the behavior that moved. When coverage depends on a human connecting a diff to a user journey, what gets tested depends on who reviewed the PR that day, not on what the change put at risk. That is the gap Ito fills. ## How it works. Set up once: connect your repo and add any plain-English instructions about what matters most to your team. From there, every PR runs on its own. ### Step 1: Ito reads the PR When a PR opens, Ito reads the full diff as its primary source, along with the PR description. If the description names specific tests to run, Ito runs those. When a linked ticket is available, it uses that as added context. Ito builds a picture of what this change was intended to do, not only which files it touched. ### Step 2: Ito builds the test plan Ito decides what to test in three ways: - **The change itself.** The diff and PR description, with no setup required. This works on the first PR, before you have configured anything. The plan is weighted toward what the change actually touches, so a PR that touches authentication gets different coverage than one that touches a billing flow. No two plans are the same. - **Your priorities.** Instructions set at the repo, user, or organization level for what matters most: security, accessibility, mobile usability, performance, whatever your team prioritizes. - **What Ito has learned.** Which bugs your team fixes, which you close as won't fix, and the feedback you give on past plans. Ito uses that signal to weight what it tests on future PRs, so **teams that ship auth-heavy changes get sharper auth coverage over time**. The plans get more accurate the more you use it. ### Step 3: Tests start running right away As soon as the plan is built, tests begin executing, each one tagged by category. You can watch the plan and its progress in the PR comment and in the Ito dashboard. If you want Ito to cover something more, you can ask for additional tests after the run. What happens during execution is covered in the next section. ## What Ito plans on every PR The categories are not a fixed checklist. A PR that modifies an auth flow gets adversarial tests, permission boundary checks, and session handling edge cases. A PR that touches a billing calculation gets logic tests focused on pricing rules, state transitions, and role-based gating. A UI-only change gets happy-path coverage and UX checks. Some example categories are: **Happy-path tests** The core user journeys still work end to end: signing in, completing a key action, submitting a form, navigating between pages. **Edge case tests** Unusual but valid inputs real users will hit: empty states, very long strings, special characters, expired sessions, slow connections, browser back and forward, refresh mid-flow. **Adversarial tests** The ways a careless or impatient user will break things: double submits, rapid repeated clicks, navigating to states that should not be reachable, attempting actions without permission. **Logic tests** The business rules behind the UI: pricing math, conditional display, role-based gating, state transitions, totals and counters reflecting the right values. **Accessibility tests** Keyboard navigation, focus order, ARIA labels, color contrast, screen reader behavior on interactive elements. **Mobile tests** The same journeys under a mobile viewport: responsive layout, touch targets, and mobile-specific behaviors like sticky headers or bottom navigation. **UX tests** The things a careful reviewer would catch by eye: confusing copy, broken affordances, layout regressions. > ""It helped me test things I never would have thought of. So many edge cases and adversarial tests."" > > Founder > ""The worst part of my day is we have these bug bashes where I have to manually list out each specific thing to test and the path to get there."" > > Engineering Manager ## Questions teams ask before using Ito. Common questions from engineering managers, senior engineers, and QA leads. **Q: How does Ito decide which tests to run for a given PR?** A: Three ways, and all three work together. First, the diff and PR description: this is zero-shot and works with no setup, on the first PR. Second, instructions your team sets at the repo, user, or organization level to prioritize what matters most, like security, mobile usability, or accessibility. Third, what Ito learns over time from the bugs your team fixes or closes as won't fix and the feedback you give on test plans. The result is a plan targeted at the real risk surface of that change, personalized to your team and sharper with every run. **Q: What is the output of the test planning phase?** A: A full test plan tagged by category and visible in your dashboard and in the PR comment. After execution, you get a complete report: tests run, issues found with severity classification, reproduction steps, video evidence, screenshots, and the specific lines of code responsible for each failure. **Q: How are issues classified?** A: When Ito finds a problem, it is classified independently by severity based on business impact (Critical, High, Medium, Low) and by whether it is attributable to the current PR or pre-existing in the base branch. PR-attributable failures determine pass/fail status. Pre-existing issues surface as additional findings without failing the PR. **Q: Can we control what Ito tests?** A: Yes. Add plain-English instructions at the org or repo level to tell Ito which flows to prioritize, deprioritize, or skip on specific PR types. Individual engineers can also name specific tests directly in the PR description to focus a run, and can ask Ito to run additional tests after a run completes. ## Your first PR tested within 60 minutes. Connect your repo and Ito starts testing pull requests right away. Each PR includes a full QA report with video, screenshots, and failure details directly in the PR. --- # Evidence Rich Results ## Every failure lands in your PR with everything needed for you or your agent to fix it. Ito posts a full test summary on every PR: which flows passed, which failed, and exactly what it did to find out. Every failure comes with run-time evidence, the exact lines responsible, the business impact, and the steps to reproduce it. A developer can open the comment and start fixing without waiting for anyone to investigate or rerun the scenario. Evidence Rich Results Start testing PRs → Watch 2 min demo A flagged bug still has to be reproduced before anyone can fix it.: When a reviewer spots a potential issue, someone has to load the environment, walk through the flow, and document what they saw in enough detail for a developer to act on it. In teams shipping at AI velocity, that loop is where review cycles stall. And when the reproduction is manual, some issues do not get reproduced at all. They get flagged, deprioritized, and rediscovered in production, where the fix costs far more. An engineering manager described it plainly: "The worst part of my day is we have these bug bashes where I have to manually list out each specific thing to test and the path to get there." Manual reports are inconsistent, and inconsistent reports are not measurable.: When a QA engineer documents a bug, the output depends on who did the documenting. Some write detailed reproduction steps. Others flag a symptom and move on. Some categorize severity. Others leave it to the developer to decide. There is no guarantee that the reviewer who finds the issue delivers the same package as the one who found the last one. That inconsistency has two costs. First, some findings are not actionable because they lack the context to reproduce and fix. Second, results that are formatted differently every time cannot be compared across PRs: you cannot tell whether your failure rate is going up, which category of bug keeps appearing, or whether a new development practice changed anything. When every PR gets the same report, the results become comparable. That is what makes the data usable beyond the individual PR. ## What lands on every PR **What Ito posts on every PR** On every PR, Ito posts a full report to the PR comment and the Ito dashboard before anyone merges. The report covers which flows held up and which ones broke, a severity rating for every failure, and whether each issue is attributable to the current PR or pre-existing in the base branch. **What every failure includes** When a flow breaks, Ito posts the evidence in the PR comment: - A video replay of what the agent did and what the application returned - Screenshots at the moment of failure - The exact lines of code responsible - The reproduction steps, so any developer can rerun the scenario without setup - Additional artifacts to help understand the issue and share it with your AI tool of choice to resolve it **What happens after you push the fix** Severity ratings tell you what to fix before merge and what can wait. Once you or your agent push the fix, Ito re-runs the affected scenario to confirm it is resolved and posts the result in the same PR thread. Many customers set up agentic loops to automatically address Ito's feedback before merge. ## What this changes for your team The back-and-forth that slows review cycles down disappears when the developer sees the failure in the PR comment with everything needed to fix it, without waiting for a reviewer to investigate or a QA engineer to document the steps. Every PR gets the same report, the same evidence format, and the same severity classification. At organizations shipping hundreds of PRs per week, that consistency means results are comparable across PRs: failure rates by category, patterns that repeat across engineers or flows, and changes in bug distribution over time. The data becomes something to act on, not just something to get through. The most expensive version of this problem is the bug that reaches production. Whether your team operates under a contractual SLA, cannot afford transactional mistakes, or wants to avoid the on-call incident, Ito's pre-merge evidence prevents production fire drills. ## What teams say **From a healthcare software company** An engineering manager's team opened a PR for a data migration. The migration looked clean in code review. Ito tested the rollback path, which no one had manually exercised, and found that a merged migration could crash mid-rollback and leave the database in a broken partial state during an incident. The engineering manager validated the finding against production telemetry and ranked it as the highest-priority issue from the review. The same run surfaced a second issue: a support workflow used to regenerate and email receipts would have crashed for certain historical purchase records. The engineering manager confirmed the bug had not reached production and used Ito's finding to identify the same unsafe pattern elsewhere in the codebase. Both findings came with runtime evidence, reproduction steps, and the exact lines responsible. No bug bash. No manual reproduction. No back-and-forth. ## Questions teams ask before using Ito. Common questions from engineering managers, senior engineers, and QA leads. **Q: What does the PR comment look like?** A: The comment Ito posts on your PR shows which flows passed and which failed, with severity ratings for each failure and a link to the full report in the Ito dashboard. When a flow breaks, the comment includes the video, screenshots, reproduction steps, and the lines of code responsible. **Q: How does Ito classify severity?** A: Every issue is classified independently as Critical, High, Medium, or Low based on the impact of the failure on user-facing behavior. Ito also distinguishes between issues introduced by the current PR and pre-existing issues in the base branch. PR-attributable failures determine pass/fail status. Pre-existing issues surface as additional findings without failing the PR. **Q: What if a finding turns out to be a false positive?** A: Give it a thumbs down in the dashboard. That feedback shapes what Ito prioritizes for your team over time. The goal is to surface what is real and let engineers decide what to act on. **Q: Does Ito re-run after a fix?** A: Yes. Once you push a fix, Ito re-runs the affected scenario to confirm it is resolved. The result appears in the same PR thread so reviewers can see the fix was verified before they merge. **Q: How does this compare to what a code review bot produces?** A: A code review bot reads the diff and reasons about what might be wrong. Ito runs the application and shows you what is wrong, with video and reproduction steps. The two belong on the same PR: the bot explains the code, Ito shows what the running app did with it. ## Your first PR tested within 60 minutes. Connect your repo and Ito starts testing pull requests right away. Each PR includes a full QA report with video, screenshots, and failure details directly in the PR. --- # Smoke Testing ## Automatic smoke test every night, every release, or on-demand. Based on your trigger, Ito maps the application changes and tests for any regressions. No test cases to write. No suite to maintain. Targeted test plans Automate smoke testing → Watch 2 min demo ## How it works. Set up once: connect your repo and add any plain-English instructions about what matters most to your team. From there, every smoke test runs automatically. ### Step 1: Ito understands the release When ready to release, Ito reads the full changes, along with the PR descriptions of related issues. Ito builds a full picture of what the change were intended to do, not only which files were touched. ### Step 2: Ito builds a smoke test plan Ito decides what to test in three ways: - **The changes themselves.** The diffs and PR descriptions, with no setup required. Ito reads all the code and issues. - **Your priorities.** Instructions set at the repo, user, or organization level for what matters most: security, accessibility, mobile usability, performance, whatever your team prioritizes. - **What Ito has learned.** Which bugs your team fixes, which you close as won't fix, and the feedback you give on past plans. Ito uses that signal to weight what it tests on future PRs, so **teams that ship auth-heavy changes get sharper auth coverage over time**. The plans get more accurate the more you use it. ### Step 3: Tests run every time You configure when you want trigger Ito, then tests begin executing. You can watch the plan and its progress as Ito searches for regressions. If you want Ito to cover something more, you can ask for additional tests. > ""Ito saves my team by smoke testing every release. It always catches something"" > > VP Engineering > ""We have Ito on autopilot then do auto-cleanup from the issues it finds on release. It paid for itself in the second month by catching a major regression."" > > CTO / Founder ## Questions teams ask before using Ito. Common questions from engineering managers, senior engineers, and QA leads. **Q: How does Ito decide which regressions tests to run during the smoke test?** A: Three ways, and all three work together. First, the actual code changes and relevant PR description: this is zero-shot and works with no setup. Second, instructions your team sets at the repo, user, or organization level to prioritize what matters most, like security, mobile usability, or accessibility. Third, what Ito learns over time from the bugs your team fixes or closes as won't fix and the feedback you give on test plans. The result is a plan targeted at the real risk surface of that change, personalized to your team and sharper with every run. **Q: What do the smoke test plans look like?** A: A full test plan tagged by category and visible in your dashboard and in GitHub. After execution, you get a complete report: tests run, issues found with severity classification, reproduction steps, video evidence, screenshots, and the specific lines of code responsible for each failure. **Q: How are issues classified?** A: When Ito finds a problem, it is classified independently by severity based on business impact (Critical, High, Medium, Low) and by whether it is attributable to the current release or pre-existing. **Q: Can we control what Ito tests?** A: Yes. Add plain-English instructions at the org or repo level to tell Ito which flows to prioritize, deprioritize, or skip on specific test types. You can ask Ito to run additional tests after a run completes. **Q: What are the triggers for the smoke test?** A: Ito is configurable so you can trigger smoke tests as you see fit. Ito tests every PR by default. Layer in smoke tests on timed cadence, based off a GitHub workflow, or on-demand triggers like Slack. ## Your first PR tested within 60 minutes. Connect your repo and Ito starts testing pull requests right away. Each PR includes a full QA report with video, screenshots, and failure details directly in the PR. --- # Product Demo Videos ## A clean demo video of every change. Automatically. For every pull request, Ito records a polished walkthrough of what changed. A clear, happy-path demo of the new feature working, posted right in the PR. No screen recording, no staging setup, no narration. Just proof the feature works, ready to share with your whole team. Get a demo on your next PR → no credit card required ## How it works. Connect your repo once. On every pull request, a dedicated demo agent produces a clean recording of the change. ### Step 1: Ito learns the change Ito reads the pull request and the code to understand what the feature is meant to do, then works out how to reach it in a running version of your app. ### Step 2: Ito rehearses until it's clean A dedicated demo agent walks the happy path, works through any bottlenecks, and only once it can complete the whole flow cleanly does it go back to the start and record. The result is a polished walkthrough, not a recording of an agent debugging. ### Step 3: The demo lands in your PR The finished video is posted right in the GitHub PR comment, alongside Ito’s test results. Engineers, PMs, designers, and execs can watch the feature work and share it, without leaving GitHub or spinning up an environment. No more bugging you for a screen recording showing off every feature. ## What engineering leaders say. > ""The demo video on every PR means I can quickly see and share what my team shipped without pulling teeth."" > > VP Engineering > ""Our PMs and designers watch the feature work the moment the PR opens. It's eliminates a lot of coordination tax."" > > Engineering Manager > ""I used to screen-record every UI change for reviewers. Now Ito does it for me."" > > Staff Software Engineer ## Questions teams ask about demo videos. Common questions from engineering managers, PMs, and designers. **Q: How is this different from the videos Ito already records during testing?** A: Ito's test-run videos are evidence collected while the agent probes your app for bugs — so they show the messy reality of testing: retries, edge cases, and failures. The demo video is a separate, purpose-built recording of the happy path: a clean walkthrough of the feature working as intended, meant to be watched and shared. **Q: Where does the demo video show up?** A: Right in the GitHub PR comment, next to Ito’s test results. Your team can watch the feature work without leaving their workflow or spinning up a preview environment — and share it with PMs, designers, or other stakeholders. **Q: Do I need a staging environment or to record anything myself?** A: No. Ito runs your app in an isolated environment and records the walkthrough for you. It replaces the manual screen recording engineers do today when they want to show a reviewer how a change works. **Q: What kinds of changes get a demo video?** A: Any pull request that introduces a user-facing change. Ito focuses the walkthrough on the flow that changed, so reviewers see exactly what’s new. ## See every change before you approve it. Connect your repo and Ito posts a clean demo video of what changed on every pull request — so reviewers, PMs, and designers can watch the feature work without leaving GitHub. Get Started no credit card required --- # Bug Bashes ## Run a full bug bash for deep feature testing. Automatically. A bug bash means pulling your whole team into a room for an afternoon to click through the app and hunt for bugs before a big feature release. Ito runs that same sweep on demand - exercising every critical flow in a real browser and reporting exactly what breaks, with video and reproduction steps. Start a bug bash → no credit card required ## How it works. Point Ito at your app and the release you're about to ship. From there the whole bug bash runs automatically — no test cases to write, no suite to maintain. ### Step 1: Ito maps your app Ito reads your codebase and the release, then builds a picture of the critical user journeys across your whole product — not just the files that changed — so the bug bash covers what actually matters to your users. ### Step 2: Ito builds the bug bash plan Ito decides what to hammer in three ways: - **The release itself.** The diffs and PR descriptions, so the flows most affected by this change get the hardest look. - **Your priorities.** Plain-English instructions at the repo or org level for what matters most: security, accessibility, mobile usability, performance — whatever your team cares about. - **What Ito has learned.** Which bugs your team fixes, which you close as won’t fix, and the feedback you give on past runs. The plan gets sharper every bug bash. ### Step 3: Ito runs the bug bash on demand Trigger a bug bash before any release. Ito exercises every flow in parallel in a real browser while you watch it work. Each issue comes back classified by severity with video, screenshots, reproduction steps, and the exact lines of code responsible — and you can ask Ito to cover more anytime. ## Questions teams ask before their first bug bash. Common questions from engineering managers, senior engineers, and QA leads. **Q: How is an Ito bug bash different from testing each PR?** A: Per-PR testing focuses on what a single change could break. A bug bash is a full sweep across a new feature's critical flows — the kind of thing teams normally schedule before a big release and staff with the whole engineering team. Ito runs that entire sweep on demand, in parallel, so you can do it as often as you ship. **Q: How long does an automated bug bash take?** A: Instead of blocking your team for an afternoon, Ito runs flows in parallel and reports as it goes. You watch the plan execute in real time and get a complete report when it finishes — typically in a fraction of the time a manual bug bash takes. **Q: Can we control what the bug bash covers?** A: Yes. Add plain-English instructions at the org or repo level to tell Ito which flows to prioritize, deprioritize, or skip. After a run completes you can ask Ito to cover more. **Q: What do we get when the bug bash finishes?** A: A complete report: every issue found, classified by severity (Critical, High, Medium, Low) and by whether it is attributable to the current release or pre-existing — each with reproduction steps, video evidence, screenshots, and the specific lines of code responsible. ## Replace your next bug bash with a single click. Connect your repo and run a full bug bash across your app on demand — no scheduling, no war room, no test cases to write. Every issue comes back with video, screenshots, and the exact code responsible. Get Started no credit card required --- # Blog ## Build an AI-native engineering team that ships fast Practical how-tos, model benchmarks, and research on AI-driven development for fast-moving teams. ## Posts - [8 AI Code Review Tools, Compared on Setup, Output and Features](/blog/ai-code-review-tools): Compare 8 AI code review tools by setup, features, review speed, and pricing. See what evidence each provides and which fits your team's workflow. (Evan Marshall · 2026-10-15) - [AI Code Review on GitHub: Automating PR Review Without Writing a Test Suite](/blog/ai-code-review-github): Get behavioral feedback on every GitHub pull request before you write a test suite. Learn how to set up runtime PR review and where human judgment still decides the merge. (Evan Marshall · 2026-10-02) - [The Electrification of Factories: What Software Teams Can Learn From the Transition](/blog/electrification-of-factories-software-teams): Factory electrification reshaped production only after owners redesigned the floor. AI software teams need the same shift in workflow, review, and verification. (Evan Marshall · 2026-09-18) - [Bug Catch of the Week: How a Failed Database Swap Could Delete Your Only Copy](/blog/failed-database-file-replacement-data-loss): DoltLite's replacement flow could delete the old database before the new copy was safe. Here's how runtime testing exposed the failure path before release. (Evan Marshall · 2026-09-04) - [How to Build Quality Into Your Software Factory](/blog/how-to-build-quality-into-your-software-factory): Software factories need speed & quality. Ship fewer bugs by testing in layers; fast tests first, then deeper validation (Barron Caster · 2026-08-28) - [The AI Model Plateau: Why Your Infrastructure Matters More Than the Next Release](/blog/ai-model-plateau-why-infrastructure-matters-more-next-release): AI model routing and evaluation now decide your production results. Learn how companies are adapting to make the most of the highly competitive LLM landscape. (Evan Marshall · 2026-08-21) - [Why AI Code Review Needs to Read Your Code and Run It](/blog/ai-code-review-read-your-code-run-it): A practical guide to static and runtime analysis in AI code review, including what each approach catches, where it falls short, and why teams use both. (Evan Marshall · 2026-08-14) - [What Happened When DoltHub Ran Ito on 43 Pull Requests](/blog/what-happened-dolthub-ran-ito-43-pull-requests): DoltHub reported a 2:1 fixed-to-dismissed ratio for Ito bug findings, and surfaced pre-existing issues in 26 of 43 tested PRs over six weeks. (Evan Marshall · 2026-08-04) - [MTTR is the Wrong Metric for AI-Era Engineering Teams](/blog/mttr-wrong-metric-ai-engineering-teams): AI tools produce 41% more bugs and 98% more pull requests. MTTR can't keep up. Here's how MTTF shifts your team from incident response to prevention. (Evan Marshall · 2026-07-23) - [Your fastest model is probably not your fastest model](/blog/tokens-per-second-measures-emission-not-work): Tokens per second measures how fast a model emits, not how fast it finishes. Our 13-model probe shows the gap that flips both the speed + cost leaderboards. (Evan Marshall · 2026-07-21) - [Moo: Giving Your Agents the Runtime Isolation git worktrees Need](/blog/moo-worktree-isolated-environments): git worktree isolates your files. Moo isolates the database, ports, and services, saved per commit. Together, they give your agents fully isolated machines. Learn how to use Moo, the benefits, and why your agents need it. (Evan Marshall · 2026-07-14) - [What we learned after moving 100B tokens/week to open-weight models](/blog/measuring-100b-tokens-week-open-weight-models): Ito moved production QA agents from GPT-5.3-codex to open-weights. Here's what we saved, why inference providers matter as much as models, and what to check. (Evan Marshall · 2026-07-13) - [Elevate Software Testing: 10 Best Practices for 2026](/blog/software-testing-best-practices): Boost your Test. Learn 10 software testing best practices for 2026, covering pre-merge, scriptless, and agentic methods for managers. (Barron Caster · 2026-06-23) - [Static Analysis Misses Bugs That Reach Production](/blog/static-analysis-misses-bugs-that-reach-production): Agents can stare at code all day long, but they will not find all the issues because they are probabilistic unit tests. You need to actually run your application to get runtime evidence and proof that things behave as your organization expects them to for full integration and end-to-end tests. (Evan Marshall · 2026-06-12) - [Your AI-scaled engineering org needs big-org processes](/blog/your-ai-scaled-engineering-org-needs-big-org-processes): When developers are 3–5x more productive with AI, your org is effectively that much bigger. Your operations need to follow suit. (Evan Marshall · 2026-05-05) - [What is agentic QA? The complete guide](/blog/what-is-agentic-qa-the-complete-guide): How autonomous AI agents are replacing brittle E2E scripts with behavioral testing that actually validates the user experience. (Evan Marshall · 2026-04-15) - [AI-Driven Testing: Why Your QA Still Runs Like It's 2015](/blog/ai-driven-testing-why-your-qa-still-runs-like-its-2015): Discover how AI-driven testing replaces brittle QA automation, cuts bottlenecks, and helps modern teams ship faster with more confidence. (Evan Marshall · 2026-03-20) ## Your first PR tested within 60 minutes. Connect your repo and Ito starts testing pull requests right away. Each PR includes a full QA report with video, screenshots, and failure details directly in the PR. --- # Elevate Software Testing: 10 Best Practices for 2026 Boost your Test. Learn 10 software testing best practices for 2026, covering pre-merge, scriptless, and agentic methods for managers. By Barron Caster · 2026-06-23 Your QA strategy is probably slower than your delivery pipeline. That sounds backwards, but the evidence points that way. The 2024 State of Testing Report found that 85% of organizations now use some form of automated testing, and 63% say at least half of their regression and smoke tests run automatically on every build or merge, according to [Tricentis on software testing best practices](https://www.tricentis.com/learn/testing-best-practices). Automation is mainstream. Slow, late, manual QA as the primary quality gate shouldn't be. At the same time, a lot of teams still ship defects they could have caught earlier. Independent research from CISQ and CAST, based on 1,100 enterprise applications across 21 countries, found that high-quality systems using structured testing practices reached 71% method-level test coverage, while systems without those practices averaged 45% and showed substantially higher defect densities, according to the Testing Best Practice research summary. That gap is big enough to affect release confidence, incident load, and engineering time. The problem isn't that teams don't care about quality. It's that traditional QA processes weren't built for AI-assisted coding, high PR volume, and teams shipping all day. If your developers can generate code faster than your test process can validate behavior, QA becomes the bottleneck. This guide lays out 10 software testing best practices that fit modern engineering. They focus on pre-merge validation, execution-based testing, evidence-rich feedback, and smarter prioritization so quality keeps up with velocity. ## Table of Contents - [1. Shift-Left Testing Pre-Merge Validation](#1-shift-left-testing-pre-merge-validation) - [Catch risk before it compounds](#catch-risk-before-it-compounds) - [2. Behavioral Testing with Real Application Execution](#2-behavioral-testing-with-real-application-execution) - [Run the product, don't just inspect it](#run-the-product-dont-just-inspect-it) - [What evidence should look like](#what-evidence-should-look-like) - [3. Scriptless Test Automation](#3-scriptless-test-automation) - [Reduce maintenance before you scale automation](#reduce-maintenance-before-you-scale-automation) - [4. Code-Aware Test Prioritization](#4-code-aware-test-prioritization) - [Select by impact, not habit](#select-by-impact-not-habit) - [5. Full-Stack Integration Testing](#5-full-stack-integration-testing) - [Test the seams where releases usually fail](#test-the-seams-where-releases-usually-fail) - [6. Evidence-Rich Test Failure Reporting](#6-evidence-rich-test-failure-reporting) - [Make failures cheap to diagnose](#make-failures-cheap-to-diagnose) - [7. Isolated Sandbox Environments for Test Execution](#7-isolated-sandbox-environments-for-test-execution) - [Stop debugging polluted environments](#stop-debugging-polluted-environments) - [8. Edge Case and Adversarial Scenario Testing](#8-edge-case-and-adversarial-scenario-testing) - [Test the failure modes that map to business risk](#test-the-failure-modes-that-map-to-business-risk) - [9. Continuous Regression Detection and Prevention](#9-continuous-regression-detection-and-prevention) - [10. Authentication and Role-Based Access Testing](#10-authentication-and-role-based-access-testing) - [Auth bugs are product bugs and security bugs](#auth-bugs-are-product-bugs-and-security-bugs) - [10-Point Comparison: Software Testing Best Practices](#from-qa-bottleneck-to-quality-flywheel) - [From QA Bottleneck to Quality Flywheel](#from-qa-bottleneck-to-quality-flywheel) ## 1. Shift-Left Testing Pre-Merge Validation Teams that wait until staging or release candidate review to find breakage are choosing expensive feedback. In a fast-moving repo, every late defect drags in more context, more owners, and more uncertainty. Pre-merge validation cuts that off before bad assumptions harden into merged code. Google, GitHub, and Stripe all rely on strong checks before code lands because merge-time is where you still have author context, a clear diff, and a contained blast radius. The modern version of shift-left isn't just more unit tests. It's validating behavior on the pull request itself. ### Catch risk before it compounds The practical move is simple. Treat the pull request as the main quality checkpoint, not a paperwork stop on the way to QA. Run tests against the actual change, with enough realism to catch regressions and enough focus to avoid turning every PR into a long queue. A lot of teams still overrate static analysis here. It matters, but it won't tell you whether the user can complete the flow after the code compiles. That's the gap described in [why static analysis misses bugs that reach production](https://www.ito.ai/blog/static-analysis-misses-bugs-that-reach-production). > **Practical rule:** Start with non-blocking PR checks if your team doesn't trust the signal yet. Then promote the checks that prove reliable into merge gates. A good pre-merge setup usually includes: - **Critical user flows first:** Validate auth, checkout, account updates, billing changes, and other paths tied directly to revenue or support risk. - **Clear completion expectations:** Engineers should know when results will appear and where failures surface, ideally inside the PR. - **PR-scoped selection:** Use the diff and PR description to decide what deserves full behavioral validation and what only needs narrower checks. What doesn't work is pushing every possible test into pre-merge. That creates a fake sense of rigor and a real delivery bottleneck. ## 2. Behavioral Testing with Real Application Execution A lot of test suites prove the code was arranged correctly. Fewer prove the product works. That distinction matters more now because modern systems fail across boundaries: browser state, API contracts, async jobs, data assumptions, permissions, and third-party behavior. Behavioral testing means running the built application in conditions close enough to reality that the result means something. Netflix, Airbnb, and Shopify all depend on forms of full-flow validation because isolated test layers don't catch enough on their own. ### Run the product, don't just inspect it This matters even more in high-velocity teams. Existing guidance often says "run all relevant tests per commit," but practical constraints get in the way. The verified research notes that teams typically see only 20–25% of builds pass all tests even when coverage is high, and that 30–50% of CI pipelines exceed 10 minutes per run, which pushes developers to skip or defer checks under pressure. That trade-off is why execution-based QA has to be selective and behavioral. Build from source. Launch the app. Authenticate. Click through real flows. Verify backend effects, not just front-end rendering. If a checkout button looks enabled but the order isn't persisted, the test should fail. ### What evidence should look like Good behavioral testing also leaves a trail developers can act on. Video, screenshots, logs, API traces, and the exact point of failure turn a flaky "failed end-to-end run" into something engineers can fix quickly. Here's a short demo of the kind of browser-level execution teams increasingly expect from modern QA tooling: When this is done well, QA stops being a separate event. It becomes a direct check on whether the change preserved user-visible behavior. ## 3. Scriptless Test Automation Teams often don't have an automation problem. They have a maintenance problem. They write a large suite in Selenium, Cypress, or Playwright, feel good for a quarter, then spend the next quarter fixing selectors, repairing fixtures, and arguing about whether a broken test means the product is broken. That cost is not trivial. The verified research notes that maintenance-intensive suites can see 15–25% of test failures caused by environmental or configuration issues rather than true defects. It also notes that teams often spend a meaningful share of QA and engineering time maintaining or rerunning fragile automation instead of expanding useful coverage. ### Reduce maintenance before you scale automation Scriptless automation is one way out. The point isn't "no effort." The point is to stop hand-authoring brittle UI scripts for every meaningful path. Tools like Applitools, Mabl, and newer agentic systems reduce dependence on exact selectors and fixed scripts by relying more on visual context, inferred intent, and behavior-driven definitions. The strongest reason to adopt scriptless approaches isn't convenience. It's control over maintenance overhead. That's the heart of [AI-driven testing and why QA still runs like it's 2015](https://www.ito.ai/blog/ai-driven-testing-why-your-qa-still-runs-like-its-2015). Use it where it pays off most: - **High-value journeys:** Automate flows users rely on constantly, not every pixel-level interaction in the product. - **Natural-language intent:** Define expected outcomes in business terms so the test remains useful when implementation details change. - **Visual and state assertions:** Check what the user sees and what the system records. What doesn't work is replacing all automation with magic AI claims. Scriptless testing still needs clear expectations, sane environments, and review when failures appear. ## 4. Code-Aware Test Prioritization Running the full suite on every change sounds responsible. In practice, it often means developers wait too long, ignore results, or rerun pipelines until they get lucky. Test selection has to get smarter as engineering output rises. Recent market analysis projects the global software testing market at about USD 48.17 billion in 2025 and USD 93.94 billion by 2030, a projected 14.29% CAGR, according to [TestGrid's software testing statistics roundup](https://testgrid.io/blog/software-testing-statistics/). That growth is tied to CI/CD, cloud-native delivery, and more pressure to validate changes inside the PR workflow, not after it. ### Select by impact, not habit The same verified analysis notes that risk-based testing can deliver up to 30–40% faster test execution cycles when teams classify effort by feature criticality and failure impact. That's the operating model engineering leaders should care about. A code-aware approach means reading the diff as a statement of risk. A pricing change should trigger checkout and billing coverage. An auth middleware change should trigger login, session, and permission flows. A copy-only edit shouldn't wait behind a broad integration suite unless the rendering path itself is sensitive. > Run the narrowest test set that still protects critical behavior. Anything else turns quality into queue management. Agentic systems improve this by combining the diff, dependency impact, and PR description to infer what to test. That's where [agentic QA](https://www.ito.ai/blog/what-is-agentic-qa-the-complete-guide) starts to matter. It narrows the gap between engineering speed and quality signal without forcing humans to hand-curate every run. ## 5. Full-Stack Integration Testing The defects that hurt most usually live in the seams. Frontend sends a shape the backend no longer accepts. A background job finishes later than the UI assumes. A role check passes in one service and fails in another. Unit tests can all pass while the user still gets a broken workflow. This is why full-stack integration testing belongs on every serious list of software testing best practices. Airbnb, Stripe, and Amazon all depend on cross-layer validation because products don't fail as isolated functions. They fail as systems. ### Test the seams where releases usually fail Integration testing should cover the entire transaction, not just the visible response. If a user updates a subscription, verify the UI message, the API call, the database record, the queued work, and any outbound event or email trigger that should follow. If one of those breaks, the feature is broken. Useful full-stack scenarios usually include: - **Cross-layer assertions:** Confirm both interface behavior and backend state changes. - **Role-sensitive flows:** Run the same workflow under different permissions where business logic changes. - **Async follow-through:** Check delayed effects such as jobs, notifications, and retries. - **Realistic dependencies:** Use production-like database and service behavior instead of oversimplified mocks for critical paths. What doesn't work is pretending that a passing API contract test proves the customer journey is safe. It proves one layer answered as expected. That's not the same thing. ## 6. Evidence-Rich Test Failure Reporting A failed test without context is just another ticket in the backlog. Engineers lose time reproducing the issue, QA loses credibility, and the organization starts treating automated checks as noise. The fix is straightforward. Every serious failure should come with enough evidence that the developer can understand the issue from the PR, not by setting aside half a day to replay the scenario manually. ### Make failures cheap to diagnose Cypress, BrowserStack, and Sauce Labs all moved in this direction because raw pass-fail output isn't enough. Strong reporting includes screenshots at key transitions, video for failed runs, browser console output, backend logs, request traces, and concise reproduction steps. The report should answer four questions fast: - **What broke:** The visible symptom and affected user path. - **Where it failed:** The step, page, service, or backend action involved. - **How to reproduce it:** Clear actions, inputs, and environment context. - **Why it matters:** Severity in business terms, not just technical labels. > The best automated test report feels like a bug report written by a careful engineer who already did the first round of debugging. What doesn't work is dumping logs into an artifact store and calling that observability. If humans can't triage it quickly, the quality signal arrives too late. ## 7. Isolated Sandbox Environments for Test Execution A surprising amount of QA pain has nothing to do with the test logic. The environment is dirty, shared state leaks between runs, credentials are stale, a seeded record wasn't reset, or a previous job left the system half-configured. Teams fix this by moving test execution into isolated, single-use environments. GitHub Actions, Docker-heavy CI setups, and Kubernetes-based internal platforms all rely on this pattern because repeatability matters more than elegance. ### Stop debugging polluted environments An isolated sandbox gives every run a clean starting point. Build artifacts, services, test data, secrets, and browser state are created for that run, then destroyed when the job ends. That reduces false failures and makes parallel execution much easier to trust. The practical rules are boring and important: - **Create fresh state:** No shared browser sessions, reused local databases, or persistent test tenants unless the scenario explicitly requires them. - **Inject secrets at runtime:** Keep credentials out of images and repositories. - **Use deterministic seeding:** If randomness matters, control it so engineers can reproduce failures. - **Destroy fast:** Tear down resources immediately after execution to reduce cost and contamination. This is also one of the cleanest ways to support execution-based and agentic QA. If a system is going to build and run software on every pull request, isolation isn't a nice-to-have. It's the basis for trust. ## 8. Edge Case and Adversarial Scenario Testing Happy-path tests do not protect a high-velocity product. They confirm the demo works. Outages usually start in the conditions teams did not model: retries after partial failure, expired auth during a long session, malformed payloads that slip past the client, timezone drift, race conditions, and third-party dependencies that respond slowly or fail halfway through a transaction. Leaders managing AI-scaled delivery need this discipline even more. More code shipped per day means more interaction surfaces, more state transitions, and more ways for a small defect to turn into a customer-facing incident. If QA only validates expected behavior, engineering velocity goes up while trust in releases goes down. ### Test the failure modes that map to business risk Edge-case coverage needs a clear structure. Tie scenarios to the failures that cost the business money, time, or credibility: invalid inputs, concurrency conflicts, timeout handling, degraded networks, upstream service errors, boundary values, locale and timezone differences, and abusive or out-of-order user behavior. One rule works well in practice. Every production bug becomes a permanent scenario, then the team expands around the surrounding conditions. If duplicate charges appeared during retry logic, keep the original regression test, then add cases for latency, partial acknowledgments, duplicate submissions, and idempotency failures. That is how a bug fix turns into stronger system behavior instead of a one-off patch. Useful techniques include: - **Property-based generation:** Use tools like Hypothesis or QuickCheck where they fit to generate many valid and invalid input combinations. - **Sanitized real-world patterns:** Build cases from actual usage shapes and production failure history, not toy examples. - **Adversarial sequences:** Execute actions in conflicting, repeated, or unexpected orders to expose state bugs. - **Stress at known limits:** Verify behavior near quotas, payload ceilings, rate limits, and retry thresholds. Agentic and execution-based QA are especially effective here. An agent can explore stateful paths, vary inputs, and probe failure handling faster than a manually scripted suite, but only if leaders define the guardrails. The goal is not random exploration. The goal is targeted pressure on the parts of the system most likely to break under real usage. ## 9. Continuous Regression Detection and Prevention Regression detection is a release control system, not a cleanup task. High-velocity teams ship too often, and AI-assisted development changes too much code at once, to wait for a scheduled regression pass. Every merge can shift behavior in ways the author did not intend. Teams that catch those shifts within hours keep velocity. Teams that catch them days later burn time in triage, reruns, and rollback debates. The goal is simple. Detect meaningful behavior drift early, prove whether it matters, and stop the same failure from returning. That takes more than rerunning a large suite on a timer. Strong regression programs watch risk continuously and respond based on code change, execution evidence, and failure history. They combine stable automated coverage for known customer-critical paths with targeted checks for the parts of the product that changed most. That is where execution-based and agentic QA start to matter. Instead of waiting for a human to notice a pattern, the system can execute affected flows, compare outcomes against a known-good baseline, and surface regressions with enough context to act. A practical operating model looks like this: - **Define business-critical baselines:** Track the flows that cannot break without affecting revenue, trust, access, or compliance. - **Trigger checks from change signals:** Run focused regression coverage based on modified services, UI areas, contracts, and dependencies. - **Turn every escaped bug into a permanent guardrail:** Keep the original failing scenario and add nearby variants when the failure mode suggests a broader weakness. - **Separate flaky signals from product regressions:** Quarantine unstable tests fast, fix them on purpose, and keep them from polluting release decisions. - **Review regression results by product impact:** Tag failures by feature, user role, tenant scope, and customer journey so the team sees what is at risk. This is also where access paths often get missed. A regression suite that verifies a feature for one happy-path user but ignores role boundaries is incomplete. Teams that need to [manage permissions with RBAC](https://audityour.app/blog/what-is-role-based-access-control) should tag regressions by role and rerun the same workflows across the permission model that matters in production. One pattern shows up in nearly every slow QA organization. People stop trusting failures. Once that happens, engineers rerun jobs until they get green, QA spends time proving whether the signal is real, and release decisions drift from evidence to intuition. Regression prevention depends on restoring suite authority. Keep the signal clean, keep the scope risk-based, and make every failure explain itself with execution evidence. The payoff is operational, not theoretical. Fewer late surprises. Faster root cause isolation. Less time spent arguing over whether a failure matters, and more time fixing the ones that do. ## 10. Authentication and Role-Based Access Testing Auth failures are among the fastest ways to break user trust. They also create confusion because the symptom often looks like a simple product bug until someone realizes the underlying issue is permission logic, session handling, token expiry, or a missed role boundary. These tests belong inside normal product QA, not on a security island. If a customer can't access their data, if an admin can see the wrong tenant, or if MFA breaks on a common browser path, you've got a release problem. ### Auth bugs are product bugs and security bugs Strong auth testing covers more than successful login. It should exercise session creation, logout, timeout, token refresh, password reset, MFA prompts, access denial, role inheritance, and cross-tenant isolation where relevant. GitHub branch permissions, AWS IAM workflows, and large SaaS admin models all show how quickly this gets complicated. A practical setup includes: - **Dedicated test identities:** Maintain accounts for each role and permission tier. - **Intentional unauthorized attempts:** Verify that restricted actions fail cleanly and visibly. - **Session lifecycle checks:** Test expiration, re-authentication, and stale token handling. - **Audit expectations:** Confirm that auth-sensitive events are logged where your process requires them. When your app has more than one role, your team should also [manage permissions with RBAC](https://audityour.app/blog/what-is-role-based-access-control) in a way that's testable. If permissions are inconsistent or implicit, QA ends up guessing what "correct" access even means. ## From QA Bottleneck to Quality Flywheel The best software testing best practices don't add ceremony. They remove wasted motion. That's the shift leaders need to make now. When QA sits at the end of the process, every defect arrives with extra cost. The engineer has moved on. The branch has drifted. The reviewer no longer remembers the trade-offs. The test report is vague. The team starts negotiating around quality because the process is too slow to support the pace they want. That's how QA becomes a bottleneck even inside an otherwise fast engineering organization. Modern teams need something different. They need quality checks that happen before merge, in realistic environments, against real behavior, with evidence strong enough that engineers can act immediately. They need test selection based on code impact and business risk, not on habit. They need automation that doesn't collapse under its own maintenance burden. And they need environments clean enough that a failing check means something. There's also a leadership point here. AI coding tools increase output. They don't increase certainty. If your developers can produce more code but your testing model still depends on broad manual review, late-stage QA passes, or brittle scripted suites, your risk grows faster than your confidence. That gap is where production regressions, release hesitation, and team frustration pile up. The more useful model is a quality flywheel. Better pre-merge validation catches issues early. Early issues are cheaper to fix. Faster fixes keep PRs small and reviewable. Smaller PRs improve test targeting. Better targeting keeps feedback fast. Fast feedback makes developers more willing to trust and use the system. Quality stops fighting velocity and starts reinforcing it. You don't need to rebuild everything at once. Start where the pain is loudest. If regressions keep escaping, focus on pre-merge behavioral testing and continuous regression detection. If your suite is expensive to maintain, focus on scriptless automation and code-aware prioritization. If failures cause long debugging loops, improve evidence-rich reporting and isolated execution environments. For many teams, the fastest practical upgrade is execution-based QA on pull requests. That's why platforms like Ito are getting attention. They build and run the application for each PR, test behavior across the full stack, and return evidence directly where developers already work. That model fits how modern teams ship. As you modernize, don't forget the access layer. Strong [RBAC best practices](https://envmanager.com/blog/role-based-access-control-software) belong inside the quality strategy, not beside it. If you want QA that keeps up with AI-scaled engineering, [Ito](https://www.ito.ai/) is built for that job. It runs scriptless, execution-based behavioral tests on every pull request, focuses on the user flows most likely to break, and delivers evidence-rich results directly in GitHub so your team can catch regressions before code reaches production. --- # Moo: Giving Your Agents the Runtime Isolation git worktrees Need git worktree isolates your files. Moo isolates the database, ports, and services, saved per commit. Together, they give your agents fully isolated machines. Learn how to use Moo, the benefits, and why your agents need it. By Evan Marshall · 2026-07-14 > *"Believe it or not, my agents are writing better code with Moo."* Our staff engineer, John Stephens, sent me that message last Friday after his first week using Moo on real tickets. I'd built it expecting the main feedback to be about parallel agents, fewer collisions, or less wasted time untangling environments. That wasn't his strongest impression. What he said was that the code coming out of his agents was better. Not faster. *Better.* Here's why: When an agent is working directly on your machine, it knows your database is real. It's careful. It won't wipe a table to test something. It won't run a destructive migration twice just to see what breaks. It has every reason to play it safe, because the downside lands on you, and it's been trained to be cautious. Give that same agent its own disposable machine and the calculation changes completely. It can seed data, drop tables, run migrations forward and backward, do whatever it actually needs to verify the feature is working. The worst case is moo drop and start over, not an afternoon untangling your local environment with multiple agents failing on their tasks in parallel. ### Better testing = better code What John found was that more unrestricted testing led to better implementations and better PRs. The agents caught more issues and more thoughtfully designed the features due to a lack of fear of conflicts. And the most exciting part? Moo's configuration means you can do this for all your parallel agentic work. Imagine 8 tasks all running with the same fervor. That's the idea behind Moo. As we’ve been building [Ito](https://www.ito.ai?utm_source=blog&utm_medium=post+mention&utm_campaign=moo+post) (a tool that runs your code in a sandbox to catch bugs that only show up at runtime), we learned how important running isolated environments is for every branch. Moo came out of solving that for ourselves. We're now open-sourcing it, because we know the underlying problem isn't specific to us. ### The gap left by worktrees git worktree gives every branch or agent its own folder. Most software developers coding with AI use this every day to allow multiple agents to build multiple things at the same time. The problem is that worktrees do nothing for your database, ports, installed packages, nor the background services your app depends on to actually run. Your agents get separate files, but a shared runtime. They step on each other's database, fight over port 3000, and silently stomp each other's dependencies. And when you roll back to an old commit to investigate a bug, the database still has today's schema, so the old code doesn't behave the way it did when that commit was current. Worktrees fix the file half of the problem, but the runtime half stays broken. ## How Moo 🐄 helps you **Git versions your files. Moo versions your machine.** *(Note: The name is inspired by copy-on-write (C-O-W), which makes forking fast. That then lends itself to calling this "moo".)* Moo gives every one of your branches, worktrees, or agent attempts its own hardware-isolated Linux machine: its own database, ports, packages, and services. The machine's state is saved per commit and restored by git checkout. And best of all, it’s extremely easy to start using. ### How to Moo: Moo is just four commands: new, run, save, drop. You call it like this: > moo new feat/billing moo run feat/billing -- npm run migrate git commit -am "add billing migration" moo save feat/billing Each machine is a full Linux VM. It forks in under a second. Six machines on a 20 GB base image cost megabytes, not gigabytes, because they share storage using copy-on-write. When the machine's state is saved per commit, git checkout doesn't just rewind your code. It rewinds the database and environment to match: > git checkout HEAD^ moo new feat/billing The migration is gone. The database matches what the code expected at that commit. **The machine becomes a versioned artifact of your repo, exactly like your code is.** ### *But wait, there's more...* Most people try Moo because of how it helps them run parallel agents. They keep using it for two reasons they didn't expect going in. **1) Time travel.** Rolling back to a specific commit now actually works. The runtime follows the code, so you're not debugging against a database that's three migrations ahead of where your code expects to be. Say a bug slipped to production last Tuesday. Your customer reports it Wednesday. You've shipped two more migrations since then. You checkout Tuesday's commit, start the app, and the numbers are wrong, but not the same wrong as what your customer saw. That's because your local database has *today's* schema, not *Tuesday's.* **With Moo, all you have to do is:** > git checkout abc1234 # last Tuesday's commit moo new debug/tuesday # boots the exact runtime from that commit moo run debug/tuesday -- npm start You're now running exactly what your customer ran. Same code, same schema, same seed state. The bug reproduces. Now, you can actually fix it. **2) Runtime-aware bisects.** A bug that only shows up after a specific migration runs is normally brutal to track down with git bisect, because bisect rewinds the code, but leaves your database wherever it is today. With Moo, each step of the bisect boots the runtime that existed at that exact commit: > git bisect start git bisect bad HEAD git bisect good v1.2.0 git bisect run bash -c 'moo new probe && moo run probe -- npm test' Every commit bisect probes, Moo restores the database state that existed at that commit. If the bug lives in migration state, not just code, bisect finds it the same way it would find any other regression. **A bug tied to database state used to mean manually reconstructing an environment at every step. Now, you run git bisect and the runtime follows the code.** ### Two questions you might be asking... While we enjoy using Moo, we can understand if there are some skeptics out there. To address the common questions we get about Moo: ***1) Why not just spin up a cloud VM per agent, or use devcontainers / Docker Compose?*** You can, and a lot of teams do. The difference is that none of those are connected to your git history. Docker Compose gives you isolated services, but it doesn't know what commit you're on. Rolling back the environment means tearing down and rebuilding, not git checkout. Moo is local and git-connected. If you break an environment, you can now reset it the same way you'd reset bad code. No infrastructure to provision or tear down for every agent you spin up. ***2) Do I still need*** git worktree ***with Moo?*** Yes, and this is worth being clear about: Moo and git worktree solve different halves of the same problem. git worktree isolates files. Moo isolates the runtime: database, ports, packages, services. You need both. The full setup is one command each. Add a worktree, give it a machine: > git worktree add ../app-agent-b -b agent/b moo new agent-b from base Files isolated by worktree, runtime isolated by Moo. It’s that easy, and it’s why John's running eight of them at a time now. ### Try it for yourself The repo is at [github.com/heyito/moo](https://github.com/heyito/moo). Go try it, and tell us what's missing. Or, you can install it now with: > curl -fsSL https://github.com/heyito/moo/releases/latest/download/install.sh | sh *Note: Moo is alpha software. Right now it only runs on macOS Apple Silicon. Linux is planned.* --- # The AI Model Plateau: Why Your Infrastructure Matters More Than the Next Release AI model routing and evaluation now decide your production results. Learn how companies are adapting to make the most of the highly competitive LLM landscape. By Evan Marshall · 2026-08-21 The days of 1:1 model comparisons are over. At one point, it felt like a game of leapfrog, with each release from OpenAI beating the last release from Anthropic, until Anthropic jumped ahead again. Some people stuck to their preference, knowing the latest release would temporarily put them back on top, while others switched back and forth. Now, with open-source options like KimiK3 competing on intelligence, and OpenAI and Anthropic bragging about pricing optimization as often as they do model quality, the field is murkier. Sheer model quality is no longer the differentiator it once was, which means your choice of model isn’t either. The value chain is shifting, and pipelines are increasingly becoming the primary constraint. How do you route work? What do you cache? How do you evaluate the results? How quickly can you adapt your harness when something new or better comes along? The companies that can answer these questions will have better, cheaper results than the companies that can’t, even if they work with the same models and in the same problem space. The best returns will come from investing in your infrastructure and in the continuous optimization necessary to remain loosely coupled as the field keeps changing. In short: Keep your options open, and don’t tie your harness too tightly to one specific model. That’s the TL;DR, but we’ve come to this opinion only after a lot of experimentation. **Why we’re writing this**: Ito is the only AI code review that actually runs your code (where we use roughly 100 billion tokens a week). The numbers here come from our own experience experimenting with many different models and pipeline configurations. ## Listen to the labs: Progress is plateauing The vendors describe their own releases in increasingly modest terms over time, and the features they lead with tend to focus on workflow and efficiency rather than on output quality. To be clear: a plateau is not a collapse. Progress is still happening (as our experience switching to Luna demonstrates), but it is becoming more incremental and less paradigm-shifting. Take, for example, Anthropic's [Opus 4.8 announcement](https://www.anthropic.com/news/claude-opus-4-8) (published in May of 2026), which says, *"Users will find Opus 4.8 to be a modest but tangible improvement on its predecessor."* Simon Willison, co-creator of Django, pored through the spec sheet and agreed, [writing](https://simonwillison.net/2026/May/28/claude-opus-4-8/), *"Not much has changed since 4.7. [...] Both the reliable knowledge cutoff and the training data cutoff are January 2026, the same as for 4.7. The context window is still 1,000,000 tokens, and the max output is 128,000 tokens."* The big feature accompanying Opus 4.8 was dynamic workflows. Dynamic workflows let Claude plan a job, dispatch hundreds of parallel subagents, and verify the results before reporting back. Cool? Of course. But if the model itself were dramatically better, they wouldn't need workflow tooling to paper over its limitations. The case is similar with OpenAI. In its GPT-5.6 [announcement](https://openai.com/index/advancing-the-price-performance-frontier-with-gpt-5-6/) (published in July of 2026), they focus on efficiency rather than capability, writing, *"By making every layer more efficient, OpenAI is delivering stronger performance per dollar."* Their [companion engineering post](https://openai.com/index/gpt-5-6-frontier-intelligence-efficiency/) walks through where the gains came from: - Better load balancing across geographies and accelerator types. - Kernel rewrites that cut end-to-end serving cost by 20%. - Speculative decoding improvements worth more than 15% on token generation. Price cuts are exciting, and they can make a big difference in what users can afford to actually do. But a price cut is not a model improvement, and it results from fixing routing and caching. It’s incremental and iterative. The plateau even shows up within product lines. On [Artificial Analysis](https://artificialanalysis.ai/models#intelligence), the intelligence gap between Luna and Terra is only a few points. Opus to Fable looks similar. The tiers are converging on quality and separating on cost and speed. Your choice of one model over another is no longer about intelligence gains. ## Their benchmarks are not your benchmarks Each model provider compares their models against benchmarks to disprove this, to signal that model choice remains the most important decision. But their benchmarks don’t necessarily correlate with what you need the models to do. That’s why we [ran our own research](https://www.ito.ai/blog/tokens-per-second-measures-emission-not-work): the benchmark results didn't predict what we saw in production. We tested 14 models on a single fixed coding task, with 6 runs each. KAT-Coder-Pro V2.5 was the third-fastest emitter at 113 tokens per second and finished tenth because it burned 5,536 tokens. GPT-5.5 needed over 3 times fewer tokens at only 1,777. Meanwhile, the once-hyped Kimi K3 came in last at 11.5x Luna's wall clock. Within a single product line, Luna took 15 seconds and $0.012, Terra took 28 seconds and $0.042, and Sol took 59 seconds and $0.079. Paying 5x more per token bought 3.8x more waiting time. Price per token also failed to predict price per task. Kimi K2.7 Code has more expensive tokens than KAT-Coder and finished the job at 37% less cost. GLM-5.2's tokens cost 2.2x MiniMax M3's, yet per task the two landed at almost identical costs. And on the extremes, DeepSeek V4 Pro finished at a cost of only $0.0017, roughly *73 times* cheaper than Claude Fable 5, despite the task being identical. Our data isn’t unique in this regard. Artificial Analysis [compares](https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index) output tokens per Intelligence Index task, and the verbosity ordering matches what we measured: MiniMax M3 at 24.0k, Kimi K2.7 Code at 17.7k, GPT-5.5 high at 10.1k. DoorDash came to the same conclusion running their [code review agent in production](https://careersatdoordash.com/blog/doordash-built-an-ai-code-reviewer-engineers-actually-listen-to/). In the engineering blog post on the topic, the authors write, *“The cheapest model is not always the cheapest review. Several stages produce structured JSON. For simple schemas, a faster model works well. For more complex schemas, weaker models sometimes produced invalid output and retried multiple times, while a stronger model produced valid output on the first attempt.”* They hit on a good summation, which applies well beyond building code review agents: *“The unit that matters is not token price. It is cost per successful review.”* That’s why you need to try the models against your system and use cases, not just look at benchmarks and leaderboards. The more you do so, the more you’ll find that your pipeline influences your results as much (or more) than the model leaderboards and benchmarks. ## The constraint moved from the model to your pipeline Many companies (including ours) just had an especially vivid moment comparing models to see how the results varied. It feels like everyone spent the last month or two furiously trying new open source models, then trying the newest cool models again, only to end up back with OpenAI. OpenAI won this round because their speed and cost are (so far) impossible for even open source to beat. Our collective surprise points to an underlying point: Model improvements, until you measure them in the real world, are hard to predict. That’s why your pipelines are increasingly the primary constraint. If you can’t easily test new models, swap them in and out for different tasks, and optimize the pipeline itself, you’ll be left behind; if your competitors can easily swap models and you can’t, they will have have major gains in speed, cost, and quality that you’ll struggle to keep up with. Now, if you want to better keep up, we’ve found three questions to ask that can guide your efforts. ### 1. Which tasks go to which model? Different models handle the same tasks differently. Given the same task, some will perform better or worse (or fail altogether). Some will also cost more or fewer tokens. Further complicating things, often the quality doesn’t correlate well with the cost. DoorDash's [DashBench numbers](https://careersatdoordash.com/blog/how-we-learned-to-trust-our-ai-code-reviewer-at-doordash/), for example, show how much your results can vary depending on the model mix inside your architecture: | Configuration | Weighted recall | Weighted F1 | Cost / PR | | --- | --- | --- | --- | | Kimi K2.6 scout + Claude Fable 5 reviewer | 65.2% | 75.3% | $3.81 | | Claude Sonnet 4.6 scout + Opus 4.8 reviewer (their production config) | 53.6% | 66.3% | $3.91 | | Claude Sonnet 5 scout + Claude Sonnet 5 reviewer | 40.1% | 52.8% | $6.55 | | Composer 2.5 scout + GPT 5.5 medium reviewer | 18.0% (92.2% precision) | 30.1% | $3.53 | For context, weighted recall is the share of known issues the system catches, with higher-severity issues counting more. Weighted F1 balances precision and recall, giving more weight to higher-severity issues. Of these configurations, the Sonnet 5 pairing costs the most but has the weakest coverage. The Composer pairing is most precise, but catches far fewer of the known issues. No configuration dominates, so the right choice depends on what you are optimizing for. As the DoorDash engineers write, *'Best' is meaningless until you say best at what, in which cases, at what cost.* ### 2. Which probabilistic steps can you make deterministic? Not every step needs to be probabilistic. Thoughtworks calls this pattern “[feedback sensors for coding agents](https://www.thoughtworks.com/radar/techniques/feedback-sensors-for-coding-agents)”: compilers, linters, type checkers, and test suites wired into agent workflows so failures trigger auto-correction before humans need to see anything. DoorDash applies similar logic to what they put in front of the model. When they designed their AI code reviewer, they ensured that every potential review rule survived a filter before it made it into a profile. They write: *“If CI would already catch it, drop it. If the LLM knows it from general training, drop it. If we can't point to a concrete file-and-line in the codebase as evidence, drop it.”* The stuff that remains is review knowledge that's actually specific to their codebase. Only after that do they start routing, as they write, *"The routing is a big part of why acceptance rates hold up across 56 very different repos. The agent isn't applying a single universal standard. It's applying the standard that matters for that specific change."* ### 3. Can you optimize the provider layer? The provider layer has more optimization space than you might think. Researchers from the University of Pennsylvania, Columbia University, and OpenMesh AI built a [continuous benchmark](https://arxiv.org/pdf/2605.00300) that measures inference at endpoint granularity across five axes: - Output speed - Time to first token - Workload-blended price - Effective context - Quality on the live endpoint They found, *“The same model name on different providers is not the same product.”* In their research, open weights such as Llama 3.3 70B and gpt-oss-120B were often served by 20 or more providers. As a result, they write, *“Quantization and serving choices change the model’s behavior in measurable, sometimes undisclosed, ways.”* Despite that, leaderboards tend to hide this variation by aggregating at the model or provider level. That means there’s optimization potential, and often quite a lot of it, at the provider layer. Our own experience demonstrated a [similar pattern](https://www.ito.ai/blog/measuring-100b-tokens-week-open-weight-models). We tested GLM-5.2 and MiniMax M3 on Vercel AI Gateway, OpenRouter, and Together AI. Vercel had by far the worst cache hit rate for both models, and scored the worst on our internal evals against the same underlying providers reached through OpenRouter. Same model, very different performance. Cache behavior turned out to be the biggest point of divergence. About 97% of our input tokens are cache reads, so a provider that degraded caching upgraded the bill. If you like a model, but find yourself experiencing performance issues, you may want to try another provider before you swap models. ## Build infrastructure that enables hot swapping Vendor (and model) lock-in is one of the biggest risks AI-enabled companies face. Building tightly coupled infrastructure that doesn’t support hot swapping different models means you can’t be on the bleeding edge today and will be behind the times tomorrow. To keep your infrastructure flexible, ensure you can evaluate candidate models effectively, write good tests, and encourage experimentation so you’re not wedded to any component of your stack. ### Evaluate candidate models by specific agent job, and test the results and quality by task “Best model” is not a useful category, nor an effective way to experiment and build today. The better question is: “*What is the best model for this step, and at this quality floor?*” Fortunately, this change in framework doesn’t require a big expenditure. The cost of tokens is dropping, and that benefits both what you do in production and your experiments. Remember: our experiment where we ran the same test across thirteen different models? It cost us *three dollars.* Total. And the learnings from our small experiment proved out when we started using the best-scoring models in production. DoorDash gets at the same idea from the measurement side, [writing](https://careersatdoordash.com/blog/how-we-learned-to-trust-our-ai-code-reviewer-at-doordash/), “*The useful question stops being 'which model is best?’ It becomes: for a given architecture, context policy, tool policy, runtime budget, and model mix, what tradeoff between coverage, precision, cost, and latency do we actually get, and where does it fail?*” The key phrase here is “actually get.” Experiments like ours close the gap between what’s hyped on X or wowing on benchmarks, and what actually will work best for your use case. You need to experiment to find out. ### Write tests that mimic tasks you need AI to do in your system Start simple. Pick an agent step, and build a test around what that step really does in production. For us, that meant a real test batch running against a real browser in a devcontainer. That was our largest LLM cost. The tests need to fail for reasons you can act on. Capture the input, the tokens it burned, the wall clock, and the output, and then score the output against something you wrote in advance. Include cases where the right behavior is doing nothing. Then, keep the set alive. Retire cases that stop discriminating between models, and add production misses as new cases. That corpus turns into a durable asset for you. Models will keep getting better, and your eval set is how you find out whether the improvement means something for your workload. We only learned that leaderboard rankings didn’t survive contact against our own workload when we tested the models ourselves. ### Encourage a culture of experimentation Don't shoot the messenger because one of your AI-pilled staff is excited about a new model. Ask how you can let them experiment safely. That might mean offering them a smaller product surface. That might mean working with a subset of customers who opt into the latest and greatest, knowing there will be some issues. Or you can hold semi-regular hackathons where it’s safe to try new tooling, setups, and models. The goal is to make bad results cheap, then reward the behavior of experimenting in your actions and in words. If a developer experiments and the results show their idea isn’t worth pursuing, the results are still useful. Now, the next three developers who might have tried the same idea know not to repeat it. Empower experimentation with real budget, real access, and a slot in the roadmap. After all, breakthroughs can only happen when you put in the time. It’s similar to [increasing your luck surface area](https://www.codusoperandi.com/posts/increasing-your-luck-surface-area). If you don’t experiment, breakthroughs will never happen. If you do experiment, breakthroughs become at least possible and, over a long enough period of time, inevitable (albeit unpredictable). That was our experience. Ito's average runtime dropped by 50% over 4 weeks due to model switches, while costs fell by over 10x. We learned a lesson we intend to repeat, and we hope you do, too. ## Avoid tightly coupled systems (again) Have you heard this one before? Avoid tight coupling. Again. Tight coupling feels efficient right up until the thing you coupled to changes underneath you. This is the same lesson the industry has learned repeatedly with databases, cloud providers, and orchestrators. The only new dynamic here is the half-life. Model generations turn over in weeks now, not years. Remember that this is a good thing. You win either way. If the plateauing trend holds, you can route queries and swap model versions as necessary, optimizing costs and improving results while everyone else waits for the next big leap. And if a genuinely significant shift does happen, your pipeline is ready to integrate it and take advantage as soon as it’s released. There’s a balance to strike. Some of this is outside of your control. After all, when OpenAI dropped Luna, our average runtime fell by 50%, and costs fell by more than 10x. We could have never planned for that. What you can control is your pipeline, and that’s why it will be a key differentiator going forward. Theoretical improvements are one thing. Putting them into practice is another at a rate that keeps up with the pace of AI is another. Is your team ready? > **Struggling to keep up with the pace of AI code generation?** Get AI code review that runs your code by [signing up for a free trial of Ito here.](https://app.ito.ai/auth/signup?utm_source=blog&utm_medium=post+mention&utm_campaign=ai+model+plateau&utm_term=free+trial+end+of+post) ## Frequently asked questions ### Why should I evaluate models on my own workload instead of public benchmarks? Public LLM benchmarks measure a fixed task under a fixed setup. Your agent has its own prompts, tools, context policy, cache behavior, structured-output requirements, and failure modes. A model that leads a public benchmark can still cost more, retry more often, or miss the issues that matter to your users. Run candidate models against representative production tasks and measure quality, latency, and cost per successful task to know if a model is actually a good fit for your product or use case. ### Does model routing replace the need for a strong frontier model? No. Routing gives each task the model that meets its quality floor at the right cost and latency. Some steps need frontier-level reasoning. Others are deterministic, well-scoped, or easy to validate and can run on a faster, lower intelligence model. The work you need to do is to create the evals and guardrails that tell you which category a step belongs in, and then match it to the right model. ### When should a team try another provider before changing models? Try another provider when the same model shows unexpected latency, cost, or quality changes. Providers can differ in caching, quantization, serving configuration, geographic routing, and endpoint reliability. It’s a good idea to compare the same workload across a few providers before replacing a model that otherwise meets your quality bar. --- # Static Analysis Misses Bugs That Reach Production Agents can stare at code all day long, but they will not find all the issues because they are probabilistic unit tests. You need to actually run your application to get runtime evidence and proof that things behave as your organization expects them to for full integration and end-to-end tests. By Evan Marshall · 2026-06-12 The Static Analysis Ceiling: Missing Bugs That Reach Production *Code review analyzes your diff. Behavioral tests check if users will run into bugs. Teams need both.* AI coding completely changed the risk equation for teams. Teams are generating code faster than they can confirm it works. Yet they can’t move slower because customer expectations have never been higher and competition has never been more fierce. Yet real business risk can be sitting in a single pull request, and current tools aren’t enough to catch it. Static code analysis is just one step. These code review tools read the change where behavioral testing runs the change. Both look at the same code change from different layers, and the difference is that one mirrors how users will use your application. Code review answers the question “does this diff look reasonable?” That’s not what actually matters to your users. They care about your software helping them achieve their outcome. > **KEY POINTS** → Code review answers whether the diff looks reasonable. Behavioral testing answers whether a user can still complete what they came to do once the change ships. → Static analysis reads source and cannot observe what the application does at runtime. Auth, data states, race conditions, and live integrations sit in that blind spot. → Static analysis code review tools like CodeRabbit and Greptile study the diff. They belong alongside behavioral testing on the same PR, not in competition with it. → Behavioral testing is end-to-end testing written around user intent rather than selectors, so an agent navigates the flow and there is no script library to maintain. ## The rise of AI code review AI review tools made the reading step faster and more consistent. CodeRabbit, Greptile, and the tools in their category pull in repository context, surface logic that looks suspect, flag API contract changes, and point a human reviewer at the parts of a diff worth their attention. As coding agents push more code into review than teams used to see, that consistency keeps the review queue moving. They ensure no unit tests would be breaking for your team. These tools hit a structural limit. They can reason about code but don’t observe what a running application does to a user's task. Seeing what happens from runtime behavior and being able to confirm it with evidence leads to catching over 30% more issues. ## What static analysis can't catch Reading a change and running it surface different information. Runtime bugs can look completely reasonable in the diff yet break the moment you run it or a real user touches it. A database migration for a fintech company that looks good but during the rollout would cause double entries for all active customers. Every file involved looks correct on its own, the unit tests pass, and the issue only occurs during the actual migration. A returning customer opens the app to finish a purchase, and an expired session drops them into a login loop instead of letting them back in, because the token refresh check runs after the redirect instead of before it. The order is what they came for, and the broken sign-in is where it stalls. Anyone with a fresh session sees nothing wrong, and the test against the refresh helper is green. The loop only shows up when the app runs with a stale token and real routing. A drag-and-drop reorder looks correct in the UI, then reverts after the user navigates away and comes back, because the reorder updates local state and never calls the API. A reviewer can see that branch in the diff. Confirming the outcome takes leaving the page and returning, which is something only a running app does. None of these are careless. Each one depends on state, timing, browser behavior, or an integration path that no amount of reading can observe. This is the structural limit of static analysis: it can only inspect the code, and the failure is in what the code does. ## The Evolution to Behavioral Testing Behavioral testing confirms that a person can complete a real task in your app, by running the app and exercising the flow the way a user would. It ties code changes to business outcomes and user flows, not flaky test suites. An agent opens the affected paths, clicks through them, fills in inputs, waits for the application to respond, and then checks the outcome that matters. Did the migration work. Does the new integration cover all edge cases. Did the order go through. Did the returning user get back to what they were doing. Did the API survive the upgrade. It can observe UI state, network behavior, session handling, and the result the user is left with. The question behavioral testing answers is the one code review cannot reach: can the user still do what they came to do once the change ships. Screenshot diffing compares pixels and tells you the page looks the same, which is a different claim from confirming a user completed the task. Behavioral testing is about the outcome, not the appearance. The unit being validated is the user's goal, not any single step. A person signs in to reach something on the other side of the login, so a working sign-in matters only because it clears the path to the task waiting behind it. Behavioral testing checks that step as part of completing the task, rather than as a box to tick in isolation, which is also where it parts ways with scripted suites, which often assert that login works in isolation and call the flow covered. ## Testing accelerates, and the landscape is changing Behavioral testing is a shift-left end-to-end testing because it runs the whole application in a real environment. The approach differs from other end-to-end (e2e) test products in how the test is written and what it asserts against. Scripted end-to-end testing in CI with Playwright, Cypress, or Selenium works well when the UI is stable and the team has capacity to maintain the suite. The cost is that the assertions are tied to selectors and DOM structure, so the tests break when the markup moves and someone has to keep them current. Record-and-replay lowers the authoring cost of these e2e tests by capturing a session instead of hand-writing it, but it inherits the same brittleness, since the recording is coupled to the underlying structure. Manual QA brings judgment that automation does not. This is crucial in testing edge cases and exploratory work. E2e test suites for this type of work aren’t worth the effort to build, and asking engineers to build them is how the queue backs even further up. Agentic QA solutions like Ito generate the test from the code change itself. The agent navigates like a user and validates the behavioral outcome, so there is no library of selectors to maintain. The test is written around what the user is trying to accomplish, not around the structure the engineer happened to ship. Same scope, different question: scripted suites confirm the code does what it was scripted to do, while behavioral testing confirms the user can finish the task they came for. ## Where Static Analysis ends and Ito begins CodeRabbit and Ito answer different questions on the same PR, and they sit together rather than compete. CodeRabbit reviews the diff and catches code-level problems: shaky logic, a missing edge case, an unclear pattern, an API contract that changed. Ito runs the changed application in a real environment and catches behavior-level problems: the checkout that dies on submit, the returning user locked out by an expired session, the change that saved on screen and never persisted. Both post their findings to the PR. Before anyone merges, the reviewer sees the code analysis alongside evidence of what the running app did with a real flow. That is a stronger basis for a merge decision than either half alone, and the reviewer still makes the call, now with more of the picture in front of them. ## Building a complete QA stack Each layer of a QA stack catches a different failure class, and the layers are additive rather than interchangeable. Linting handles style and syntax. Unit tests check logic in isolation, one function at a time. AI code review reasons about the diff and flags code-level risk. None of these sees whether a real user can complete a task in the running app, and that is what behavioral testing covers. A team that skips a layer is not removing those failures from the world. It is choosing not to see them until a user does. For teams shipping at the speed coding agents now allow, the layer most often missing is the last one, the one that confirms the people using the product can still finish what they started. The forward-looking version of this is a development loop where behavioral validation is a property of every change, not a phase bolted on at the end. A developer opens a PR, and before the reviewer clicks in, the impacted flows have already run in a real browser and the results are waiting in the thread. The reviewer starts from behavioral evidence instead of from a guess. The bottleneck in modern software development has moved downstream, from how fast a team can write code to how quickly it can trust the code it wrote. Faros, analyzing 22,000 engineers in 2026, found the strain landing exactly there: as AI adoption deepened, bugs per developer rose 54% and the workflow slowed most at the point of review. Behavioral testing is what turns a diff that looks safe into evidence that a real person can finish the task it touches. > **How this looks in practice** Ito connects to your GitHub repo and configures an isolated environment for each pull request, so there is no shared state between runs. Adding testing credentials such as secrets, variables, or seed data is optional. On every PR, Ito deploys the change, runs the user flows it affects, and posts the results before anyone merges. When a flow breaks, the PR comment includes which flow failed, what the agent did, what the app returned, a video and screenshots of the failure, the steps to reproduce it, and the lines responsible. [Get started →](https://app.ito.ai/auth/signup) ### **By the numbers** Faros tracked 22,000 developers across 4,000 teams over two years. As AI adoption rose, median PR review time increased fivefold and incidents per PR tripled. The volume of code going into review grew. The ability to trust it before it shipped did not keep pace. Code review did not absorb the difference just slowed down under the weight of it. ### Your team reviews the code. Who confirms a user can still get through the app? Connect your repo and add behavioral testing to every pull request. Ito configures the environment, so there is no infrastructure to stand up first. [Get started →](https://app.ito.ai/auth/signup) ## Frequently asked questions ### Why do some bugs slip past static code analysis and only show up in production? Static analysis inspects source without running the application, so it cannot observe runtime state, timing, real data, or live integrations that often trigger production-only failures. ### What kinds of bugs do static analysis tools typically miss? Issues that depend on runtime configuration, concurrency and timing, migrations, session or token handling, UI state that never persists, and third-party integration behaviors are commonly missed. ### Does behavioral testing replace code review? No. Code review evaluates the quality of the diff and the code itself, while behavioral testing validates that a real user can complete their task when the change runs, and both belong on the same pull request. ### How does behavioral testing differ from traditional end-to-end tests? Both run the whole app, but behavioral tests are written around user goals and use agents to navigate flows rather than relying on brittle DOM selectors, which reduces maintenance overhead. ### How does Ito integrate into a pull request workflow? Ito connects to your repository, deploys each PR into an isolated environment, runs the impacted user flows, and posts the results back to the PR before anyone merges. ### How long does it take for Ito to test a pull request? Ito can run the affected flows and post a full testing report within about an hour of opening the PR. ### What information does Ito provide when a flow fails? When a flow fails, Ito includes which flow failed, what the agent did, the app response, a video and screenshots, reproduction steps, and the lines of code responsible. ### Sources 1. Faros (2026): *[The Acceleration Whiplash](https://pages.faros.ai/hubfs/AI_Engineering_Report_2026_The_Acceleration_Whiplash_Faros.pdf)* --- # 8 AI Code Review Tools, Compared on Setup, Output and Features Compare 8 AI code review tools by setup, features, review speed, and pricing. See what evidence each provides and which fits your team's workflow. By Evan Marshall Every engineering team eventually hits the same moment. The number of PRs needing review grows faster than the team’s capacity to review and approve them, and nobody wants to respond to that by skipping review and shipping straight to production. The question becomes which parts of that load can move to an automated reviewer. Automated review adds another layer to your workflow. The right tool should return evidence your team can act on and keep costs predictable as PR volume grows. **You’re probably dealing with one of two situations right now.** Maybe your team adopted AI-assisted development and PR volume is rising fast enough that automated review needs to be part of your broader AI tool stack. Or maybe you already have a reviewer in place, and you’re past that stage. You want more coverage, you’re looking to delegate even more to AI, and you might even have enough confidence to let low-risk changes merge without a human involved. That second situation depends on your test coverage and the quality of the review evidence the tool returns. Either way, you’re looking for a tool, so we did the legwork so you don’t have to. We evaluated 8 tools across 5 axes covering setup requirements, output evidence, feature controls, review latency, and billing at volume. One thing worth knowing before you start comparing: installing a GitHub App is only the first step. Environment setup, credentials, and data access can add time before the tool returns useful findings. ## Our picks by review job The table below shows what each tool offers and who it's best for. It includes our own product, Ito, which runs your app to check how code changes behave and shows you how to reproduce the bugs it finds. ## How we compare the automated code review tools For each tool, we recorded setup prerequisites separately from vendor-reported install times. A GitHub App can be installed in 2 minutes, but having secrets, private package access, and seed data configured for a useful run can take much longer. We keep each vendor's severity labels and identify published latency figures as vendor estimates. If a vendor does not publish a latency figure, we say so. Billing units vary by tool. Some charge per seat, some per review run, some per token. The cost arithmetic in the decision section uses consistent assumptions so the numbers are comparable. For practitioner evidence, we use [DoltHub's published review of Ito](https://www.dolthub.com/blog/2026-07-20-ito-ai-qa-for-doltgresql/) on 92 DoltgreSQL pull requests over 6 weeks, 43 of which were substantive enough for a confirmed-versus-dismissed accounting. That article is public and attributed to DoltHub. For the other 7 tools, we use vendor documentation, pricing pages, and public PR examples. These examples show individual findings rather than a multi-week evaluation like DoltHub's. ## The 8 AI code review tools for engineering teams Each entry below covers the same ground, including setup, output, latency, and pricing, followed by a limitation and a best-fit call, so the eight tools compare on equal terms. ### Ito [Ito builds your application](https://ito.ai/) from source on every pull request, runs the flows the changed code touches, and returns what it observed rather than a reading of the diff. Web app changes can return a video replay of the flow, while backend and API changes return scripts, logs, and text evidence. **Key numbers:** - **Setup:** Install that GitHub App at [app.ito.ai](https://app.ito.ai/auth/login), complete your GitHub OAuth, and select your repository. Depending on your stack, to get the best test results, you may also need to provide sandbox secrets, private-package access, and seed data. - **Platforms: **GitHub. - **Output: **PR summary, failed-test details, separately labeled Additional Findings, and severity ratings of Critical, High, Medium, or Minor. UI flows get a video replay, and every finding includes reproduction steps. Follow-up commits get a diff report with a Bugs Fixed section for tests that failed in the prior run and now pass. - **Latency: **20 to 90 minutes, depending on your codebase’s complexity. - **Pricing: **Pro plan at $40/seat/month, with 50 reviews included per seat and $1 per additional review. The first 20 reviews are free with no credit card required, and qualifying non-commercial MIT/Apache projects get free access. Ito reports pre-existing findings without letting them determine whether the run passes or fails. **Example: **[DoltHub's review of Ito](https://www.dolthub.com/blog/2026-07-20-ito-ai-qa-for-doltgresql/) gives a good example of what Ito's evidence looks like in practice. Ito found that a named window reference (OVER w, defined with WINDOW w AS (...)) returned a full-partition sum instead of a running sum. The equivalent inline OVER (PARTITION BY ... ORDER BY ...) returned the correct value. That kind of bug is dangerous because it doesn’t immediately look like a bug. The query runs clean and returns a result, just the wrong one. And anything built on that aggregate, a running balance, a total, a rank, etc., carries the error forward with nothing in the output to flag it. It only shows up once the database actually executes the query. The reproduction steps pointed straight at the lines responsible. **Best for.** Teams that want to expand their automated testing to cover runtime issues, so manual QA and brittle Playwright tests aren’t the only way to catch them. [Try your first 20 reviews free with Ito, with no credit card required.](https://ito.ai/pricing) ### CodeRabbit [CodeRabbit](https://ito.ai/compare/ito-vs-coderabbit) reviews pull requests across four Git platforms and posts inline findings, suggested fixes, and a walkthrough summary. Its open-source plan runs Team-level features free on public repositories. Open-source projects get its Team-tier feature set on public repositories automatically, with no application required. **Key numbers:** - **Setup: **Install the GitHub App, GitLab app, Azure DevOps integration, or Bitbucket connection. Optional .coderabbit.yaml for version-controlled configuration. Reviews trigger automatically on new PRs and incrementally on new commits. - **Platforms:** GitHub, GitLab, Azure DevOps, Bitbucket Cloud, and Bitbucket Data Center. - **Output: **AI-generated PR summary and walkthrough, including sequence diagrams where relevant, inline comments on changed lines, one-click fix suggestions, and interactive chat within PR comments. - **Latency: **Not published by vendor. - **Pricing (annual billing): **Essentials $24/developer/month, Team $48/developer/month, Advanced $72/developer/month. The free plan works on unlimited public and private repositories, but is limited to PR summaries. Open-source projects on public repos get the Team feature set automatically, no credit card or application required, under a separate rate-limit tier that scales with the repo’s popularity rather than Team’s standard limits. A 14-day trial and qualifying open-source access are available as separate access types. - **Rate limits (PR reviews per developer/hour):** Free 1, OSS 1–10 (varies by repo popularity), Essentials 5, Team 8, Advanced 10, Enterprise 12. CodeRabbit covers four major Git platforms, including Bitbucket Data Center, without requiring self-hosting to get there (unlike others such as Qodo). Self-hosting is also available for Enterprise customers with 500 or more seats. What sets CodeRabbit’s open-source offer apart is how it’s granted, not how many platforms it covers. CodeRabbit is the only tool in this comparison that gives open-source projects full feature access with no application or eligibility review. Qodo, Ito, Graphite, CodeAnt AI, and GitHub Copilot all gate their equivalent programs behind applying or qualifying. **Example: **In [one public pull request](https://github.com/HadesArchitect/ExpoPlusCodeRabbit/pull/37#pullrequestreview-4670717373), CodeRabbit flagged a Functional Correctness bug in a nextIndex function. Operator precedence makes current + 1 % length evaluate as current + (1 % length), so the index never wraps. The suggested fix wraps the addition in parentheses and adds a length check. **Best for.** Teams that want a consistent source review across multiple supported Git providers. [Compare Ito vs. CodeRabbit to see where runtime validation adds coverage.](https://ito.ai/compare/ito-vs-coderabbit) ### Qodo [Qodo](https://ito.ai/compare/ito-vs-qodo) links review findings to tickets, learns rules from your PR history, and lets teams codify engineering standards through a .pr_agent.toml file or an optional REVIEW.md layer. Specialized agents check each change in parallel for correctness, standards compliance, architecture, and risk. A judge agent then combines their findings, removes duplicates, and filters out low-confidence results before they reach the PR. **Key numbers:** - **Setup: **Connect to GitHub, GitLab, Bitbucket, or Azure DevOps through the Qodo portal. Settings in repo-, project-, or org-level .pr_agent.toml files override portal settings. REVIEW.md (Research Preview) adds plain-language review guidelines but is off by default; enable it in the portal or toml. - **Platforms: **GitHub Cloud and Enterprise, GitLab Cloud and Server, Bitbucket Cloud, Azure DevOps, and Gerrit. - **Output:** AI-generated PR description, severity-ranked findings with suggested fixes or an agent prompt, a walkthrough with a Mermaid flowchart of the change's logic, and ticket compliance checks when requirements are linked. Incremental reviews trigger on each push when enabled. - **Latency: **Not published by the vendor. - **Pricing:** $0.012 per pooled credit. Credit packs: 2,500 credits for $30/month (approximately 18 reviews), 5,000 credits for $60/month (approximately 36 reviews), 20,000 credits for $240/month (approximately 144 reviews). Rule Miner, in beta, looks at your team’s past PRs and writes review rules for you, which helps if your team’s review style isn’t written down anywhere. It works with GitHub, GitLab, and Azure DevOps, but not Bitbucket, so Bitbucket teams can use Qodo’s reviewer but not this feature. **Example:** In [one public pull request](https://github.com/ether/ep_comments_page/pull/441#issuecomment-4639660605), Qodo's review caught a missing event parameter that could crash modal navigation and a helper function whose undefined return values could abort navigation entirely. Both findings included an agent-remediation prompt and were later confirmed resolved. **Best for.** Teams that want to write down their review standards and enforce them, or check each change against the ticket it’s meant to address. [Compare Ito vs. Qodo to see how runtime evidence pairs with standards-based source review.](https://ito.ai/compare/ito-vs-qodo) ### Graphite AI reviews [Graphite's AI reviewer](https://graphite.com/pricing), Graphite Agent, sits inside a PR workflow built around stacked pull requests and a merge queue. Teams can adopt it for ordinary PRs on their existing branching model, though its value compounds for teams that already use stacks. **Key numbers:** - **Setup: **GitHub authorization. Graphite says its workflow installation takes under 5 minutes. - **Platforms: **GitHub. GitHub Enterprise Server support is listed as an Enterprise-plan feature and isn't available on Hobby, Starter, or Team. - **Output:** Inline findings with suggested fixes, Graphite Chat for per-file and per-change questions, a PR inbox with customizable filters, and review analytics. - **Latency: **Graphite claims its AI reviewer cuts feedback time from about an hour to about 90 seconds compared to waiting on a human reviewer. - **Pricing (annual billing): **Hobby (free, limited AI reviews), Starter $20/user/month (limited AI reviews), Team $40/user/month (unlimited AI reviews, merge queue, automations), Enterprise (custom pricing, adds GitHub Enterprise Server support, advanced merge queue, SAML, and audit logging). The merge queue is stack-aware. When you merge a stack, the queue handles dependency ordering and rebases branches automatically to keep them green. That's workflow infrastructure, and it changes the value calculation for teams that ship frequently. Evaluating Graphite as a reviewer means evaluating the entire workflow package. The Team plan bundles AI review with the CLI-driven stacking workflow, even if you only need review. Its pricing does not clearly separate the 2. **Example: **In [one public pull request](https://github.com/oven-sh/bun/pull/20945#discussion_r2198264284), Graphite Agent flagged that a PATH environment variable built from Bun.which('node') could start with a stray colon if the call returned null, and suggested a null check before assembling the string. Other comments on the same PR caught OS-architecture text mismatched between README files and their packages, plus a workflow path referencing the wrong file extension. **Best for.** Teams that want AI review integrated into a PR workflow and merge queue, especially those using or evaluating stacked pull requests. [Check Ito's current pricing and try your first 20 reviews free, with no credit card required.](https://ito.ai/pricing) ### CodeAnt AI CodeAnt AI reviews pull requests and can run SAST, secrets detection, and dependency scanning alongside the review comment. Teams evaluating both source review and application security checks can run them from a single tool. **Key numbers:** - **Setup: **Connect via GitHub, GitLab, Bitbucket, or Azure DevOps. PR review activates automatically on connection. A CLI is available for pre-commit hooks and CI pipeline integration. - **Platforms: **GitHub, GitLab, Bitbucket, Azure DevOps. - **Output:** Inline PR review comments labeled with a severity and an occurrence likelihood (for example, Major, Occurrence: Sometimes), one-click Fix in Cursor and Fix in VS Code Claude links, PR-level SAST findings, secrets detection, and configurable PR blocking gates for critical security findings. Custom instruction prompts let teams encode their own review standards per repository. - **Latency: **Not published by vendor. - **Pricing:** Premium includes unlimited PR reviews at $24/user/month with annual billing or $30/user/month with monthly billing. Its 14-day free trial requires no credit card and is capped at 100 PR reviews. Enterprise pricing is custom and includes on-premises/VPC deployment. Exploit simulation and attack-path mapping are part of CodeAnt's separate AI Pentesting product, outside the review plan's PR-level SAST and security gates. **Example: **In [one public pull request](https://github.com/Mihir-Rabari/inboxfm-connect/pull/301#discussion_r4124972065), CodeAnt AI flagged a test that validated a separately hardcoded snippet instead of the actual page output, so the test could still pass after the real URL, headers, or payload changed. **Best for.** Teams evaluating source review and application-security checks together. [Check Ito's current pricing and try your first 20 reviews free, with no credit card required.](https://ito.ai/pricing) ### GitHub Copilot code review [GitHub Copilot code review](https://github.com/features/copilot) is built into GitHub.com with no separate reviewer app to install. Teams already paying for Copilot can enable it from repository settings without additional procurement. **Key numbers:** - **Setup:** Enable through repository or project settings in GitHub. For GitHub.com, no separate app installation is required. - **Platforms: **GitHub.com, GitHub CLI, GitHub Mobile, VS Code, Visual Studio, Xcode, and JetBrains IDEs, plus Azure DevOps in public preview. - **Output:** Inline suggestions on changed lines, manual reviewer assignment or automatic review on new PRs, Lite or Balanced effort levels. Agent skills, repository instructions, and MCP context (including default GitHub and Playwright MCP servers) can expand what the review accesses. - **Latency: **Not published by vendor. - **Pricing: **Credit-based. Copilot plans are Pro $10/month, Pro+ $39/month, and Max $100/month, each including a monthly AI credit allowance worth more than the subscription price (Pro $15, Pro+ $70, Max $200). One credit equals $0.01 USD. Lite reviews consume an estimated $0.05 to $1.00 of that allowance per review, and Balanced reviews consume $0.25 to $5.00. Copilot Free does not include reviews. When Copilot review is configured as a required reviewer and enabled for automatic approval, it can satisfy a required-approval rule in GitHub's branch protection settings. CodeRabbit and Qodo each document similar approval-review mechanisms, each gated behind a separate setting. Automatic approval is itself in public preview, and GitHub dismisses the approval if new commits are pushed after it's granted. **Example: **In [one public pull request](https://github.com/ThomasMichon/copilot-extensions/pull/4472#discussion_r4128301284), Copilot flagged a description claiming that a policy engine's review_blocking=False state made the review verdict irrelevant. The repository's CONTRIBUTING.md still required a passing verdict and prohibited merging with open Medium or High findings. Several of its comments cited exact line numbers in the repo's AGENTS.md and CONTRIBUTING.md, grounding findings in the project's own written policy rather than generic style rules. **Best for.** Teams that want review integrated into existing GitHub workflows and Copilot policies. [Check Ito's current pricing and try your first 20 reviews free, with no credit card required.](https://ito.ai/pricing) ### Cursor Bugbot Bugbot reviews pull requests and links findings directly into Cursor's editing environment. When a finding appears in the PR, it includes a "Fix in Cursor" link that opens the relevant code with a pre-populated fix prompt. **Key numbers:** - **Setup: **Connect GitHub, GitLab, Bitbucket, or Azure DevOps repositories in the Cursor dashboard, then enable Bugbot in Automations. - **Platforms: **GitHub (including GitHub Enterprise Server), GitLab (including GitLab Self-Hosted), Bitbucket (including Bitbucket Data Center), and Azure DevOps, same as CodeRabbit. - **Output: **Inline PR comments with suggested fixes, a "Fix in Cursor" link per finding, and a severity label (Low, Medium, and High). Reviews run at Low, Default, High, or Smart effort, incrementally by default (focused on commits since last review), with .cursor/BUGBOT.md or team rules for custom guidelines. Autofix (beta) can generate a patch automatically. - **Latency: **Cursor's June 2026 changelog reports average review time now runs about 90 seconds, down from about 5 minutes. - **Pricing: **Usage-based since the May 2026 pricing update, effective for existing customers at renewal after June 8, 2026. Bugbot bills from Cursor plan usage or on-demand spend. Average run cost is approximately $1.00 to $1.50, varying with PR size and complexity, and higher effort levels increase cost. A legacy seat-plan page still exists for accounts that haven't transitioned. Going from finding to fix means opening the specific file in Cursor with context pre-loaded, which cuts triage-to-remediation time for teams already in that editor. Findings default to a neutral check conclusion. A neutral check doesn't block a merge on its own, and configuring a gate that fails on unresolved findings requires separate setup with availability varying by plan. **Example: **In [one public pull request](https://github.com/PrimeIntellect-ai/prime-agent/pull/3031#discussion_r4128464954), Bugbot flagged a selection-highlight bug. A fallback chain filtered Color::Reset out of the theme's accent color but not out of selectedBg. An explicitly empty selectedBg could therefore paint nothing, making a selected row look unselected. The maintainer confirmed and fixed it. The maintainer disputed and disproved a separate, higher-severity finding in the same PR about a monotonic clock value underflowing. It applied only to a build target the repository does not ship on. This is a reminder to verify Bugbot's findings, just as you would any other AI reviewer's. **Best for.** Teams that prefer Cursor's review and remediation workflow. [Compare Ito vs. Cursor Bugbot to see where runtime evidence adds coverage.](https://ito.ai/compare/ito-vs-cursor-bugbot) ### Claude Code Review Claude Code Review runs parallel agents that analyze the diff in the context of the full codebase, then passes findings through a verification step to filter false positives. The managed service is in research preview for Claude Team and Enterprise plans. **Key numbers:** - **Setup:** Configure through Claude Code admin settings at claude.ai/admin-settings/claude-code. Install the Claude GitHub App to your organization, select repositories, and set a review behavior per repository (once per PR, on every push, or manual). Spend caps are configurable and can cause reviews to be skipped if the cap is reached. - **Platforms:** GitHub.com and GitHub Enterprise Server. The managed service is GitHub-exclusive. - **Output: **Inline findings labeled by severity (🔴 Important, 🟡 Nit, 🟣 Pre-existing), a check-run summary with a neutral conclusion, and a verification pass that re-checks candidate findings against actual code behavior. Reads CLAUDE.md for project context, where violations surface as Nit findings, and an optional REVIEW.md for review-specific instructions that reach the finding and verification agents directly. - **Latency: **Approximately 20 minutes to results (Anthropic's published average). - **Pricing: **$15 to $25 per review on average, based on Anthropic's published figures. Cost scales with PR size and the number of issues requiring verification. Reviewing on every push multiplies runs and cost. Zero Data Retention plans are ineligible. The Pre-existing label distinguishes carry-forward issues from new ones, which makes the review output easier to triage. **Example: **In [one public pull request](https://github.com/oven-sh/bun/pull/37664#discussion_r4090841864), Claude Code Review found that a TLS client wrap built with an explicit secureContext dropped the caller's servername. That left the ClientHello without SNI, so servers that relied on SNI would return the wrong certificate or refuse the handshake. Its comment included a collapsible reasoning section that verified the exact code path line by line before posting. The maintainer confirmed and fixed it, adding a regression test asserting the SNI value reaches the ClientHello. The check's neutral conclusion isn't configurable, per Anthropic's docs, so gating a merge on Claude's findings means building that logic outside this check entirely. **Best for.** Teams selectively commissioning more involved reviews and budgeting per run. [Compare Ito vs. Claude Code Review to see how runtime evidence complements source analysis.](https://ito.ai/compare/ito-vs-claude-code-review) ## Choose for your review volume and merge workflow If you’re not sure which AI code review tool is best for you and your team, choose based on the setup needed for a first useful run, how soon your team needs results, and the cost per run at your actual PR volume. ### Setup and timing More setup doesn't automatically mean better reviews. The setup depends on what the tool needs to access to do its job. Ito requires a buildable, runnable environment, which means secrets, dependencies, and sometimes seed data needs to be available before your first useful run. Most production apps need at least some configuration beyond defaults. The benefit is true observed behavior rather than predicted patterns. CodeRabbit, Qodo, Graphite, CodeAnt AI, Copilot, Bugbot, and Claude Code Review all work from the diff and codebase context with no running environment required, which means faster first-run setup and faster results. For distributed teams, timing matters as much as setup. Ito takes 20 to 90 minutes, and Claude Code Review averages about 20 minutes. Graphite says its AI reviewer gets you feedback in about 90 seconds instead of waiting on a person. That's their own estimate, so it'll vary depending on your repo and how big the change is. If your team is spread across time zones, a code review has a short window to finish before the next person starts work. If it's late, they lose time waiting. If it's unclear, they have to wait even longer for the author to wake up and answer questions. For example, when it’s 6 pm in New York, it’s 6 am in Singapore. If you push code before you log off and your teammate in Singapore starts work an hour later, a 20-minute review will be ready for them, but a 90-minute review won’t. ### Cost calculation Every tool in this list charges differently. Some charge per seat, some by usage, and some combine the two, so the advertised prices don’t tell you much on their own. The easiest way to compare them is to run the same team through each one and see what the monthly bill looks like. Picture a 10-engineer team shipping 200 PRs a month, about 20 per engineer. Not all 200 PRs are the same size, and that’s what should decide which tool runs where: 1. **Run Ito on everything:** If you’d rather not sort PRs by size before deciding what gets reviewed, Ito is built to run on all 200 without that step. 10 seats at $40/month is $400, which covers 500 reviews, 50 per seat. Using 200 of those 500 puts you at $2 per review in practice. Push past 500 in a busier month, and each extra review is $1. The $0.80 figure some people quote ($40 / 50) only holds if you use every included review each month, which most teams won’t. 2. **Save Claude Code Review for the PRs that need it:** Most teams don’t run this on every PR. They trigger it manually, or set it to run once per PR, on the changes that are large, cross-cutting, or touch something risky. Say that’s 20 of the 200 PRs this month, about 1 in 10. At $15 to $25 per review, that’s $300 to $500 in usage on top of the subscription. Running it on all 200 PRs would cost $3,000 to $5,000, which is why the tool is built to be selective rather than automatic. 3. **Split Copilot by effort level:** Most PRs get a quick Lite pass. A smaller set of harder changes get Balanced. Say 160 of the 200 PRs run Lite and 40 run Balanced. That’s $8 to $160 for the Lite runs and $10 to $200 for the Balanced ones, so $18 to $360 for the month before subtracting whatever your plan’s credit allowance already covers. If you don’t want the overhead of triaging PRs by size, Ito’s flat per-seat cost is simple to plan around. If your team ships a lot of small, fast PRs and only occasionally needs a deeper pass, Copilot’s effort split or Claude Code Review’s selective trigger will likely cost less, since you’re not paying full price on routine changes that didn’t need it. It's important to know that re-reviews change this math. None of the above assumes a PR stays still after it opens. A PR with three pushes and two rebases can trigger several review runs, not one, on any tool that reviews automatically on new commits. This is the real reason Claude Code Review gives you a trigger choice: once after PR creation, after every push, or manual. “After every push” catches bugs introduced by later commits, but it also multiplies your run count, and your bill by however many times the PR gets updated. Ito and the other diff-based tools carry the same exposure. A PR with five pushes is actually five reviews, not one. When you estimate monthly cost, use your average commits per PR, not just your PR count, or you’ll underestimate what you’ll actually spend. ## Merge behavior A tool can flag a bug, post a comment about it, and still let the PR merge. Those are three different things: what it finds, what it shows you, and what actually stops a merge. That matters because you want to control which bugs block a merge and which ones don’t. Block everything, and your team gets stuck fighting false alarms and nitpicks. Block nothing, and a real bug can slip through because nobody had to deal with the comment. Most teams want a middle ground: block the serious stuff, leave the rest as a note. Claude Code Review's check always completes neutral, and this isn't configurable. Bugbot defaults to neutral, with configuration-dependent options for fail-on-unresolved. Copilot, CodeRabbit, and Qodo can each satisfy a required-approval rule through their own approving-review setting, but each requires deliberate activation and isn't on by default. Ito posts pass/fail wording in its PR comment, but blocking a merge through a required GitHub status check takes separate configuration that you will need to do manually. You should verify per vendor and per plan before assuming a tool's review comment creates a merge gate. ### Evidence and access How much access a tool needs depends on what kind of review it’s doing. If a tool only reads your diff and codebase, repo access is all it needs. Give it read access to look at the code, and write access if it posts comments on your PRs. This covers most of the tools in this comparison. Runtime analysis asks for more (see [our post on static versus runtime analysis](https://www.ito.ai/blog/ai-code-review-read-your-code-run-it) for the full difference). To actually boot your app and run the code, Ito needs access to secrets, build dependencies, and any external services your app calls on startup. That has real security implications, and it’s one we take seriously. Ito should never get access to your production environment or production credentials. Use staging or development credentials and a dedicated test environment, not your live systems or live data. A tool with only diff access can tell you the code looks right, but one that runs your app can tell you it behaves right. The evidence an AI reviewer can return is limited by what it’s allowed to see. ## What a review score can't tell you Vendors and third parties publish benchmark scores that show how well AI reviewers catch security issues and code quality problems in public test datasets. These provide helpful information, but you should read them carefully. A score only tells you how a tool did on one specific test. Change the dataset, the setup, or the version of the tool, and you could get a different number. Even when the dataset is public, the vendor still picked which samples to use, how to grade the results, and how to set up the tool. So don't put scores from different tests side by side as if they were measured the same way. Benchmarks also have a limit to what they measure. A security benchmark checks whether a tool spots known problems in a set of labeled examples. It doesn't check if your app still works after the change. For example, can a user still log in or check out? That's a different question, and it's the one runtime testing answers. Neither approach catches every bug. Reports from other teams on forums are useful too, as long as you know who they came from. A team saying a tool raised too many false alarms or kept missing a certain kind of bug is a real signal. But it happened on their codebase and with their workflow, which may not look like yours. The best test is your own code. Rather than going by a published score or how many comments a tool leaves, try it on a handful of typical PRs and see whether its findings are correct and worth acting on. ## Where Ito fits Most of the tools in this comparison review your code by reading the diff. Ito builds and runs your app, so it can catch bugs that only show up when the code executes, like a query that runs without errors but returns the wrong result. DoltHub ran Ito on 92 pull requests over six weeks while building DoltgreSQL, their Postgres-compatible database. 43 of those PRs had findings worth judging, and the team confirmed more than twice as many findings as they dismissed. On 26 of those 43 PRs, Ito also found bugs that were already in the codebase before the change, issues that had been sitting there unnoticed. The SQL bug from earlier is a good example. A query using a named window returned the wrong total, but it ran without errors, so nothing in the code looked off. Ito caught it by running the query and checking the result, then gave DoltHub's team the steps to reproduce it so they could confirm it and fix it. If a bug slipping past code review could cause real problems for your team, Ito adds a layer of coverage the source-review tools don't. [Try it on a few of your own PRs first. Your first 20 reviews are free, with no credit card required.](https://ito.ai/pricing) ## What else to know before adding a reviewer ### Should an AI reviewer replace human approval? For most changes, no. AI review is meant to be one input among several, and the vendors say so themselves. GitHub, for example, recommends that teams check Copilot's findings and pair them with human review. For small, low-risk changes, though, some teams now skip the human step. If every automated check passes, the PR merges on its own. This only makes sense when the codebase has strong test coverage and the change doesn't touch anything critical. Compliance is catching up too, with some SOC 2 advisors relaxing the rule that every PR needs a human sign-off. Everything outside that narrow group still needs a person. That includes major changes, anything on a critical path, and code written with AI help by non-technical teammates. Automation can clear the easy PRs, so your reviewers have more time for these. ### How do you trial a second AI code reviewer without doubling the noise? Start with a defined subset of PRs rather than enabling org-wide on day one. Pick changes where you have enough context to judge the findings, not greenfield features where you can't tell a false alarm from a real catch. Track 3 things: (1) useful unique findings the first reviewer missed, (2) false alarms that required triage time, and (3) how long you spent acting on the output. Add a second layer when you can point to a specific missing job, not because the industry says two layers are standard. ### What about the merge-without-human path for agent-written code? Teams running AI coding agents are finding that agent-generated PRs need the same review discipline as human-written ones, sometimes more, because an agent won't catch its own wrong assumptions the way a human reviewer might. That doesn’t mean every agent-written PR needs a human before it merges. Frontier teams are letting agent review clear the low-risk changes on its own: copy edits, markdown updates, small UI tweaks like moving a button. The blast radius is small, so the review just confirms nothing broke. At Ito, 10–15% of our own PRs skip human review for this reason. It’s a good tradeoff. It frees up reviewer time for the PRs that actually need a close look. Those bigger PRs are where runtime evidence helps most. It can confirm the agent’s code actually does what the PR says, not just that it’s clean syntax. --- # The Electrification of Factories: What Software Teams Can Learn From the Transition Factory electrification reshaped production only after owners redesigned the floor. AI software teams need the same shift in workflow, review, and verification. By Evan Marshall · 2026-09-18 In June, Anthropic bragged that Spotify is shipping over 4,500 PRs per day thanks to Claude. Yet, 5.5 million views and countless comments roasting them later, the case study became an example of industry-wide frustration rather than success. Teams are producing more code with AI, but the gains in shipped product quality and organizational throughput are much harder to see. That bears out in anecdata, such as the comments from frustrated Spotify users and in research: Linear found that Silicon Valley product teams, its primary customer base, see a 111% increase in opened PRs and triple their weekly PRs if they connect a coding agent. And yet, who can say their product is three times better than it once was? Who's even used a product that's three times as good as it was before AI? This long road from new technology to real gains isn't new. We've heard “AI is the new electricity,” but what we're not seeing yet is electrification. Electricity, like AI, seems like a clear technological step change. But at the turn of the 19th century, electrification took decades to reach widespread adoption and truly significant gains in output. We should expect to see the same pattern with AI, so we set out to learn from the history of electrification and why factory owners had such a hard time turning this new technology into power and profit. The truth: The missing ingredient wasn't a new technology. It was a different factory floor. ## The long road to electrification Electricity. It's hard to imagine a time without it. And yet, when electricity emerged, people didn't get a lightbulb-above-their-heads epiphany and start integrating it into everything. Even factory owners didn't immediately see the benefits of what seems so obvious to us now. Looking back, we see factory owners making limited improvements, hitting constraints, and iterating until they finally experienced breakthroughs. Those breakthroughs were never electricity itself, but the factory design and workflows that channeled electricity in new ways. For this history, we rely primarily on Warren D. Devine, Jr., an engineer the United States government tasked with researching the rise of electrification. His analysis identifies five primary stages, moving from steam and water power to true electrification. ### Stage 1: Steam and water power through a central line shaft “By 1929, just 45 years after their first use in a factory, electric motors accounted for over three-quarters of total power capacity used to drive machinery,” Devine writes. That's a steep rise. But it took almost half a century. What happened? Or, more accurately, what didn't happen? Before electrification, factory production machines were connected directly to the power sources that drove them. Not by wires, of course, but by pulleys and leather belts that were suspended from the ceiling and ran all the way to a central water wheel or steam engine. It's as chaotic as you can imagine: Devine writes, “The entire network of line shafts and countershafts rotated continuously, from the time the steam engine was started up in the morning until it was shut down at night, no matter how many machines were actually being used.” If any part of this system broke, the entire production had to be paused until repairs were made. At this point, a failure in the central system could stop the entire floor, and the factory layout followed the machinery rather than the most efficient flow of materials or workers. The old production system was an interconnected constraint, not a power source factory owners could just swap out. That shaped how factory owners thought about electricity once they had access to it. ### Stage 2: An electric motor replaces the central steam engine The first way people think to use a new technology is to use it like the things they already use. That applies to asking AI to write your code the way you do, and to replacing a steam engine with an electric motor spinning the same rotors. Humans are, by nature, skeptical of new technologies and new systems. No one wants to (nor often can afford to) change too much at once. As Devine writes regarding electrification, we see “the usual juxtaposition of a new technology upon the framework of an old one.” In the 1890s, for example, clothing and textile manufacturers took to electricity relatively quickly. But those factories used a single large electric motor to turn the same shafts and belts. Power became electric, but the factory still had the same layout, downtime dependencies, and mechanical losses. Factory owners hadn't yet thought about the new and novel ways to use electricity. That meant that electrification didn't immediately lead to much in the way of useful gains and impact. This is a close equivalent to giving every developer an AI coding assistant while preserving every downstream process. We shouldn't be surprised, in either case, that the gains weren't significant. A new technology layered on an old workflow provides an incremental benefit, not a new operating model. ### Stage 3: The group drive shifts factory layouts In the early 1900s, factory owners finally started designing factories around electric power distribution, treating electrification as a first principle rather than a bolted-on improvement. Electrification led to smaller machines, so they could cluster inter-related machines together on the same group drive, reducing reliance on one central drive and giving operators more control. Unlike in the steam engine era, tools were starting to drive workflows rather than workflows being limited by power constraints. With electricity, factory owners could consolidate operations and increase specialization, and these changes (enabled but not caused by electricity) started to trigger factory-level transformation. “Electrification of mechanical drive and factory reorganization went hand-in-hand,” writes Devine. We again find a parallel between electrification and AI: Breaking one monolithic workflow into smaller units is progress, but it retains some inherited constraints. Still, this stage proved to be a critical intermediate step to unit drives, where things really take off. ### Stage 4: Unit drives increased workflow flexibility Factory owners only started to realize electrification's full potential when they started redesigning how workers used power within their factories. With the electric unit drive, factory workers no longer needed to rely on a central machine. Instead: - Each machine got its own motor. - Operators could turn machines on and off, reposition them, and control them independently. - Operators no longer needed to rely on a centralized engine or the shafts, belts, and structural supports once required to transmit power. If one machine needed worked on, upgraded, or changed out, no other machine would be impacted. What now seems so obvious to do became inevitable only when factory owners realized electricity brought flexibility, not just cost savings. “Because of this flexibility,” Devine writes, “Manufacturers could turn their attention away from problems of power production and distribution and toward improving the overall efficiency of their operations.” ### Stage 5: The factory is redesigned around new capabilities Once machines no longer need to cluster around shafts or connect to central engines, factories could use lighter building materials, single-story and linear layouts, more efficient materials handling, and flexible machine placement. It was only with the electric unit drive, Devine writes, that “processes could be arranged within factories to maximize throughput; plants could be more readily expanded; and a better working environment and improved machine control increased both quantity and quality of output.” In other words, the full economic benefit of electrification followed from factory reorganization, not from lower power bills. For example, the images below show two different hat factories, giving you a sense of how much the layout and working environment could change after the introduction of electric motors. Tim Harford, an economic journalist, explains, “You couldn't get these results simply by ripping out the steam engine and replacing it with an electric motor. You needed to change everything: the architecture and the production process.” ## What does this mean for software factories? The takeaway for companies looking to benefit from AI is that it's not as simple (and has never been as simple) as swapping in a new technology. Gains will be uneven, unpredictable, and even disappointing until you redesign and rebuild your processes. A conventional SDLC with AI coding agents integrated is essentially a factory at the electric line-shaft stage. Teams that add agents to isolated steps, such as drafting tests or triaging tickets, are experimenting with group drive-level work. The next step is getting to the AI equivalent of unit drive. What work can run independently with clear goals, safe environments, and reliable verification? Answering that question will require a redesigned software factory that changes the flow of work, the role of review, and the conditions for human intervention. Turning from the distant past to the near future, we can see emerging patterns as companies experiment with redesigning their factory floors. These are the kinds of ideas you can take back to your team to start experiencing some of the real gains from the use of AI. ### Example 1: Change where work begins Traditionally, a customer reports an issue, the complaint populates in a Slack channel, and a human writes a ticket based on the message, eventually leading to a developer's task and a developer's PR. If we rearrange how work flows and change where work begins, an agent could turn a Slack conversation into a draft issue, investigate the relevant code paths, propose a plan, and prepare a PR in an isolated environment. Then a human developer could decide whether the task matters and review only the work output that requires judgment. Linear did a similar redesign when it built self-driving bug management. Now, a Linear Agent creates a new issue, automatically adds context, and tasks a Cursor Agent to draft a fix. It would be nice if simply having AI write the code was a major gain for teams. It is a new way of working, but that alone does not product significant gains. Instead, the biggest gains come from removing administrative steps, finding places where AI can work independently across multiple stages of work, and only involving developers where and when they need to be. While some companies may do this inefficiently through endless meetings, for many teams these improvements are measured in only minutes gained. ### Example 2: Make verification a continuous, independent loop Many companies have embraced AI, but now face verification bottlenecks. An agent writes a PR and a human reviews it, but the work gets stuck in a queue that seems to constantly get longer. And if you simply skip your engineers reviewing it, or they get tired and a bit sloppy, problems are found later in production at the expense of your customers' experience. But if we rearrange things, the bottleneck resolves. An agent writes a change, starts a production-like environment, runs targeted behavioral checks, fixes failures, and repeats until it has evidence that the change works. Human review comes only after, and the review focuses on product judgment, architecture choices, risk, and exceptions rather than mechanically checking every routine PR for basic issues. Addy Osmani, for example, uses AI to generate more code but then makes human review more selective. “Agents do the first pass,” he writes, “and humans cover the blast radius.” More generation requires more independent verification. Maintaining the same level of verification just leads to a bottleneck that prevents experiencing the speed gains teams (and AI-frenzied executives) are seeking. ### Example 3: Run cheap experiments that were previously not worth a developer's time It wasn't that long ago that we were all used to development resources being scarce, and experimentation was necessarily limited. Product improvements often had to be small, and they actively competed with core roadmap work. When we treat AI as a first principle and redesign the factory floor around it, experimentation becomes cheap. Teams can give agents bounded, reversible experiments to run overnight. Humans can assess the evidence and keep only the strongest results. For example, developers at Gamma, an AI design platform, will have an idea in the morning, a prototype to test by the afternoon, and an idea worth building or killing by the end of the day. Lower marginal cost changes the kinds of work that are rational to attempt, just as distributed electric motors changed which factory layouts were rational. Just remember: with this change comes responsibility to clean up after yourself. All that experimentation can lead to a better product, if you remember to remove the things you added that don't work. That didn't use to be much of a problem, because you simply couldn't run that many experiments. Now, with this new process, it becomes essential, or you'll very quickly have a bloated mess of a product. ### Example 4: Separate work by risk, rather than treating every PR identically Typically, most, if not all, work passes through the same basic review process, whether that's a documentation correction, or a change to a core authentication system. That can mean slowing down the former and missing checks on the latter. But if we redesign and separate work by risk, we can maintain different throughput levels. Low-risk, well-instrumented changes progress through automated implementation and verification. No need for a human. Ship it as fast as AI can write it and run basic checks. Meanwhile, high-risk changes trigger stronger controls, deeper review, and explicit human approval. We can speed up on one end and de-risk on the other. Augment Code, an AI coding platform, lays out a workflow that emphasizes risk-gated handoff between humans and agents. Every PR gets classified before work begins, and the loop learns across every step. This approach helped their team triple their code output, and dropped merge times by two-thirds. Similarly, Duckbill, a cost management services provider, found itself with a painful backlog of over sixty open PRs for a team of five. To resolve it, they implemented a risk-based system (along with stronger guardrails) to let agents do the code review for them. This work led to a 94% increase in PRs merged per week, and a 17% increase in PRs merged within one hour. As you build your software factory, you need to create different lanes for different risk levels, not blanket autonomy. Just as electrification let workers use new, more flexible electric tooling where it was beneficial, development teams can rely on AI more for some tasks and use more human judgment for others. ### Example 5: Change the plant from “smarter operators” to a harness Today, many teams try to deliver more output by giving individual developers better models and better prompts. When model gains were more often seismic, this often made sense, but model progress is plateauing. Now, it's more practical to build a harness than to wait for the next 3-4 point gain in the IQ of your model of choice. If we restructure things, you can treat the agent like a machine on your factory floor. You can write agent skills, add sensors (such as tests, linters, and evals), bound retries, set permissions and escalation paths, and keep memory outside the model. The model becomes the motor, and the harness becomes the wiring and your interface. Lamis Mukta, a member of Anthropic's technical staff, laid this out in a talk, where she explained how to build a graph that runs itself. If a loop is one agent doing one job continuously, a graph is a team of agents, orchestrated by something like OpenAI's Codex or Claude's dynamic workflows, that splits the work, runs tasks in parallel, and checks the results as they go. Remember, after electrification was first introduced, the real step change came after rearranging the factory floor to take advantage of newly cheap, controllable power. Similarly, after coding agents, the step change won't result from a better prompt or a slightly smarter model. It will result from the graph, the loops, the permissions, and the evals you place around the model. ## Iteration is the only way to see the future No one wants to be the Paul Krugman of AI. Back in 1998, he predicted the Internet would have little impact on the economy, and he's been an example of the famously wrong ever since. The good news is that hardly anyone in the software world faces this fate. The bad news is that merely refusing to be a luddite, to experiment with AI instead of avoiding it, isn't enough. The history of electrification is full of factory owners who tried electric motors but got stuck in the second stage; replacing steam and water power with electric motors that turned the same central shafts didn't make enough of a difference on its own. Many teams and vibe coders are experimenting, and they're showing us glimpses of what the future will be for everyone. Some are heading down dead ends, and others are sketching the blueprints that will one day be used to design new factories. These structural changes are coming sooner than you might expect. The tools will keep getting cheaper, more available, and more capable. The companies that benefit most will be the ones that use this period to redesign how work moves through your factory. An iterative approach lets you try more ideas and see what works and what doesn't as we all navigate the idea maze of AI software development. Electricity required iteration, too, but AI is advancing faster. Don't get left behind with a more efficient steam engine. ## Frequently asked questions ### Why don't coding agents make engineering teams ship faster? A coding agent speeds up code production. Your company still ships at the pace of its slowest necessary step. That typically happens during review, verification, CI, product decisions, or release management. More generated pull requests can make that constraint worse by adding work to a later step in your software development lifecycle. The throughput gain arrives when you redesign the flow around the new volume: smaller bounded tasks, independent evidence loops, limits on work waiting for review, and clear human gates for decisions that need judgment. ### After an agent runs the tests, what still needs human review? Humans should review changes with a large blast radius: work that touches critical workflows, changes core architecture, affects security or customer data, or could impact key customers. Small, low-risk, reversible changes with clear verification evidence from AI tools can move through an automated lane without the need for human review. This helps prevent a review bottleneck from building up, because it saves your engineers from reviewing things that were easy to verify without them. ### When can I let a coding agent ship without a human approving it? Let an agent ship without human approval when the change has a low blast radius, is easy to reverse, and has verification evidence that shows an AI checked the work and approved it. A copy correction or a small, well-covered UI change (like a button name or color change) can move fast. Changes to authentication, billing, data, critical workflows, or a key customer's experience need a human reviewer, even when every test passes. Set those risk lanes *before* agents start work so they do not decide for themselves when a change is safe to ship. Don't make your customers become your QA team. ### Why can't I solve this with better prompts and models? A better model can write stronger code, but it still needs a workflow that tells it what it can and cannot do, how to prove the result, and when to stop. Without that structure, agents lose context, retry the wrong thing, build beyond the scope you wanted them to work in, and hand you a result that is hard to trust. Give agents clear goals, bounded permissions, isolated environments, and checks that produce evidence you can review. That structure is what makes their work reliable enough to run at scale and goes well beyond the prompts you wrote or the model you choose to use for a task. --- # Why AI Code Review Needs to Read Your Code and Run It A practical guide to static and runtime analysis in AI code review, including what each approach catches, where it falls short, and why teams use both. By Evan Marshall · 2026-08-14 Teams are shipping more code than ever. Not because anyone suddenly got faster at typing, but because AI can generate code faster than any individual engineer can. That speed creates a new constraint: every AI-generated code change still needs to be reviewed, tested, and trusted before it reaches main. For many teams, the practical choice has become either merging on faith, or accepting a major pre-merge bottleneck. AI code review helps break that bottleneck: - [Reviewing and testing every change](https://www.ito.ai/why-we-built-ito) by hand takes more time than most teams have. - Code review is tedious and easy to defer, so it is a natural task to offload to a system that does not get tired or frustrated. - AI can review far faster than a human and catch issues before they become production bugs. Yet, deciding you're going to have AI help review all the code you're generated is just the first step. You need to decide what kind of AI code review you want to use. The two main approaches are called static analysis and runtime analysis. Let's take a look at where each fits and why the strongest reviews use both together. ## The two main approaches to AI code review AI code review generally falls into one of two categories: - **Static code analysis **reads your source code without running it. It identifies bugs, quality issues, and risky patterns by examining your code’s structure, known failure patterns, and underlying logic. - **Runtime analysis **runs your application and evaluates how it behaves in real scenarios. Instead of inferring what the code will do, it validates your user flows by using your product as a customer would. These two approaches test different layers of the same problem, which is why they are at their best when they work together. ## TL;DR: Static code analysis vs runtime code analysis If you only have a minute, here is how the two approaches compare: | | Static code analysis | Runtime code analysis | | --- | --- | --- | | What it does | Examines source code by reading and analyzing it | Executes the application and observes real behavior | | How it works | Parses code, dependencies, and PR diffs, then applies AI reasoning on top of static rules | Builds and runs the app, then explores impacted user flows like a real user | | Key benefits | Fast, cheap, early feedback; scales reviewer expertise across many PRs | Catches the unanticipated, high-severity bugs that only surface at runtime | | Limitations | Cannot validate real-world performance, timing, or end-to-end behavior | Needs reproducible execution environments and prioritized, high-impact flows to test | | Ideal use cases | Shift-left reviews, large codebases, rapid multi-contributor development | Pre-merge validation of critical user-facing flows, complex apps | Note: Static analysis is where most teams start, and many AI-forward companies already have covered. Runtime analysis is usually the gap. If getting that side right is what you are after, we built Ito specifically to help you. ## An overview of static code analysis for AI code reviews Let's start with the more familiar approach: static analysis. ### What is static code analysis? AI-powered static code analysis examines source code without executing it, using AI models alongside code-aware techniques to identify likely bugs, security risks, and quality problems. Now, you might be wondering how this is different than a linter. The key is intent. A linter enforces a defined set of style and code-quality rules, such as unused variables, missing semicolons, or improper formatting. Syntax errors are usually caught earlier by the parser or compiler. Traditional static analysis scanners go deeper, tracing known bug patterns and security flaws through your code. AI-enhanced static analysis can use the surrounding code and repository context to infer likely intent and follow how logic flows through a change. That lets it surface subtler issues that no fixed rule would catch; these issues are the kind you would only notice on a very careful read. The goal is to expose hidden risk early, while the code is still a diff in a PR. By analyzing structure, dependencies, and patterns before anything runs, static analysis helps you fix problems early in your review process, often before the first human reviewer takes a look at your PR. Speed is a key benefit here, because they can often identify issues within a few minutes of putting up your PR for review. ### How it works AI static reviewers typically start with your pull request diff, then use available context on your repository to assess the change against the surrounding code. That can help identify how a local edit may affect other parts of your codebase. Depending on the tool, that analysis may also incorporate traditional static-analysis techniques such as abstract syntax tree parsing, dependency graphs, and control-flow tracing, alongside AI-based reasoning. That broader context helps the tool look beyond individual lines and flag issues such as risky coupling across files, dependency problems, and deviations from established patterns. Instead of a raw warning, it can provide a plain-English explanation and, in some cases, a suggested fix. A quick example: a PR adds a function that pulls a user's orders for a dashboard. It compiles, the types line up, and nothing looks wrong. An AI static reviewer reads the diff against the rest of your repo and notices the function queries the database inside a loop, once per order, instead of batching a single query; no linter rule was broken. It flagged the pattern as a performance risk and suggested a batched query. That is the kind of catch static analysis is good at. ### When to apply it Static analysis should be used in large codebases, where manual review stops scaling. When lots of pull requests are open at once, AI static analysis spreads reviewer-level expertise across all of them in parallel. It also shines when fast feedback matters. You can identify and fix issues within minutes of pushing, long before you request a review from a peer or open the merge. In rapid development cycles, where multiple contributors are pushing changes constantly, that early pass can surface likely problems before your PR reaches the main branch, keeping your review queue more manageable. ### Biggest limitations [Static analysis](https://ito.ai/blog/static-analysis-misses-bugs-that-reach-production) is genuinely useful, but it has real limits, and most of them come from the fact that it never runs your code. The first is consistency. AI reasoning quality can vary, so the same tool might produce conflicting or overly cautious feedback depending on how it interpreted a given change. The second is scope. A static reviewer is limited to the code and context it can access. It may not know the product decision behind a change, an architectural direction that has not yet been captured in the repository, or the tradeoffs your team has agreed to make. That is a big part of why human review still matters. The rest of the limitations show up the moment code stops being analyzed and starts executing in a real environment: - **Hidden integration failures**, where services that appear correct in isolation fail when they interact through a distributed system or API. - **Asynchronous issues**, where timing, race conditions, or event ordering create behavior that looks fine in the code but fails under real execution. - **User-flow inconsistencies**, where the system technically runs but does not produce the end-to-end behavior users expect. - **Performance and load gaps**, because spotting a risky pattern in the code, like the query-in-a-loop from earlier, is not the same as measuring latency, throughput, or how the system holds up under pressure. Static analysis can flag the risk, but measuring the impact requires running the app under real conditions. None of these are failures of static analysis so much as things it was never designed to see. To catch these issues, you need to run the app under real conditions, with all of your integrations, inputs, timing, and load. That's where runtime analysis comes in. ## Understanding AI runtime code analysis Static analysis is strong at what it does, but it has key blind spots. It can inspect code, but it cannot observe what happens when that code runs in a real environment. ### What is AI-powered runtime analysis? AI-powered runtime analysis is execution-based evaluation of your code. Rather than inferring behavior from the source, it observes and tests what the system does while it actually runs. It exercises the parts of your codebase affected by your PR, including the frontend, backend, databases, and external-service calls, then [uses AI to generate test scenarios](https://ito.ai/blog/what-is-agentic-qa-the-complete-guide) around the user flows and execution paths that matter. ### How runtime code analysis works in AI code review Runtime analysis starts in the same place static does: with the pull request. AI agents read the PR diff and its surrounding context, then generate a [targeted test plan](https://www.ito.ai/features/targeted-test-plans) focused only on the user flows and system areas the change affects. One of the quieter advantages here is context. That context determines what gets run. Runtime analysis is a broad idea. When you start running the code, you could run tests in a variety of ways including using Playwright and Cypress suites, creating integration tests, load tests, and perform manual QA. The biggest difference between these options is how much you have to build and maintain by hand, and how much the tool can do on its own. Ito sits at the far end of that range. Ito runs tests by looking at the diff and the PR description as the starting point for determining what to test. You can also give the agent [plain-English instructions](https://www.ito.ai/docs/settings/test-instructions) about which flows matter, applied across every run, or scoped to specific repositories and authors. That context shapes which user flows get built, exercised, and validated, so the review lines up with what you were trying to ship, not just the lines that moved. From there, instead of scripts you write and keep green, Ito builds and runs a full instance of your application from source in a [controlled, single-use sandbox](https://www.ito.ai/features/containerized-test-execution), spun up fresh for that pull request and thrown away afterward. Then it explores that instance the way a real user would: clicking through the UI, firing API requests, and triggering backend workflows to test end-to-end behavior across your stack. ### When should you use runtime analysis? Runtime analysis belongs in pre-merge validation, where it tests whether a change has broken critical, user-facing flows before you merge your code. It is most valuable for teams shipping at AI speed with complex apps, where manual verification cannot keep up, and end-to-end suites are too cumbersome to maintain. With this approach, you can catch problems early, so you neither burn customers with production bugs, nor burn out your engineers having them do too much manual testing themselves. ### The biggest limitations Runtime analysis comes with trade-offs worth knowing up front. The hardest part is the setup. You can wire this up yourself with Playwright or Cypress, but those suites are yours to maintain. Then, as your product changes, keeping them green can become a significant engineering cost. If you want something more sophisticated and custom to your company, you may end up building your own system. However, building and running your app in a clean, reliable environment takes real engineering time. Most teams would rather stay focused on building their product for customers, not maintaining an internal tool like this. Coverage is another issue. The results are only as good as the paths your runtime analysis chooses to test. To save time, you can focus on your most important flows, but that means you may miss key edge cases. Yet, even if you built the perfect system for yourself, and had limited maintenance demands, your work is not done. You still need to gather the evidence and make sense of any issues you do find. That means you need to develop clear [severity ratings](https://www.ito.ai/docs/concepts/severity-levels) and quality evidence packages to help you and your AI code gen of choice figure out what needs to be fixed. ## Ito's approach to AI code review through runtime analysis Ito is an AI-powered code review tool built to perform runtime analysis on your PRs. Instead of only reading your source code, it executes your application and validates the user flows each pull request affects. For a concrete example, see [what happened when DoltHub ran Ito on 43 pull requests](https://ito.ai/blog/what-happened-dolthub-ran-ito-43-pull-requests). We built Ito to take the manual review work off your plate. When your team ships code faster than you can check it, you end up with a backlog no matter what you do; either new features and functionality are delayed actually reaching customers while you wait for more review, or you ship it without sufficient checks, and now you have to deal with a wave of new bug reports and customer issues to fix. That's why having AI runtime analysis in addition to a static analysis tool is so helpful; it gives you more validation and confidence before shipping, without overburdening your software engineers, nor your QA team. Best of all, Ito handles many of the runtime limitations we covered above: - It [sets up the test environment for you](https://www.ito.ai/features/containerized-test-execution), so you skip the setup that makes in-house systems so painful to create and maintain. - It takes [plain-English instructions](https://www.ito.ai/docs/settings/test-instructions), so you tell it which flows matter most instead of leaving it to guess. - It [ranks issues by severity](https://www.ito.ai/docs/concepts/severity-levels), so you get a clear list of what to fix first instead of a wall of noise. - It writes [evidence-rich reports](https://www.ito.ai/features/evidence-rich-results) with video, logs, screenshots, steps to reproduce, and [the exact files and lines](https://www.ito.ai/docs/how-it-works) behind each failure, so you have what you need to fix it fast. - It goes past the happy path on purpose, into the edge cases and adversarial flows that are easy to miss. The result is similar to having a QA engineer on every pull request, running your app, trying to break it, and handing your team a report they can act on right away. The difference is Ito is tireless, can test many PRs in parallel, and delivers results in a clear, actionable way. You can literally copy and paste a prompt for your agent to let it validate it's a real bug and propose a fix. **Want to see for yourself? **[Start a free Ito trial](https://app.ito.ai/auth/signup) and experience AI-powered runtime code review firsthand with 10 free code reviews of your PRs. ## Static code analysis + runtime code analysis: final thoughts The two approaches answer two different questions: - Static analysis asks whether the code looks correct before it runs. - Runtime analysis asks whether the application behaves correctly when it does. Because they address different layers of code quality, they are complementary rather than rivals. Static analysis finds likely risks at a low cost and high speed. You can often get results in minutes. Runtime analysis tests the changed behavior and produces evidence about failures that only appear in execution. You may have to wait longer for it, but you'll end up catching results you'd otherwise only find in production after they've burned a customer. Together, they combine to give you the best possible coverage of your code changes, without taxing your team and making them spend all their time testing. ## Frequently asked questions ### What is the difference between AI code review and traditional code review? Traditional code review relies on humans reading every pull request line by line. AI code review hands the first pass to a system that reads your changes, reasons about them, and, in the strongest setups, runs them, catching issues in minutes so human reviewers can focus on judgment calls instead of routine checks. ### Static code analysis vs runtime code analysis: what's the difference for AI code reviews? Static analysis inspects your source code without running it, catching bugs and risky patterns early. Runtime analysis executes your application and validates how it behaves. Static reviews the code, runtime reviews the running product. Used together, they cover both layers of quality. ### Should an AI code reviewer be static or dynamic? Ideally both. Static analysis gives fast, cheap feedback on code quality before execution. Dynamic, or runtime, analysis catches behavioral regressions that only appear when the app runs. Choosing just one leaves a gap, so the strongest setups combine them across the review workflow. ### What does Ito offer as an AI runtime code analysis solution? Ito executes your application in an isolated environment and validates the user flows affected by each pull request. It provisions environments automatically, takes plain-English priorities, assigns issue severity, and produces evidence-rich reports with video, logs, screenshots, and reproduction steps before every merge. ### Can runtime analysis replace an existing end-to-end test suite? No. A maintained end-to-end suite gives you repeatable regression coverage for critical workflows, release gates, and known edge cases. Runtime analysis is strongest as an additional pre-merge layer: it uses the pull request context to decide what changed, exercises those affected flows in a real environment, and investigates behavior that a fixed suite may not cover. Keep the tests that protect your highest-value paths. Use runtime analysis to broaden the review around each change, and catch issues before you merge. --- # What is agentic QA? The complete guide How autonomous AI agents are replacing brittle E2E scripts with behavioral testing that actually validates the user experience. By Evan Marshall · 2026-04-15 Your test suite passed, but once it gets to real users, things break. This is the fundamental disconnect in modern software engineering: the cost of producing code is collapsing, making the value in validation higher than ever. We’ve automated our builds and deployments in an attempt to move away from manual testing. But instead of automating old processes, what if we reimagined QA from the ground up for the agentic era? This is where agentic QA comes in: an autonomous, closed development loop where any builder, human or agent, plugs into a verification layer that doesn't just read code, but test drives it instantly to ensure intent matches reality. ## What is agentic QA? **Agentic QA** is the shift from semantic review to behavioral validation. While traditional tools read your code to judge if it *should* work, agentic QA actually runs your application to tell you if it *does* work. At its core, an agentic QA platform performs three autonomous actions on every pull request: 1. **Observes** — It reads your code changes to infer which user flows are impacted. 2. **Executes** — It deploys your app in a real, isolated browser environment and navigates it like a human user. 3. **Validates** — It verifies behavioral outcomes, like a checkout flow or an API sequence, rather than just checking if a specific piece of code exists. ### Agentic QA vs. "traditional AI testing" The mistake many teams make is automating the old playbook instead of rethinking what validation means. - **Traditional AI testing** typically uses AI for "self-healing" locators or stabilizing record-and-replay scripts. It still requires a human to define the steps. - **Agentic QA** is different. The agent *is* the tester. It doesn't just read your code; it "test drives" your app by executing it in real-time, providing evidence rather than just opinions ## How agentic QA differs from traditional test automation The shift from instructions to intent changes the entire economics of software quality. While scripted tests are an asset that quickly turns into liability (maintenance), agentic QA is an infrastructure that scales with your code. | | Code Review | Agentic QA | | --- | --- | --- | | Question it answers | Does this diff look reasonable? | What happened when the changed app ran? | | Works on | The code at rest | The running application | | Surfaces | Logic, style, and readability issues a reader can spot | Behavioral failures that appear only at runtime | | Evidence it produces | Comments on lines of code | Screenshots, video, logs, and reproduction steps | | Covers every PR by | Reading faster | Running the affected flows automatically | ## The three pillars of agentic QA To be truly agentic, a QA agent must operate across three layers of the software lifecycle: ### 1. Code-aware inference The agent starts at the pull request. By reading the PR diff and description, it understands the *intent* of the change. It doesn't just run a "smoke suite"; it intelligently maps the code changes to the specific user flows they impact. ### 2. Real browser execution This is the "test drive" of your code. Ito provisions an **ephemeral, isolated test environment** for every PR, running the actual application with the diff applied. The agent then navigates the UI or calls APIs exactly as a consumer would. ### 3. Visual evidence Unlike a terminal output that just says **FAIL**, agentic QA returns proof of what happened. This includes video, screenshots, console logs, and network traces posted directly to your PR. It doesn't just tell you it’s broken; it provides the reproduction steps to fix it. ## When does agentic QA make sense? Not every team needs a **QA agent** today. However, the spectrum of QA from manual to agentic usually shifts when you hit these signals: - **Manual QA (Stage 1) —** Best for very early exploration where humans are still figuring out what to build. - **Static Review (Stage 2) —** commoditized pattern matching that catches syntax issues but remains blind to real-world interaction. - **Agentic Review (Stage 3) —** Necessary when you ship >5 PRs/day and your manual QA hand-off takes >2 days, creating a bottleneck. For fast-shipping teams, agentic QA is the only way to verify code fast enough to ship it. ## How to evaluate an agentic QA platform If you are looking to adopt an AI QA platform for your [engineering](https://www.ito.ai/for-engineering-teams) or [QA team](https://www.ito.ai/for-qa-teams), use this checklist to separate the agents from the wrappers: ## Agentic QA by the numbers - **75%** — of teams spend more time maintaining tests than shipping without agentic QA. - **70%** — fewer production regressions with agentic QA. - **5 min** — typical agentic QA setup time (1-click GitHub install) with Ito. ## Frequently asked questions ### What is the difference between agentic QA and AI test automation? Agentic QA is behavioral and execution-based; traditional AI automation typically just stabilizes legacy scripts or locators. ### Do I need to write test scripts for agentic QA? No, agentic QA platforms like Ito infer what to test from your code and validate behavioral outcomes without human-authored scrip. ### Can agentic QA replace my existing E2E test suite? It can replace or supplement it. While Stage 2 tools give opinions, Stage 3 agents provide evidence. Most teams let Ito handle high-churn PR flows while transitioning away from brittle legacy suites. --- # AI-Driven Testing: Why Your QA Still Runs Like It's 2015 Discover how AI-driven testing replaces brittle QA automation, cuts bottlenecks, and helps modern teams ship faster with more confidence. By Evan Marshall · 2026-03-20 ## How intelligent testing systems are replacing brittle automation to match the speed of modern software development Discover why traditional test automation fails at scale and how AI-driven testing fundamentally changes quality assurance. Learn how chaos engineering creates resilient systems that keep pace with rapid development. ## The QA Bottleneck Nobody Wants to Admit Your engineering team ships faster than ever. Your CI/CD pipeline runs in minutes. Yet somehow, QA still takes days. Here's the uncomfortable truth: most teams have automated everything except the part that actually breaks releases. They've optimized deployment while leaving quality assurance stuck in 2015. The result? Engineers wait. Releases slip. And that "high-velocity" team you built moves at the speed of your slowest manual tester. ## Why Traditional QA Automation Fails at Scale The conventional wisdom says: write more tests, run them in parallel, hire more QA engineers. This approach worked when releases happened monthly. It breaks down when you're merging dozens of pull requests daily. Traditional test automation frameworks require constant maintenance. Every UI change means updating selectors. Every new feature means writing new test scripts. [Three-quarters of teams using code-based test automation](https://www.rainforestqa.com/blog/ai-in-software-testing-report-2025) have already recognized this ceiling, which is why they're adopting AI testing tools for writing and maintenance. The manual vs automated testing debate misses the point entirely. The real question is: can your testing keep pace with your development? ## The Shift That Changes Everything AI-driven testing doesn't just run your existing tests faster. It fundamentally changes what testing means. Here's what I actually believe: the future of QA isn't more automation of the same old approach. It's intelligent systems that understand your code changes and test what matters, when it matters. This isn't incremental improvement. It's a different category of solution. ## What Intelligent QA Actually Looks Like Consider what happens when a developer opens a pull request today. In most organizations, they wait. They context-switch. They forget what they were working on by the time feedback arrives. Now consider an alternative: within minutes of pushing code, an AI agent analyzes the changes, runs real-browser tests against the affected functionality, and delivers video-backed reports showing exactly what passed or failed. This isn't theoretical. [Organizations using AI-powered testing report 40% reduction in testing costs](https://testquality.com/how-ai-is-transforming-software-testing/) and up to 30% productivity gains. The efficiency comes from testing smarter, not just testing more. ### The Chaos Engineering Connection Here's where it gets interesting. The same principles driving chaos engineering in CI/CD apply to intelligent QA. Both approaches share a core belief: you can't test your way to resilience by only checking happy paths. Traditional QA asks: "Does this feature work as specified?" Intelligent QA asks: "What breaks when this code changes?" The difference sounds subtle. In practice, it's the difference between catching regressions before users do and apologizing after. Performance testing automation benefits from the same shift. Instead of running the same load tests on every build, AI-driven systems can identify which changes actually affect performance and focus testing resources accordingly. ### The Numbers Tell the Story The market has noticed. [The global AI-enabled testing market generated USD 498.4 million in 2023](https://www.grandviewresearch.com/horizon/outlook/ai-enabled-testing-market-size/global) and is expected to reach USD 1,627.2 million by 2030. That's 18.4% compound annual growth. [75% of organizations identify AI-driven testing as pivotal to their 2025 strategy](https://www.qable.io/blog/is-ai-really-helping-to-improve-the-testing). Yet only 16% have actually adopted it. The gap between intention and action reveals the real challenge: most teams know they need to change but don't know how to start. [84% of developers are using or planning to use AI tools](https://survey.stackoverflow.co/2025/ai) in their development process, up from 76% last year. The shift is happening whether QA teams are ready or not. ## If This Is True, What Changes? If intelligent QA becomes the standard, several things follow. First, the QA bottleneck disappears as a scaling constraint. Engineering teams can grow without proportionally growing QA headcount. The math changes from "one QA engineer per five developers" to "one platform serving the entire team." Second, shift-left testing becomes practical rather than aspirational. When testing happens in minutes rather than hours, developers actually wait for results. They fix issues while context is fresh. The feedback loop tightens. Third, release confidence becomes measurable. Video-backed test reports don't just tell you something failed. They show you exactly what happened. Debugging time drops. Finger-pointing stops. The cost of ignoring this shift? Your competitors ship daily while you ship weekly. They catch regressions in PR review while you catch them in production. ## A Better Way to Think About QA Stop thinking of QA as a gate at the end of development. Start thinking of it as continuous validation woven into every code change. The old model: developers write code, QA tests it, bugs bounce back, everyone waits. The new model: code-aware AI agents test changes as they're proposed, developers get instant feedback, regressions never reach main. QA shouldn't be a phase. It should be a property of your pipeline. This reframe matters because it changes what you optimize for. Instead of asking "how do we test faster?" you ask "how do we build quality into every commit?" The answers look completely different. ## The Path Forward The transition to automated QA doesn't require ripping out your existing infrastructure. It requires rethinking what you're trying to achieve. You're not trying to run more tests. You're trying to ship with confidence. You're not trying to eliminate manual QA. You're trying to eliminate the bottleneck that manual QA creates. The teams that figure this out first will define the next era of software development. The rest will wonder why they can't keep up. ## Frequently asked questions ### How can parallel testing improve the speed of software testing? Parallel testing runs multiple test suites simultaneously rather than sequentially. Combined with AI-driven test prioritization, it typically reduces total test time from hours to minutes while maintaining coverage. ### When should I implement continuous testing in my QA strategy? Implement continuous testing when manual QA becomes your release bottleneck, typically when your team exceeds 10 engineers or ships more than weekly. The earlier you integrate it, the less technical debt accumulates. ### What is the impact of manual testing on overall testing cycle speed? Manual testing typically adds 2-5 days to release cycles and scales linearly with team size. Organizations report 40% cost reductions and 30% productivity gains after transitioning to AI-powered testing approaches. ### Sources 1. [https://www.rainforestqa.com/blog/ai-in-software-testing-report-2025](https://www.rainforestqa.com/blog/ai-in-software-testing-report-2025) 2. [https://testquality.com/how-ai-is-transforming-software-testing/](https://testquality.com/how-ai-is-transforming-software-testing/) 3. [https://www.grandviewresearch.com/horizon/outlook/ai-enabled-testing-market-size/global](https://www.grandviewresearch.com/horizon/outlook/ai-enabled-testing-market-size/global) 4. [https://www.qable.io/blog/is-ai-really-helping-to-improve-the-testing](https://www.qable.io/blog/is-ai-really-helping-to-improve-the-testing) 5. [https://survey.stackoverflow.co/2025/ai](https://survey.stackoverflow.co/2025/ai) --- # MTTR is the Wrong Metric for AI-Era Engineering Teams AI tools produce 41% more bugs and 98% more pull requests. MTTR can't keep up. Here's how MTTF shifts your team from incident response to prevention. By Evan Marshall · 2026-07-23 AI makes code generation and deployment faster, but our safeguards haven’t caught up. For over a decade, mean time to recovery, or MTTR, was the industry’s primary protective measure against bad code. Organizations assumed failure was inevitable and used MTTR to measure how fast they could recover. The smaller your MTTR, the better. MTTR, however, doesn’t map to AI-era failure modes. It doesn’t meaningfully capture the speed of AI-generated code production, nor the potential blast radius of autonomous agents. Every engineering team is already rethinking their deployment process, so it’s time to rethink recovery, too. Enter mean time to failure, or MTTF. MTTF first emerged in hardware contexts, where it referred to the average operating time before a non-repairable component broke and required replacement. In software, we can use the same framework to focus on the component *we* can’t have broken: Your customer’s experience. ## DORA, SRE, and the assumption of failure MTTR might feel like it’s been around forever, but it’s only been around for about a decade. It has its roots in DORA and site reliability engineering (SRE), both of which were responses to the rise of distributed computing. MTTR was a valid response to a real shift in how we produce software, but with the rise of AI, old ideas working from previously valid principles can cause damage. That’s why it’s worth rewinding to see why MTTR emerged, and why it no longer suits what we need today. ### DORA: DevOps Research & Assistance The first study in [what would become DORA](https://dora.dev/insights/dora-metrics-history/) attempted to tie IT performance to organizational performance. The goal was ambitious: How could organizations link sometimes abstract-feeling technical goals to real business value? The research turned up four values, split into two categories: - 1 Throughput measures: Deployment frequency and deployment lead time. - 2 Stability measures: MTTR and change fail rate. Note that, in the years since, there have been updates. In 2023, DORA renamed MTTR to failed deployment recovery time to focus on failures caused by software changes, rather than events like data outages. And in 2024, DORA added a fifth metric: deployment rework rate. By then, however, MTTR had already become the de facto term. In the years since, DORA has become the preeminent methodology for measuring performance, replacing (and in many cases upgrading) other methods. These five metrics subsequently became canonized, and even organizations that don’t use DORA know to prize these metrics. ([Source](https://www.aviator.co/blog/everything-wrong-with-dora-metrics/)) As DORA metrics were taking off, Google was formalizing a new role and philosophy: Site Reliability Engineering, or SRE. The SRE approach emphasizes automation, reliability, and hands-on incident management. This places SRE engineers in charge of building fault-tolerant systems and minimizing failure. MTTR became a key metric for Google’s SREs, resulting in MTTR’s spread as other organizations adopted SRE roles. As the metric spread, business leaders keyed in, too. Many organizations defaulted to these metrics, along with the frameworks they implied, to build consensus across teams. Over time, organizations could capture failure in one simple number, and trust everyone understood what they meant. Shared understandings can be dangerous, however, when new concepts unseat previously foundational assumptions. ## The limitations of MTTR MTTR is a widely accepted standard. The problem with standards is that the map can be mistaken for the territory. What happens then when the territory changes underneath this map? ### Goodhart’s Law and its consequences When Anthropic published a case study with Spotify, the online response wasn’t entirely positive, with developers pointing out that velocity doesn’t equal value, and fewer bugs don’t mean a better experience. As Mitchell Hashimoto, cofounder of HashiCorp, writes in a [representative post](https://x.com/mitchellh/status/2071971627748020409), *“This is the problem with a lot of AI reporting. It reports completely meaningless metrics.”* Production deploys per day, proportion of AI-assisted PRs, MTTR: They’re all classic cases of Goodhart’s Law, which states, *“When a measure becomes a target, it ceases to be a good measure.”* Goodhart’s Law is a common citation in these conversations, but it can be easy to dismiss. As Hillel Wayne, a software development consultant, [writes](https://buttondown.com/hillelwayne/archive/goodharts-law-in-software-engineering/), *“The weak version of this is that people have perverse incentives to game the metrics.” Measure development productivity by lines of code, for example, and you’ll find yourself with verbose code, not greater productivity.* Wayne, instead, emphasizes a strong version: *“Even 100% honest pursuit of a metric, taken far enough, is harmful to your goals, and this is an inescapable consequence of the difference between metrics and values.” This version requires no bad faith or gaming. An honest pursuit of one metric among others has a way of twisting things in unintended ways.”* The philosophy underpinning MTTR (that you can't prevent every failure) isn’t wrong, but it licensed the treatment of customer pain as a cost of doing business. As development accelerates, MTTR, followed faithfully and innocently, can cause teams to rely on customers having negative experiences, especially as they scale up velocity. ### The pain of MTTR in the age of AI MTTR sanitizes what failure means. Ultimately, customers had to experience a failure, report it, file a ticket, and wait for it to be fixed. That’s true no matter how low your MTTR is. Depending on your scale, that could mean tens, hundreds, thousands, or millions of people having bad experiences. And because there’s typically a delay between a bug happening and a user reporting it, users can live with bad experiences for long periods. Worse, only a fraction of users ever report bugs, meaning many will suffer through the experience, feeling discouraged or becoming churn risks. With AI, the risk and scale of this failure pattern is only increasing. According to research from [DORA and Google](https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report), an increase in AI adoption has been accompanied by a 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability. And they’re not alone. Research from Uplevel complements these results, showing a [41% increase in bug rate](https://uplevelteam.com/blog/genai-developers) resulting from the use of generative AI tools. If you measure success based on MTTR, you're allowing your customers to experience paper cuts frequently and big pains periodically. If you accept that pattern in the name of fixing it fast, you accept a bad experience in disguise. This was just as true before AI as it is now, but with AI, velocity *and* bugs are rising. You’re putting more work in front of your customer than ever, and it's buggier than ever. Bad experiences will occur more frequently. Even if AI helps you resolve them, you're still making your customers play quality assurance for you. Sadly, the pain is also disproportionate in a way no company wants: The customers suffering the most will be your power users and early adopters. Organizations end up in a situation where AI transforms everything except their ability to reliably provide quality user experiences. Fixing it fast still means you're on day zero, again and again and again. That's no way to build a product, and it's certainly not a good way to measure quality. That's why MTTR has to go. ## MTTF: An old metric with a new life You’ve likely heard of MTTF, or mean time to failure. MTTF has its origins in hardware, where organizations used it to measure the average time before components broke down and required replacement. Reframed for software, MTTR has a new lease on life and the capacity to change how we think of failure in the wake of AI. Here, failure isn’t an outage or a dip in reliability but a customer having some level of negative experience. This shift in metrics also requires a shift in emphasis away from post-deployment response to pre-deployment review and [runtime analysis](https://www.ito.ai/blog/static-analysis-misses-bugs-that-reach-production). ### Code review is the current bottleneck With AI, we can accelerate sheer code output. However, a 10x increase in code also requires a 10x increase in code review capacity. Unfortunately, as an industry, we’ve spent much more time increasing code production than improving our review processes. Anyone familiar with systems building knows what happens next. In *The Phoenix Project*, written almost 15 years ago, the authors write, *“Any improvement made after the bottleneck is useless because it will always remain starved, waiting for work from the bottleneck. And any improvements made before the bottleneck merely results in more inventory piling up at the bottleneck.”* Right now, we’re just putting more pressure on the code review bottleneck and hoping for increased throughput anyway. The results speak for themselves. [Faros AI’s 2025 telemetry study of 10,000+ developers](https://arxiv.org/pdf/2605.01160) found that “A 98% increase in merged pull requests coincides with a 91% increase in review time and flat organizational delivery metrics.” The *New York Times*, providing an even more vivid example, spoke with a [financial services company](https://www.nytimes.com/2026/04/06/technology/ai-code-overload.html) that went from producing 25,000 lines of code per month to 250,000, resulting in a backlog of 1 million lines of unreviewed code. Some metrics moved (PRs and lines of code), but did the most important ones change for the better? ### The blast radius of bad code is growing Despite the bottleneck, the risk of rushing code review is enormous. Research from [METR](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) shows that the length of tasks that AI agents can complete on their own is doubling about every 7 months. As true AI-generated code scales, the volume of potential bugs increases (recall the 41% increase in bug rate cited earlier). With [agents creating code autonomously](https://www.ito.ai/blog/what-is-agentic-qa-the-complete-guide), including bugs, the blast radius of failures can increase in tandem. More bugs across more code can result in domino effects that spread across systems, and because agents author most of the code, the failures can be inscrutable to human incident response teams. Code review processes, as they stand today, are not up to the task. You try to catch things before they ship, knowing that you likely won’t catch everything. That leaves you acknowledging that you’ll only catch the issue later, in production. From there, you can roll back, fix, and keep compressing, but *never eliminate*, MTTR. This is especially risky when MTTR fails to capture a complete view of poor customer experiences. ### Do you know what you’re recovering from? According to research from [VOID](https://www.infoq.com/articles/incident-metrics-void/), there is no correlation between incident duration and incident severity, meaning MTTR can be misleading. According to the company’s research, organizations *“can have long or short incidents that are very minor, existentially critical, and nearly every combination in between. Not only can duration not tell a team how reliable or effective they are, but it also doesn’t convey anything useful about the event's impact or the effort required to deal with the incident.”* As a result, MTTR captures skewed data when we need precise data more than ever. The more code you generate, the more bad experiences MTTR will fail to capture and prevent. You’ll never know which issues were fine and only seen by one small customer, and which ones hit your core user base on a core workflow, or a key buyer on the day they’re making a purchasing decision. ### New metrics, new workflows MTTF implies a new way of thinking. Suppose there’s a bug in the code. Do we care? The answer requires a follow-up question: Does the bug cause a bad customer experience? If no, we don’t care; if yes, we do care, and we care enormously. The throughline is not quality for quality’s sake, but confidence that customers will not have bad experiences. Short-term, that means accelerating developers with AI, relieving the code review bottleneck without sacrificing or driving your engineers crazy with endless code review, and trusting that user experiences will remain good throughout. Long-term, this enables teams to ask difficult questions without worrying about losing quality. For example, a common debate among many engineering teams is whether you should let non-technical people vibe code to production? The cost of doing this can be high, but with the right safeguards, so can the rewards. With MTTF, you safeguard the shift because engineers won't have to measure how fast they can fix what the vibe coders mess up. They can instead ensure the right guardrails are in place, so that only the good results of vibe coding make it to production. MTTR, however, isn’t a hot-swappable metric. Industry workflows will have to change as organizations shift to MTTF. For example, when developers have [agents working in parallel](https://www.ito.ai/blog/your-ai-scaled-engineering-org-needs-big-org-processes) with them and each other, teams will have to think about the pace of work differently. Rushing to production – shipping fast and rolling back later – won’t be the default right move anymore. Not when code review is as scalable as code generation. To build the bridge to this future state, organizations can use a combination of tools to provide multiple code review passes: 1. Run [static analysis tools](https://www.ito.ai/blog/static-analysis-misses-bugs-that-reach-production) to find issues in the code. Fix the code until the static analysis greenlights it. 2. Run Ito to detect behavioral issues through runtime analysis. Fix the code until Ito greenlights it. 3. Ask a human to check the code. Fix the code until the reviewer is happy. 4. Run smoke tests and/or E2E tests as a final pass. 5. Merge. Each pass will catch different issues, and in combination safeguard your customers from having bad experiences when you ship. Now, you do have to remember that MTTF is not a silver bullet. Don’t treat MTTF as another singular metric, or it will become another victim of Goodhart’s Law. Start by defining the unit: a customer-impacting failure is any discrete moment a real user hits a degraded experience. MTTF is the interval between those moments. Instead of reducing it to a single, simplistic number, track it as an impact-weighted distribution, separating the cohorts that suffer first and churn fastest (power users, new users, high-value accounts) and the workflows you can't afford to break (checkout, login). Then, pair that lagging indicator with a leading one: how often a customer-facing issue slips past static analysis, runtime analysis, human review, and smoke and E2E tests. Together, you can build a scorecard that lets you track and improve MTTF. By putting this into practice, you’ll stop asking how fast you cleaned up after a customer got hurt, and start asking how long it's been since a customer got hurt at all. ## Skills, not cycles Before AI, the industry mapped software development linearly, with the SDLC aiming to reduce MTTR (treating failure as built-in). With AI, we can rethink these foundational assumptions and view our organization as building skills rather than following cycles. For example, in a video game like Runescape, you build skills over time, and the decisions you make at the early levels can shape the decisions you’re able to make at the later levels. As you progress in your skill tree, your early decisions compound, and by the end game, you can find yourself grateful or regretful, victorious or struggling. If you want to reach a particular end state, you need to know where you’re going when you start. Thinking ahead to the end state and building experiences that compound instead of subtract is the framework we need to embrace. Prevention compounds, while recovery resets you to day zero again. And that’s the core benefit of our work: catching issues *before* they hit your customers, not helping you respond faster once they do. Ito is the MTTF layer that static code review misses, but your customers experience if you don’t catch them. What will your engineering team prioritize? ## Common questions ### What is MTTF in software engineering? MTTF stands for mean time to failure. In this context, failure refers to a customer having a negative experience. MTTF measures how long you go before that happens. The longer the interval, the better you're doing. ### What’s the difference between MTTR and MTTF? MTTR measures how fast you recover after something breaks; MTTF measures how long you go before a customer has a bad experience. MTTR assumes failure is inevitable and asks only how quickly you can clean up from each incident. MTTF asks how you drive the failures that reach customers toward zero. MTTR pulls attention toward having a great post-deployment response, while MTTF draws attention to pre-deployment review and runtime analysis to prevent issues from ever happening to your customers in the first place. ### Why is MTTR the wrong metric for engineering teams? Unfortunately, MTTR sanitizes what failure means, because it makes your customer’s pain acceptable, and implicitly accepts that they will regularly have bad experiences. The data itself is also skewed: there is no distinction between incident duration and incident severity, meaning a team can have long incidents that are minor and short incidents that are existentially critical. As AI accelerates code output, both in number of PRs, and size of the changes it can manage, it also widens the blast radius of bad code. That makes MTTR a metric with diminishing returns, because it is increasingly harmful as your AI code output increases. ### What does 'mean time to failure' mean for customer experience? Mean time to failure means the customer experience becomes the definition of failure, rather than a downstream consequence. The goal is confidence that customers will not have bad experiences, and you measure how often it happens. The greater your mean time to failure is, the less frequently bad experiences occur for your customers. Or put more simply: Prevention compounds, while recovery from a bad experience resets you to day zero. --- # Your AI-scaled engineering org needs big-org processes When developers are 3–5x more productive with AI, your org is effectively that much bigger. Your operations need to follow suit. By Evan Marshall · 2026-05-05 Engineering leaders are celebrating a real productivity gain. Developers using AI coding tools are shipping 3–5x more code than they were two years ago. AI isn't just a linter; it's a force multiplier. Here is the problem: almost no one is running their org like it’s 3–5x bigger. Faros (2026) calls this "acceleration whiplash,” a phenomenon where engineering throughput is up, but bugs, incidents, and rework are rising even faster. When you hire your way from 50 engineers to 250, operational upgrades are forced. You adopt Architecture Decision Records (ADRs). You formalize Change Management. You build a dedicated department for QA automation. You do this because you can no longer rely on the person across the room remembering why a specific trade-off was made in 2024. With AI-scaled productivity, the output scales, but the headcount stays flat. Nobody forces the operational upgrade, leaving you running 250-person output on 50-person processes. That’s a silent ticking clock. ## What actually changes when a 50-person team becomes a 250-person team In a growing company, scale forces change. With AI, that force is absent, but the consequences of staying small-minded are the same. Recent data shows that despite a **34% increase in task completion**, median review times have increased **5X** because organizations cannot safely absorb the volume. 1. **Documentation moves from "nice-to-have" to non-negotiable** At 50 people, context lives in Slack threads. At 250, it doesn't. At a larger scale, you have to treat every merge like someone who wasn't in the room needs to understand it six months later. With AI-generated code, there is an additional risk: there may not be a human who fully "knows" why something was written a certain way. It’s estimated that 60% of AI-generated code is now accepted into codebases, meaning AI has moved from assistant to author. To counter this, teams at scale must implement mandatory PR evidence. Every pull request testing cycle should include: - **Decision records:** Not just what changed, but why. - **Visual proof:** Screenshots and videos of the behavioral change. 2. **Testing must scale with output, not headcount** A 10-person team with one QA person can barely keep up. That same ratio breaks catastrophically when PR volume triples. If your team’s output is equivalent to a 250-person org, your testing infrastructure needs to handle that volume. The data confirms the danger of falling behind: the incidents-to-PR ratio has more than tripled under high AI adoption. You need automated, high-coverage behavioral testing before things merge, not just regression testing after the fact. 3. **Code review rigor must increase, not stay flat** More code means more surface area for errors to compound. At large orgs, engineers expect code review to be thorough and sometimes slow. That’s a feature, not a bug. However, the "whiplash" is causing a breakdown in this gate: 31.3% more PRs are now merging without any review entirely. When volume increases 5x, "it looked fine to me" is not a sufficient review. You need specific evidence that the intent of the code matches the reality of the user experience. 4. **The incident blast radius expands** At a 20-person company, a production bug is painful but recoverable. At 100, it's a multi-team incident with formal postmortems. If your team is producing code at a 250-person rate, a production incident carries 250-person consequences: more surface area affected, more complex rollbacks, and higher costs of mitigation. Monthly incidents are up nearly 58% as AI-generated code reaches production systems. ## The mindset shift for engineering leaders The core reframe is simple: Stop asking "how do I get my engineers to move faster?" Instead, ask: "If I woke up tomorrow and had 5x more engineers on my team, what would I immediately need to change about how we work?" Because in terms of output, you already woke up with that team. To change your mindset, use the AI-scaled readiness checklist: - **Audit your PR evidence** — Does every merged PR leave a thorough, human-readable trail? - **Measure QA vs. volume** — Is your testing coverage keeping pace with your current PR volume, or your historical headcount? - **Pressure-test docs** — Would your current documentation make sense if the team doubled tomorrow? - **Recalculate bug costs** — Think about the cost of an incident at your output scale, not your headcount scale. Bugs per developer are already up 54%, so the infrastructure of accountability must match. ## Bridging the quality infrastructure gap The companies that win in the AI era won't be the ones that just shipped the most code. They will be the ones that figured out how to run a 500-person engineering organization with 50 people, both in output *and* process maturity. If your team is moving at AI speed, you can’t rely on manual QA to hold the line. You need tools that scale with your output and that help reduce manual testing. Ito is designed for this exact gap: it gives AI-scaled teams the automated QA testing infrastructure of a much larger org by validating user flows on every PR, providing the video evidence your reviewers need to approve with confidence. ## Engineering maturity at AI scale: by the numbers - **3-5x** more code output per developer with AI coding tools. - **242.7%** increase in the incident-to-PR ratio when organizations fail to scale their quality gates. - **67.4%** increase in daily PR contexts per developer, leading to massive cognitive load and "thrashing". - **5 min** to set up automated PR testing with Ito (1-click GitHub install). ## Frequently asked questions ### Does scaling processes mean slowing down individual developers? Not necessarily. The goal is that the system moves faster even if individual PRs get more scrutiny. That is the tradeoff every large org makes: slower per-PR, but faster per-quarter because you spend less time on production fires. ### How is this different from just hiring more QA? Hiring QA scales linearly, but tooling scales with output. If AI makes your developers 5x more productive, your QA automation approach needs to handle that without linearly scaling headcount. Automated behavioral testing handles the volume without the hiring constraint. ### What's the first process to upgrade when adopting AI coding tools? Testing and PR evidence. These two processes represent the most immediate risk when there is a gap between 50-person practices and 250-person output. Without behavioral testing, you inherit the full blast radius of your output scale. ### Sources 1. Faros (2026): [The Acceleration Whiplash](https://pages.faros.ai/hubfs/AI_Engineering_Report_2026_The_Acceleration_Whiplash_Faros.pdf) — Analysis from 22,000 on the real-world impact of AI adoption. 2. Github (2024): [Research: quantifying GitHub Copilot’s impact on developer productivity and happiness](https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/)** **— Widely-referenced research on the impact of AI adoption on developer productivity. --- # Bug Catch of the Week: How a Failed Database Swap Could Delete Your Only Copy DoltLite's replacement flow could delete the old database before the new copy was safe. Here's how runtime testing exposed the failure path before release. By Evan Marshall · 2026-09-04 **Your database rewrite failed. The Good:** It returned an error. **The Bad:** It had already deleted the only known-good copy. Ouch. That was the failure Ito caught in DoltLite *before* merge (thankfully!). A file replacement error left the system without the original database or a reliable recovery copy. The team confirmed the finding and fixed the replacement flow. This is the first **Bug Catch of the Week**, a semi-regular series about verified, critical and high severity bugs caught in the wild *before* they merge. Each entry follows a real **runtime analysis** finding: Ito runs the code on a pull request, exercises the affected path, and shows what static **analysis alone** could not verify. ## The bug catch at a glance - **Product:**[ DoltLite](https://github.com/dolthub/doltlite), an open-source embedded database. - **Severity:** Critical. The affected path replaces the database file during backup and garbage collection. - **The catch:** A replacement failure could leave neither the original database file nor a completed replacement available for recovery. - **Potential production consequence:** Stored database contents could be lost after an interrupted replacement. - **Evidence:**[ the merged pull request](https://github.com/dolthub/doltlite/pull/2171), [Ito's test result](https://app.ito.ai/pull-requests/d8c38841-515a-4f0f-87f9-f80c222355d8), and [the fix commit](https://github.com/dolthub/doltlite/commit/d3305aab79). ## The impossible-looking symptom A refactor moved database replacement into the platform-specific code. The happy path still worked, but that doesn't mean it was always safe. Then Ito forced a file-operation failure. The replacement itself reported an error, as it should. But when Ito inspected the state after an injected failure, the system could have already removed the existing database. A recovery operation had failed, yet the only proven-good copy was no longer there. That is the scariest part of this kind of issue. A normal rewrite can look correct right up until a file operation goes wrong. The failure result does not guarantee that the database is safe to retry. ## What the first explanation missed A competent reviewer could see error handling, a cleanup path, and a successful replacement test. Those facts establish that the happy path worked. They do not establish the order in which the old file, temporary file, and replacement are handled after an I/O error. The missing condition was a failure after replacement had begun. The old flow could delete the existing destination before opening and synchronizing the replacement. If a later write, truncate, sync, or open operation failed, cleanup could delete the destination again. The operation correctly returned an error, but the recovery guarantee had already been broken. ## The investigation Ito tested the replacement operation under injected file-operation failures, then checked the state of both files. 1. The expected outcome was that the original database and the temporary replacement would remain available for recovery. 2. The failed run showed that the operation could return an error after the original had been removed, without a reliable replacement to fall back to. 3. The public implementation explained the result: deletion could happen before the later file operations that still had failure paths. 4. Dolthub founder and maintainer Tim Sehn confirmed the finding and changed the ordering, then added fault-injection coverage for the failure cases. ## The mechanism and consequence Here is the failure chain: the replacement code removed database.db before the replacement file was safely installed. It then opened, wrote, truncated, or synchronized the replacement. An I/O error at any of those steps stopped the operation after the known-good database had already been deleted. The cleanup path could then remove the destination again. The error message was correct. The state left behind was the problem. The fix changed the order of operations. It avoids deleting the destination first and retains the synchronized temporary replacement when the replacement fails, leaving a known recovery path instead of an ambiguous one. Fortunately, no production data loss was reported. This was caught before merge. The potential consequence still made it a critical finding: a database rewrite that fails must leave a safe recovery path. Otherwise, it can delete the only database file available to that operation and cause data loss. ## Why reading the code was not enough A code review can identify error handling and confirm that the normal copy succeeds. Static analysis can also flag a suspicious operation order. Neither can prove what remains on disk when the filesystem fails halfway through a replacement. [Runtime analysis and static code review answer different questions.](https://www.ito.ai/blog/ai-code-review-read-your-code-run-it) The decisive test asks the filesystem to fail at a particular step, then checks the original database, temporary replacement, and recovery state. Ito ran the pull request's replacement code under those conditions. The failure could only be proven before production by running the replacement under a forced I/O error and checking what was left on disk. Ito did that before merge. The maintainers fixed the order of operations and added fault-injection coverage for the failure cases. The Key Lesson: The failure path is part of your product, and it needs to be tested before your users find it. **Want a closer look at the evidence?** See it all below: - [DoltLite pull request #2171](https://github.com/dolthub/doltlite/pull/2171) - [Fix commit](https://github.com/dolthub/doltlite/commit/d3305aab79) d3305aab79 - [Ito test result for the pull request](https://app.ito.ai/pull-requests/d8c38841-515a-4f0f-87f9-f80c222355d8) *Want Ito to run your code before your next merge? SIgn up to *[*run Ito on your PRs.*](https://app.ito.ai/auth/signup?utm_source=blog&utm_medium=post+mention&utm_campaign=bug+of+week+1+-+dolthub+database+drop&utm_term=free+trial+end+of+post) --- # What Happened When DoltHub Ran Ito on 43 Pull Requests DoltHub reported a 2:1 fixed-to-dismissed ratio for Ito bug findings, and surfaced pre-existing issues in 26 of 43 tested PRs over six weeks. By Evan Marshall · 2026-08-04 Finding one bug can be luck, but a code review tool that keeps finding real problems is a lifesaver. That is [what DoltHub saw](https://www.dolthub.com/blog/2026-07-20-ito-ai-qa-for-doltgresql/) after six weeks using Ito on Doltgres, one of their open source repos. In the review threads where its engineers judged Ito's findings, they confirmed more than two valid bugs worth fixing for every one they chose to ignore. They also estimated that Ito's findings led to them fixing issues before production for roughly one out of every three pull requests. That is the kind of proof you want to grow your confidence both in using an AI code review tool, and safely merging to production. ## DoltHub did the work. Go read their full breakdown. Jason Fulghum and the DoltHub team published the full story themselves: the SQL queries Ito ran, the bugs it found, the evidence in the pull request comments, the false starts, and the numbers behind their conclusion. We are grateful they took the time to write it. This short post gives you the headline. Their post gives you the full, unfiltered proof. [**Read DoltHub's complete technical review of Ito.**](https://www.dolthub.com/blog/2026-07-20-ito-ai-qa-for-doltgresql/) ## What six weeks of real Ito use looked like Ito commented on 92 Doltgres pull requests**.** Many were automated dependency bumps or release metadata changes, so DoltHub set those aside. That left 43 pull requests where there was real logic for Ito to run and test. Of the 38 of those pull requests that had merged or closed, Ito reported at least one issue on 18. Eight of those showed a fix and re-check cycle directly in the pull request comments before merge. DoltHub also counted fixes that happened later or were moved into its backlog. With those included, it estimated that Ito's findings directly led to improvements made on roughly 33% of the pull requests it could meaningfully test. That is the outcome you want from a review tool. It should keep giving your team things worth fixing as you build new features, fix bugs, and scale your product without being too noisy. **The takeaway: Ito gave DoltHub a repeatable way to catch and fix problems before merge.** ## Ito found more than regressions in the changed code **In 26 of those 43 pull requests, Ito surfaced at least one issue DoltHub labeled as pre-existing and unrelated to the code in that pull request.** That does not mean every one of those issues was a confirmed bug or needed a fix. DoltHub was clear about that. It does show what happens when Ito runs your product and explore the edges around a change instead of staring only at the changed lines. Some problems live nearby. A pull request is often the reason you finally look closely enough to find them. **The takeaway: running the code can uncover problems that a diff-only review never gets near.** ## The two SQL bugs show why this matters **DoltHub documented two separate bugs in SQL window functions that Ito found while reviewing one pull request.** Both were deep in a shared query processor used across its database products. The first bug returned the full sum for every row when a query used a named SQL window. It should have returned a running sum. The bug was especially nasty because DoltHub already had test coverage for the syntax. The test expected the wrong answer. The second bug involved inheriting or overriding a named window's ORDER BY clause. It created the same wrong full-partition result through a different path. Neither problem was obvious from the pull request diff. Ito ran Doltgres, opened a PostgreSQL connection, issued the queries, and showed the wrong output next to the correct baseline. You can see the exact SQL and results in [DoltHub's first bug write-up](https://www.dolthub.com/blog/2026-07-20-ito-ai-qa-for-doltgresql/#bug-named-window-reference-over-w-produces-a-full-partition-sum-instead-of-a-running-sum-within-each-partition) and [second bug write-up](https://www.dolthub.com/blog/2026-07-20-ito-ai-qa-for-doltgresql/#bug-inheriting-or-overriding-a-named-windows-order-by-drops-it). **The takeaway: a green test suite is only as good as the behavior it actually checks. Dolthub shows how Ito can fill in the gaps in your test suite to catch important issues.** ### DoltHub's results came with clear limits **DoltHub found Ito especially useful when it was testing well-documented SQL behavior and edge cases.** It also said Ito was less helpful on internal performance work that did not directly change SQL behavior, and sometimes added comments that did not warrant action. That is the honest version of this story. Ito is not a replacement for human review. DoltHub kept human reviewers in the loop, but Ito made their lives easier by providing greater test coverage than they can get from static analysis and their own alone. Ito also got better over the 6 weeks. DoltHub found a problem where Ito fell back to an older binary after a branch build failed, which led to false alarms. We fixed it so failed builds now fail clearly instead of testing the wrong artifact. DoltHub said it had not seen the issue again, and noted how responsive we are: > “When we’ve hit an occasional issue using the tool, the Ito team has been responsive and quick to roll out a solution to improve our experience, including making their product work well for non-web-UI products, like Doltgres.” Jason Fulghum, Software Engineer at DoltHub Good tooling earns trust by being useful, by showing its work, and by getting better with every bit of feedback from customers. ### Try Ito on your own pull request The fastest way to see whether Ito helps your team like we helped DoltHub is to put it on a pull request and see the evidence yourself. Ito runs your code, tests the behavior around the change, and reports what it found right in the comments of your PR in Github. And you get 10 free reviews when you sign up below: [**Try Ito free on your next pull request.**](https://app.ito.ai/auth/signup?utm_source=blog&utm_medium=post+mention&utm_campaign=dolthub+case+study&utm_term=free+trial+end+of+post) ## Common Questions about Ito you may have ### Can Ito find a bug when the existing test suite passes? Yes. For example, DoltHub had test coverage for named-window SQL syntax Ito tested, but the test expected the wrong result. Ito ran the named-window query against a live Doltgres server and compared it with the equivalent inline query. The output exposed the mismatch. A passing test suite confirms its current assertions. It does not guarantee those assertions describe the correct behavior. ### Can Ito test backend services and databases without a browser UI? Yes. For example, Doltgres is an open source database server, so Ito started the server, connected through PostgreSQL, and issued SQL queries. Its evidence was a reproduction script and the observed output, not a screen recording. Ito ensures the execution method matches the product's real interface to give the best, most actionable result whether your PR surface area is front-end-heavy, back-end-heavy, or a mix of both. --- # AI Code Review on GitHub: Automating PR Review Without Writing a Test Suite Get behavioral feedback on every GitHub pull request before you write a test suite. Learn how to set up runtime PR review and where human judgment still decides the merge. By Evan Marshall · 2026-10-02 Every senior engineer runs the same mental exercise when a PR lands in their queue. They scan the diff, translate the change into a list of things that could break, and wonder whether the author tested any of them. Dozens of times a week, reviewers mentally work out what a code change could break and what they need to test. It's expert labor, and it's almost entirely invisible, and until recently the main way to make it reusable was to author a test suite upfront. But writing, maintaining, and populating an environment for a full suite takes time that many teams don't have before they need feedback. Reviewing code, proposing a test plan, and executing that plan are three separate activities. A tool that reads the diff and leaves inline comments handles the first activity. A tool that translates PR context into scenarios handles the second. A tool that builds and runs the application against those scenarios handles the third. Knowing which of those your team needs, and where the human reviewer still belongs, makes the difference between useful automation and a green badge that doesn't mean anything. In this guide, we’ll cover a GitHub-specific workflow, a shortlist of 7 compatible tools, and a concrete example that connects a code change to observed behavioral evidence. By the end, you'll know what to connect, what context to supply, what actually runs, and what stays a human decision to help you approve PRs and merge them with more confidence, faster. ## How to automate GitHub PR review without a test suite You can automate parts of PR review without writing a test suite first. But checking how a change actually behaves requires a runnable application and a plan for testing it. Building that plan is the exercise you've been running in your head on every PR, and you can hand it off now. You still make the merge call, but you no longer have to come up with the test scenarios yourself or write a suite before anyone can check the change. Here's how the workflow runs. 1. Connect your GitHub repository to the review tool. 2. Write PR descriptions that explain intent, expected behavior, and scope, so the tool has material to work from. 3. Configure the environment your application needs to run, including credentials, seed data, and service access. 4. Trigger a review, observe the run state, and read the findings with reproduction evidence. 5. Use the results alongside your existing CI checks and human approval to make the merge decision. The sections below walk through each step using [Ito](https://ito.ai/docs), which generates a targeted test plan from PR context and executes the changed application in an isolated environment. That walkthrough answers the problem-solving question directly, showing how you can catch runtime bugs before merging a GitHub pull request even without a pre-written test suite. Review tools that read your code can catch logic errors, naming problems, and pattern violations in the diff. Tools that run your application can catch behavioral regressions, data integrity failures, and broken user flows that only appear when the code executes. Both signals matter, and the setup steps, latency, and evidence each approach returns differ, which is why the tool list below covers both. ## What are your options for AI code review on GitHub? Most AI review tools for GitHub read the diff and leave comments on it. Ito, our product, builds the changed application and runs a generated test plan against it. The tools below connect to GitHub in 3 ways. Copilot is built into GitHub, most others install as hosted GitHub Apps, and ai-review runs as an Actions workflow you maintain yourself. Before you connect a tool to your GitHub repository, here's what to expect from each option. For setup steps, pricing, and a fuller comparison, see our roundup of AI code review tools for GitHub. ## Set up Ito for automated GitHub PR review The 5 steps below follow one repository from GitHub App connection through test planning, environment configuration, findings review, and the merge decision. You'll move between GitHub's own installation flow and our dashboard along the way, but there's no review-workflow YAML to author and no test scripts to write before the first run. ### Step 1: Connect your GitHub repository to Ito Go to [app.ito.ai](http://app.ito.ai) and select "Sign up with GitHub." Ito uses GitHub OAuth for authentication, so no separate password is needed. After authentication, the onboarding wizard opens at /auth/setup. In the Connect step, you'll see your existing Ito installations if any exist. Select one, or click "Install App" to go to GitHub's installation page. Choose the organization or personal account, then grant access to specific repositories rather than all repositories in the organization. Click Install, then Continue back in the Ito wizard. Two things can slow this step down. If you don't have admin or owner rights on the GitHub organization, GitHub queues the install request instead of completing it, and onboarding shows "Waiting on organization owner approval" until an owner approves it. And because onboarding lists only your own open or recent PRs, you'll need at least one in the repository you're connecting to, ready to be built and used for testing. Once onboarding completes, confirm the repository appears in the Repositories list in our dashboard before moving to settings. For an existing installation where you need to adjust repository access later, go to Repositories > Manage GitHub Access. An authorized GitHub administrator can add or remove repositories there without repeating the full onboarding flow. This integration uses the [Ito GitHub App](https://github.com/apps/itoqa), so there's no review-workflow YAML to maintain. ### Step 2: Give Ito context for an AI-generated test plan Ito builds its test plan from the diff and the PR description, so the plan is only as good as the context you give it. Write the description for someone who has to test the change, not just review it: explain what the change does, how the affected flows should behave afterward, and what's deliberately out of scope. We don't read issue trackers directly, so ticket context needs to reach the PR description one way or another. If you use Linear, linking an issue by putting its ID in the branch name or PR title, or writing a magic word like Fixes ENG-123 in the description, is enough. [Linear's GitHub integration](https://linear.app/docs/github-integration) then posts the issue's title and description as a PR comment, which we read as additional context. For other trackers, pasting the relevant ticket details straight into the PR description is the reliable path. For priorities that apply across every PR, rather than only one, use [Test Instructions](https://www.ito.ai/docs/settings/test-instructions) instead. A line like "always verify authenticated flows" or "skip the admin dashboard for now" applies org-wide by default and can be scoped to a specific repository or PR author. Treat it as a starting point rather than a finished configuration: Ito calls this a new and evolving capability, and plans improve as your team closes findings as won't-fix and Ito learns from what gets fixed. To see what good context produces, take [DoltgreSQL PR #2913](https://github.com/dolthub/doltgresql/pull/2913). The PR moved window function support into the Doltgres layer specifically so its behavior would match PostgreSQL's. A description stating that intent, plus the expected SQL output, gives Ito exactly the material it needs to generate targeted scenarios instead of generic ones. ### Step 3: Configure the environment and run your first review Before Ito can execute anything, it needs to run your application the way a person would. Open [Settings > Context & Secrets](https://app.ito.ai/settings/context-secrets) in our dashboard, select the repository, and supply what it needs: - Variables for non-sensitive configuration like a base URL or feature flags. - Secrets for credentials, API keys, and private-package tokens. Use dedicated test accounts and nonproduction credentials: our tests can exercise adversarial behavior, so credentials that touch production data don't belong here. - Seed data, if the application needs specific records to reach the states the test plan exercises. With that in place, open [Settings > Automation](https://app.ito.ai/settings/automation) to turn reviews on for the repository. To change these settings, you need to be a Github organization admin, an Ito Manager, or a personal-account owner. Reviews only run for PRs opened by active team members, so check that the PR author qualifies before you turn Reviews on. We post a summary on every PR where a run completes, including all-pass runs, with a message like "15 test cases ran, 15 passed." Leave Comments off and results stay in the dashboard only. From here, opening or updating an eligible PR is all it takes. Then, you can watch its state move from Pending to Running in our dashboard. Each run is tied to a specific commit SHA, so pushing a newer commit while a run is executing cancels the stale run and starts a fresh one against the new commit. > **An important note:** a build failure or an inaccessible service isn't a behavioral finding; it's an environment problem. A run that can't build the application or reach a required service returns no evidence either way, so you should treat that outcome as something to fix, not a pass. returned 30 for id 1, while the equivalent inline query returned 10 for the same row. Under PostgreSQL semantics those two queries should return identical results, and the inline query's output was the correct one. DoltHub had test coverage for this syntax already, it was just asserting the incorrect result, which is exactly the kind of gap that reading the diff alone won't surface. Findings like this one, and Additional Findings, both carry one of 4 severities: - **Critical:** Core functionality or key user flows are blocked, or data integrity is at risk. - **High:** Important features are significantly degraded, but the application isn't completely blocked. - **Medium:** Usability is affected, but core functionality remains accessible. - **Low:** Limited impact, often cosmetic or triggered only in rare conditions. Severity doesn't decide whether a run passes: a run fails if any PR-attributed test case fails, regardless of that case's severity, so a failed run can still be a Low-severity bug. As a rule of thumb, fix Critical issues before merge and treat High issues as usually worth fixing before shipping as well. Additional Findings sit outside that pass/fail calculation because they're pre-existing issues, not something the current PR introduced. That doesn't mean ignore them. A Critical pre-existing issue is still worth assigning an owner, even if it isn't blocking this particular merge. When you push a fix, Ito starts a new run against the new commit. Compare that run's results to the one before it, and use the Origin labels, New, Regression, Still broken (verified), Still broken (inherited), plus the separate Bugs Fixed section, to see at a glance whether the fix actually landed. Here’s how the diff summary from Ito would look like on a PR after your commit fixes all the bugs: ### Step 5: Use Ito's results in your GitHub merge decision Ito's review informs the merge decision; it doesn't replace GitHub's required human approvals or your existing CI checks. A "safe to merge" verdict in our dashboard is guidance, not an approving GitHub review, and it only becomes a merge gate if your team has separately configured repository rules to read our check status. The table below covers the signals you'll actually see and what each one calls for: | | Ito review signal | What it means | What to verify before merge | | --- | --- | --- | --- | | | Failed run with a PR-attributable finding | At least one tested scenario related to the PR's changes failed. This is not proof of a configured GitHub merge block. | Reproduce or investigate the finding, check the latest commit's rerun, and obtain the required human approval. | | | Passed run | The scenarios generated from this PR's context completed without failure at the reviewed commit. | Confirm you're looking at the latest commit's results. Check independent GitHub approval or CI requirements. | | | Additional Findings | Pre-existing issues that do not set the run's pass/fail outcome. | Assess severity, assign ownership, and decide whether a Critical or High pre-existing issue warrants action before merge. | | | Review without usable execution evidence | Pending, Running, and Cancelled states, plus build failures and inaccessible services, are all distinct from a passed run. None is a clean behavioral pass. | Obtain current, usable execution evidence rather than inferring a clean result from an incomplete run. | The one habit worth keeping regardless of the signal: always check the latest commit's completed run. An old passed run from before your fix tells you nothing about whether the fix worked. Record ownership and rationale in your team's PR discussion or issue tracker, and if you want Ito to stop re-reporting a finding you've consciously accepted, turn on reviewer dismissal. ## Keep the tests that protect known behavior None of this replaces a test suite. It changes what your suite is for. Keep deterministic tests for the checks a runtime review shouldn't be responsible for, like security boundaries and data integrity rules. They also catch regressions on every push in milliseconds, while a runtime review takes 30 minutes to 2 hours. With automated review, you can get feedback on a PR before you've written any of those tests. The DoltgreSQL example shows why both still matter. The t_inherit bug, where the named-window derivation failed to preserve its ordering, was a genuine coverage gap. DoltHub's team wrote regression tests for it once the fix was confirmed, which is exactly how a suite is supposed to grow. The t_named bug is the harder case: DoltHub already had a test for that syntax, it just asserted the wrong result. Neither bug is an argument against the suite. The suite tells you what the team intended; the runtime finding told them where that intention and the actual behavior had quietly diverged. That's the honest limitation on both sides. A generated test plan only covers what its context pointed it toward. A source reviewer flags false positives and sometimes proposes the wrong fix. A test suite can assert an incorrect result for years and pass every single CI run, the way t_named's did. None of that is a reason to drop any of the three signals, source review, runtime execution, deterministic tests. It's a reason not to trust any one of them past what it actually checked. For a fuller view of how these fit together across your development process, see our post on [how to build quality into your software factory](https://www.ito.ai/blog/how-to-build-quality-into-your-software-factory). ## Where Ito fits in your GitHub review workflow Ito's job is the middle step: turning intent into a plan, then a plan into evidence. Your team still writes the PR description and any Test Instructions that state what matters. We translate that into scenarios, run them against the actual application, and hand back findings with reproduction evidence. Your engineers and your CI pipeline still own everything upstream and downstream of that. That means the planning work doesn't disappear; it moves. Writing a PR description that says what changed and how it should behave was already part of a good PR. Now, that description also drives what gets tested. And because runtime evidence takes real setup and a real run to produce, it arrives after source review is already underway, not before. Read the diff first, let the run complete, then merge with both in hand. The way to find out if that trade-off works for your team is a single-repository pilot: run it for a few weeks, and track which findings were reproducible, how much time and money you spent on environment setup, and whether the review time fits how your team actually merges. Judge it on findings that turned out to be real and verifiable, not on how many comments it left. [Follow our quickstart guide to connect one GitHub repository and run a review on a representative pull request.](https://ito.ai/docs) --- # What we learned after moving 100B tokens/week to open-weight models Ito moved production QA agents from GPT-5.3-codex to open-weights. Here's what we saved, why inference providers matter as much as models, and what to check. By Evan Marshall · 2026-07-13 Ito runs a QA pipeline against every pull request in the repositories we watch: test-runner agents execute planned test batches in devcontainers with a real browser. That agent is our largest LLM cost. From March through June it ran on GPT-5.3-codex; in late June we migrated it to MiniMax M3, through which it now pushes roughly 100 billion tokens a week. Because we meter every agent step, the migration gave us a clean natural experiment: over half a trillion tokens of production traffic, both models on the same workload. Terms, since we use them throughout: a *run* is everything we do for one pull request; a *step* is one agent execution within a run (for the test-runner, one batch of planned tests, so a run contains several); a *turn* is one model call within a step. This post is the guide we wanted when we started: what open-weight economics look like in production, and where the difficulty actually lives. It is not the model. ## The savings are real Median LLM cost per pipeline run fell 55%. The whole distribution moved, not just the median: the per-step median fell 60%, the 90th percentile 36%, the 99th 17%. The cutover took 11 days, staged repository by repository, with a same-day revert path. Weekly spend fell roughly 28% while workload volume grew about 8%. The aggregate hides a wide spread. Within the same repository, MiniMax's median step cost ranged from 19% to 87% of its codex-era median. Context-heavy repositories with long agent sessions benefit most, because cheap cache reads dominate their bill. ## Cheap tokens are not efficient tokens On matched workloads (156 succeeded steps per model, drawn from the same fixed set of repositories), the model we adopted does more of everything except cost: | Median per test batch | GPT-5.3-codex | MiniMax M3 | Ratio | | --- | --- | --- | --- | | Agent turns | 94 | 116 | 1.23x | | Output + reasoning tokens | 21,500 | 46,800 | 2.18x | | Input tokens (incl. cache reads) | 5.4M | 13.5M | 2.51x | | Cost (indexed to codex = 1.0) | 1.0 | 0.46 | 0.46x | An evaluation ranked on turns or tokens to completion would have chosen codex. Our invoices chose MiniMax: at 3–12x lower per-token prices, the inefficiency is absorbed with room to spare. One accounting note: codex's effective output rate ran about 58% above its list price, because reasoning tokens are billed as output but excluded from visible token counts. Derive effective per-token rates from your invoices, not from pricing pages. ## The same model is not the same model on a different provider This is the part no public benchmark told us. “Open weights” suggests a commodity: the same model wherever you buy it. In practice, sampling settings, quantization, context handling, and above all prompt-cache behavior vary by provider, and most of those parameters are not visible to you. We tested GLM-5.2 and MiniMax M3 on Vercel AI Gateway, OpenRouter, and Together AI. Vercel had by far the worst cache hit rate for both models, and scored worst on our internal evals against the same underlying providers reached through OpenRouter. Aggregators add a second copy of the problem. OpenRouter can route to Together AI, but the cache rate and reliability we measured through that path were dismal compared to calling Together AI directly. Same weights, same named provider, different behavior, depending on the layer in between. Our working hypothesis is that serving settings do not survive the trip through aggregation layers intact. We ended up cutting out the middlemen and integrating with Together AI directly. For agent workloads, cache behavior is not a nicety. About 97% of our input tokens are cache reads; a provider that quietly degrades caching multiplies your bill. One constraint worth stating plainly: our shortlist was limited to providers that meet our SOC 2 commitments for handling customer code. A provider you cannot clear through compliance review is not on your list, whatever the benchmarks say. ## What the savings cost Median step duration rose from 23 to 29 minutes, with a heavier right tail. Steps ending in a failed state roughly doubled, from 2.4% to 4.5% of terminal states. Retries absorb most of these before they are user-visible, but they are part of why realized savings (55%) are smaller than the raw price gap (3–12x). For background QA pipelines, six extra median minutes cost no one's attention. An agent in a developer's inner loop would weigh the same numbers differently. ## The playbook 1. **Shortlist providers through compliance first.** SOC 2 and data-handling commitments eliminate options before performance is ever measured. 2. **Evaluate the provider, not just the model.** Run your own evals per provider and per routing path. Assume nothing transfers. 3. **Watch cache hit rate above everything.** For agent workloads it is the largest cost lever, and the first thing aggregation layers degrade. 4. **Go direct once you have chosen.** Every layer between you and the weights is a place where serving settings get lost. 5. **Derive effective per-token rates from invoices.** List prices miss hidden reasoning billing. 6. **Stage the rollout on production traffic, per workload.** Our per-repository savings varied 4x. Keep a same-day revert path. *Methodology: cost figures from our production billing artifacts and database, March 19 – July 12, 2026. The rollout was staged by repository rather than randomized; workload-sensitive comparisons are within-repository before/after. Turn and token comparisons use a matched sample of 156 steps per model drawn from the same fixed set of repositories. Step durations include non-LLM work, unchanged across the migration. Effective per-token rates estimated by least-squares regression of step cost on token counts (residual error under 2%).* --- # How to Build Quality Into Your Software Factory Software factories need speed & quality. Ship fewer bugs by testing in layers; fast tests first, then deeper validation By Barron Caster · 2026-08-28 Your board wants "more AI." Your CEO wants the team to build, "one of those software factories". So you add coding agents, automate the backlog, and push more pull requests into your merge queue. It works, at first. Your PR count has gone up. The roadmap is moving faster. Your reports show that your factory is producing more than ever. Now, your quality bill is arriving. Your reviewers are skimming changes they cannot fully understand. Your engineers are manually clicking through flows before merge because they do not trust the checks. Customers find bugs that passed CI. Your factory is producing more code, but more code is not necessarily better software. Testing is a big part of why. Whether it's due to a lack of testing or blind spots in the tests you have, it's often the biggest reason your software factory isn't delivering the impact you want (customer wins) instead of simply vanity metrics (increased velocity and a board off your back). ## Your factory needs speed *and* stability A software factory is a delivery system built for throughput. Tickets come in. Agents write code. PRs open. CI gates the merge. Done right, it's a machine. [DORA's four key metrics](https://dora.dev/guides/dora-metrics/) measure speed and stability together: deployment frequency, change lead time, [change failure rate](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance), and recovery time. Shipping ten PRs a day only helps if those PRs do not create ten more emergencies for customers and on-call engineers. A factory that optimizes only for throughput is going to have major quality problems. Merge volume can rise while change failure rate climbs right alongside it. As AI coding tools push PR volumes higher, the pressure on the stability side gets worse, not better. Faros' [2026 AI Engineering Report](https://www.faros.ai/research/ai-acceleration-whiplash) tracked 22,000 developers across 4,000 teams and found the strain building downstream: bugs per developer rose 54%, median review time grew fivefold, and incidents per PR more than tripled as AI adoption increased. You cannot ask reviewers to manually click every affected path every single time. There aren't enough hours in the day, nor patience from your team. You also cannot make every pull request wait for the perfect regression suite to run a million tests. A full regression suite on every change turns the merge queue into a traffic jam and trains your people to bypass it. The answer is a [testing stack](https://www.ito.ai/blog/your-ai-scaled-engineering-org-needs-big-org-processes) that runs the right tests in the right order: find and fix the cheapest failures first, then run the heavier tests as your confidence in the PR grows. ### What automakers can teach software teams Toyota learned this same lesson decades ago. The Toyota Production System was developed to ensure consistent, quality cars at a high rate. Despite the lofty goal of building millions of cars per year, it has just two pillars: Just-in-Time, which keeps production moving without waste, and [jidoka](https://global.toyota/en/company/vision-and-philosophy/production-system/index.html), which builds quality controls into the line itself. When a machine detects an abnormality, it stops. When a worker sees one, they can stop the line and signal the problem with an andon board. The defect is fixed at its source before more work gets added on top of it. That is what makes it a production system, not simply a faster assembly line. Toyota did not treat quality as a final inspection after the car was built. Quality controls are part of every stage of making the car. Software factories need the same design. More output only helps when the factory also detects failures early enough to fix them before they reach customers. ## Build your testing stack from cheapest to most complete The testing stack is the software equivalent of Toyota's quality controls. You want the cheapest checks to find the simple failures early, before they absorb more engineering time to fix or review. Then you add slower, more complete tests as the cost of a missed failure rises. Here is a practical order. Times are planning ranges, not service-level commitments. Your codebase, test environment, and change risk will change the numbers. | Testing layer | Typical time | Where and when | Primary job | | --- | --- | --- | --- | | 1. Lint and types | Under 60 seconds | Local machine, pre-commit, or draft PR | Catch syntax, type errors, broken imports, and formatting | | 2. Unit tests | 1 to 5 minutes | Local machine, pre-push, or draft PR | Check known logic inside a function or component | | 3. Static and AI code review | 5 to 10 minutes | Draft PR or ready-for-review PR | Flag risky patterns, inconsistent changes, and edge cases in the diff | | 4. Integration tests | 5 to 15 minutes | Draft PR or targeted CI check | Check contracts between services, databases, queues, and APIs | | 5. Runtime analysis | 20 to 30 minutes | Ready-for-review, high-risk PR | Check whether a real user can complete the job the change affects | | 6. Human review | 15 to 60+ minutes | Ready-for-review PR, based on risk | Validate intent, architecture, tradeoffs, and risks automation cannot judge | | 7. Full regression, performance, and security testing | 30 minutes to hours | Scheduled run or pre-release | Find broad regressions, load limits, and security exposure | The first two layers should usually happen before a pull request is ready for a teammate to review. They give the author immediate feedback and avoid spending CI time or reviewer attention on a missing import or failed branch of known logic. Static review, integration tests, and targeted checks can then start on a draft PR, while runtime analysis and human review make the most sense once the change has the basics covered and fixed. Human review belongs here, but it has a different job from testing. A reviewer should assess whether the team is building the right thing and whether the design is safe to maintain. They should not be asked to manually repeat every automated test. It's not a good use of their time, and it's one of the most common complaints we hear from engineers; they got into their profession to build, not be testers all day, every day. These runtime and human tests are more expensive because they are testing more of the real system. Integration tests need services and data. Runtime analysis needs a running application, user state, and sometimes a browser. Full regression and performance testing can take even longer and require even more data and configuration. The key is to recognize that each layer has blind spots. Lint cannot prove logic. Unit tests cannot prove that services work together. Static review cannot prove what happens when the app runs. Runtime analysis cannot cover every path. The stack works because the various layers of tests combine to cover each other's gaps, preventing bugs from reaching production. **Order matters. You should run the cheap tests broadly and first. Then, you run the expensive tests where the consequences justify them.** ## A passing test can still check the wrong thing Earlier this year, DoltHub was running Ito on Doltgres, its open-source PostgreSQL-compatible database. One PR touched a shared query processor. The diff was clean. Lint passed. Static tests were green. The problem: DoltHub already had test coverage for the exact SQL syntax Ito was about to test. The test expected the wrong answer. So it had been passing. Ito opened a PostgreSQL connection, ran a query using a named SQL window, and compared the output against the expected running sum. The query returned the full partition sum for every row. The test said that was correct. It wasn't. Nothing in the diff would have told you this. Static review could not flag it. A unit test that expects the wrong answer passes every time. The failure appeared only when someone ran the product and compared its result with the behavior PostgreSQL requires. If you'd like to go deeper on this example, [DoltHub's full writeup](https://www.ito.ai/blog/what-happened-dolthub-ran-ito-43-pull-requests) includes the SQL queries, the wrong output, and the evidence Ito posted in the PR thread. This is why a factory needs more than one green check. The unit test did its job as written. The runtime test exposed the job it failed to cover. **A green test suite is only as good as the behavior it actually checks. That's why having each layer matters.** **Each one plays a part in giving you full coverage before you ship a change to your customers that should improve their experience, not hinder it.** ## Where runtime analysis belongs in your stack Runtime analysis should run after your cheap checks pass and before your change merges. That order matters. There is no reason to spend browser time on a PR with a type error. There is also no reason to wait until a nightly suite or a customer report to discover that a valid-looking change broke checkout. The goal is not full-product testing on every push. The goal is to run the user paths your current change puts at risk, while you can still fix what the test finds. ### Isolate each PR's environment Shared staging breaks PR-level testing. When two PRs share an environment, their data bleeds together, feature flags conflict, and failures are often just collisions between your branches, not real bugs. That means each of your PRs needs its own environment so the signal is actually about that PR. This is why our open source project, [Moo](https://www.ito.ai/blog/moo-worktree-isolated-environments), ties agent runtime isolation directly to git worktrees. A Playwright suite in shared staging can be useful, but it is a snapshot of staging. It is not reliable evidence about one specific PR. It's also why Ito spins up a separate environment for every PR we test. ### Scope to the risk in the diff Running the full product on every PR turns this layer into a bottleneck. Pricing logic changes warrants checkout flow tests. A copy change does not. Code-aware scoping means you reserve the slower set of checks for when a change is large or risky enough to deserve a deeper set of testing. Simple, low-risk changes get released faster by skipping some of these. ### Assert user outcomes, not just DOM state "Button visible" tells you the button rendered. "Order persisted and confirmation shown" tells you the user finished your flow. Proper testing should care about the job the user came to do, not just that a screen loads at all. Testing that an element exists is testing the UI layer. Testing that the behavior is correct is testing the product. To ship with confidence, especially at AI speeds of development, you need both. ### Make your PR thread the source of truth A runtime check that posts a red X and nothing else trains your engineers to ignore it. Useful evidence includes the flow name, the steps taken, what the app returned, a reproduction path, and screenshots or video. Whatever evidence you have, it needs to be clear, easy to understand, actionable so you fix it, and located in a place that's easy to reference and share. This matters even after you merge. When an agent loop runs out of control, or a customer escalates a bug three weeks later, the PR thread is where you reconstruct what happened. Screenshots, logs, and reproduction steps in the PR thread create an audit trail the diff alone can't give you. GitHub Actions, checks, and status APIs all have hooks to automate this. Use them. A well-instrumented PR thread becomes a debugging tool, a quality record, and an agent feedback loop all at once. ## Make your test layers work together The testing stack works best when layers have clear boundaries and clear owners. A reasonable pull request sequence looks like this: - First: lint, types, and unit tests stop cheap failures within seconds or minutes. - Next: static review examines the diff for risks those checks do not model. - Then: integration and selective runtime code review test the services and user paths the change can break. - Finally: scheduled or pre-release suites look for the broad failures that are too expensive to run on every PR, but are essential to prevent bugs reaching production. You can run some of these in parallel. The important decision is your budget. Do not make a browser check block every typo. Do not let a sensitive billing change skip a browser check because the unit suite happened to pass. Start new runtime checks as advisory findings, not blocking. Engineers need to trust a signal before it has veto power over their merges. Run it for a few weeks, fix flakey tests, improve your coverage of critical areas, then promote the reliable checks into your merge gates. Treat these flakey tests like infrastructure debt. If the runtime station goes noisy, engineers route around it. Someone has to own flake burn-down the way a platform team owns flaky CI. **A testing layer that engineers cannot trust will not protect the factory.** ## Where Ito can help you Ito supplies the selective runtime layer for teams that do not want to build and maintain that scaffolding themselves. On each PR we review, Ito spins up an isolated environment, runs the user flows most likely affected by the diff, and posts the evidence in the GitHub review thread: what it tested, what it found, screenshots, reproduction steps, and logs. It works beside your linting, unit tests, integration tests, and static review tools. [The DoltHub result](https://www.ito.ai/blog/what-happened-dolthub-ran-ito-43-pull-requests) is the most detailed public picture of what this looks like on a real codebase over time. ## Conclusion Your software factory runs on throughput. But throughput without a working product is debt accumulation at scale. What your board / CEO is asking for is proof that all the AI investment is leading to real gains. That comes from not just shipping more faster, but doing so with quality in mind. Unfortunately, manual testing tires and frustrates your engineers while slowing down your factory. Meanwhile, waiting and fixing bugs in production hurts your customers and burns out your team who is on-call. The happy medium is building a complete testing stack: catch the broad, cheap failures first, then spend more effort on the remaining risks that survive. Lint, unit tests, static review, integration tests, runtime analysis, and deep regression suites each serve a purpose. The right order and knowing when to use each test keeps costs down. The complete stack keeps bugs from escaping through the blind spots between layers. That is how you build a factory that ships fast and keeps working as you scale. ## Common Questions About Testing in Software Factories ### Can AI code review tools like CodeRabbit or GitHub Copilot catch runtime bugs? No. CodeRabbit, GitHub Copilot code review, and Greptile analyze the diff statically: they read the source, apply pattern matching and LLM reasoning, and flag what looks risky. None of them execute the code. They are useful testing layers, but they cannot confirm runtime behavior. Reading code and running code catch different classes of bugs. A strong factory uses both. ### Is running Playwright tests on every PR in GitHub Actions enough? Is it the same thing? If your Playwright suite runs in an isolated per-PR environment and posts evidence in the PR thread, yes, that's the pattern. Most setups miss at least one of those: they run in shared staging (not isolated per PR), report pass/fail with no evidence package to debug from, or cover the whole product rather than the paths the diff touches. Runtime analysis is a specific type of tests on our code changes. Playwright is one valid way to implement it, but requires your ongoing maintenace and test planning. Ito automatically generates a test plan for you by learning about your repos, analyzing the diff, and reviewing other context like a test plan and PR comments. ### What's the difference between a PR runtime check and a nightly e2e regression suite? Unit of work and timing differ between the two. A nightly suite covers the full product on a schedule, after changes have already merged. A PR runtime check covers the paths most likely affected by the diff, before merge, while the author still has context. They serve different parts of your testing stack. Nightly suites catch regressions across the full surface. PR checks catch the specific failure the current diff introduced. Detection before merge is cheaper than detection after, so choose wisely. ### How do you keep PR runtime checks from going flaky and getting ignored? Three rules can help you: scope your tests to behavioral outcomes rather than brittle DOM selectors. When a designer changes the markup without changing the user's job, those selectors can fail even though the flow still works. Run each PR in its own isolated environment to eliminate shared-state collisions that aren't real bugs you need to fix before merge. Assign a clear owner for flake burn-down. A check that fails 20% of the time for unrelated reasons trains engineers to ignore it. Fix the source of the noise and treat that work like any other infrastructure maintenance. ### Which tests should run locally before I open a pull request? Run the checks that give you fast, deterministic feedback: formatting, linting, types, and the unit tests for the code you changed. A reviewer should not spend time discovering a missing import or a failed branch of known logic. Run those checks on your local machine, before making a commit, hook, or draft PR. Save slower environment and browser tests for a change that has earned that cost, and do those once your change is a PR. ### Should human reviewers run the code themselves? Usually, no. A healthy CI system should build the application and run the checks that verify basic behavior. Human review is more valuable when it evaluates intent, architecture, tradeoffs, and risk. For a complex or high-risk change with missing coverage, a reviewer may still need to run the product or request a better test. That is a sign your testing stack has a gap, because with the rise in volume of PRs, it's increasingly difficult to do that for every PR. ### How do you decide whether a runtime test is worth the extra 20 to 30 minutes? Look at the consequence of being wrong. A copy-only change does not warrant a checkout test. A change to pricing, permissions, authentication, saved state, or an external integration does. Use the diff to identify the user job at risk, then run the smallest realistic flow that proves everything still works. The point to do only the tests necessary to build with confidence. --- # Your fastest model is probably not your fastest model Tokens per second measures how fast a model emits, not how fast it finishes. Our 13-model probe shows the gap that flips both the speed + cost leaderboards. By Evan Marshall · 2026-07-21 What shows up on LLM leaderboards and benchmark pages doesn't always match what you experience in practice. For instance, the third-fastest model we tested actually finished ***tenth.*** Unfortunately, the LLM leaderboards we all eagerly check as all the new models drop don't measure task completion time, and the gap between what they show and what your task actually takes is bigger than most teams expect. Two numbers dominate model selection pages: tokens per second and dollars per million tokens. Both are denominated in tokens (roughly a word or word fragment), yet tokens aren't a unit of work. We asked fourteen models to do the same job and they used anywhere from 1,800 to 6,400 tokens to finish it, which obviously makes a big difference in both cost and time to complete the task. The truth is: a model that emits fast but thinks in paragraphs can take longer, and cost more, than one that emits slowly and gets to the point. That's why we made our own benchmark to test all these new models like Luna, Grok 4.5, DeepSeek V4, MiniMax M3, Kimi K3, and Fable. Our test structure: We measured this the way a customer would experience it: One fixed task (review a Python function, write an improved version with tests) sent to fourteen current models through OpenRouter. Specifically: - For each model, we gave them three runs capped short to probe speed, and three runs uncapped, so each model finished naturally. - Speed and latency come from OpenRouter's own per-generation statistics, so they reflect the provider the router actually selected. - Token counts are native counts including reasoning tokens. - Prices are based on OpenRouter's list price. Best of all, this is the kind of test anyone can afford to run, even for your side project: The whole experiment cost under three dollars. Here's what we learned... ## Dividing by tokens per task reorders the leaderboard | | $/Mtok out | tps (measured) | Tokens for task | Seconds to finish | Cost to finish | | --- | --- | --- | --- | --- | --- | | GPT-5.6 Luna | $6.00 | 131 | 2,011 | 15 | $0.012 | | Grok 4.5 | $6.00 | 114 | 2,607 | 23 | $0.016 | | Claude Sonnet 5 | $10.00 | 92 | 2,403 | 26 | $0.024 | | GPT-5.6 Terra | $15.00 | 101 | 2,777 | 28 | $0.042 | | DeepSeek V4 Pro | $0.87 | 65 | 1,973 | 31 | $0.0017 | | Claude Sonnet 4.6 | $15.00 | 73 | 2,235 | 31 | $0.034 | | Claude Fable 5 | $50.00 | 76 | 2,479 | 33 | $0.124 | | GPT-5.5 | $30.00 | 50 | 1,777 | 35 | $0.053 | | MiniMax M3 | $1.20 | 90* Our internal production median on Together AI; the OpenRouter median across its routed providers was 53 tokens/sec. | 4,364 | 48 | $0.0052 | | KAT-Coder-Pro V2.5 | $2.96 | 113 | 5,536 | 49 | $0.016 | | GLM-5.2 | $2.64 | 38 | 2,060 | 54 | $0.0055 | | GPT-5.6 Sol | $30.00 | 45 | 2,644 | 59 | $0.079 | | Kimi K2.7 Code | $3.49 | 41 | 2,952 | 71 | $0.010 | | Kimi K3 | $15.00 | 36 | 6,384 | 176 | $0.096 | Surprisingly, GPT-5.6 Luna won both columns: fastest emitter (131 tps) and fastest to finish (15s). Below it the two rankings disagree, in places by seven positions. KAT-Coder-Pro V2.5, the third-fastest emitter at 113 tps, finished tenth: it generated 5,536 tokens where GPT-5.5 needed only 1,777. That verbosity (using over 3X the tokens) costs more time than a fast emission rate saves. Meanwhile, despite a lot of hype, Kimi K3 was both the slowest emitter and the most verbose model in the set, finished *last* at 11.5x Luna's time. **The harsh reality: Speed rankings on model pages don't predict the finish order on real tasks. Verbosity determines how long a model actually takes.** The reordering happens inside a single product line, too. GPT-5.6 ships in three tiers, and on this task the cheapest finished first: - Luna took 15s and $0.012 - Terra 28s and $0.042 - Sol, the $30/Mtok flagship tier, 59s and $0.079 Paying 5x more per token bought 3.8x more waiting. Slower and more expensive. This is why you have to take a more nuanced approach to choosing the right model for the right task. ## Cheap tokens are not cheap tasks The same split we just saw also applies to price. Kimi K2.7 Code's tokens cost more than KAT's ($3.49 vs $2.96 per Mtok), but Kimi finished the task for 37% less, because it used half the tokens. GLM-5.2's tokens cost 2.2x MiniMax M3's, yet per task the two are within 4% of each other, because of the difference in verbosity. You see this same effect across other model comparisons: - Claude Sonnet 4.6's tokens cost 12.5x M3's, but per finished task the gap is 6.4x. - Kimi K3's tokens cost 3.3x less than Claude Fable 5's ($15 vs $50 per Mtok), but because K3 used the most tokens in the set, the two are within 30% of each other. As you can see, a cheap token rate doesn't mean a cheap task. In our next analysis, you can see that neither ranking survives: the cheapest-per-token model isn't the cheapest per task, and the fastest emitter isn't the fastest to finish. DeepSeek V4 Pro is both cheap per token and terse, so it finished the task for $0.0017, roughly 20x less than Claude Sonnet 4.6 and 73x less than Claude Fable 5, the most expensive finish in the set. Where public data allows a clean comparison, it agrees with our probe. Artificial Analysis publishes output tokens per [Intelligence Index task](https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index): MiniMax M3 at 24.0k, Kimi K2.7 Code at 17.7k, GPT-5.5 (high) at 10.1k, the same verbosity ordering we measured. For several of the newer closed models, no public per-task figure exists, or the published data uses reasoning-effort settings that don't match what a router actually serves. That's part of why we ran this ourselves. For [Ito](https://www.ito.ai?utm_source=blog&utm_medium=post+mention&utm_campaign=july+tokens+analysis), the stakes are high. Tokens per task isn't an academic metric for us; it's how we decide which models stay in production. Our QA agents run the same categories of tasks on every pull request, and a model that uses 3x the tokens to complete the same step costs 3x more and takes 3x longer per review cycle. **The per-token price on the spec sheet tells you the cost of one token, not one task. You need to run your actual workload to find out the real difference.** ## The router adds a second distribution on top The speed column above is a median hiding a wide spread. Across six runs, Kimi K2.7 Code measured 223 tps when OpenRouter routed it to Together and 28 tps on DeepInfra. MiniMax M3 ranged from 25 tps (Parasail) to 130 tps (MiniMax's own API). GLM-5.2 hit six different providers in six runs and ranged 20 to 85 tps. First-party serving narrows the spread but doesn't close it: Sonnet 4.6 and KAT-Coder held within about 15%, while GPT-5.6 Sol varied 2.4x between OpenAI and Azure and GPT-5.6 Luna varied 1.7x across runs on OpenAI's own endpoint. A leaderboard tps entry is a claim about one provider on one day. If your traffic goes through a router, your realized speed is a draw from a distribution the leaderboard doesn't show. For open-weight models the spread of that distribution was larger than most gaps between models. **If you're routing through OpenRouter or a similar service, the speed you see in production isn't the speed on the leaderboard. It's whatever provider got selected for that call.** ## What you should measure instead Once both axes are per-task, your picture clears up. Of the ten models (fourteen minus the four without a published Intelligence Index), six are on the frontier: no single model is cheaper, smarter, and faster than DeepSeek V4 Pro, MiniMax M3, GLM-5.2, GPT-5.6 Luna, Grok 4.5, or Claude Fable 5. For the other four, there's always at least one model in the set that beats them on all three axes. Grok 4.5 alone beats both GPT-5.5 and GPT-5.6 Sol. Tokens per task is as easy to publish as tokens per second, and it's the denominator that turns both speed and price into decision-grade numbers: seconds to finish and dollars to finish. Until a more realistic standard emerges, the probe above is a cheap alternative that's easy to reproduce: choose a batch of the latest models, assign them all one task you care about, give them six runs each, then analyze your results. And you can do all of it for less than $3 of token spend. **To truly know which models are best for your company, measure it yourself.** **The leaderboard won't tell you what you need to know, but a simple test you run yourself will.** It costs just a few dollars to find out. ## Common questions ### What was your methodology for your tests? *Our tests included 90 completions (15 models x 6 runs) through the OpenRouter API, July 14-16, 2026, from one fixed coding prompt; GPT-5.3-codex was measured but excluded from the analysis as a prior-generation model. * *Speed pass: max_tokens 1,200. Completion pass: max_tokens 16,000; runs that hit the cap (one, KAT-Coder-Pro V2.5) are excluded from token medians. Throughput and latency are OpenRouter's own generation statistics (GET /api/v1/generation), token counts are native counts including reasoning tokens, prices are OpenRouter list on measurement day. * *One exception: MiniMax M3's headline throughput is our internal production median on Together AI (90 tokens/sec) in place of the OpenRouter median (53 tokens/sec across three routed providers); the provider-spread figures for MiniMax remain the raw OpenRouter runs. Kimi K3 was measured on July 16 through Moonshot AI, its only OpenRouter provider, with retries around launch-window rate limits; retries do not affect the per-generation statistics. * *Work-normalized throughput uses GPT-5.6 Terra's tokens for the task as the baseline. Tokens per task is task-dependent; matched samples on our production QA workload show the same 2x to 3x verbosity gaps between models doing identical steps. n=3 per cell is a probe, not a benchmark; the provider spread within one model was the largest source of variance we observed. * *Reasoning-effort settings were router defaults throughout, which is what most API traffic gets but not what leaderboard "max" variants measure. Artificial Analysis figures from their public model pages, July 15, 2026.* ### How do I calculate the actual cost of an LLM API call for my workload? Run your real task on each candidate model (or a representative sample of 3-6 prompts from your workload), let each model finish, and record the total tokens used per run. Multiply the median token count by the per-token price. That's your per-task cost. The tokens-per-million price on the model's spec sheet only tells you the cost of one *token*, not the cost of one *task*. On our fixed benchmark, the model with the lowest per-token price ($0.87/Mtok) finished 5th fastest and second cheapest per task. Per-token price and per-task cost disagreed on every model in the set. ### Why did my fast model take longer to finish my task than a slower one? Because tokens per second measures emission rate: the speed a model generates output once it starts. It doesn't measure how many tokens the model uses to complete a task. A model emitting 113 tokens per second that needs 5,500 tokens to answer your prompt takes 49 seconds. A model emitting 50 tokens per second that needs 1,777 tokens takes 35 seconds. Verbosity determines task time, not emission speed. You can test this yourself: run the same prompt on two models and measure wall time rather than watching the token counter. ### What should I actually measure when comparing LLM APIs? Measure seconds to complete your specific task and dollars to complete your specific task. Pick a real prompt from your workload, run it on each candidate model, record wall time and total tokens used, and multiply tokens by the per-token price. That gives you two numbers that are directly comparable across all models. Tokens per task is as easy to publish as tokens per second; it just requires actually running the task rather than measuring burst generation speed. The probe described in this post cost about three dollars across fourteen models and six runs each. ### Is a higher-tier LLM always worth the premium? No. In our benchmark, GPT-5.6 Sol (the $30/Mtok premium tier) took 59 seconds and cost $0.079 to complete the same task that GPT-5.6 Luna (the $6/Mtok entry tier) finished in 15 seconds for $0.012. That's 3.8x more waiting and 6.6x more cost for the premium. The explanation is token verbosity: Sol used 2,644 tokens where Luna used 2,011 on the same prompt. Whether the premium tier gives you better output on your specific task is a separate question. But on speed and cost, the higher tier doesn't win by default. --- # Ito vs Mabl Both run real tests against your application. The difference is the model: Ito puts an autonomous QA agent on every pull request, while Mabl gives you a broad low-code platform to build and maintain test suites. Here's how to choose. ### Ito Agentic, pre-merge QA. Autonomous tests on every PR. ### Mabl Low-code test automation platform for QA and engineering teams. ## Ito vs Mabl at a glance Ito and Mabl both belong to the modern, AI-assisted end of QA — neither is a static analyzer or a linter. Both spin up a real browser and exercise your application the way a user would. Where they diverge is philosophy: Mabl is a **platform you operate**, and Ito is an **agent that operates for you**, gated to the pull request. | | Ito | Mabl | | --- | --- | --- | | Category | Agentic pre-merge QA | Low-code test automation platform | | Primary trigger | Every pull request, before merge | Scheduled runs & CI/CD jobs (typically post-merge) | | Testing philosophy | Shift-left - catches regressions on every PR, before merge | Shift-right - runs after merge via CI/CD or scheduled | | Who creates tests | The agent - autonomously | A person, in the low-code trainer | | Test authoring required | None (scriptless) | Low-code authoring | | Test maintenance | Agent adapts as the app changes | AI auto-healing on a maintained suite | | Primary audience | Engineering teams (developer-first) | QA teams & engineering | | Setup model | Connect repo, agent starts | Build a test suite first | | Output per run | QA report with video + screenshots, per PR | Test results, diagnostics, dashboards | | Breadth | Focused on behavioral PR validation | Broad: UI, API, performance, accessibility | | Pricing model | Per-seat, public ($40/seat/mo Pro) Per ito.ai pricing, 2026. | Quote-based / custom Mabl pricing is not published publicly; figures must be confirmed with Mabl. See Sources. | ## The core difference: when testing happens The most consequential difference between the two tools isn't a feature — it's *timing*. Mabl is designed as a cross-functional automation platform. Teams build a suite of tests and run it on a schedule, against a staging deploy, or as a step in a CI/CD pipeline. That model is powerful for broad regression coverage, but the feedback usually arrives **after** code has merged, when a fix means a new branch and another cycle. Ito is built around a single moment: the pull request. When a PR opens, the agent tests the change against the running application and posts a QA report back on the PR — before anyone hits merge. Bugs are caught at the exact point where they're cheapest to fix, inside the review the developer is already doing. ## Who writes the tests With Mabl, a person creates tests in the low-code trainer by recording and configuring flows, and the platform's AI helps keep them stable as selectors and layouts drift. It's faster than hand-writing Selenium, but the test suite is still an asset your team owns, grows, and curates. Ito removes that step. You connect your repository and the agent explores the running application to generate behavioral coverage on its own. There's no suite to build before you get value, and no backlog of test cases waiting to be written. As the product changes, the agent adapts its coverage rather than asking a human to re-record flows. ## Setup and developer workflow Mabl integrates with the major CI/CD platforms and lives alongside your pipeline, with results surfaced in its own dashboards and workspace. Onboarding centers on connecting environments and authoring your first set of journeys. Ito is GitHub-native and lives where developers already work. Setup is a one-click install on the repo; from there the agent picks up pull requests automatically and reports inline. There's no separate place to babysit — the QA report shows up on the PR next to the diff and the review comments. For an engineering team that lives in pull requests, that's the difference between adopting a new tool and adopting a new *workflow*. Ito slots into the one you already have. ## Maintenance and flaky tests Flaky, brittle tests are the tax every automation tool tries to reduce. Mabl's answer is auto-healing: when a selector changes, its AI attempts to repair the affected test so the suite keeps running. That meaningfully cuts maintenance versus traditional frameworks — but it's still maintenance *on a suite of test assets* that someone owns. Ito's answer is to not have a hand-built suite in the first place. Because the agent regenerates and adapts coverage against the live application on each run, there's no growing library of scripts to keep green. The maintenance burden that defines most QA programs largely moves off your team's plate. This is also where breadth cuts the other way. Mabl's maintained-suite model is exactly what you want when you need **deterministic, repeatable** coverage of specific certified flows, API contracts, or performance budgets. If those are core requirements, that's a point in Mabl's favor — see below. ## Pricing and cost of ownership Ito publishes its pricing: a per-seat Pro plan at $40/seat/month, with a custom Team tier.1 That transparency makes it easy to model cost as your team grows. Mabl uses a quote-based model; pricing isn't published publicly and depends on usage and plan.2 When you compare the two, weigh more than the sticker price: the *cost of ownership* of any automation platform includes the engineering time spent authoring and maintaining tests. Ito's model is designed to push that ongoing human cost toward zero, which is often the larger line item over a year than licensing. ## Which one should you choose? These tools optimize for different jobs. Here's the honest split. Many teams don't treat it as either/or at first. A common pattern is Ito as the fast, autonomous pre-merge gate on every pull request, with an existing Mabl suite covering scheduled regression runs and specialized API or performance checks — then consolidating as Ito's coverage compounds. ## Frequently asked questions ### Is Ito a Mabl alternative? Yes, for teams that want QA tied to the pull request. Both Ito and Mabl run real browser-based tests against your application. The difference is the model: Ito is an agentic, pre-merge QA platform that autonomously generates and runs tests on every PR, while Mabl is a broader low-code test-automation platform where teams build and maintain test suites that typically run on a schedule or in CI. ### What is the main difference between Ito and Mabl? Where the work goes. With Mabl, a person authors flows in a low-code trainer and the platform helps maintain them. With Ito, an agent explores your app and writes the behavioral tests itself, gated to each pull request before merge. Ito removes the test-authoring step; Mabl gives you a broader low-code platform with API, performance, and accessibility testing. ### Does Ito require writing or maintaining test scripts? No. Ito is scriptless. You connect your repository and the agent explores the running application to generate behavioral coverage, then adapts as the product changes. There is no test suite to build or keep green by hand. ### Can I use Ito and Mabl together? Some teams do. A common pattern is Ito as the fast, autonomous pre-merge gate on every PR, with an existing Mabl suite covering scheduled regression runs, cross-browser matrices, or API and performance checks. Over time, many teams consolidate as Ito coverage grows. ### Sources 1. [Ito — Pricing](https://www.ito.ai/pricing) (per-seat Pro tier, 2026) 2. [Mabl — Official site](https://www.mabl.com/) (features & positioning — verify before publish) 3. [Mabl — Pricing](https://www.mabl.com/pricing) (quote-based; confirm current terms) --- # Ito vs Cursor Bugbot (2026): Reviewing the Code You Just Generated In-editor review vs the merge gate ### Ito Runtime AI code review at the merge gate. Works with any editor or coding agent. ### Cursor Bugbot Static review inside Cursor. Catches logic errors as your agent writes them. ## Cursor Bugbot vs Ito at a glance | | ITO | CURSOR BUGBOT | | --- | --- | --- | | Where it runs | The pull request | Inside Cursor, and on PRs | | When it runs | At the merge gate | As you write, and at review | | Analysis type | Runtime execution | Static | | Executes your code | Yes | No | | Catches | Broken flows, runtime errors, dead integrations, regressions in files the PR never touched | Logic errors, edge cases, bugs in freshly generated code | | Evidence | Video, logs, repro steps, a failing test | Inline explanation and a suggested fix | | Requires your team uses Cursor | No. Any editor, any agent. | Yes | | Covers code written elsewhere | All of it. Human, Claude Code, Copilot, anything. | Strongest on code written in Cursor | | Pricing | $40/seat/mo, unlimited repos Per ito.ai pricing, August 2026. | Add-on to a Cursor subscription Read from cursor.com/pricing. Confirm current terms. | ## Why agentic coding broke code review for AI generated code Agentic coding didn't just change how code gets written. It broke the assumption review was built on. Code review was designed around a rate limit that no longer exists. One human could only open so many pull requests a week, so another human could plausibly read them. Teams running coding agents removed that limit on the writing side and kept it entirely on the reviewing side. You get a queue that grows faster than anyone can drain it, reviewed by people who know they can't read it all properly. Approval rates hold steady. Attention per PR collapses. Nobody says this in standup. Bugbot answers part of this well. Catching a bug in the editor seconds after generation is the cheapest place to catch anything, and staying inside Cursor means no context switch. If your team lives in Cursor, that loop is fast. It's still reading, though. And agent-written code has a specific signature. It's syntactically clean, plausibly structured, and confidently wrong about something it couldn't verify. An API shape that changed. An assumption about data it never saw. A flow it never walked. That code reads fine. It just doesn't work. ## Two layers, not two options ### Bugbot at authoring time: 1. Sees the code the moment your agent writes it 2. Flags logic errors and edge cases inline 3. No context switch. You're already in the editor. 4. Blind to anything only visible at runtime ### Ito at the merge gate: 1. Builds the app once the PR opens 2. Drives the flows your change touched 3. Catches regressions in files the PR never edited 4. Works no matter who or what wrote the code ## Six bugs an agent writes, and where each one gets caught Agent-written code fails in a specific way. It is clean, plausible, and confidently wrong about something it could not check. | | BUGBOT | ITO | | --- | --- | --- | | Off-by-one in generated pagination — Visible in the source the moment it is written. | Caught | Caught | | Unreachable branch in generated logic — Static analysis is good at this. | Caught | Missed | | Hallucinated method on a real library — Plausible name, does not exist. Sometimes caught in the editor, always caught at runtime. | Caught | Caught | | Assumed an API field that was renamed — The agent never saw the current response. | Missed | Caught | | Regression in a flow the PR never touched — Outside the diff entirely, so there is nothing to read. | Missed | Caught | | Code written outside Cursor — Claude Code, Copilot, a person. Bugbot is strongest on what it watched get written. | Missed | Caught | The overlap is real and useful. So is the gap. If you ship agent PRs daily with only one of these two, the missing layer is the one costing you. ## What this looks like across 43 real pull requests The case for a merge gate, in one team's numbers rather than in theory. DoltHub ran Ito across 43 pull requests over six weeks. Fixed versus dismissed is the number to watch, because at high PR volume a noisy tool does not just waste time. It trains your team to stop reading. Eighty-one percent were unique to runtime, which means authoring-time review had no line of sight to them. When agents are opening pull requests faster than anyone reads them, that is the share of risk moving straight through to main. ## A real pull request, in public, that you can go and read This is what the merge gate is for. One public pull request, four defects, none of them visible at the moment the code was written. Chatwoot is a widely used open-source customer engagement platform. In July 2026 a pull request touched their Freshdesk data import. Nothing here was caught at authoring time, because at authoring time there was nothing wrong to see. Import counts silently regressed under concurrency, Concurrent workers could push progress metrics backwards while cursors advanced. The numbers went down and nothing errored. Rate-limit handler ignored the provider's retry delay, Freshdesk returned a requested backoff. The code did not honor it. Placeholder inbox creation bypassed after_create callbacks — Working hours never initialized, because the callback that sets them never fired. Non-atomic claim on DataImportItem — Two workers could claim the same item, producing duplicate deliveries. Final run: 1 fixed, 14 passing. Safe to merge. The maintainer fixed all four, using row-level locking and monotonic cursor comparisons for the concurrency bug. Merged 5 August 2026. Link to PR: [https://github.com/chatwoot/chatwoot/pull/15261](https://github.com/chatwoot/chatwoot/pull/15261) None of these four are authoring-time bugs. They are integration-time and concurrency-time bugs, and they only exist once the system runs with more than one worker against a live provider. An in-editor reviewer watching the code get written has no vantage point on any of them. That is not a criticism of in-editor review. It is a description of where the two layers sit. ## When Cursor Bugbot is the better buy If you only get one, here's when it should be Bugbot. ### Why most teams end up with both These aren't substitutes, and the pricing doesn't force the choice that two seat-priced review bots would. Bugbot tightens the loop where code gets written. Ito holds the line where code gets merged. If you're shipping agent-written PRs daily with only one of the two, the missing one is costing you more than the one you have is saving. ## Frequently asked questions ### Do I need Cursor to use Ito? No. You connect a GitHub repository and it works regardless of which editor or agent wrote the code. Cursor, Claude Code, Copilot, or a person. ### Is Ito a Cursor Bugbot alternative? They overlap on intent, not method. Bugbot does static review, mostly inside Cursor. Your app gets built and run at the pull request. Plenty of teams use both. ### Does Bugbot run my application? No. Bugbot analyzes code statically. It doesn't build or execute your product, so runtime failures sit outside what it can see. ### Which is better for AI-generated code? They catch different failures in it. Bugbot is strong on logic errors visible in the source. Ito is strong on the signature failure of agent code: confidently written, structurally sound, and wrong about something it couldn't verify without running. ### How much does Cursor Bugbot cost? Cursor Bugbot pricing is structured as an add-on to a Cursor subscription, so your cost depends on your existing Cursor plan and seat count. Ito is $40 per seat per month with unlimited repos and does not require any other tool. Check both vendors for current pricing. ### What is Cursor Bugbot? Cursor Bugbot is Cursor's code review feature. It reviews the code your agent just generated, flags logic errors and edge cases inline, and also comments on pull requests. It performs static analysis, so it reads code without executing it. ### What are the alternatives to Cursor Bugbot? If you want review that works regardless of editor, CodeRabbit or Greptile cover static review across teams not standardized on Cursor. If you want the layer Bugbot cannot reach, Ito builds and runs your app at the pull request and catches failures that never appear in source. --- # Greptile Alternatives: Ito vs Greptile Compared Greptile indexes your whole repo to review a diff. Ito builds the app and runs it. ### Ito Runtime AI code review. Executes your application on every pull request. ### Greptile Static AI code review with whole-codebase indexing. Reviews the diff against a graph of your repo. ## Greptile vs Ito at a glance Both are good at their job. The jobs aren't the same. | | ITO | GREPTILE | | --- | --- | --- | | Analysis type | Runtime execution | Static, full-codebase context | | What it inspects | Your running application | The diff plus a graph of your whole repo | | Executes your code | Yes. Builds and runs the app. | No | | Core strength | Finds behavior nobody predicted | Understands ripple effects across files | | Catches | Broken flows, runtime errors, dead integrations, migration and auth failures | Cross-file logic errors, missed call sites, convention breaches | | Evidence in the PR | Video, console and network logs, repro steps, a failing test | Inline comments with codebase-aware reasoning | | What it learns | How your app behaves, by running it | How your codebase is written, by indexing it | | Test scripts required | None. It's scriptless. | N/A. It doesn't test. | | Platforms | GitHub. GitLab on the roadmap. | GitHub, GitLab | | Pricing | $40/seat/mo, unlimited repos Per ito.ai pricing, August 2026. | From $30/dev/mo Read from the vendor's public pricing page, August 2026. Verify before relying on either figure. | ## Context is not the same thing as consequence Greptile solves a real problem most review tools ignore. It just isn't the problem that causes most incidents. Diff-only review has tunnel vision. The tool sees 20 changed lines and none of the code depending on them. Greptile fixes that by indexing your repository and reasoning across it. Rename a method, miss a call site in a service you've never opened, and Greptile is the tool most likely to catch it. On a large monorepo that's worth paying for. There's a ceiling, though, and it isn't about model quality. Everything Greptile knows, it knows from source. Your payment provider's sandbox behavior isn't in your repository. Neither is the shape of your production data. Neither is queue latency under load, or how your session cookie survives a redirect, or the fact that a dependency quietly changed a default two minor versions ago. Those aren't edge cases. That's most of what breaks after a clean review. And none of it is knowable from code, however completely you index it. ## One pull request, two tools A change to discount logic in checkout. ### Greptile: 1. Indexes the repo, finds every caller of the discount function 2. Spots a second call site in billing still on the old signature 3. Flags error handling that doesn't match the rest of the module 4. Can't tell you whether checkout still completes ### Ito: 1. Builds the app, opens a real browser 2. Adds an item, applies the discount, goes to checkout 3. Total renders fine. Payment step now 500s. 4. Posts the video, the stack trace, and a test that reproduces it ## Six real bugs, and which tool sees them Greptile has the best whole-codebase reasoning in the category. Watch where that stops helping. | | GREPTILE | ITO | | --- | --- | --- | | Missed call site in an untouched service — A signature changed and one caller three files away still uses the old one. | Caught | Missed | | Error handling that breaks module convention — Correct code that contradicts how the rest of the codebase does it. | Caught | Missed | | Checkout renders but payment 500s — Every file is internally consistent. The running system is not. | Missed | Caught | | Third-party sandbox changed its response — The contract lives outside your repository, so indexing cannot reach it. | Missed | Caught | | Queue backs up under real load — Latency is a property of the running system, not of the source. | Missed | Caught | | Dependency changed a default two minors ago — Your code is unchanged. Its behavior is not. | Missed | Caught | The first two are exactly why teams buy Greptile, and Ito will not catch either. The last four are unknowable from source at any indexing depth. That is the honest division of labor between the two. ## The failure, on video A static reviewer tells you a line looks wrong. This shows you the product not working. Ito records the session. You watch the checkout fail rather than reading a description of it. ## What this looks like across 43 real pull requests Worth reading with codebase indexing in mind, because it puts a number on what indexing cannot reach. DoltHub ran Ito across 43 pull requests over six weeks. The ratio that matters is fixed versus dismissed, since a tool producing findings nobody acts on is a tool your team will quietly mute by week three. - 2:1 Fixed to dismissed, across 43 PRs - 81% Of bugs found were unique to runtime - 50% Fewer regressions reaching production Eighty-one percent of those bugs existed only at runtime. Greptile's whole-codebase context genuinely widens what static review can see, and it is the best implementation of that idea available. It does not move this number, because the constraint is the method rather than the depth of the index. "Nothing in the code diff in the PR that Ito reviewed had any direct sign of this bug. If Ito had only been looking at the diff lines from the PR, then it wouldn't have caught this." — Jason Fulghum, CEO at DoltHub ## A real pull request, in public, that you can go and read Worth reading closely if you are weighing full-codebase indexing against execution, because this is a case where indexing everything would still have come up short. Chatwoot is a widely used open-source customer engagement platform. In July 2026 a pull request touched their Freshdesk data import. Every file involved was internally consistent. A tool with a complete graph of the repository would have found nothing to flag. - Import counts silently regressed under concurrency — Concurrent workers could push progress metrics backwards while cursors advanced. The numbers went down and nothing errored. - Rate-limit handler ignored the provider's retry delay — Freshdesk returned a requested backoff. The code did not honor it. - Placeholder inbox creation bypassed after_create callbacks — Working hours never initialized, because the callback that sets them never fired. - Non-atomic claim on DataImportItem — Two workers could claim the same item, producing duplicate deliveries. Final run: 1 fixed, 14 passing. Safe to merge. The maintainer fixed all four, using row-level locking and monotonic cursor comparisons for the concurrency bug. Merged 5 August 2026. Link to PR: [https://github.com/chatwoot/chatwoot/pull/15261](https://github.com/chatwoot/chatwoot/pull/15261) Now apply full-codebase context to that list. Indexing every file tells you who calls what. It does not tell you that two workers raced, that a callback silently failed to fire, or that Freshdesk changed the backoff it asks for. Three of the four live outside the repository entirely. That is the ceiling on static analysis, and it is a property of the method rather than of how well any tool implements it. ## When Greptile is the better buy There are real cases where Greptile should be your first purchase. Greptile makes static review as good as static review gets. For some teams that's enough. If your incidents look like 'we didn't realize that would affect X,' buy Greptile. If they look like 'it passed review and broke anyway,' reading the code harder was never the fix. ## The main Greptile alternatives, compared briefly If you are shortlisting Greptile alternatives rather than only comparing two tools, here is the honest short version of the field. **CodeRabbit**: The cheapest broad option and the widest platform support. It covers GitHub, GitLab, Bitbucket and Azure DevOps, has a real free tier, and is free on open-source repositories. Less whole-codebase context than Greptile, which is the trade you are making. **Cursor Bugbot**: Review inside the editor rather than on the pull request. The fastest feedback of any option here if your team already works in Cursor, and much weaker on code written anywhere else. **Qodo**: Reviews the pull request and generates tests for the change. Reasonable if your underlying problem is a thin test suite rather than review quality, though generated tests need maintenance like any other. **Ito**: The one option here that is not a like-for-like swap, because it builds and runs your application instead of reading it. Most teams add it alongside a static reviewer rather than replacing one, since the two catch different things. **SonarQube and Semgrep**: Where you land when the requirement turns out to be governance or security rules you own, rather than conversational review. ## Frequently asked questions ### Is Ito a Greptile alternative? Partly. Both post on pull requests and both try to catch bugs before merge, so they compete for the same slot. But Greptile analyzes source with full-repository context and Ito executes your application. Teams with budget for one pick based on whether their bugs are context failures or behavioral failures. ### What does Greptile do that Ito doesn't? Greptile reasons across your entire codebase. It can see that a change breaks an assumption in a file nobody touched, or contradicts a convention used elsewhere. The diff is read only to decide which of your flows to exercise. It isn't doing full-codebase static reasoning. ### What does Ito do that Greptile doesn't? Your application gets built and run in an isolated container, then driven in a real browser. That surfaces runtime errors, broken user journeys, failed integrations and migration problems. None of those are visible in source, no matter how much context you index. ### Can I run Greptile and Ito on the same repo? Yes. They post different signals to the same pull request and don't conflict. Greptile comments on your code. Ito reports on your running app. ### Which is better for a monorepo? Greptile has the clearer edge on pure code comprehension in a large monorepo. Ito's value there depends on whether the services can be built and exercised, which is usually a config question rather than a blocker. ### How does Greptile pricing compare to Ito? Greptile pricing starts around $30 per developer per month. Ito is $40 per seat per month with unlimited repositories. Check both pricing pages before you decide. This category changes pricing often. ### What is Greptile? Greptile is an AI code review tool that indexes your entire repository and reviews each pull request against a graph of the whole codebase. That gives it unusually strong cross-file context, so it can flag a change that breaks an assumption in a file the pull request never touched. It performs static analysis and does not run your application. ### What are the best Greptile alternatives? It depends what you want Greptile to do. For broad multi-platform coverage at lower cost, CodeRabbit. For in-editor feedback on Cursor teams, Cursor Bugbot. For catching regressions that only appear when the app runs, Ito, which is a runtime tool rather than another static reviewer and is usually added alongside rather than instead. --- # Ito vs Qodo: Generating Tests vs Running the App Qodo writes tests for your changed code. Ito runs your app and tests the flows. Both target the same gap. Here's which one fits which failure mode. ### Ito Builds and runs your application on the pull request, drives the affected flows, reports what broke. ### Qodo Reviews the diff and generates test cases for the changed code. Formerly CodiumAI. ## What each one does on a pull request Qodo analyses the diff, reviews it conversationally, and generates test cases for the changed code. You get review comments plus a set of tests you can commit. The tests then live in your repository and run in CI forever. Ito reads the diff to work out which user flows the change affects, builds the application in an isolated container, drives those flows in a browser, and reports what broke with video and logs. Nothing is added to your repository. Both are trying to close the same gap: the change looked fine, nobody knows if it works. ## The case for generating tests It is a good case and worth stating without hedging. **The tests persist.** Run Qodo once, commit the tests, and they protect that code path for as long as it exists. Ito re-runs on every pull request and produces nothing durable. If you value accumulating a safety net over time, generation is the model that does that. **They run everywhere.** Committed tests run in any CI, on any machine, offline, in a fork. There is no dependency on a vendor being up. **Coverage on code without a runnable surface.** A parser, a serialisation library, a pure calculation module. These have no user flow to drive. Generated unit tests are the only automated option that reaches them, and Ito has very little to offer here. **It is cheaper to start.** Qodo has a free tier. There is no environment to configure and no container to make work. If your gap is "we have no tests," generation addresses that directly and nothing else on this page does. ## The case against generating tests Two problems, one well known and one not. ### The known one: maintenance Generated tests are tests. They break when the code changes, they need updating, and the maintenance cost is the same as hand-written tests. The World Quality Report has roughly half of QA leaders naming AI-generated test maintenance as their biggest challenge, which is a striking finding for a technology sold as a labour saver. Generating a thousand tests is fast. Owning a thousand tests is not. ### The less obvious one: generated tests encode current behaviour, including bugs A test generator reads what the function does and asserts that it does that. If the function has an off-by-one, the generated test asserts the off-by-one result and passes forever. You now have a green suite that actively defends a defect, which is worse than having no test, because the green is load-bearing in someone's decision to merge. A human writing a test starts from an idea of what the code should do. Generation starts from what it does. That distinction sounds academic until you are looking at a passing suite and a production incident. There is one case where this is a feature rather than a bug: characterisation tests before a refactor, where you explicitly want to freeze existing behaviour so you can detect any change. Qodo is genuinely good for that. ## Four shapes that get through both static review and generated tests | Bug | Generated test | Running the app | | --- | --- | --- | | Two workers claim the same row — Each operation individually synchronised, the sequence not. No unit test exercises two workers. | Missed | Caught | | A callback that never fires — The object was created through a path that skips it. The unit test creates it through the normal path. | Missed | Caught | | A provider renamed a response field — Your code is unchanged. Your mocked test still passes, because the mock still returns the old shape. | Missed | Caught | | A migration that locks under load — Takes an ACCESS EXCLUSIVE lock. Runs in 40ms against 200 local rows, holds for minutes against 40 million. | Missed | Caught | The third one is worth dwelling on, because it is the specific failure mode of tested-but-mocked code. The more thoroughly you mock, the more confidently you test your own assumptions rather than reality. ## The case against running the application Our side of it, stated properly. ### Do not buy Ito if any of these are true **Your app cannot build unattended.** Most applications almost do. The gap is usually a migration expecting seeded data, an environment variable that only exists in someone's shell, or a service assuming a sibling is up. Budget a day, sometimes a week. **You want something durable.** Stop paying and you have nothing. Qodo leaves you with tests. **You need editor-speed feedback.** Building and driving an app is minutes. A unit test is milliseconds. **You ship a library or SDK.** No runnable surface, essentially nothing to exercise. Qodo is the right tool and we are not. **You are not on GitHub.** Qodo covers more platforms. ## Side by side | | Ito | Qodo | | --- | --- | --- | | Approach | Builds and runs the application | Reviews the diff and generates tests | | Leaves artifacts in your repo | No | Yes, test files | | Works without a runnable app | No | Yes | | Catches concurrency and integration failures | Yes | Only if a test happens to exercise them | | Ongoing maintenance burden | None | Generated tests need upkeep like any test | | Feedback speed | Minutes | Seconds once written | | Setup cost | App must build in a container | Minimal | | Free tier | Trial | Yes | | Platforms | GitHub | GitHub, GitLab, Bitbucket | | Pricing | $40/seat/mo, unlimited repos | Free tier, paid plans | ## Which to buy Buy both if you have both problems, which is common. They do not overlap much and they are not competing for the same seat in the pipeline. The question that separates them: when something broke last quarter, would a unit test have caught it? If yes, your gap is coverage and Qodo addresses it. If a unit test would have passed while the system was broken, more tests will not help. ## Frequently asked questions ### Is Qodo the same as CodiumAI? Yes. Qodo is the current name; the product was previously CodiumAI. Older comparisons and documentation use the old name. ### Does Qodo replace writing tests? It replaces writing the first draft. The tests still need reviewing, because a generated test asserts what the code currently does rather than what it should do, and it still needs maintaining when the code changes. ### Can I use Ito and Qodo together? Yes, and they are unusually complementary. Qodo builds unit coverage and Ito verifies behaviour at the pull request. They post different signals and do not conflict. ### Does Ito generate tests I can keep? It attaches a failing test to a finding so you can reproduce it, but it does not build a suite you own and maintain. If accumulating durable coverage is the goal, that is Qodo's model rather than ours. ### Which is cheaper? Qodo, particularly at the free tier. Ito is $40 per seat per month with unlimited repositories. The comparison changes if you are weighing either against maintaining an E2E suite by hand, which is the cost both are actually displacing. ### Does Qodo run my application? No. It performs static analysis and generates tests. The tests run in your CI; Qodo itself does not execute your product. --- # Ito vs Claude Code Review: How Each Reviews a PR Both run on every pull request. Claude Code Review reads your code and leaves inline comments. Ito runs your application and reports what actually broke. Here is how they differ, and why most teams want both. ### Ito Runs your app and tests behavior on every PR. ### Claude Code Review Analyzes the diff using static analysis. ## Ito vs Claude Code Review at a glance Both tools show up on the pull request before merge. Both use agents. The similarity ends there. Claude Code Review is a **static analyzer** - an agent that reads your diff in the context of the codebase and leaves high-signal inline comments. Ito is a **runtime analyzer** - it spins up your application, exercises the change the way a user would, and reports what actually broke. | | Ito | Claude Code Review | | --- | --- | --- | | Category | Agentic pre-merge QA | Multi-agent code review | | What it inspects | The running application | The source diff + codebase context | | Primary trigger | Every pull request, before merge | Every pull request (or on demand) | | Output on the PR | QA report with video + screenshots | Inline review comments + severity ranking | | What it catches | Behavioral regressions, broken flows, UI failures | Logic errors, security issues, edge cases in code | | Runs the application? | Yes. Real application against a live build | No. Analyzes code without executing it | | Who writes the checks | The agent from the diff, your test instructions, and organization preferences | Specialized review agents (configurable via CLAUDE.md / REVIEW.md) | | Primary audience | Engineering teams (developer-first) | Engineering teams on Claude Team / Enterprise | | Setup model | Connect repo, agent starts | Enable in Claude Code settings + GitHub App | | Pricing model | Per-seat ($40/seat/mo Pro) Per ito.ai pricing, 2026. | Token usage (~$15–25 avg per PR) (scales with PR size) Per Anthropic; research preview on Team/Enterprise. Confirm current terms. | ## The core difference: reading code vs. running it The most consequential difference is *what each tool looks at*. Claude Code Review dispatches specialized agents against your pull request. They analyze the diff in the context of your full codebase, hunt for logic errors, security vulnerabilities, broken edge cases, and subtle regressions, then verify candidates to cut false positives. Findings land as inline comments on the exact lines, with severity ranking and a summary overview. It does **not** approve PRs; that is still a human call. Average review time is around 20 minutes. Ito is built around a different question: *does this change still work when a user tries it?* When a PR opens, the agent boots the application, explores the affected flows in a real sandbox/browser, and posts a report back on the PR, along with evidence like video and logs, before anyone hits merge. Bugs are caught at the moment they are cheapest to fix. ## How each reviews a PR ### Claude Code Review on a PR Once enabled for a repo, Claude Code Review typically runs when a PR opens (admins can also trigger on every push or only on demand via @claude review). Multiple agents analyze the change in parallel on Anthropic infrastructure. Results are deduplicated, ranked by severity, and posted as inline comments on the specific lines where issues were found, with a summary in the review body. Teams can steer review focus with **CLAUDE.md** or **REVIEW.md** instructions — useful for enforcing internal conventions. What you get is a deeper, slower pass than a lightweight bot comment: Anthropic positions it for depth over speed, and bills on token usage (often averaging $15–25 per PR depending on size). ### Ito on a PR Ito is GitHub-native. After a one-click install on the repo, the agent picks up pull requests automatically. It **exercises the change** against a running build, exploring flows and validating behavior the way a careful QA engineer would. The output is a full runtime analysis report on the PR: what was tested, what failed, and evidence (video + screenshots) so the author can reproduce the issue without guessing. There is no separate suite to author first - the agent generates and adapts behavioral coverage as the product changes. ## What each one catches (and misses) Claude Code Review is strong at the things a sharp senior engineer notices while reading: incorrect conditionals, insecure patterns, misuse of APIs, regressions that are visible in the diff if you know the surrounding code. It will not tell you that the checkout button now fails for logged-out users because a CSS z-index change buried it, or that an API contract still type-checks while the staging UI returns empty states. Ito is strong at the opposite class of failure: flows that break only when the app runs. Auth redirects, form validation edge cases, broken navigation after a refactor, integrations that return unexpected data. It will not replace a deep read of a security-sensitive diff - that is still a job for humans and code-review agents. ## Setup and developer workflow Claude Code Review lives inside the Claude Code / Anthropic surface: an admin enables it for Team or Enterprise, installs the GitHub App, and selects repositories. Developers then get automatic reviews (or on-demand via comments). Customization is file-based, where project instructions steer what agents look for. Ito is also GitHub-native and lives where developers already work. Setup is a repo install; from there the agent picks up PRs and reports inline. There is no separate place to babysit, the QA report shows up next to the diff and the review comments. For an engineering team that lives in pull requests, the ideal stack often looks like: Claude (or another code-review agent) for *reading* the change, and Ito for *running* it. ## Pricing and cost of ownership Ito publishes its pricing: a per-seat Pro plan at $40/seat/month, with a custom Team tier. That transparency makes it easy to model cost as your team grows. Claude Code Review is billed on token usage and generally averages $15–25 per pull request, scaling with PR size and complexity. It is available as a research preview on Claude Team and Enterprise plans. When you compare them, weigh more than sticker price. Claude Code Review cost scales with how much code you push and how deep you want reviews. Ito cost of ownership is designed around removing the ongoing human burden of building and maintaining a behavioral test suite, often the larger line item over a year than licensing. ## Which one should you choose? These tools optimize for different jobs. Here is the honest split. Many teams do not treat it as either/or. A common pattern is Claude Code Review (or another AI reviewer) for code-level findings, with Ito as the autonomous pre-merge QA gate that proves the change behaves correctly in a real browser — then consolidating tools as coverage compounds. ## Frequently asked questions ### Is Ito an alternative to Claude Code Review? No, they address different failure modes. Claude Code Review analyzes your pull request source for logic errors, security issues, and regressions in code. Ito runs your application and catches behavioral bugs that only appear at runtime. Most teams that ship quickly benefit from both. ### What is the main difference between Ito and Claude Code Review? What each one inspects. Claude Code Review reads the diff and codebase context, then posts inline review comments. Ito exercises the running application in a real browser and posts a QA report with video and screenshots. One is a code reviewer; the other is a QA agent. ### Does Claude Code Review run my application? No. Claude Code Review analyzes code without executing your app. If you need runtime validation of user flows, that is Ito's job. ### Can I use Ito and Claude Code Review together? Yes, and that is often the best setup. Use Claude Code Review for deep code-level findings on the diff, and Ito as the autonomous pre-merge QA gate that proves behavior still works. They post different signals on the same PR. ### Does Ito require writing or maintaining test scripts? No. Ito is scriptless. You connect your repository and the agent explores the running application to generate behavioral coverage, then adapts as the product changes. There is no test suite to build or keep green by hand. ### Sources 1. [Ito — Pricing](https://www.ito.ai/pricing) (per-seat Pro tier, 2026) 2. [Anthropic — Code Review for Claude Code](https://claude.com/blog/code-review) (product announcement) 3. [Claude Code docs — Code Review](https://code.claude.com/docs/en/code-review) (setup, triggers, behavior — verify before publish) --- # Ito vs CodeRabbit: Reading the Diff vs Running the Code Both show up on every pull request. CodeRabbit reads the diff and leaves review comments. Ito boots the app, runs through the change, and reports what failed. Same PR, different jobs. ### Ito Runs your app and tests behavior on every PR. ### CodeRabbit Analyzes the diff and posts inline findings with suggested fixes. ## Ito vs CodeRabbit at a glance CodeRabbit and Ito both attach to the pull request before merge. CodeRabbit is a static analysis reviewer: it summarizes the change, flags issues in the source, and suggests fixes. Ito is runtime reviewer: it starts the application, exercises the affected flows in a real environment, and reports what broke with logs, video and screenshots. | | Ito | CodeRabbit | | --- | --- | --- | | Category | Runtime analysis | Static analysis | | What it inspects | The running application | The source diff | | Primary trigger | Every commit | Every commit | | Output on the PR | Report with runtime evidence (logs, video), reproduction steps, and inline comments | Summary, inline comments, suggested fixes | | What it catches | Runtime errors, behavioral regressions, broken flows, broken integrations | Logic bugs, security issues, style and best-practice problems | | Runs the application? | Yes: real sandbox and browser against a live build | No: analyzes code without executing the product | | Extra surfaces | GitHub-native PR reports | PR reviews plus IDE and CLI reviews | | Primary audience | Engineering teams (developer-first) | Engineering teams that want automated review coverage | | Setup model | Connect repo, agent starts | Connect repo, agent starts | | Pricing model | Per PR author ($40/seat/mo Pro) Per ito.ai pricing, 2026. | Per PR author ($24/dev/mo Pro annual) (or $30 monthly) Per CodeRabbit public pricing, 2026. Free tier and open-source plans available. Confirm current terms. | ## The core difference: reading code vs running it CodeRabbit works on the text of the change. After you connect a repository, it watches for new pull requests and commits, builds context from the diff and surrounding code, and posts a review: a summary of what changed, inline comments on specific lines, and often a suggested fix you can apply in one click. Ito asks a different question on the same PR: does this change still work when someone uses the product? When a PR opens, Ito boots the application, explores the affected flows, and posts a QA report before merge. The evidence is a recording of what happened, not a comment on a line of source. ## How each reviews a PR ### CodeRabbit on a PR On a new pull request, CodeRabbit runs a full review. On later commits, it usually reviews the new changes again so the thread stays current. The output includes a plain-language walkthrough, severity-tagged findings, and suggested patches. Teams can steer behavior with rules and style guides, and they can reply to comments so later reviews learn from that feedback. CodeRabbit also ships reviews outside the PR: an IDE extension and a CLI for feedback before you push. That is useful when the goal is faster code review, not runtime proof. ### Ito on a PR Ito installs on the GitHub repo and picks up pull requests automatically. It does not scan the diff for suspicious patterns. It **exercises the change** against a running build: clicking through flows, checking behavior, and attaching evidence when something fails. The QA report lands on the PR next to the review comments. Authors get what was tested, what failed, and video or screenshots they can replay. There is no behavioral suite to author first. The agent generates and adapts coverage as the product changes. ## What each one catches (and misses) CodeRabbit is strong at issues visible in source: bad conditionals, insecure patterns, API misuse, missing edge cases, style drift from team rules. It will not tell you that checkout fails for logged-out users because a CSS change buried the button, or that a typed API response leaves the UI empty in staging. Ito is strong at runtime failures: broken auth redirects, form validation that only fails with real data, navigation that dies after a refactor, integrations that return unexpected payloads. It will not replace a careful read of a security-sensitive diff. That job still belongs to humans and code-review tools. ## Setup and developer workflow CodeRabbit connects to GitHub, GitLab, Bitbucket, or Azure DevOps. After install, reviews start on new PRs with little configuration. Teams that want more control add path filters, custom rules, and learned preferences from comment replies. Ito is GitHub-native. Setup is a repo install. From there the agent attaches to PRs and reports inline. There is no separate dashboard your team has to babysit for day-to-day use. The QA report shows up beside the diff. A common stack looks like this: CodeRabbit for reading the change, Ito for running it. One tool covers code-level findings. The other covers behavioral regressions. ## Pricing and cost of ownership Ito publishes a per-seat Pro plan at $40/seat/month, plus a custom Team tier. That makes it easy to model cost as headcount grows. CodeRabbit bills per developer who creates pull requests. Public pricing lists a free tier for summarization and IDE reviews, Pro at about $24/developer/month on annual billing ($30 monthly), and free Pro features for public open-source repos. Confirm current terms on CodeRabbit pricing before you budget. Cost of ownership is wider than license price. CodeRabbit reduces time humans spend on first-pass review. Ito reduces time spent building and maintaining a behavioral test suite. Those are different line items, and both can matter more over a year than the seat fee. ## Which one should you choose? These tools optimize for different jobs. Plenty of teams run both. CodeRabbit handles code-level findings on the diff. Ito is the pre-merge QA gate that proves behavior in a real browser. The signals sit next to each other on the same pull request. ## Frequently asked questions ### Is Ito an alternative to CodeRabbit? Only if you treat them as the same job. CodeRabbit reviews source for bugs, security issues, and style problems. Ito runs the application and catches behavioral failures. Most teams that move quickly benefit from both. ### What is the main difference between Ito and CodeRabbit? What each inspects. CodeRabbit reads the diff and posts review comments, often with suggested fixes. Ito exercises the running app in a real browser and posts a QA report with video and screenshots. ### Does CodeRabbit run my application? No. CodeRabbit analyzes code without executing your product. If you need runtime validation of user flows, that is Ito's job. ### Can I use Ito and CodeRabbit together? Yes. A common setup is CodeRabbit for code-level findings and Ito as the autonomous pre-merge runtime analysis gate. They post different signals on the same PR. ### Does Ito require writing or maintaining test scripts? No. Ito is scriptless. Connect the repository and the agent explores the running application to generate behavioral coverage, then adapts as the product changes. ### Sources 1. [Ito. Pricing](https://www.ito.ai/pricing) (per-seat Pro tier, 2026) 2. [CodeRabbit docs. Code review overview](https://docs.coderabbit.ai/guides/code-review-overview) (how reviews work) 3. [CodeRabbit. Pricing](https://www.coderabbit.ai/pricing) (per-seat Pro and free tiers. verify before publish) --- # Terms of Service Effective Date: March 2, 2026 ### Introduction This Terms of Service is a contract entered into by and between You (“you” or “User”) and Demox Labs, Inc. (“Demox Labs,” “We,” or “us”), the company that develops and operates Ito.ai. These terms and conditions (together with our Privacy Policy, these “Terms of Service” or “Terms”) govern your access to and use of [https://www.ito.ai](https://www.ito.ai/) (our “Website”), our web application (our “App”), and any software, application, content, functionality, and services (collectively, the “Services”) offered through Ito.ai, whether as a guest or registered user. Please read these Terms of Service carefully before you start to use or access our Services. By using our Services, you accept and agree to be bound and abide by these Terms. If you are not eligible or do not agree to these Terms of Service, then you do not have permission to use the Service and you must not access or use our Services. ARBITRATION NOTICE. Except for certain kinds of disputes described in Section 21, you agree that disputes arising under these Terms will be resolved by binding, individual arbitration, and BY ACCEPTING THESE TERMS, YOU AND DEMOX LABS ARE EACH WAIVING THE RIGHT TO A TRIAL BY JURY OR TO PARTICIPATE IN ANY CLASS ACTION OR REPRESENTATIVE PROCEEDING. Your ability to use or access the Services is dependent on third parties, such as GitHub. You acknowledge and agree that your ability to access and use the Services is governed by the terms of these third parties, and those terms may change at their discretion. ### 1. Age Restrictions The Website and Services are intended for users 13 and older. By accessing or using the Services, You represent and warrant that you are at least thirteen (13) years old and that you possess the legal right and ability to enter into this Terms of Service and to use the Services in accordance with these Terms. ### 2. Changes to Terms of Service We may revise and update these Terms of Service from time to time in our sole discretion by posting a revised version on the Website. All changes are effective immediately when we post them. Demox Labs may provide notice of any material changes, at our sole discretion, by posting the updated Terms of Service on the Website. Any revisions to the Terms of Service will take effect on the noted Effective Date, located at the top of these Terms. ### 3. About Ito.ai Ito.ai is an AI-driven quality assurance automation platform developed and operated by Demox Labs, Inc. The Ito.ai Service automatically provisions isolated development environments, executes end-to-end tests on pull requests, and provides feedback with visual evidence including screenshots and video recordings. Throughout these Terms, references to “Ito.ai” refer to the product and Services; references to “Demox Labs” or “we” refer to Demox Labs, Inc. as the legal entity. ### 4. User Accounts You may register for an Ito.ai user account (“Account”). In order to register an Account, you will be required to connect to the Services via your account with GitHub (“Third-Party Account”). By connecting a Third-Party Account to the Services, you authorize Demox Labs to access your Third-Party Account, including your profile information, the primary email address associated with your Third-Party Account, profile information from the organization of which you are a part (such as organization name, description), and information about the GitHub repositories to which you have access. You control the scope of the authority granted to Demox Labs to the extent permitted by GitHub. By providing Demox Labs access to your Third-Party Account, you authorize Demox Labs to act on your behalf to retrieve information from the applicable Third-Party Account for purposes of providing the Services under these Terms. You are responsible for maintaining the security and confidentiality of your Account information. You agree that you are solely responsible for any and all losses incurred by us or any other user or visitor to the Services due to someone else using your Account as a result of your failing to keep your account information secure and confidential. You represent and warrant that you have all necessary rights, consents, authorizations and permissions to grant Demox Labs access to your Third-Party Account, including for the purposes described in these Terms, without any breach by you of any of the terms and conditions that govern your agreement with the applicable Third-Party Account provider, and without subjecting Demox Labs to any payment obligations, usage limitations or other liabilities. Authorized Users You are responsible and liable for all uses of the Services resulting from access provided by you, directly or indirectly, whether such access or use is permitted by or in violation of this Agreement. Without limiting the generality of the foregoing, you are responsible for all acts and omissions of anyone authorized to access or use the Services on your behalf (“Authorized User(s)”), and any act or omission by an Authorized User that would constitute a breach of this Agreement if taken by you will be deemed a breach of this Agreement by you. You shall use reasonable efforts to make all Authorized Users aware of this Agreement’s provisions as applicable to such Authorized User’s use of the Services and shall cause Authorized Users to comply with such provisions. ### 5. Prohibited Uses You may use our Services only for lawful purposes and in accordance with these Terms of Service. You agree not to use the Services: - In any way that violates any applicable federal, state, local, or international law or regulation (including, without limitation, any laws regarding the export of data or software to and from the US or other countries). - For the purpose of exploiting, harming, or attempting to exploit or harm minors in any way by exposing them to inappropriate content, asking for personally identifiable information, or otherwise. - To send, knowingly receive, upload, download, use, or re-use any material that does not comply with these Terms of Service. - To transmit, or procure the sending of, any advertising or promotional material, including any “junk mail,” “chain letter,” “spam,” or any other similar solicitation. - To impersonate or attempt to impersonate Ito, an Ito employee, another user, or any other person or entity. - To violate, encourage others to violate, or provide instructions on how to violate, any right of a third party, including by infringing or misappropriating any third-party intellectual property right. - To engage in any other conduct that restricts or inhibits anyone's use or enjoyment of the Services, or which, as determined by us, may harm Ito or users of the Services, or expose them to liability. - To submit repositories or code containing malicious software, viruses, Trojan horses, worms, logic bombs, cryptocurrency miners, or other material that is malicious or technologically harmful. - To attempt to gain unauthorized access to, interfere with, damage, or disrupt any parts of the Services, the server on which the Services is stored, or any server, computer, or database connected to the Services. - To attack the Services via a denial-of-service attack or a distributed denial-of-service attack. ### 6. Code Execution and Environment Provisioning You acknowledge that the Services execute code from your repositories in isolated development container environments hosted on Demox Labs’ infrastructure. This includes execution of Dockerfiles, devcontainer configurations, package installation scripts, build scripts, and other code necessary to provision and run your application. You represent and warrant that you have all necessary rights to execute such code and that the code does not contain malicious components. You agree to indemnify Demox Labs for any damages arising from the execution of code you submit to the Services. ### 7. LLM Providers The Ito.ai Services use artificial intelligence powered by third-party large language model providers, which may include OpenAI, Anthropic, and other providers. Demox Labs reserves the right to add, remove, or change LLM providers at any time without notice. When using our Services, you agree to abide by the usage policies and terms of service of our LLM providers, and you agree not to use Ito.ai in any way that is prohibited by these model providers, including: - Illegal activity. - Child Sexual Abuse Material or any content that exploits or harms children. - Generation of hateful, harassing, or violent content. - Generation of malware. - Activity that has high risk of physical harm. - Activity that has high risk of economic harm. - Fraudulent or deceptive activity. - Activity that violates people's privacy. - High risk government decision-making. ### 8. Free Trial Demox Labs may offer a free trial period of seven (7) days (“Free Trial”) to new users who have not previously registered for an Account. The Free Trial begins on the date you register for an Account and expires at the end of the seventh calendar day. No credit card is required to begin the Free Trial, but you must provide valid payment information before or at the time the Free Trial expires in order to continue accessing the Services. At the end of the Free Trial, your account will automatically transition to a paid Subscription unless you cancel before the Free Trial expires. Demox Labs reserves the right to modify or terminate the Free Trial offering at any time without notice. Free Trials are available only once per person or organization; creating duplicate accounts to obtain additional Free Trial periods is a violation of these Terms. Users who were previously enrolled in a Beta program are not eligible for a Free Trial unless expressly authorized by Demox Labs in writing. ### 9. Service Availability Commercially Reasonable Efforts Demox Labs will use commercially reasonable efforts to make the Ito.ai Services available on a continuous basis. However, Demox Labs does not guarantee any specific uptime percentage or service level. The Services may be temporarily unavailable due to scheduled maintenance, unscheduled outages, third-party service failures, or other factors within or outside of Demox Labs’ control. Demox Labs will use commercially reasonable efforts to provide advance notice of scheduled maintenance where practicable. Force Majeure Demox Labs shall not be liable for any delay or failure to perform its obligations under these Terms to the extent that such delay or failure is caused by circumstances beyond Demox Labs’ reasonable control, including without limitation: acts of God, natural disasters, pandemic or epidemic, war, terrorism, riots, civil unrest, government action or regulation, labor disputes, power outages, internet or telecommunications failures, denial-of-service attacks, failures of third-party infrastructure or services (including cloud providers and LLM providers), or any other cause beyond Demox Labs’ reasonable control (“Force Majeure Event”). In the event of a Force Majeure Event, Demox Labs’ obligations under these Terms shall be suspended for the duration of the event, and Demox Labs will use commercially reasonable efforts to resume performance as soon as practicable. ### 10. Terms of Service Violations and Termination Term These Terms are effective beginning when you accept the Terms or first access or use the Service, and ending when terminated as described in the Section titled “Termination” below. Termination Any violation of these Terms of Service shall result in immediate account termination without prior warning to you and without refund applied to your Account. Additionally, any violation of these restrictions may further subject you to liability for violation of Demox Labs’ intellectual property rights and further claims and damages. We may choose to suspend or terminate your Account or ability to access or use the Services at any time, for any or no reason, at our sole discretion, and without notice or liability of any kind. You agree that any violation by you of these Terms of Service will constitute an unlawful and unfair business practice, and will cause irreparable harm to us, for which monetary damages would be inadequate; and you consent to our obtaining any injunctive or equitable relief that we deem necessary or appropriate in such circumstances. These remedies are in addition to any other remedies we may have at law or in equity. You may terminate these Terms at any time and for any reason by deleting your Account and discontinuing your use of all Services. You may delete your Account by accessing your Account settings on the Site or by contacting us at: support@ito.ai. Usage Limits and Service Suspension Demox Labs may establish usage limits for the Services, including without limitation limits on the number of automated test runs, pull requests processed, compute hours consumed, or other resource utilization metrics (“Usage Limits”). Usage Limits applicable to your subscription tier will be posted at https://www.ito.ai/pricing or communicated to you directly. If your usage exceeds applicable Usage Limits, Demox Labs reserves the right to: (a) throttle or suspend your access to the Services, with or without prior notice; (b) require you to upgrade to a higher subscription tier; or (c) charge overage fees at then-current rates. In cases where Demox Labs determines, in its sole discretion, that your usage poses risk to the stability, security, or availability of the Services for other users, Demox Labs may suspend your access immediately without prior notice and without liability. Demox Labs will use commercially reasonable efforts to restore suspended access promptly upon resolution of the underlying issue or upon your upgrade to an appropriate subscription tier. Effect of Termination Upon termination of these Terms: (a) your license rights will terminate and you must immediately cease all use of the Service; (b) you will no longer be authorized to access your account or the Service; (c) you must pay Demox Labs any unpaid amount that was due prior to termination; and (d) all payment obligations accrued prior to termination and those provisions which by their nature are intended to survive any termination or expiration of these Terms will survive. You are solely responsible for retaining copies of any User Content you upload to the Service since upon termination of your account, you may lose access rights to any User Content you uploaded to the Service. If your account has been terminated for a breach of these Terms, then you are prohibited from creating a new account on the Service using a different name, email address or other forms of account verification. ### 11. Privacy Policy Your use of the Services may involve the transmission of your personal information to us. Our policies regarding the collection, use, disclosure, and protection of such personal information are governed according to our Privacy Policy, as made available at https://www.ito.ai/privacy-policy (the “Privacy Policy”). Please read the Ito.ai Privacy Policy carefully. The Privacy Policy is incorporated by this reference into, and made a part of, these Terms. ### 12. Intellectual Property Rights The Services and their entire contents, features, and functionality (including but not limited to the Website, App, software, applications, text, displays, images, video, and audio, and the design, selection, and arrangement thereof) are owned by Demox Labs, its licensors, or other providers of such material and are protected by United States and international copyright, trademark, patent, trade secret, and other intellectual property or proprietary rights laws. Demox Labs hereby grants you a non-exclusive, non-transferable, non-sublicensable, limited, revocable license to access and use the Service solely for your internal business operations as set forth in these Terms of Service and expressly conditioned upon your Account remaining active, in good standing, and in full compliance with these Terms of Service. You must not reproduce, distribute, modify, create derivative works of, publicly display, publicly perform, republish, download, store, or transmit any of the material on our Services except as permitted by this Terms of Service. Further you must not use the Services to create or in relation to any product or service that competes with the Services. The Demox Labs intellectual property rights, including name, copyrights, patents, trade secrets, logo, trademarks and all related intellectual property rights are property of Demox Labs or its affiliates or licensors. You must not use such property without the prior written permission of Demox Labs. If you choose to provide input and suggestions regarding existing functionalities, problems with or proposed modifications or improvements to the Service (“Feedback”), then you hereby grant Demox Labs an unrestricted, perpetual, irrevocable, non-exclusive, fully-paid, royalty-free right and license to exploit the Feedback in any manner and for any purpose, including to improve the Service and create other products and services. ### 13. User Content “User Content” refers to any information, data, or content that you upload, post, input, or submit while using the Service, such as submitting code repositories for testing or adding content to your profile. Demox Labs treats your User Content as confidential information. Demox Labs is not obligated to back up your User Content, and it may be deleted without notice. It is your responsibility to create and maintain backup copies of User Content if you want to. Ito.ai will use your User Content to generate test results, feedback, suggestions, and other outputs (“Output”). Subject to your compliance with these Terms, Demox Labs hereby assigns to you all of its rights, title and interest (if any) in and to the Output resulting from your use of the Service. This assignment is limited to intellectual property rights in the Output and does not constitute any representation or warranty regarding the accuracy, completeness, or fitness of the Output for any purpose. In particular, the assignment of Output does not imply that the Output is free from errors, false positives, false negatives, or omissions. Your use of Output is subject to the disclaimers set forth in Section 18 (Warranty Disclaimer). The Service may provide the same or similar Output to others, and Demox Labs’ assignment to you does not apply to outputs resulting from other users’ use of the Service. You agree that Demox Labs may use Output to (a) provide, maintain, protect and improve the Services; (b) comply with applicable law; and (c) enforce these Terms. You are solely responsible for your use of any Outputs. By providing User Content via the Service, you affirm, represent, and warrant to Demox Labs that: - You are the creator and owner of the User Content, or have the necessary licenses, rights, consents, and permissions to authorize Demox Labs to use your User Content as necessary to exercise the licenses granted by you in the manner contemplated by these Terms; - Your User Content and use of your User Content as contemplated by these Terms does not and will not: (a) infringe, violate, misappropriate, or otherwise breach any third-party right, including any copyright, trademark, patent, trade secret, moral right, privacy right, right of publicity, or any other intellectual property, contract, or proprietary right; (b) slander, defame, libel, or invade the right of privacy, publicity or other proprietary rights of any other person; or (c) cause Demox Labs to violate any law or regulation; and - Your User Content could not be deemed by a reasonable person to be objectionable, profane, indecent, pornographic, harassing, threatening, embarrassing, hateful, or otherwise inappropriate. If you become aware of the service being used for illegal purposes, you should notify Demox Labs at support@ito.ai. ### 14. Code Handling and Privacy Your proprietary code remains confidential with Demox Labs. Pull request diffs and contextual information may be shared with our LLM providers solely for the purpose of generating test results for your use. Demox Labs does not use your raw proprietary code to train AI models. However, Demox Labs may use aggregated, anonymized, and de-identified data derived from test results, performance metrics, and usage patterns to train, evaluate, and improve AI models and the Services generally. Such aggregated data will not contain code identifiable to you or your organization. Demox Labs cannot control or warrant the data handling practices of third-party LLM providers beyond the contractual commitments those providers make to Demox Labs. You should review the applicable terms of our LLM providers before using the Services. Test artifacts, including screenshots, video recordings, and logs generated by the Service, are retained to support your ability to review results and debug issues. You may submit a request for deletion of test artifacts at any time by contacting support@ito.ai. We will review each request and respond within a reasonable timeframe. Demox Labs may use aggregated and anonymized test results and performance metrics to improve the Service. Demox Labs may also use publicly available open-source project code to train and improve our systems. ### 15. Confidential Information From time to time during the Term of this Agreement, You, or Demox Labs (“Disclosing Party”) may disclose or make available to the other party (“Receiving Party”), information about Disclosing Party or Disclosing Party’s affiliates’ business affairs, products, confidential intellectual property, trade secrets, financial information, third-party confidential information, and other sensitive or proprietary information, whether disclosed or accessed in written, electronic, or any other form or media, that is identified as confidential at the time of disclosure or should be reasonably known by Receiving Party to be confidential or proprietary due to the nature of the information disclosed and the circumstances surrounding the disclosure. The Receiving Party shall maintain in confidence all confidential and proprietary information and shall not disclose confidential or proprietary information to any person or entity, except to the employees, agents, or subcontractors who have a legitimate need to know, to perform their obligations hereunder and who are required to protect the confidential or proprietary information in a manner no less stringent than required under this Agreement. ‍ Confidential and proprietary information does not include information that: (a) is or becomes publicly known through no fault of Demox Labs, our service providers, or service integration providers, or their representatives; (b) already rightfully known to the Receiving Party at the time of disclosure; (c) rightfully obtained and on a non-confidential basis from a third party without breach of any confidentiality obligation; or (d) independently developed by or on behalf of the Receiving Party without access to or use of any confidential and proprietary information of the Disclosing Party. ‍ Notwithstanding the foregoing, Demox Labs, our service providers, or service integration providers, or their representatives may be required to disclose your confidential and proprietary information (a) to comply with the order of a court or other governmental body, or as otherwise necessary to comply with applicable law, with written notice to you and a reasonable effort to obtain a protective order where permitted; or (b) to establish Demox Labs’ rights under this Agreement, including to make required court filings. ‍ Demox Labs’ software, applications, scripts, code, plug-ins and technology incorporated in the Services, the design and layout of the Ito.ai platform user interface, all pricing information relating to the Services, and the terms and conditions of this Agreement shall be deemed confidential information of Demox Labs without any marking or further designation. ### 16. Purchases and Subscriptions Subscription Plans Demox Labs offers the following standard subscription plans: - Monthly Plan: $150.00 USD per month, billed on a recurring monthly basis. - Annual Plan: $127.50 USD per month ($1,530.00 USD billed annually), representing a 15% discount from the Monthly Plan rate. Annual Plans are billed in full at the start of each annual Billing Cycle and are non-refundable except as expressly stated in these Terms. All Subscriptions automatically renew at the end of each Billing Cycle at the then-current rate for your plan unless cancelled before the renewal date. Demox Labs reserves the right to modify pricing at any time; any price change will take effect at the start of your next Billing Cycle. Order Forms and Custom Agreements Customers who enter into a direct sales agreement or order form with Demox Labs (“Order Form”) will be subject to the pricing, seat counts, contract length, and other commercial terms specified in that Order Form. In the event of any conflict between an Order Form and these Terms of Service, the Order Form shall govern solely with respect to the commercial terms expressly addressed therein. All other provisions of these Terms of Service remain in full force and effect. Order Forms are not transferable and apply only to the organization identified therein. Customers operating under an Order Form are not eligible for the standard Free Trial described in Section 8 unless expressly stated in the Order Form. Subscriptions All Subscriptions are payable in accordance with payment terms in effect at the time the subscription becomes payable. Payment can be made by credit card, debit card, or other means that we may make available. If you purchase a Subscription, you will be billed on a recurring and periodic basis (“Billing Cycle”). Billing Cycles are set on a monthly or annual basis, depending on your Subscription choice. You agree that by purchasing a Subscription, the Subscription will automatically renew, and you will be charged until you cancel. Fees and Payment If you purchase the Services (including on a Subscription-basis), you agree to pay Demox Labs the applicable fees and taxes in U.S. Dollars. Failure to pay these fees and taxes will result in the termination of your access to the paid Services, including any Subscription Services. Our pricing is set out on https://www.ito.ai/pricing. Fee Changes Demox Labs, in its sole discretion and at any time, may modify Subscription fees. Any Subscription fee change will become effective at the end of the then-current Billing Cycle. Your continued use of the Service after a Subscription fee change comes into effect constitutes your agreement to pay the modified Subscription fee amount. Refunds and Subscription Cancellations Subscriptions can be cancelled at any time before the next subscription cycle, and the cancellation will take effect from the next subscription cycle. You may cancel your Subscription through our web app by accessing subscription management, or by contacting Demox Labs at support@ito.ai. Refunds may be issued at Demox Labs’ discretion. ### 17. Third-Party Materials The Services may display, incorporate, permit access to or make available content, data, information, applications, systems, materials and other resources of third parties (“Third Party Materials”) or provide links to certain third-party websites. By using the Services, you acknowledge and agree that Demox Labs is not responsible for examining or evaluating the content, accuracy, completeness, availability, timeliness, validity, copyright compliance, legality, decency, quality, security or any other aspect of such Third-Party Materials or third-party websites. Third Party Materials and third-party websites are not under Demox Labs’ control, and, to the fullest extent permitted by law, Demox Labs is not responsible to you for any Third Party Materials, third-party websites, or third-party services. ### 18. Warranty Disclaimer WE PROVIDE THE SERVICES “AS IS” AND WITHOUT ANY REPRESENTATION OR WARRANTY, EXPRESS, IMPLIED, OR STATUTORY. WE SPECIFICALLY DISCLAIM ANY IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. WITHOUT LIMITING THE FOREGOING, WE DO NOT REPRESENT OR WARRANT THAT THE SERVICES WILL BE ACCESSIBLE WITHOUT INTERRUPTION OR THAT THE SERVICES, ANY CONTENT FROM THE WEBSITE, OR THE SERVER THAT MAKES THE SERVICES AVAILABLE ARE FREE FROM ERRORS, DEFECTS, DESIGN FLAWS, OMISSIONS, VIRUSES, OR OTHER HARMFUL COMPONENTS. YOUR USE OF THE SERVICES IS AT YOUR OWN RISK. DEMOX LABS USES ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING TO PROVIDE THE ITO.AI SERVICE. YOU ACKNOWLEDGE AND AGREE THAT THE TECHNOLOGY USED TO PROVIDE THE SERVICE IS EXPERIMENTAL, RAPIDLY EVOLVING, AND SUBJECT TO UNEXPECTED OUTPUTS AND RESULTS. THE SERVICES MAY PROVIDE RESULTS THAT CONTAIN ERRORS, OMISSIONS, OR DO NOT ACCURATELY REFLECT THE STATE OF YOUR CODE OR APPLICATION. THE SERVICES MAY FAIL TO DETECT BUGS, SECURITY VULNERABILITIES, OR OTHER ISSUES IN YOUR CODE. THE ASSIGNMENT OF OUTPUT TO YOU UNDER SECTION 13 IS LIMITED TO INTELLECTUAL PROPERTY RIGHTS AND DOES NOT CONSTITUTE ANY REPRESENTATION OR WARRANTY REGARDING THE ACCURACY, COMPLETENESS, QUALITY, OR FITNESS OF THE OUTPUT FOR ANY PURPOSE. DEMOX LABS WILL NOT BE LIABLE FOR ANY MISTAKES, INACCURACIES, OMISSIONS, FALSE NEGATIVES, OR FALSE POSITIVES IN ANY OUTPUT GENERATED BY THE SERVICE. YOU RELY UPON ANY OUTPUT AT YOUR SOLE RISK. THE LIMITATIONS, EXCLUSIONS AND DISCLAIMERS IN THIS SECTION 18 APPLY TO THE FULLEST EXTENT PERMITTED BY LAW. SOME STATES DO NOT ALLOW THE DISCLAIMER OF IMPLIED WARRANTIES, IN WHICH CASE PORTIONS OF THIS DISCLAIMER MAY NOT APPLY TO YOU. ### 19. Limitation of Liability TO THE FULLEST EXTENT PERMITTED BY LAW, IN NO EVENT WILL DEMOX LABS BE LIABLE, WHETHER IN CONTRACT, WARRANTY, TORT (INCLUDING NEGLIGENCE, WHETHER ACTIVE, PASSIVE, OR IMPUTED), PRODUCT LIABILITY, STRICT LIABILITY, OR OTHER THEORY, TO YOU OR ANY OTHER PERSON FOR ANY INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE, OR CONSEQUENTIAL DAMAGES, ARISING OUT OF OR IN CONNECTION WITH ANY USE OF, THE INABILITY TO USE, OR THE RESULTS OF USE OF THE SERVICES, EVEN IF WE HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. ‍ TO THE FULLEST EXTENT PERMITTED BY LAW, DEMOX LABS’ TOTAL LIABILITY TO YOU FOR ALL DAMAGES, LOSSES OR CAUSES OF ACTION ARISING OUT OF OR RELATING TO THE USE OF OR INABILITY TO USE ANY PORTION OF THE SERVICE OR OTHERWISE UNDER THESE TERMS, WHETHER IN CONTRACT, TORT, OR OTHERWISE, SHALL NOT EXCEED THE GREATER OF (a) THE AMOUNT YOU HAVE PAID DEMOX LABS IN THE SIX (6) MONTHS PRECEDING THE CLAIM, OR (b) ONE HUNDRED DOLLARS ($100). SOME JURISDICTIONS DO NOT ALLOW THE EXCLUSION OR LIMITATION OF LIABILITY FOR CONSEQUENTIAL OR INCIDENTAL DAMAGES, IN WHICH CASE PORTIONS OF THIS LIMITATION MAY NOT APPLY TO YOU. ‍ Claims arising out of or related to these Terms or the Services shall be subject to the statutes of limitations provided under applicable California law. ### 20. Indemnification You agree to indemnify and hold Demox Labs, its parents, subsidiaries, affiliates, any related companies, suppliers, licensors and partners, and the officers, directors, employees, agents and representatives of each of them harmless, including costs, liabilities and legal fees, from any claim or demand made by any third party due to or arising out of (i) your access to or use of the Services, (ii) any violation of these Terms of Service (including negligent or wrongful conduct) by you, (iii) the infringement by you, or any third party using your account, of any intellectual property or other right of any person or entity, (iv) your User Content, including any code you submit to the Services that causes damage to Demox Labs’ infrastructure or third parties, or (v) any damages arising from the execution of code from your repositories on Demox Labs’ infrastructure. ### 21. Marketing Demox Labs may publicly refer to Customer as a customer of Ito.ai, including on the Ito.ai website and in sales presentations, and may use Customer’s logo for such purposes. Similarly, Customer may publicly refer to itself as a customer of Ito.ai, including on Customer’s website. 22. Dispute Resolution and Arbitration Generally Except as described in the Sections titled “Exceptions” and “Opt-Out” below, you and Demox Labs agree that every dispute arising in connection with these Terms, the Service, or communications from us will be resolved through binding arbitration. Arbitration uses a neutral arbitrator instead of a judge or jury, is less formal than a court proceeding, may allow for more limited discovery than in court, and is subject to very limited review by courts. This agreement to arbitrate disputes includes all claims whether based in contract, tort, statute, fraud, misrepresentation, or any other legal theory, and regardless of whether a claim arises during or after the termination of these Terms. YOU UNDERSTAND AND AGREE THAT, BY ENTERING INTO THESE TERMS, YOU AND DEMOX LABS ARE EACH WAIVING THE RIGHT TO A TRIAL BY JURY OR TO PARTICIPATE IN A CLASS ACTION. Exceptions Although we are agreeing to arbitrate most disputes between us, nothing in these Terms will be deemed to waive, preclude, or otherwise limit the right of either party to: (a) bring an individual action in small claims court; (b) pursue an enforcement action through the applicable federal, state, or local agency if that action is available; (c) seek injunctive relief in a court of law in aid of arbitration; or (d) file suit in a court of law to address an intellectual property infringement claim. Opt-Out If you do not wish to resolve disputes by binding arbitration, you may opt out of the provisions of this Section 22 within 30 days after the date that you agree to these Terms by sending an email to support@ito.ai with the subject line “Arbitration Opt-Out”. Your opt-out notice must include your full legal name and the email address associated with your Account. Once Demox Labs receives your opt-out notice, this Section 22 will be void and any action arising out of these Terms will be resolved as set forth in Section 23. The remaining provisions of these Terms will not be affected by your opt-out notice. Arbitrator This arbitration agreement, and any arbitration between us, is subject to the Federal Arbitration Act and will be administered by JAMS under the rules applicable to consumer disputes (collectively, “JAMS Rules”) as modified by these Terms. The JAMS Rules and filing forms are available online at www.jamsadr.com, by calling JAMS at +1-800-352-5267, or by contacting Demox Labs. No Class Actions YOU AND DEMOX LABS AGREE THAT EACH MAY BRING CLAIMS AGAINST THE OTHER ONLY IN YOUR OR ITS INDIVIDUAL CAPACITY AND NOT AS A PLAINTIFF OR CLASS MEMBER IN ANY PURPORTED CLASS OR REPRESENTATIVE PROCEEDING. Further, unless both you and Demox Labs agree otherwise, the arbitrator may not consolidate more than one person’s claims, and may not otherwise preside over any form of a representative or class proceeding. Enforceability If the Section titled “No Class Actions” or the entirety of this Section 22 is found to be unenforceable, or if Demox Labs receives an opt-out notice from you, then the entirety of this Section 22 will be null and void and, in that case, the exclusive jurisdiction and venue described in Section 23 will govern any action arising out of or related to these Terms. ### 23. Governing Law These Terms of Service and any claim arising out of these Terms will be governed by and construed in accordance with the laws of the State of California, without regard to its conflict of law provisions. Any disputes not subject to arbitration under Section 22 shall be resolved in the state or federal courts located in Contra Costa County, California, and you hereby consent to personal jurisdiction in such courts. ### 24. Survival After this Terms of Service terminates, the terms of this agreement that expressly or by their nature contemplate performance after termination or expiration will survive and continue in full force and effect. Without limiting any other provisions of the Terms of Service, the termination of these Terms for any reason will not release you from any obligations incurred prior to termination of the Terms or that thereafter may accrue in respect of any act or omission prior to such termination. ### 25. Assignability You may not assign the Terms of Service, or any of its rights or obligations hereunder, without Demox Labs’ prior written consent in the form of a written instrument signed by a duly authorized representative of Demox Labs. Demox Labs may freely assign this Terms of Service without your consent. Any attempted assignment or transfer in violation of this subsection will be null and void. ### 26. Waiver and Severability No waiver by Demox Labs of any term or condition set out in these Terms of Service shall be deemed a further or continuing waiver of such term or condition or a waiver of any other term or condition, and any failure of Demox Labs to assert a right or provision under these Terms of Service shall not constitute a waiver of such right or provision. If any provision of these Terms of Service is held by a court or other tribunal of competent jurisdiction to be invalid, illegal, or unenforceable for any reason, such provision shall be eliminated or limited to the minimum extent such that the remaining provisions of the Terms of Service will continue in full force and effect. ### 27. Access of the Site Outside the United States Given the global nature of the Internet, you agree to comply with all local rules, including, without limitation, rules about the Internet, data, email, privacy, copyright and trademark infringement. The Service is intended for visitors located within the United States. Demox Labs makes no representation that the Service is appropriate or available for use outside of the United States. Access to the Service from countries or territories or by individuals where such access is illegal is prohibited. ### 28. Consent to Use Electronic Records In connection with the Terms of Service, you may be entitled to receive certain records from Demox Labs or our affiliates, such as contracts, notices, and communications, in writing. To facilitate your use of the Services, you give us permission to provide these records to you electronically instead of in paper form. ### 29. Entire Agreement This Terms of Service and the Ito.ai Privacy Policy (available at https://www.ito.ai/privacy-policy) constitute the sole and entire agreement between you and Demox Labs and supersede all prior and contemporaneous understandings, agreements, representations, and warranties, both written and oral, to the extent they relate in any way to the Services. Where a customer has entered into a signed Order Form with Demox Labs, the Order Form and these Terms of Service together constitute the entire agreement with respect to the subject matter therein, and the Order Form shall control in the event of any conflict as set forth in Section 16. ### 30. Contact Information For questions or concerns related to these Terms, please contact us at: support@ito.ai Demox Labs, Inc. 1547 Palos Verdes Mall, #124 Walnut Creek, CA 94597 ### 31. Miscellaneous Communications We may send you emails concerning our products and services, as well as those of third parties. You may opt out of promotional emails by following the unsubscribe instructions in the promotional email itself. Modification of the Service Demox Labs reserves the right to modify or discontinue all or any portion of the Service at any time (including by limiting or discontinuing certain features of the Service), temporarily or permanently, without notice to you. Demox Labs will have no liability for any change to the Service, including any paid-for functionalities of the Service, or any suspension or termination of your access to or use of the Service. Notice to California Residents If you are a California resident, then under California Civil Code Section 1789.3, you may contact the Complaint Assistance Unit of the Division of Consumer Services of the California Department of Consumer Affairs in writing at 1625 N. Market Blvd., Suite N 112, Sacramento, California 95834, or by telephone at +1-800-952-5210 in order to resolve a complaint regarding the Service or to receive further information regarding use of the Service. No Support We are under no obligation to provide support for the Service. In instances where we may offer support, the support will be subject to published policies. --- # Privacy Policy Last Modified: March 2, 2026 ### 1. Introduction Demox Labs, Inc. (“Demox Labs,” “we,” “our,” or “us”) is the company that develops and operates Ito.ai. Demox Labs respects your privacy and is committed to protecting it through our compliance with this Privacy Policy (“Policy”). A core element of our mission is our commitment to protect your personal information and to be transparent about the data we collect about you, how it is used, and with whom it is shared. Throughout this Policy, references to “Ito.ai” refer to the product and Services; references to “Demox Labs” or “we” refer to Demox Labs, Inc. as the legal entity. This Policy describes our practices for collecting, using, maintaining, protecting, and disclosing your information through https://www.ito.ai (our “Website”), our applications (our “Apps”) and our products, services, technology platforms and related applications (collectively, the “Service(s)”). Please read this Policy carefully to understand our policies and practices regarding your information and how we will treat it. If you do not agree with our terms, your choice is not to use our Services. By accessing or using our Services, you agree to this Privacy Policy. This Policy may change from time to time (see Changes to This Policy). Your continued access of our Services after we make changes is deemed to be acceptance of those changes, so please check the Last Modified date at the top of this Policy to ensure that you are viewing the most current version. ### 2. Information We Collect About You and How We Collect It We collect only the minimum amount of information needed to provide you with our Services. For example, we collect basic contact information when you sign up for an account. We do not ask you for any information if you choose not to register for an account; however, you will not be able to use our Services without registering for an account. Other types of information we collect relate to how you use our Website or Apps, which helps us improve our Services. Information You Provide to Us - Personal Information: In the course of registering an account on our Services, we collect information that identifies you as a specific individual and can be used to contact or identify you (“Personal Information”). Examples of Personal Information we collect for these purposes include your name and email address. - Payment Information: We may process your payment information, such as credit card, billing address, and other financial information necessary to purchase or otherwise use our Services. Please note that we do not collect or store your Payment Information. Rather, your Payment Information is collected and stored by our payment processor, Stripe. By submitting your Payment Information, you consent to our providing your Payment Information to Stripe as reasonably necessary to support and process your transactions as well as your credit card issuer and banking institution. Stripe’s privacy policy is available at https://stripe.com/privacy. - User Contributions: You may have the ability to interact with parts of our Website or with us through third-party platforms, such as the ability to post on third-party forums or to submit tickets on our Website. Your feedback or posts may be published or displayed publicly or transmitted to third parties (collectively, “User Contributions”). Your User Contributions are posted on and transmitted to others at your own risk. We cannot control the actions of other users of the Website or third-party platforms with whom you may choose to share your User Contributions. Information Collected Automatically As you navigate through and interact with our Website, we and our third-party service providers, including analytics and third-party content providers, may automatically collect certain information from you whenever you access or interact with the Service. - Usage Information: Details of your visits to our Website, including which links you clicked on, content response times, location data, logs, and other similar communication data and statistics about your interactions. - Device Information: Information about your computer and internet connection, including your Internet Protocol address, operating system, and browser type. - Non-Identifying Information: We may collect non-identifying or non-personal information when you use our Website, such as zip codes, demographic data, time zone, publicly available data, and general information regarding your use of the Service. We may combine this automatically collected log information with other information we collect about you. We do this to improve services we offer you, analytics, and site functionality. Cookies and Other Automatic Data Collection Technologies Demox Labs and its partners use cookies or similar technologies, which store certain information on your computer and allow us to, among other things, analyze trends, administer the Website, and gather demographic information about our user base as a whole. The technology used to collect information automatically from Ito.ai users may include the following: - Cookies: Like many websites, we and our operational partners, affiliates, analytics, and service providers use “cookies” to collect information. A cookie is a small data file that we transfer to your computer’s hard disk for record-keeping purposes. We use both persistent cookies that remain on your computer or similar device (such as to save your registration ID and login password for future logins to the Service) and session ID cookies, which expire at the end of your browser session. You can control the use of cookies at the individual browser level, but if you choose to disable cookies, it may limit your use of certain features or functionality of the Service. - Web Beacons: We and our operational partners, affiliates, analytics, and service providers may also employ software technology known as “web beacons” and/or “tracking tags” to help us keep track of what content on our Service is effective. Web beacons are small graphics with a unique identifier that may be invisible to you and are used to track the online activity of Internet users. - Embedded Scripts: We and our operational partners, affiliates, analytics, and service providers may also employ software technology known as an Embedded Script. An Embedded Script is programming code that is designed to collect information about your interactions with the Service, such as the links you click on. The code is temporarily downloaded onto your computer or other device and is deactivated or deleted when you disconnect from the Service. Analytics Services We use Google Analytics and PostHog to help us understand how users interact with our Services. These services collect information about your use of our Website and Services, including your IP address, browser type, pages visited, and time spent on pages. For more information about how Google collects and processes data, please visit https://policies.google.com/privacy. For information about PostHog’s privacy practices, please visit https://posthog.com/privacy. Information Received from Third Parties We may receive information about you from third parties. For example, we and our partners, affiliates, and service providers may use a variety of other technologies (such as tags) that collect statistical data relating to your Website activity for security and fraud detection purposes. You may choose to elect that certain third parties share information with us, for example, when you choose to access the Services through another service, such as through Single Sign-on (e.g., GitHub). GitHub Details: We collect data necessary to enable your Ito.ai account to interface with your GitHub account, such as your GitHub token, your user organization, your GitHub username, and your user role. We integrate with GitHub’s API and Open Authentication (OAuth) system. We do not have access to your GitHub credentials. GitHub’s privacy policy is available at https://docs.github.com/en/site-policy/privacy-policies/github-privacy-statement. ### 3. Code and Repository Data Ito.ai accesses your code repositories to provide automated QA testing services. This section describes how Demox Labs handles your code and related data. Code Access and Processing When you connect a repository to Ito.ai, Demox Labs accesses your code solely to provision development environments and execute tests. We do not use your source code for any purpose beyond active test execution and do not intentionally retain copies of your source code beyond what is necessary for that purpose. Data Sent to LLM Providers To generate test cases and analyze code changes, Demox Labs sends pull request diffs and contextual information to our large language model (LLM) providers, which may include OpenAI, Anthropic, and others. We send only PR diffs and context, not your complete codebase. Demox Labs does not use your raw proprietary code to train AI models. Demox Labs cannot control or warrant the independent data handling practices of our LLM providers beyond the contractual commitments those providers make to Demox Labs. You should review the applicable terms and privacy policies of our LLM providers. OpenAI’s privacy policy is available at https://openai.com/privacy. Anthropic’s privacy policy is available at https://www.anthropic.com/privacy. Test Artifacts Ito.ai generates test artifacts including screenshots, video recordings, and logs during test execution. These artifacts are stored in Amazon Web Services (AWS) S3 to allow you to review test results and debug issues. You may submit a request for deletion of test artifacts at any time by contacting support@ito.ai. We will review each request and respond within a reasonable timeframe. Development Environments Ito.ai provisions isolated development container environments on Demox Labs’ AWS infrastructure to execute tests. These environments are transient and are terminated after test execution completes. Aggregated and Anonymized Data By using the Services, you agree that Demox Labs may collect and use aggregated, anonymized, and de-identified data derived from your use of the Services to train, evaluate, and improve our AI models and overall service quality. This aggregated data may include: - Test pass/fail patterns and statistics - UI interaction sequences and navigation patterns - Test execution timing and performance data - Error and failure classification data This data is processed to remove any information that could identify you, your organization, or your specific codebase. Aggregated data will not contain code identifiable to you or your organization. Open Source Code Demox Labs may use publicly available open-source project code to train and improve our systems. The data handling practices described in this section regarding proprietary code do not apply to open-source projects that are publicly available. ### 4. Do Not Track Signal Do Not Track (DNT) is a privacy preference that users can set in some web browsers, allowing users to opt out of tracking by websites and online services. We do not currently respond to DNT signals. However, we do not track users across third-party websites to provide targeted advertising. ### 5. Children's Privacy The Services are general audience and intended for users 13 and older. Demox Labs does not knowingly collect Personal Information from anyone younger than age 13. If we learn that we have collected Personal Information from a child under 13, we will delete that information as quickly as possible. If you believe that we might have any information from a child under 13, please contact us at support@ito.ai. 6. How We Use Your Information Demox Labs may use information that we collect about you or that you provide to us, including any Personal Information: - To provide you with an Account. - To provision development environments, execute automated tests, and provide QA feedback on your pull requests. - To deliver, provide, and process payment for the Services. - To improve our Services, including improving the Ito.ai test runner agent and AI models using aggregated and anonymized data as described in Section 3. - To address your inquiries and provide customer support. - To tailor content we display to you and offers we may present to you, both on the Service and elsewhere online. - To communicate with you, and to promote products, services, offers, and events offered by Demox Labs. - To comply with legal requirements and assist law enforcement. - To stop any activity we may consider to be, or to pose a risk of being, illegal, fraudulent, unethical, or legally actionable. - To identify Ito.ai users. - For the purposes disclosed at the time you provide your information. - As otherwise permitted with your consent. ### 7. Legal Bases To process your information as described above, we rely on the following legal bases: Contractual Necessity: To provide you with the Ito.ai Service and perform the contract that you have with us. Legitimate Interests: It is in our legitimate interests to improve and analyze our Service, promote our products, prevent fraudulent transactions, maintain security of our Services, and provide functionality. Consent: In instances where we have indicated we will ask for consent within this Privacy Policy, you may withdraw your consent at any time by contacting us at support@ito.ai. ### 8. How We Disclose and Share Your Information Demox Labs does not sell personal information to third parties. We share information we receive about you as follows: With Our Service Providers We employ third-party companies to provide Services on our behalf, to perform Service-related operations (e.g., maintenance services, database management, web analytics, server hosting, fraud detection, and improvement of Ito.ai’s features) or to assist us in providing and analyzing how our Service is used. These third parties may have access to your Personal Information in order to perform these tasks on our behalf. - Stripe for payment processing. Stripe’s privacy policy: https://stripe.com/privacy - Amazon Web Services (AWS) for cloud infrastructure and storage. AWS privacy policy: https://aws.amazon.com/privacy/ - Google Analytics for website analytics. Google’s privacy policy: https://policies.google.com/privacy - PostHog for product analytics. PostHog’s privacy policy: https://posthog.com/privacy For Our Service Integrations We integrate with the following services to provide you with the best possible functionality: - GitHub for repository access and pull request integration. GitHub's privacy policy: https://docs.github.com/en/site-policy/privacy-policies/github-privacy-statement - OpenAI for AI-powered test generation. OpenAI's privacy policy: https://openai.com/privacy - Anthropic for AI-powered test generation. Anthropic's privacy policy: https://www.anthropic.com/privacy Demox Labs does not use your raw proprietary code to train AI models. However, Demox Labs may use aggregated, anonymized, and de-identified data derived from test results and usage patterns to improve our AI models and Services as described in Section 3. Demox Labs cannot control or warrant the independent data handling practices of OpenAI or Anthropic beyond the contractual commitments those providers make to Demox Labs. You should review the applicable privacy policies of those providers directly. For Corporate Transactions Demox Labs may share information, including Personal Information, with any current or future subsidiaries or affiliates, primarily for business and operational purposes, in connection with a merger, acquisition, reorganization, or sale of assets (including, in each case, as part of the due-diligence process with any potential acquiring entity) or in the event of bankruptcy. If Required By Law Demox Labs will disclose information about you to government or law enforcement officials or private parties as we, in our sole discretion, believe necessary or appropriate to respond to claims and legal process (including but not limited to subpoenas), or at the request of governmental authorities or other third parties conducting an investigation where we determine in our sole discretion the disclosure is necessary to (a) protect the property and rights of Demox Labs or a third party, (b) protect the safety of the public or any person, or (c) prevent or stop activity we may consider to be, or pose a risk of being, illegal, fraudulent, unethical, or legally actionable activity. With Your Consent You may submit Personal Information to us through a form on the Website and consent to receive communication from us or our business affiliates based on the information in the form. ### 9. Third-Party Websites and Links Our Services may contain links and/or features to other websites and/or online platforms operated by third parties. We do not control such other online platforms and are not responsible for their content, their privacy policies, or their use of your information. ### 10. Your Rights and Choices Regarding Your Information You have several ways to exercise control over your information: Account Settings: You may submit requests regarding your personal information by accessing your Account settings in our App or contacting us at support@ito.ai. We will review each request and respond within a reasonable timeframe. Contact Us: You may submit requests to access, update, or delete your personal information by contacting us at support@ito.ai. We will review each request and respond within a reasonable timeframe. Please note that certain data may be retained as required by law or as necessary to fulfill our contractual obligations to you. Email: You may opt out of receiving marketing emails from us by following the opt-out instructions provided in those emails. Please note that we reserve the right to send you certain communications relating to your account or use of the Service (for example, administrative and service announcements) via email and other means, and these transactional account messages may be unaffected if you opt out from receiving marketing communications. Test Artifacts: You may submit a request for deletion of test artifacts (screenshots, videos, logs) by contacting support@ito.ai. We will review each request and respond within a reasonable timeframe. California Residents If you are a California resident, the California Consumer Privacy Act (CCPA) may provide you with additional privacy rights with respect to our collection, use, and disclosure of your Personal Information. To the extent that Demox Labs is a covered business under the CCPA, you may contact us at support@ito.ai for more information regarding the following rights: - The right to know what Personal Information we have collected and how we have used and disclosed that Personal Information in the 12-month period preceding your request. - The right to request deletion of your Personal Information. - The right to be free from discrimination related to the exercise of any of your privacy rights. - The right to opt out of the sale of your personal information. Demox Labs does not sell personal information, nor do we share personal information with third parties for marketing purposes. To submit a request under the CCPA, please contact us at support@ito.ai. We will review your request and respond within a reasonable timeframe. We may require you to verify your identity by matching your email address or other account information to the information in our systems before processing your request. Please note that certain data may be retained as permitted or required by law. ### 11. International Jurisdictions Our servers are located in the United States, and your personal information passes through servers located in this geographic area. If you are accessing the Services from another country, please be advised that you may be transferring your personal information to such geographic areas and countries and you consent to that transfer, processing, and storage of your personal information in accordance with this Privacy Policy. You also agree to abide by the applicable US federal, state, and local laws concerning your use of the Services and your agreements with us. ### 12. Security We use technical and organizational measures designed to protect your information against unauthorized access, theft, and loss. Although we take precautions intended to help protect information that we process, no system or electronic data transmission is completely secure. Any transmission of your personal data is at your own risk, and we expect that you will use appropriate security measures to protect your personal information. You are responsible for maintaining the security of your account and the information in your account. We may suspend your use of all or part of the Services without notice if we suspect or detect any breach of security. ### 13. Data Retention Unless you submit a request that we address certain information (see Your Rights and Choices Regarding Your Information), Demox Labs will retain your personal information for the period necessary to fulfill the purposes outlined in this Privacy Policy unless a longer retention period is required or permitted by law. The criteria used to determine our retention periods include: - The length of time we have an ongoing relationship with you and provide services to you (for example, for as long as you have an account with us or keep using the Website); - Whether there is a legal obligation to which we are subject (for example, certain laws require us to keep records of your transactions for a certain period of time before we can delete them); - Whether retention is advisable considering our legal position (such as for statutes of limitations, litigation, or regulatory investigations). Test Artifacts: Screenshots, video recordings, and logs generated by test runs are retained to support your ability to review results and debug issues. You may submit a deletion request at any time by contacting support@ito.ai. Code Data: We do not use your source code for purposes beyond active test execution and do not intentionally retain copies of your source code beyond what is necessary for that purpose. Free Trial Data: If you register for a Free Trial and do not convert to a paid subscription, your account and associated personal information will be retained for the period necessary to fulfill our legal obligations and resolve any outstanding matters, after which we will address deletion requests submitted to support@ito.ai. ### 14. Changes to This Policy Demox Labs may update this Privacy Policy at any time and any changes will be effective upon posting. In the event that there are material changes to the way we treat your Personal Information, we will update the Last Modified date at the top of this Policy upon becoming effective. We may also notify you by email, in our discretion. ### 15. How to Contact Ito If you have any questions about this Privacy Policy or our privacy practices, please contact us at: ‍ Demox Labs, Inc. 1547 Palos Verdes Mall, #124 Walnut Creek, CA 94597 Email: support@ito.ai