# Bug Catch of the Week: How a Failed Database Swap Could Delete Your Only Copy DoltLite's replacement flow could delete the old database before the new copy was safe. Here's how runtime testing exposed the failure path before release. By Evan Marshall ยท 2026-09-04 **Your database rewrite failed. The Good:** It returned an error. **The Bad:** It had already deleted the only known-good copy. Ouch. That was the failure Ito caught in DoltLite *before* merge (thankfully!). A file replacement error left the system without the original database or a reliable recovery copy. The team confirmed the finding and fixed the replacement flow. This is the first **Bug Catch of the Week**, a semi-regular series about verified, critical and high severity bugs caught in the wild *before* they merge. Each entry follows a real **runtime analysis** finding: Ito runs the code on a pull request, exercises the affected path, and shows what static **analysis alone** could not verify. ## The bug catch at a glance - **Product:**[ DoltLite](https://github.com/dolthub/doltlite), an open-source embedded database. - **Severity:** Critical. The affected path replaces the database file during backup and garbage collection. - **The catch:** A replacement failure could leave neither the original database file nor a completed replacement available for recovery. - **Potential production consequence:** Stored database contents could be lost after an interrupted replacement. - **Evidence:**[ the merged pull request](https://github.com/dolthub/doltlite/pull/2171), [Ito's test result](https://app.ito.ai/pull-requests/d8c38841-515a-4f0f-87f9-f80c222355d8), and [the fix commit](https://github.com/dolthub/doltlite/commit/d3305aab79). ## The impossible-looking symptom A refactor moved database replacement into the platform-specific code. The happy path still worked, but that doesn't mean it was always safe. Then Ito forced a file-operation failure. The replacement itself reported an error, as it should. But when Ito inspected the state after an injected failure, the system could have already removed the existing database. A recovery operation had failed, yet the only proven-good copy was no longer there. That is the scariest part of this kind of issue. A normal rewrite can look correct right up until a file operation goes wrong. The failure result does not guarantee that the database is safe to retry. ## What the first explanation missed A competent reviewer could see error handling, a cleanup path, and a successful replacement test. Those facts establish that the happy path worked. They do not establish the order in which the old file, temporary file, and replacement are handled after an I/O error. The missing condition was a failure after replacement had begun. The old flow could delete the existing destination before opening and synchronizing the replacement. If a later write, truncate, sync, or open operation failed, cleanup could delete the destination again. The operation correctly returned an error, but the recovery guarantee had already been broken. ## The investigation Ito tested the replacement operation under injected file-operation failures, then checked the state of both files. 1. The expected outcome was that the original database and the temporary replacement would remain available for recovery. 2. The failed run showed that the operation could return an error after the original had been removed, without a reliable replacement to fall back to. 3. The public implementation explained the result: deletion could happen before the later file operations that still had failure paths. 4. Dolthub founder and maintainer Tim Sehn confirmed the finding and changed the ordering, then added fault-injection coverage for the failure cases. ## The mechanism and consequence Here is the failure chain: the replacement code removed database.db before the replacement file was safely installed. It then opened, wrote, truncated, or synchronized the replacement. An I/O error at any of those steps stopped the operation after the known-good database had already been deleted. The cleanup path could then remove the destination again. The error message was correct. The state left behind was the problem. The fix changed the order of operations. It avoids deleting the destination first and retains the synchronized temporary replacement when the replacement fails, leaving a known recovery path instead of an ambiguous one. Fortunately, no production data loss was reported. This was caught before merge. The potential consequence still made it a critical finding: a database rewrite that fails must leave a safe recovery path. Otherwise, it can delete the only database file available to that operation and cause data loss. ## Why reading the code was not enough A code review can identify error handling and confirm that the normal copy succeeds. Static analysis can also flag a suspicious operation order. Neither can prove what remains on disk when the filesystem fails halfway through a replacement. [Runtime analysis and static code review answer different questions.](https://www.ito.ai/blog/ai-code-review-read-your-code-run-it) The decisive test asks the filesystem to fail at a particular step, then checks the original database, temporary replacement, and recovery state. Ito ran the pull request's replacement code under those conditions. The failure could only be proven before production by running the replacement under a forced I/O error and checking what was left on disk. Ito did that before merge. The maintainers fixed the order of operations and added fault-injection coverage for the failure cases. The Key Lesson: The failure path is part of your product, and it needs to be tested before your users find it. **Want a closer look at the evidence?** See it all below: - [DoltLite pull request #2171](https://github.com/dolthub/doltlite/pull/2171) - [Fix commit](https://github.com/dolthub/doltlite/commit/d3305aab79) d3305aab79 - [Ito test result for the pull request](https://app.ito.ai/pull-requests/d8c38841-515a-4f0f-87f9-f80c222355d8) *Want Ito to run your code before your next merge? SIgn up to *[*run Ito on your PRs.*](https://app.ito.ai/auth/signup?utm_source=blog&utm_medium=post+mention&utm_campaign=bug+of+week+1+-+dolthub+database+drop&utm_term=free+trial+end+of+post)