TECHNICAL GUIDE · GLOBAL ARCHITECTURE

Forensic Data Recovery: Resolving Corrupted PostgreSQL WAL and MySQL InnoDB

Methodology for repairing split-brain replication, corrupted transaction logs, and recovering orphaned tables without backups.

Updated october 2026
8 min read
Altovar Technical Editorial Team

1. Why database corruption is different from file loss

A database stores its data in files and also in transaction logs that describe changes not yet fully written to the main data files. PostgreSQL uses the write-ahead log (WAL) for this purpose. MySQL with the InnoDB storage engine uses redo and undo logs. When a disk fails, power is lost during a write, or two replicas disagree, the logs and the data files may no longer match. The database then refuses to start, or it starts with missing or inconsistent rows.

The first priority is containment, not repair. Every write to a damaged database can shrink the set of states you can still recover. Stop the applications that write to it, avoid repeated restarts that may trigger automatic repair routines, and treat the current state as evidence. This approach keeps your options open if the first attempt fails, and it makes the recovery work reviewable afterwards.

2. Diagnosing the failure before touching anything

Start by collecting facts: the database version, the server logs, the timeline of the failure, any storage events reported by the host, and the state of every replica. Establish whether the damage is in the log, in data pages or in an index. The response differs. A damaged index can often be rebuilt from table data, while damaged table pages may need restoration from a replica or from a backup.

Work on a copy. Take a file-level or block-level copy of the data directory while the service is stopped, verify that the copy is complete, and run every experiment against that copy only. Record each command and its output. This discipline costs time at the start, but it prevents a second failure caused by a repair attempt on the only remaining copy of the data.

3. Comparing recovery options

Restoring a verified backup and replaying the archived logs is the cleanest route, but changes made after the last archived log are lost. Promoting a healthy replica is fast when the replica was in sync, yet writes accepted only by the failed node may be missing or conflicting. Repair tools can fix indexes and some page types, but they can also hide the loss of rows that nobody notices until later.

The table lists the common options. Manual extraction from files is the last resort when no backup or replica exists. It is slow, partial and dependent on expert judgment. Use it to recover what matters most, not to reconstruct the whole database, and document clearly what was not recovered, so that the business knows what it no longer has and can plan around the gap.

OptionWorks whenMain limitationRisk
Restore latest backup and replay archived logsA verified backup and complete archived logs existChanges after the last archived log are lostLow, if the backup has been tested
Promote a healthy replicaA replica was unaffected and was in syncDivergent writes may be missing or conflictingMedium, check timestamps first
Repair tools on a copyDamage is limited to indexes or known page typesNo guarantee that all rows are completeMedium, can hide data loss
Manual extraction from data filesNo backup and no usable replica existSlow, partial and needs expert judgmentHigh, last resort only

4. Dealing with split-brain replication

Split-brain happens when two nodes both accept writes and both believe they are primary. The data sets then diverge. Merging them is a business decision as much as a technical one: when two customer orders conflict, someone must decide which one is authoritative. Identify both timelines, stop writes where possible and export the differing records so that the business can compare them side by side.

Prevent recurrence by enforcing a single primary through a consensus or fencing mechanism, and by making failover a deliberate action rather than an automatic side effect of a network glitch. Document which node is authoritative and how a former primary rejoins. Usually it is rebuilt from the new primary rather than reattached in place, because its own history may contain writes that were never accepted.

5. Checklist for a recovery effort

A recovery effort under pressure tends to skip the basics. The checklist below keeps the team focused on the steps that protect the remaining data and make the result verifiable. Each step should be written down with the time it was performed and the person who approved it, so that the final report can explain what was recovered and what was not.

Share the checklist with the business owner, not only with the database team. Walk through the list together. The owner must agree on which data loss is acceptable, because that decision determines which recovery option is appropriate. Technical success is not enough if the recovered data does not support the processes that depend on it. Agree on the acceptance test before the work starts, not after.

  • Stop writes from applications and record the exact time of the stop.
  • Inventory backups, archived logs and replicas, and check their timestamps.
  • Copy the damaged data directory before any repair attempt.
  • Verify the copy and record checksums or file sizes.
  • Run repair tools only on the copy, one step at a time, with full logs.
  • Compare recovered row counts for key tables with what the business expects.
  • Decide with the business owner which data loss is acceptable, and document the decision.

6. Common mistakes and what to ask your provider

The most damaging mistake is running a repair command against the only copy. Some repair procedures discard data so that the database can start, and the discarded data cannot be recovered afterwards. Another mistake is restarting the server repeatedly while the disk is failing, which can spread the damage. Check the hardware health first, and move the data to healthy storage before further repair work.

Teams also forget to verify restored backups. A backup that has never been restored is an assumption, not a recovery plan. Test restores in an isolated environment and measure how long they take. Record the timing. Finally, do not declare success just because the database starts. Check referential integrity, the functions that depend on the data and the most important reports.

7. How to start: prepare before the next failure

The first concrete step is to prove that a restore works. Choose one important database, restore its latest backup into an isolated environment and measure the time it takes and which data you can check afterwards. Write the procedure in a place where the on-call team can find it, and make sure the steps do not depend on one person's memory.

Next, confirm that replicas are monitored for lag and that promoting a replica requires a deliberate decision. Record who is authorized to stop writes and who decides on acceptable data loss. If no usable backup exists today, that is the first finding to take to management. Recovery from corruption without a backup is slow, partial and expensive, and the cost of preventing it is almost always lower.

Frequently Asked Questions

Can a corrupted PostgreSQL or MySQL database always be repaired?

No. Some corruption can be repaired from replicas, from backups or by rebuilding indexes, but some data may be lost. The outcome depends on what was damaged, when the last good copy was made and whether the logs are complete. Treat any repair as a partial result until the business has verified it.

Is it safe to run repair tools directly on production?

Generally no. Repair procedures can discard data or change files in ways that are hard to reverse. Work on a verified copy, and keep the original untouched until the recovered result has been checked against business expectations and the reasons for every change are documented.

What is the difference between a replica and a backup?

A replica copies changes continuously, so a bad delete or a corruption is copied too. A backup is a point-in-time copy kept separately, so you can return to a state before the problem. Most organizations need both, with the backup isolated from the production administrator account.

Who should we call for a recovery?

Use a provider with database administration experience, and ask for a written plan before work starts. Agree in advance on who approves each step, because recovery decisions often involve trade-offs between completeness and speed. Ask for logs and timelines before work starts.

How long does recovery take?

It depends on the data volume, the extent of the damage and whether a clean copy exists. Recovery from a verified backup is generally much faster than forensic reconstruction. Ask the provider to estimate the work in phases and to report progress against the checklist.

How do we avoid split-brain in the future?

Use a clear primary election or fencing mechanism, and make promotion a controlled action. Monitor replication health and alert when more than one node accepts writes. Test failover regularly, not only when the first real incident happens, so that the team knows the procedure before it is needed.

About this content

General informational guide. Regulations and technical requirements change over time: always check the official sources and assess your specific case with a qualified technical or legal advisor.

GET IN TOUCH

Talk to our team.

We'll show how Altovar integrates into your infrastructure. No commitment, no sales pitch. Just a technical conversation.

BOOK A DEMO →