Synology: repairing an ext4 volume DSM called unfixable

A DS920+ with a 10.9 TB ext4 volume had been showing "File system errors" since September 30, 2025. The Storage Manager check failed every time with the same message, and Synology support recommended the only thing they ever recommend: delete the storage pool, create a new one, restore from backup. As for a manual e2fsck, they said the chance of success was low. The actual cause was an orphan inode list broken by a power cut, and the repair took a few hours and lost nothing.

The symptom

In Storage Manager both the pool and the volume sat at "Warning", data scrubbing was blocked ("Unable to run data scrubbing because the storage pool is in abnormal status"), and the system log recorded this on every attempt:

info  admin: System ran [ext4 filesystem check & repairing] on [Volume 1].
      Exit code: [File system errors cannot be fixed]

All four drives were SMART-clean (zero reallocated sectors, zero pending, zero CRC errors) and every md array reported [4/4] [UUUU]. So it was not hardware.

What the superblock actually says

Filesystem state:      clean with errors
Errors behavior:       Continue
FS Error count:        102
First error time:      Tue Sep 30 23:44:04 2025
First error function:  ext4_mb_generate_buddy
Last error time:       Mon Feb  9 16:57:41 2026
Last error function:   ext4_mb_generate_buddy
Last checked:          Sat May 10 17:56:37 2014
Mount count:           257

Last checked reading 2014 is the detail that changes everything. At the end of an e2fsck run that completes in read-write mode, the code in e2fsck/unix.c does four things in the same place: sets s_state = EXT2_VALID_FS, writes s_lastcheck = now, sets s_mnt_count = 0, and memsets the whole error counter block.

All four were untouched since 2014, the date the filesystem was created. So no e2fsck had ever run to completion on this volume in twelve years. Every DSM check in 2025 and 2026 had bailed out early.

ext4_mb_generate_buddy

The error signature is one single check in fs/ext4/mballoc.c:

if (free != grp->bb_free) {
        ext4_grp_locked_error(sb, group, 0, 0,
                              "block bitmap and bg descriptor "
                              "inconsistent: %u vs %u free clusters",
                              free, grp->bb_free);

free is the number of free bits counted in the block bitmap, bb_free is what the group descriptor claims. The two counters disagree. That is all. It is not inode, directory or data corruption. And the inode #0 / block #0 fields in the superblock are not inode zero, they are "not applicable" sentinels passed literally by the caller.

What follows is more interesting: the kernel flags the group EXT4_GROUP_INFO_BBITMAP_CORRUPT, in memory only, and quarantines it. Nothing is allocated from it any more, and ext4_free_blocks() returns early for it, so every delete in that group permanently leaks space. The correction (grp->bb_free = free) never reaches the disk, because writes to that group are disabled. Net result: it is re-detected on every mount, forever.

But that does not explain the failure

In e2fsck/problem.c, the pass 5 problems (PR_5_FREE_BLOCK_COUNT_GROUP, PR_5_BLOCK_BITMAP_HEADER and the rest) carry PR_PREEN_OK | PR_PREEN_NOMSG. Preen mode fixes them silently. They were not what stopped the check.

And the superblock only records the first and last error. Whatever was halting e2fsck did not appear anywhere in tune2fs.

Where DSM writes down what actually happened

Step one was finding out what Storage Manager runs. A poll over /proc/*/cmdline while the check was running from the UI:

while true; do
  for p in /proc/[0-9]*; do
    c=$(cat "$p/cmdline" 2>/dev/null | tr '\0' ' ')
    case "$c" in *fsck*) echo "$(date +%T) ${p##*/} $c";; esac
  done
  sleep 2
done

10:07:38 27575 /sbin/e2fsck -C-1 -pvftt /dev/mapper/cachedev_0

-p is preen. Theory confirmed on the box itself. More importantly, DSM was successfully unmounting the volume and really was running e2fsck read-write, so the problem was not that it never got started.

Step two: /usr/syno/lib/systemd/scripts/volume.sh, the script that runs the boot-time check, shows where the output goes:

/sbin/e2fsck -C-1 -pvf $ThisDev >> /var/log/fsck/${ThisRemap}.log 2>&1

ThisRemap is the device path with / replaced by ., so the file is /var/log/fsck/mapper.cachedev_0.log. The full history of every attempt had been sitting there for months. The last two lines:

Inodes that were part of a corrupted orphan linked list found.

UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)

The loop

That message maps to PR_0_ORPHAN_CLEAR_INODE, the one problem in that area that does not carry PR_PREEN_OK. Preen reaches it, calls preenhalt(), and that function exits with FSCK_UNCORRECTED (code 4) from inside itself, before the cleanup block that would have written s_lastcheck and cleared the counters. Hence Last checked: 2014.

The orphan inode list is a linked list in the superblock holding files that had been unlinked but were still open in some process when the power went. The kernel frees them on the next mount. Except that, per fs/ext4/super.c, ext4 skips orphan list processing entirely while the EXT2_ERROR_FS flag is set.

So:

the power cut breaks the orphan inode list;
EXT2_ERROR_FS gets set;
the kernel stops processing the list on every mount while the flag is set;
e2fsck -p refuses to clear the inodes unprompted and exits 4;
the flag is never cleared, so back to step 3.

It does not get worse. But by construction, it cannot resolve itself either.

Why the list broke

From dmesg, on every boot:

EXT4-fs (md0): mounted filesystem with ordered data mode. Opts: barrier=1
EXT4-fs (dm-1): barriers disabled
EXT4-fs (dm-1): mounted filesystem with writeback data mode.

Synology mounts the system partition with barrier=1 and data=ordered, and the data volume with barriers disabled and data=writeback. Neither is an ext4 default: both are passed explicitly.

The bitmap and the descriptor are modified in the same journal transaction, so with barriers on they cannot diverge. What nobarrier removes is the blkdev_issue_flush before the journal tail is advanced, and jbd2 states plainly in checkpoint.c that this flush is required for correctness. Without it, already-checkpointed metadata can still be sitting in the drive's volatile cache when the transactions are dropped from the log. Power goes, one of the two writes is lost, and there is no redo record left anywhere.

One clarification: data=writeback is not the mechanism here. That governs ordering of file data against metadata; the bitmap and the descriptor are metadata and are journalled identically under ordered and writeback. On top of that, md RAID5 on kernel 4.4 has no PPL (added in 4.12), so the write hole is uncovered as well.

The trigger, in this case, was a UPS that failed.

The repair

DSM exposes exactly what is needed, with no call for synospace --stop-all-spaces (which has left pools marked "Crashed" for some people) or tricks with disable_volumes in synoinfo.conf:

synostgvolume --enum-dep-pkgs /volume1
synostgvolume --stop-package-and-unmount /volume1

Then a dry run, which writes nothing:

e2fsck -nvf -C 0 /dev/mapper/cachedev_0

Three things in the output made the decision easy. In the block bitmap difference list, every entry carried a minus sign. A -N means "the block is marked in use on disk but no inode claims it", so it gets freed; a +N would have been the dangerous direction, a file using a block the bitmap believes is free. Passes 2, 3 and 4 were completely empty, not a single line. And there was no pass 1B/1C/1D at all, meaning zero multiply-claimed blocks.

The reason the repair is safe lies in how e2fsck works: pass 1 rebuilds block_found_map by walking every inode's blocks, and pass 5 replaces the on-disk bitmap with the computed map. It does not trust the existing bitmap, so it cannot free a block that any inode still claims.

The safety net is -z, not e2image: install_image() in misc/e2image.c explicitly refuses to restore raw and qcow2 images, so a metadata dump cannot be put back. The undo file has to live on a different filesystem: a stick formatted ext4, not FAT32, where the 4 GB per-file limit could well be reached.

e2fsck -yvf -C 0 -z /volumeUSB1/usbshare/vol1-p1.undo /dev/mapper/cachedev_0
e2fsck -yvf -C 0 -z /volumeUSB1/usbshare/vol1-p2.undo /dev/mapper/cachedev_0

The second pass is the verification. And at the end, so DSM refreshes its own state and clears the Storage Manager warning (a reboot alone does not do it):

synostgvolume --report-ext4-fsck-done \
    --path=/dev/mapper/cachedev_0 --fsck-exit-code=0

The result

Field	Before	After
Filesystem state	clean with errors	clean
FS Error count	102	field gone
Last checked	May 10, 2014	July 26, 2026
Mount count	257	0
Blocks used	1,319,495,401	1,319,396,527

17 orphan inodes released, 98,874 blocks reclaimed (about 386 MB of phantom space), and roughly a hundred extent trees optimised. /lost+found did not exist on the volume at all and was created by e2fsck. It stayed empty, so no unattached inodes to recover and not a single file lost. The second pass produced no "Fix?" line. And after a reboot, the warning: mounting fs with errors, running e2fsck is recommended line, present on every boot since October 11, 2025, was gone.

One point worth knowing when reading the output: in the dry run the counted= figures come out much lower than in the real run (group 47290: counted=14803 versus counted=31878). That is not an inconsistency. With -n, e2fsck refuses to release the orphans and keeps their blocks accounted for. Those 17 deleted-but-unreleased inodes were holding over a gigabyte hostage.

That figure does not subtract from the table above, though. The "blocks used" number from before came out of the superblock counters, which is exactly what was broken: in the quarantined groups every delete was lost without ever being marked free. The 98,874 difference is between wrong accounting and accounting rebuilt from scratch by e2fsck, not an inventory of what was released.

Three DSM gotchas

The fsck log is not in Log Center. It is /var/log/fsck/<device-with-dots>.log, and it holds the complete output of every run plus the exit code. First place to look, not last.
The e2fsck config is not in /etc. Synology built e2fsprogs with sysconfdir /sbin/etc. A hand-written /etc/e2fsck.conf is ignored outright. The same volume.sh also shows DSM mounting the swap partition (md1) as scratch_files for fsck and then rebuilding it, so Synology itself treats that as non-optional on large volumes.
The prefix in e2fsck messages is the volume label, not the binary version. Lines like 1.42.6-3810: Inode ... IGNORED. were showing up on a system where e2fsck -V reports 1.44.1. e2fsck uses ctx->device_name, and when the filesystem has a label that is the label. Synology puts the mke2fs version from creation time there. It matters, because -z only exists from e2fsprogs 1.43 onwards.

What is left

The mechanism that caused the corruption is unchanged and cannot be changed persistently: /etc/fstab is regenerated on every boot by /usr/syno/cfgen/s00_synocheckfstab, so the mount options come back. On this configuration a working UPS is not optional. It is the only defence left.

The volume also has a quirk that fsck does not fix: 731,676,672 inodes allocated against 2,524,173 used, which is 0.34%. That is what you get from running mke2fs with the default ratio of 16 KB per inode on an 11 TB volume, and it is also why any full check takes as long as it does. The inode table alone occupies around 174 GiB.

And, most relevant for next time: ext4 has no data checksums. There is no structure on disk that knows a file's contents are wrong, so the question "which files are affected" has no answer on ext4. Btrfs does have them, and on a redundant layout Synology documents automatic repair from the good copy. At the next drive replacement, the volume gets rebuilt on Btrfs.

The practical takeaway

"File system errors cannot be fixed" in the DSM interface means e2fsck exited with code 4, that is, errors left uncorrected. It does not mean unfixable errors. In preen mode, code 4 normally means exactly one thing: e2fsck hit a problem that requires a human decision and refused to answer it on its own.

That answer had been in /var/log/fsck/, one tail away, for months.