8

I have tried to remove 1 HDD from a RAID-5 but something went wrong, but I still hope I can recover my data (in fact, I have all the backups so it is just a question on mdadm possibilities)

I had a 4 x 1 Tb RAID-5, and one of the disks started to show lots of Reallocated_Sector_Ct, so I decided to remove it.

What I did:

  1. mdadm --manage /dev/md0 --fail /dev/sdc
  2. mdadm --manage /dev/md0 --remove /dev/sdc

  3. tried to run:

    root@darkstar:/home/anton# mdadm --grow /dev/md0 --raid-devices=3 mdadm: this change will reduce the size of the array. use --grow --array-size first to truncate array. e.g. mdadm --grow /dev/md0 --array-size 1953262592 
  4. mdadm --grow /dev/md0 --array-size 1953262592

and finally:

  1. mdadm --grow /dev/md0 --raid-devices=3 --backup-file=/root/grow_md1.bak

Now that the reshape and recovery is done, I cannot access my /dev/md0 (it does not mount), resize2fs /dev/md0 tells to run e2fsck first, and e2fsck tells:

The filesystem size (according to the superblock) is 732473472 blocks The physical size of the device is 488315648 blocks Either the superblock or the partition table is likely to be corrupt! 

on the other hand, mdadm -D /dev/md0 tells:

Array Size : 1953262592 (1862.78 GiB 2000.14 GB) Used Dev Size : 976631296 (931.39 GiB 1000.07 GB) 

That leaves some hope that not all my data is lost. Does anybody have some ideas on what I should do to have a valid RAID-5 array of 3 x 1 Tb disks?

3 Answers 3

9

All you should have done was your step one

mdadm --manage /dev/md0 --fail /dev/sdc 

At this point your RAID 5 array is running in degraded mode and you can replace the disk with a new one.

Unfortunately it looks like you have truncated your array's effective size from 2TB to 1TB, destroying the second half of your filesystem. Fortunately you say you have backups.

I'm slightly puzzled. With four disks in a RAID5 configuration you should have had 3TB available to you. But without seeing the results of mdadm --examine I'm not sure there's much else I can offer you.

9
  • i didn't want to replace the disk, i wanted to remove it. Commented Sep 8, 2017 at 14:27
  • root@darkstar:/home/anton# mdadm --examine /dev/md0 mdadm: No md superblock detected on /dev/md0. Commented Sep 8, 2017 at 14:28
  • @AntonS. without the mdadm details for the remaining physical devices I can't make any other suggestions. Commented Sep 8, 2017 at 14:29
  • alas, this site does not allow long comments. otherwise i'd copy everything that mdadm --examine returns. Commented Sep 8, 2017 at 14:34
  • 1
    @AntonS. comments are for commenting. Updates like this I'm asking you for, go into your question for everyone to find easily. (Don't forget the formatting.) Then I can delete my comments because they're no longer required. Commented Sep 8, 2017 at 14:38
7

You got the order backwards.

To shrink, you first shrink the filesystem (resize2fs) then second shrink the block device (mdadm). The order you did is correct for enlarging a filesystem, but backwards for shrinking one.

You've destroyed your data. To recover from this, you first confirm your backups are good. Then mkfs the array and restore from backup. If your backups aren't good, you can probably recover files that happened to be on the 1st 2TB of the filesystem. (See below)

PS: The normal way an array is managed is if a disk fails, you replace that disk with one of the same capacity or larger. mdadm --grow is not a normal part of dealing with disk failures.

Recovery

What used to be the 3rd terabyte of your filesystem has been overwritten; essentially that space is now used for the parity. (The actual sectors contain a mix of parity and data which was moved from other disks, where those sectors now contain parity.) That part of the data is gone for good; absent (possibly theoretical) high-cost approaches capable of reading the previous contents of sectors it's not recoverable.

In addition, ext4 does not keep all the metadata at the start of the filesystem; it's spread out throughout the filesystem. So you've lost a bunch of metadata as well. Importantly, if any part of the file data or metadata was in that lost third, the file will be inaccessible. Limited recovery of snippets may be possible from the 4th disk (which was probably untouched by the grow, since it was failed at the time.)

The first, and most important step, is to purchase a 4TB disk and use that to make a complete copy (image) of the filesystem. Then, set the 4 original disks aside. If there is any question of the reliability of the original disks, make a second copy and only work on one of the copies. You will also need additional disks to copy recovered files on to, including possibly multiple copies of partially damaged files.

Now you can try recovery steps on a copy. Note that most these will need to be done on a fresh copy—the steps are destructive, this is one of the many reasons to only work on a copy. Do not destroy your originals:

  1. Let e2fsck -y /path/to/copy do it's thing. Probably it'll produce something you can mount. Go ahead and do so, copy files off.

  2. Extend your copy back to the original size (should be OK for it to be sparse; truncate -s can do this). Then it'll likely mount (do so read-only). Copy what you can. Unmount it, and again let e2fsck -y do its thing again. Again mount and copy what you can.

  3. Run fsck without -y and actually go through all those messages. E.g., I'd expect it actually to give you a choice of what to do when part of a file's data is in the lost area (replace with 0s, delete file). Possibly it gives choices about lost metadata too. I'd do -y first, because it will have a lot of questions for you...

  4. If you have an old filesystem image backup, combine the 2TB you have + the missing 1TB from the backup. fsck the result, see if you can get any additional files off it. Risk that the recovered files are corrupt is quite high, though.

  5. Use programs that scan through the filesystem image for data patterns (e.g., photorec to look for JPEGs). This is the only one that doesn't strictly need a fresh copy.

  6. In theory, ¾ of the final ⅓ of the "failed" disk #4 contain some of your missing data. If you can figure out the sector/chunk mapping (I sure don't know it!) you could copy ~250GB from that disk back in to your image, and repeat all the above recovery steps to recover additional files.

Note that all these recovered files may have corruption in them (e.g,. blocks full of 0s instead of data). Verifying them is easy if you have checksums somewhere, but otherwise a tedious manual process.

We have a bunch of questions about recovering data from damaged filesystems; as long as you only work on copies you can experiment without putting your data at further risk.

4
  • are you 100% sure i have destroyed my data? if i really have destroyed it, can you tell me how i can "probably recover files that happened to be on the 1st TB of the filesystem" Commented Sep 8, 2017 at 15:10
  • 1
    @AntonS. You truncated the array, meaning "just cut it off". So anything that was stored on the 2nd half of that filesystem (data or metadata) is now gone (that second --grow that changed the number of devices overwrote it all). I'll edit in some of the data recovery steps... Commented Sep 8, 2017 at 15:13
  • ok i trunkated it but can i access what was on the first part (in fact, two thirds) of the disk (md0 array)? Commented Sep 8, 2017 at 15:17
  • @AntonS. recovery notes added. Commented Sep 8, 2017 at 15:29
1

To summarize the correct way to remove a device, first mark it as failed:

sudo mdadm /dev/md127 --fail /dev/sdc 

Estimate the new size of the filesystem after shrinking it:

sudo resize2fs -P /dev/md127 

If the disk is large, you may want to estimate how long the resize2fs command will take before committing to the operation. See Estimating the time needed for a resize2fs shrink - Server Fault for details.

Shrink the filesystem:

sudo resize2fs -p -M /dev/md127 

Verify the filesystem:

sudo e2fsck -f /dev/md127 

Check the new filesystem size (see How to find the size of a filesystem? - Ask Ubuntu):

sudo dumpe2fs -h /dev/md127 |& awk -F: '/Block count/{count=$2} /Block size/{size=$2} END{print count*size}' 

Estimate the new size of your RAID5 array, by attempting to run this command and checking the error message:

mdadm --grow --raid-devices=3 /dev/md127 

Verify the filesystem is small enough to fit. Shrink the block device:

mdadm --grow /dev/md127 --array-size new_size 

Remove the extra device:

mdadm --grow --raid-devices=3 /dev/md127 --backup-file /root/md127.backup 

Resize the filesystem to take all the available space:

resize2fs /dev/md127 

As @roaima points out, it's more common to fail a device and replace it. The approach laid out here is going to require you to bring a live system down, often unacceptable.

See also:

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.