1

I have one disk (sdd) on which I currently have stored my data. I now have got two new 3TB disks (sdb&sdc) and want to create a RAID5 array over all three disks.

  • sdb: gpt table, empty partition sdb1
  • sdc: gpt table, empty partition sdc1
  • sdd: gpt table, btrfs-partition sdd1 with my data

My plan looks like this:

  • Create RAID5 array md0 over sdb1 and sdc1
  • Create a btrfs filesystem on it.
  • Copy the data from sdd1 to md0
  • Re-partition(=wipe) sdd
  • Grow the array onto sdd1

I currently am stuck at creating the 2-disk RAID5 array. I built the array

# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=2 /dev/sdc1 /dev/sdb1 mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 512K mdadm: size set to 2900832256K mdadm: automatically enabling write-intent bitmap on large array mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started.

and /proc/mdstat shows that it's doing the initial sync: I have one disk (sdd) on which I currently have stored my data. I now have got two new 3TB disks (sdb&sdc) and want to create a RAID5 array over all three disks.

  • sdb: gpt table, empty partition sdb1
  • sdc: gpt table, empty partition sdc1
  • sdd: gpt table, btrfs-partition sdd1 with my data

My plan looks like this:

  • Create RAID5 array md0 over sdb1 and sdc1
  • Create a btrfs filesystem on it.
  • Copy the data from sdd1 to md0
  • Re-partition(=wipe) sdd
  • Grow the array onto sdd1

I currently am stuck at creating the 2-disk RAID5 array. I built the array

# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=2 /dev/sdc1 /dev/sdb1 mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 512K mdadm: size set to 2900832256K mdadm: automatically enabling write-intent bitmap on large array mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started.

and /proc/mdstat shows that it's doing the initial sync:

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] md0 : active raid5 sdb1[2] sdc1[0] 2900832256 blocks super 1.2 level 5, 512k chunk, algorithm 2 [2/1] [U_] [>....................] recovery = 0.6% (19693440/2900832256) finish=308.8min speed=155487K/sec bitmap: 0/22 pages [0KB], 65536KB chunk unused devices: <none>

top shows, that during this time, md(adm) uses ~35% CPU:

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 989 root 20 0 0 0 0 S 29.1 0.0 0:17.69 md0_raid5 994 root 20 0 0 0 0 D 6.6 0.0 0:03.54 md0_resync

So far so good. This should take ~6 hours. On my first try I had to reboot my server and thus stop the array after ~5h an the second time my sdb drive mysteriously disappeared, so I also had to restart the system.

The array started itself automatically, but the progress bar is gone: 

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] md127 : active (auto-read-only) raid5 sdb1[2] sdc1[0] 2900832256 blocks super 1.2 level 5, 512k chunk, algorithm 2 [2/1] [U_] bitmap: 0/22 pages [0KB], 65536KB chunk unused devices: <none>

and top reports no CPU-use.

So I tried stopping and assembling it manually:

~# mdadm --stop /dev/md127 mdadm: stopped /dev/md127 ~# mdadm --assemble --verbose /dev/md0 /dev/sdc1 /dev/sdb1 mdadm: looking for devices for /dev/md0 mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0. mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 1. mdadm: added /dev/sdb1 to /dev/md0 as 1 mdadm: added /dev/sdc1 to /dev/md0 as 0 mdadm: /dev/md0 has been started with 1 drive (out of 2) and 1 rebuilding.

Although it says it's rebuilding, mdstat shows no sign of that:

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] md0 : active (auto-read-only) raid5 sdc1[0] sdb1[2] 2900832256 blocks super 1.2 level 5, 512k chunk, algorithm 2 [2/1] [U_] bitmap: 0/22 pages [0KB], 65536KB chunk unused devices: <none>

also top again shows no CPU use.

So I searched the web for a method to manually force a sync and found --update=resync, but trying this also doesn't yield a victory:

~# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 ~# mdadm --assemble --verbose --force --run --update=resync /dev/md0 /dev/sdc1 /dev/sdb1 mdadm: looking for devices for /dev/md0 mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 0. mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 1. mdadm: Marking array /dev/md0 as 'clean' mdadm: added /dev/sdb1 to /dev/md0 as 1 mdadm: added /dev/sdc1 to /dev/md0 as 0 mdadm: /dev/md0 has been started with 1 drive (out of 2) and 1 rebuilding. root@server:~# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] md0 : active (auto-read-only) raid5 sdc1[0] sdb1[2] 2900832256 blocks super 1.2 level 5, 512k chunk, algorithm 2 [2/1] [U_] bitmap: 0/22 pages [0KB], 65536KB chunk unused devices: <none>

(still no CPU-use)

After two days of trying to fix it myself I would be very thankful for any help or advice

Improve this question
13
  • 1
    "I had to reboot my server and thus stop the array after ~5h an the second time my sdb drive mysteriously disappeared" -- well, that sounds like it could include just about anything, so I'd probably go with just redoing the whole thing from start. FWIW, if you're aiming for a three-drive array in the end, but need one to hold the data temporarily, I'd create the array with three drives from the start (-n 3) and just list one of them as missing. You can add the missing drive later on. Commented Dec 31, 2017 at 18:47
  • @ilkkachu My problem is just that I have no idea how to continue the sync process. -n 3 doesn't work with just two drives Commented Dec 31, 2017 at 22:04
  • 1
    strongly recommend you don't do this. btrfs still doesn't have a reliable raid5/6 capability, and btrfs on top of mdadm raid5 is no safer. recommend ZFS instead (most distros have it packaged now, including dkms packages for the kernel module). You can create a zpool with a missing vdev by creating a sparse file of the appropriate size using truncate -s, then zpool create ...., and then zpool offline the sparse file. zfs create a fs on the pool, rsync your sdd data to it, unmount the sdd filesystem then zpool replace the sparse file with sdd. Commented Jan 1, 2018 at 10:32
  • @cas I've never had problems with btrfs, and since ZFS is too demanding for my server (4GB RAM), I think md(adm) is the most secure and reliable solution for me. Commented Jan 2, 2018 at 23:42
  • It's your data, you can trust it to whatever you want, but IMO you should thoroughly research the risks of btrfs raid5/6 first (the btrfs devs warn that it's only for testing), and research the various alternatives too. BTW, just as with zfs, if you don't give individual drives/block-devs to the fs but instead layer btrfs on top of either hardware or software RAID, you lose the most important benefit of using the fs, error correction. Commented Jan 3, 2018 at 0:38

1 Answer 1

Reset to default
0

Since there's nothing on the raid array, the easy option is to just wipe everything and start over:

  • mdadm /dev/md127 --fail /dev/sdb1; mdadm /dev/md127 --fail /dev/sdc1; mdadm /dev/md127 --remove failed
  • wipefs -a /dev/sdb1; wipefs -a /dev/sdc1

Incidentally, I recommend creating a degraded RAID 5 array and then adding the third disk to it. A rebuild is usually faster than a reshape, though there's an increased chance of data loss if a disk fails during the rebuild (you do have a backup, don't you?)

  • mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdc1 /dev/sdb1 missing
  • Create the new filesystem
  • Copy the data
  • wipefs -a /dev/sdd1
  • mdadm /dev/md0 --add /dev/sdd1
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.