I had a working RAID5 array until I installed kvm and qemu and rebooted. After that the system wouldnt boot as /dev/md0 could not be mounted.
Running cat /proc/mdstat gives:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : inactive sdb[2](S) sda[0](S) 1953263024 blocks super 1.2 unused devices: <none> I did have sda, sdb and sdc in the array but it looks like only sda and sdb are there but now as spares.
Checking each of the disks gives:
sudo mdadm --examine /dev/sda /dev/sda: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : e25ff5c6:90186486:4f001b87:27056b4a Name : SAN1:0 (local to host SAN1) Creation Time : Sat Jul 16 17:13:01 2022 Raid Level : raid5 Raid Devices : 3 Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB) Array Size : 1953260544 (1862.77 GiB 2000.14 GB) Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=2480 sectors State : clean Device UUID : 16904f75:c2ddd8b0:75025adb:0a09effa Internal Bitmap : 8 sectors from superblock Update Time : Wed Jul 20 18:59:56 2022 Bad Block Log : 512 entries available at offset 16 sectors Checksum : 8d2ba8a7 - correct Events : 4167 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AAA ('A' == active, '.' == missing, 'R' == replacing) sudo mdadm --examine /dev/sdb /dev/sdb: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : e25ff5c6:90186486:4f001b87:27056b4a Name : SAN1:0 (local to host SAN1) Creation Time : Sat Jul 16 17:13:01 2022 Raid Level : raid5 Raid Devices : 3 Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB) Array Size : 1953260544 (1862.77 GiB 2000.14 GB) Used Dev Size : 1953260544 (931.39 GiB 1000.07 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262064 sectors, after=2480 sectors State : clean Device UUID : 02a449d0:be934563:ff4293f3:42e4ed52 Internal Bitmap : 8 sectors from superblock Update Time : Wed Jul 20 18:59:56 2022 Bad Block Log : 512 entries available at offset 16 sectors Checksum : aca8e53 - correct Events : 4167 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 1 Array State : AAA ('A' == active, '.' == missing, 'R' == replacing) sudo mdadm --examine /dev/sdc /dev/sdc: MBR Magic : aa55 Partition[0] : 1953525167 sectors at 1 (type ee) and sudo mdadm --examine --scan shows
ARRAY /dev/md/0 metadata=1.2 UUID=e25ff5c6:90186486:4f001b87:27056b4a name=SAN1:0 My mdadm.conf looks like:
mdadm.conf # # !NB! Run update-initramfs -u after updating this file. # !NB! This will ensure that initramfs has an uptodate copy. # # Please refer to mdadm.conf(5) for information about this file. # # by default (built-in), scan all partitions (/proc/partitions) and all # containers for MD superblocks. alternatively, specify devices to scan, using # wildcards if desired. #DEVICE partitions containers # automatically tag new arrays as belonging to the local system HOMEHOST <system> # instruct the monitoring daemon where to send mail alerts MAILADDR root # definitions of existing MD arrays #level=raid5 #num-devices=3 # This configuration was auto-generated on Fri, 15 Jul 2022 20:17:00 +0100 by mkconf ARRAY /dev/md0 uuid=e25ff5c6:90186486:4f001b87:27056b4a Any ideas how to fix the array? Ideally I'd like to do it without having to go back to my backups if possible..
I tried:
sudo mdadm --stop /dev/md0 sudo mdadm --assemble /dev/md0 /dev/sda /dev/sdb /dev/sdc --verbose and I got:
mdadm: looking for devices for /dev/md0 mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000) mdadm: no RAID superblock on /dev/sdc mdadm: /dev/sdc has no superblock - assembly aborted Update 1: Ok, I ran mdadm -D /dev/md1 and it's come back as degraded. Thats not so bad.. I just need to add the 3rd disk back in..
Update 2: After a seemingly successful rebuild I rebooted and got the same issue, to fix it again I tried:
lex@SAN1:/etc/apt$ sudo mdadm --assemble /dev/md0 alex@SAN1:/etc/apt$ sudo mdadm -D /dev/md0 /dev/md0: Version : 1.2 Raid Level : raid0 Total Devices : 2 Persistence : Superblock is persistent State : inactive Working Devices : 2 Name : SAN1:0 (local to host SAN1) UUID : e25ff5c6:90186486:4f001b87:27056b4a Events : 6058 Number Major Minor RaidDevice - 8 0 - /dev/sda - 8 16 - /dev/sdb alex@SAN1:/etc/apt$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : inactive sdb[2](S) sda[0](S) 1953263024 blocks super 1.2 unused devices: <none> alex@SAN1:/etc/apt$ sudo mdadm --stop /dev/md0 mdadm: stopped /dev/md0 alex@SAN1:/etc/apt$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] unused devices: <none> alex@SAN1:/etc/apt$ sudo mdadm -D /dev/md0 mdadm: cannot open /dev/md0: No such file or directory alex@SAN1:/etc/apt$ sudo mdadm --assemble /dev/md0 mdadm: /dev/md0 has been started with 2 drives (out of 3). alex@SAN1:/etc/apt$ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active (auto-read-only) raid5 sda[0] sdb[2] 1953260544 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_] bitmap: 0/8 pages [0KB], 65536KB chunk unused devices: <none> Update 2a:
I tried the following:
sudo gdisk /dev/sdc GPT fdisk (gdisk) version 1.0.6 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Command (? for help): x Expert command (? for help): z About to wipe out GPT on /dev/sdc. Proceed? (Y/N): y GPT data structures destroyed! You may now partition the disk using fdisk or other utilities. Blank out MBR? (Y/N): y Added to array again, lets see what happens in another 2 hours after the rebuild and another reboot.. :(
Any idea what's wrong with disk 3?
Thanks
