3

I have 2 disks, each 1TB in size, and both of them are a member of an MD RAID-1 array, as created by this command:

mdadm --create /dev/md0 /dev/sda /dev/sde --level=1 --metadata=1.0

The argument --metadata=1.0 instructs the kernel to use a specific MD RAID header that resides at the end of the physical disks.

The reason I made this decision is because disks in this array must be bootable by a UEFI that does not understand MD RAIDs; so from the view of UEFI, it can pick any member disk and boot from it just fine, as it will find GPT partition tables at the beginning of the disk just like any normal disk that is not part of a RAID setup.

The kernel, along the initramfs, are placed in the UEFI partition. UEFI loads the kernel from any member disk that the UEFI system chooses, without UEFI knowing anything about RAID. This already works, as all member disks are identical (thanks to RAID 1 mirroring over them).

Once the MD RAID-capable kernel loads, it reads the RAID metadata (at the end of the disks), and automatically assembles them into the logical mapper device file /dev/md0. This sort of works.

Currently, I have two problems:

  • Problem 1: The kernel detects the existence of the MD RAID array, which proves that the kernel is reading MD RAID headers.

    But the problem is that it only shows up 1 member disk, and doesn't show the other. Which disk is chosen shuffles randomly across reboots, but only 1 is shown.

    Logically, this would mean that the kernel is somehow not reading the metadata header of one of the other disks, hence not knowing of their existence. Even though, say, mdadm --examine /dev/sda shows that it is indeed a RAID member (suppose that /dev/sda was the unlucky one at this reboot).

    Trying to add the unlucky disk to the array, causes mdadm to say says that the disk is busy.

    Additionally, the unlucky disk does not get any mirroring to it. For example, if I edit some files in a reboot with /dev/sde showing, then if I reboot again so that the other, /dev/sda, is shown, then I would not see any of the changes that I made during the previous reboot when /dev/sde was shown. This shows that there is no mirroring.

  • Problem 2: fdisk /dev/md0 says that the backup GPT table is corrupt. The backup table exists at the end of the disks in the case of GPT.

    I don't know the cause of this. Is fdisk peeking into the end of the physical disks /dev/sda and /dev/sde? How did it even know of them (if it did)? After all, I just gave it a logical mapper device /dev/md0 that should abstract the physical member disks. I would imagine that /dev/md0 is also smaller in size as it should subtract the metadata that MD RAID puts in the physical disks.

Question: What is going on? How to resolve both of the problems?

Background: I'm on Gentoo Linux latest stable version, a fresh installation this week, using systemd and systemd-boot (formerly known as the gummiboot).

Improve this question
10
  • 1
    For problem 1, do you have dmesg / syslog of the raid assembly process? Commented Jan 15, 2023 at 9:41
  • 1
    For problem 2, always put RAID on partitions instead of bare drives. Otherwise, once anything 'fixes' GPT header on sda, it corrupts it for md0, and vice versa. Also stick to 1.2 metadata whenever possible. Using raid1 with 1.0 metadata should be limited to the efi partition alone. Commented Jan 15, 2023 at 9:52
  • 1
    Doing this might as well solve problem 1, but the root cause for problem 1 is still unclear to me. Your setup is very prone to corruption but it should work as long as nothing tries to helpfully fix it... Commented Jan 15, 2023 at 9:56
  • 2
    OK from the log my guess (only a guess!) is that your Initramfs does not handle RAID at all (this should be a configuration error). Instead it mounts the filesystem directly (from either drive) [this would not be possible with 1.2 metadata but it's possible in your case]. Then as the system boots, it tries to belatedly assemble RAID, but it can't assemble it from the drive that is already mounted / busy. So you end up with a one drive degraded RAID which isn't mounted and doesn't do anything useful. Commented Jan 15, 2023 at 15:54
  • 1
    If you're using RAID then you can only use /dev/md… if you write to /dev/sd… then your RAID will be out of sync and corruption will follow. It's different if you create two independent partitions and keep them in sync by other means (not involving RAID at all). Commented Jan 15, 2023 at 18:33

1 Answer 1

Reset to default
3

I'm just summarising the discussion in the comments section under my question as a future reference (Thanks to frostschutz).

TL;DR:

  • As for loading RAID in initramfs: since I use dracut to build my initramfs, I had to add rd.auto=1 to /etc/kernel/cmdline, so that dracut knows to mount the virtual devices, such as RAID. Default is rd.auto=0 (which is a reason the setup didn't work).
  • As for the ESP: Use two separate partitions for ESP, without RAID; e.g. mounting /boot and /boot_backup on different physical disks, /dev/sdX1 and /dev/sdY1. Then implement your own EPS synchronisation across them.

This, and links inside it, show that Lennart Poettering wants Systemd to do the extra work of checking the ESP, and to outright refuse to write to it if it is RAIDed, even if the system administrator knows what he is doing (e.g. no competing OSes on the ESP, --metadata=1.0 is used so that the UEFI system doesn't feel that it is RAID1-ed).

The reason is that such RAID1 setup will make an OS mirror an entire ESP, which is possibly also used by other OSes. Other OSes may write to their ESP partitions without this MD RAID (Linux-specific) but write to the individual disks directly, which could cause conflicts in those cases.

While I see that such a setup is not for everyone, I still don't see why it is Systemd's job to parent system administrators to forbid them from doing what they want, when they know what they're doing. This parenting seems just an extra code that lives in bootctl that should be deleted (or moved to mumctl). At most, I think that bootctl should only show a warning, or at least allow a --force option. The reason is that, there are many cases where using RAID1 on the ESP, such as an MD RAID1 with --metadata=1.0, is beneficial.

I think, the best solution is to have bootctl and systemd-boot-update.service, via configuration files, have the option of being told to update multipe ESPs for those that want it, without a RAID. This way, a backup ESP can be used, yet the local OS will only synchronise its own entries in the ESP (not an entire ESP). However, Poettering is against this too, which I don't get why.

Appendix

My specific approach so far:

  • Create /boot, /boot_primary and /boot_secondary.
  • Mount /dev/sdX1 and /dev/sdY to the primary and secondary, respectively, then mount --rbind /boot_primary /boot. Add the same to /etc/fstab.
  • To populate /boot_primary, I ran bootctl install and emerge --config gentoo-kernel (yes, I use distribution kernels and have installed sys-kernel/installkernel-systemd-boot, so these 2 commands will fully populate the primary ESP).
  • To populate /boot_secondary, Rebind /boot to be /boot_secondary, bootctl install --efi-boot-option-description='Linux Boot Manager (Secondary)' and repeated emerge --config gentoo-kernel.

Now I have 2 separate ESPs. The steps above are done manually, but are only for once during the system installation phase.

Then, later on, to update them, I extended my Gentoo updating system (which is just a Makefile) to blindly do 4 additional steps after each update:

  1. Remount /boot to rbind to /boot_secondary.
  2. bootctl update
  3. emerge --config gentoo-kernel
  4. Revert /boot back to /boot_primary.

It is not the fastest solution (speed here doesn't matter, as Gentoo's updates are slow anyway; due to compiling, how slow emerge is, etc), but I like its simplicity and how it doesn't try to be too smart about the specific formats that upstream tools use. So, in case upstream decides to change its behaviour or format, I won't have to change this code and things will work flawlessly, without me having to constantly catch up with upstream. This code should still work even I ever install another OS that happens to share the ESPs.

It's not the slowest either, as emerge --config gentoo-kernel does not recompile the kernel nor the modules, but only repeats the installation, including the boot entries (thanks to distribution kernels being so awesome).

When, say, /dev/sdX fails, then the root partition / will still work (thanks to RAID), and I'll still be able to smoothly boot into the system by using the secondary ESP. No more manual fiddling with ESP recovery tools.

Improve this answer
4
  • It's not systemd is a culprit here. It's just following the UEFI spec which was made without SW RAID in mind. So you should not RAID ESPs as per UEFI spec. That means you have to sync it by hand or by some script (the path Proxmox devs chose — it's worth looking how they handle ESP sync if you selected UEFI and zfs or btrfs SW RAID). And, honestly, other OSes and RAID? Why the hell? RAID is solution to avoid unplanned downtime for "always on" systems; those almost never reboot and thus don't have any chance to make use of multiple OS installed, so they have always precisely one OS. Commented Jan 16, 2023 at 12:10
  • I don't have other OSes and RAID. I have a single OS that's managing ESP. I only mentioned systemd's perspective on why they have gone the extra mile to implement a forceful fool-proof parenting on everyone. I think that systemd has gone a bit too far in the fool-proofness dimension. I think bootctl should only give a warning, then let us use RAID1 if we want to. There are many cases where a system administrator would benefit greatly by having ESP on an MD RAID with --metadata=1.0. Commented Jan 16, 2023 at 13:19
  • The RAID metadata location is not the main problem here. The filesystem/partition/device UUID is the problem. If you RAID them, filesystem serial will be the same on all devices. If you do the partitionable RAID, component devices will not have correct GPT structure at all (the second copy at the end will find itself in the wrong place), and everything will appear under same UUIDs. The EFI firmwares are proven to be buggy, some less, some more, and nobody knows how they react on the colliding IDs (and nobody really wants to know, all we want for it to "just work"). So stay safe, don't RAID ESP Commented Jan 17, 2023 at 5:03
  • @NikitaKipriyanov - I think you're referring to another issue: partitioning RAID's logical device (e.g. /dev/md0), but not when the RAID is built on already-partitioned devices (so /dev/md0 is only going to get a file system by mkfs.vfat). Commented Jan 19, 2023 at 15:29

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.