1

LVM only allows volume groups to be expanded by adding physical volumes. So if I have multiple volume groups which may need extra space, but it can't be predicted which will expand fastest, it's best to have several spare physical volumes available to allocate. If I am using MD raid arrays to provide physical volumes, what is the best way to set these up?

For example, assume I have 3 1Tb drives. I want to create 2 volume groups, each initially 500GB, with the option of extending either by another 500Gb at a future date,

As I see it, there are two choices:

  1. Multiple raid arrays - The disks are each divided into four partitions, each partition being 250Gb in size. Four raid arrays, each 500Gb, are then assembled using these partitions (e.g. mdadm /dev/md0 -create -lraid5 -n3 /dev/sda1 /dev/sdb1 /dev/sdc1 , mdamd /dev/md1 -create -lraid5 -n3 /dev/sda2 /dev/sdb2 /dev/sdc2 etc. One raid device is then used as the initial physical volume for each volume group. The remaining 2 spare raid devices can be added to whichever volume group may need it in future.
  2. Partitioned raid device(s) - In this case, a single 2Tb raid device is created. This is then partitioned into 4 500Gb partitions (/dev/md0p1 /dev/md0p2 /dev/md0p3 and /dev/md0p4. These are then used as the physical volumes as above, i.e. initially assign one to each VG, etc.

Obviously, there is some overhead in terms of disk space when creating multiple partitions, but this will probably be very similar for either configuration and is negligible compared to the size of the volumes.

However, is there any processing or performance overhead to using partitioned raid devices?

I've always used the first configuration and only recently considered (and experimented with) the second. I haven't noticed any difference but does anyone has any long term experience of this or can point to technical reasons why one configuration is preferred or deprecated?

Improve this question
7
  • I don't think there would be any performance overhead, maybe some administrative overhead. I can't really see how partitions would be the drag point since rotational media is probably going to be your biggest factor and this doesn't add or detract from that at all. Commented Mar 15, 2015 at 17:43
  • Also, if it's just a matter of moving a physical volume to another volume group that needs more space, you should be able to use pvs to select a device with the amount of space you want to move, pvmove any extents off it, vgreduce the source VG, then vgextend the one you're wanting to add the volume to. I would think partitioning an md device would only be useful if they were local disks or negotiating with your SAN team was so time consuming you wanted to just request big volumes up front and use them for all your storage needs. Commented Mar 15, 2015 at 17:46
  • @Bratchley I didn't think that multilevel partitioning would have a noticeable performance impact but was looking for confirmation. This is a home system and they are local disks. Over the past few years, I've been in the habit of separating different sets of files onto different volume groups, but as my setup has evolved, the majority of raid (md) devices are about 700Gb in size (after partitioning the underlying 3Tb disks into 12). I'd rather keep some of the volume groups significantly smaller than this, hence the sub-partitioning of the raid devices. Thanks for the input. Commented Mar 16, 2015 at 21:47
  • Why are you separating the volume groups out like that? Legitimately curious. I usually only create two VG's (one for OS and the rest for application data) at most so I'm wondering what the advantage you're going for is. Commented Mar 17, 2015 at 2:15
  • Mostly historical. Original VGs were OS, home directories, MythTV recordings, ISO+disk images and backup. OS/home separation was from install. Recordings were separate simply to keep a better handle on usage and allow them to be moved to bigger disks as I changed hardware. The ISO and virtual disk images were put in their own VGs so they could be on faster physical disks (I intended to put them on SSDs, but never got around to it). Backup was separate to put VG on 2 mirrored USB drives - keeping a long term backup simply meant unplugging a drive, replacing it and letting the mirror rebuild. Commented Mar 17, 2015 at 11:04

1 Answer 1

Reset to default
0

In general, you should try to have as few VGs as possible (VGs can be spanned across multiple PVs with vgextend). It makes things easier to manage when a single VG spans all PVs and you can just use pvmove to migrate LVs to a different physical device.

What I've found over the years with Software RAID, as you get into the larger disk sizes, is that splitting the larger drives up into smaller partitions and then RAID'ing across the drives works better from a practical standpoint. At the expense of a bit of complexity.

Example: (8) 4TB drives, assembled into a Software RAID10 array with (1) spare. The total net space is going to be about 13TB no matter how we slice the disks. But if we go with one big partition on each drive, rebuild time will be around 8-10 hours for the array. And if it fails or you need to reboot, mdadm will usually start over from the beginning.

But if you divide that up into (4) partitions per disk and build (4) RAID10 arrays (/dev/md101 = sd[abcdefgh]1, /dev/md102 = sd[abcdefgh]2, etc.), then each array will rebuild in about 2-2.5 hours. The total rebuild time is still 8-10 hours. And if mdadm gets interrupted by a reboot, it won't have to redo the arrays that have already completed. Note that all (4) mdadm arrays are PVs and all should belong to the same VG.

There's also a slight performance benefit, because you could force all your heavy access LVs onto the same PV (/dev/md101) which would put the blocks onto the first 25% of the platter. That may net you a small speed increase. You could also use pvmove to defragment the LVs by moving them around.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.