2

I am trying to get my personal filesystem in order

I want to tar everything and get it organized into proper directories

I have many HDDs and a few SSDs

I want to use Btrfs with it's checksum and redundancy features

The question I bring to you now is: How do I configure a data scrubbing daemon that automatically detects when I randomly lose a HDD sector causing an archive to not pass it's checksum verification, then to copy a backup onto another sector.

I'm under the impression that this backup can actually be on the same HDD. It's unlikely I'm going to lose an entire drive all at once with normal aging and wear. In the future I plan to expand my backup filesystem to multiple HDDs. But for now I'm interested in having duplicates scattered across multiple sectors of the same drive.

My end result is to never again have to deal with losing a random JPEG or text file

I do have multiple HDDs I can dedicate towards a single large redundant autocorrecting Btrfs filesystem. But I don't have multiple of the same sized drives

Please link me to some good reading material covering what I'm interested in here

Improve this question
6
  • The drives wouldn't need to be the same size to build a btrfs array. Just go to the btrfs wiki. Commented Jul 5, 2015 at 8:02
  • Yea, I am actually aware that Btrfs doesn't need same sized drives like a classic RAID1. However, wouldn't that limit my duplication benefits to the size of the smaller drive? What about "internal mirroring" where it duplicates the data on a single drive. Commented Jul 5, 2015 at 8:09
  • @nope No, using different-sized drives doesn't necessarily limit the effective capacity to the size of the smaller drive unless you have exactly 2 drives. If you have more than 2 drives, some data can be on 1+2 while other data can be on 3+4 and still other data can be on 1+4, and so on. It can potentially balance out quite well. Commented Jul 5, 2015 at 8:18
  • What about having 2 btrfs partitions on the same physical drive. Then balancing the btrfs filesystem on the two partitions? Or is there a way to do this internal mirroring with a single partition? Commented Jul 5, 2015 at 8:19
  • When you mkfs.btrfs you can tell it what kind of metadata/actual data duplication to apply, and, to some extent, where. do man mkfs.btrfs. And yes, the performance benefits are limited there, but not the duplication benefits. Besides, you can partition a disk to hold more than one member. You can also change this at will w/ a live fs with btrfs balance later on - but that can take awhile. Commented Jul 5, 2015 at 8:19

2 Answers 2

Reset to default
2

How do I configure a data scrubbing daemon that automatically detects when I randomly lose a HDD sector causing an archive to not pass it's checksum verification, then to copy a backup onto another sector.

It's built in to Btrfs. Try:

btrfs scrub start

It has never yet happened to me that an error was found, but I expect that if one is found then:

  • It would be reported in the kernel log output
  • It would be corrected if possible. That is, it would be corrected as long as the storage policy is not single.

Since you have multiple devices, you can use raid1 or raid5 or raid6 as your storage policy for both metadata and data. Be aware that raid5 and raid6 were introduced much more recently and may not be considered as stable (trustworthy) as the rest of Btrfs.

But I don't have multiple of the same sized drives

With Btrfs, that's perfectly OK, unlike with block-level RAID.

Using different-sized drives doesn't necessarily limit the effective capacity to the size of the smaller drive unless you have exactly 2 drives. If you have more than 2 drives, some data can be on 1+2 while other data can be on 3+4 and still other data can be on 1+4, and so on. It can potentially balance out quite well. If it gets out of balance over time (perhaps due to uneven data churn) you can just run btrfs balance later on – but that can take awhile.

Related: ideal btrfs storage scheme to incorporate external USB HDD as backup media

Improve this answer
3
  • great answer! thank you - however, what about having data duplicated with a single drive? would i have to use 2 (or more) partitions and raid1 between them? Commented Jul 5, 2015 at 10:25
  • Yes, you would have to do that since you would need the DUP policy for that (raid1 won't do it) and Btrfs only offers DUP for metadata, not data. Commented Jul 5, 2015 at 10:56
  • 1
    Note that running btrfs scrub start manually will not act like a daemon that detects errors in the future. You have to make sure to run a scrub periodically and check the results. Here's an example cronjob that would email you the results. Commented Nov 14, 2015 at 13:32
0

Side note about the comment telling that "Btrfs only offer DUP for metadata" that it nowadays offers support for DUP for data too (ie even before 2022). https://zejn.net/b/2017/04/30/single-device-data-redundancy-with-btrfs/

# mkfs.btrfs --data dup --metadata dup /dev/sdX
Improve this answer
1

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.