6

There was a nice question that sadly got deleted while I was writing a rather extensive answer :(

Not wanting to let that effort go to waste, let me paraphrase that question from the question text and comments:

I observe that using dd to overwrite files does increase fragmentation. I'm looking for an alternative to dd that doesn't lead to fragmentation.

As an example for how that fragmentation happens: imagine a file which occupies the entire filesystem. Start overwriting it, you'll immediately see how the partition becomes completely "free", and you'll be able to write another file to it in the meantime. Blocks are allocated dynamically and there's zero guarantee older blocks will be reused when the file gets overwritten.

I observe this behaviour on multiple file systems (ext2, 3 and 4, XFS, as well as FAT32 and NTFS) and on multiple OSes (win95 through modern Fedora)².

I'm convinced this is independent of FS and OS.

My main file system is a Bog standard ext4. FedoraROOT: 103874/1310720 files (0.2% non-contiguous), 1754833/5242880 blocks. Minimum overall fragmentation.

note that I myself cannot observe this, I'm trusting the original asker on these fragmentation claims!

Improve this question
2
  • 1
    Fragmentation is completely normal and can't be avoided in some cases. For example if the filesystem has distributed metadata (like backup superblocks), it will get in the way of large files at some point. So you might get fragmentation even if your filesystem is dedicated to a single file only. And normally no one cares because a few fragments don't affect performance, problems start when you have lots and lots of them. So if you need an unfragmented file for a special use-case you can't just toss a coin and hope for the best - you have to verify. Commented Jun 11, 2022 at 18:53
  • 2
    +1 for reposting this interesting question. However, as far as I had been able so read the question, (OP only since had not time to read comments) I had understood that its author was not complaining against dd because of fragmentation issues but because of performance issues. (using cache, reading one block, then writing it…) as well as inconveniencies in copying several files. Commented Jun 11, 2022 at 19:33

1 Answer 1

Reset to default
4

TL;DR! What's the tool I should use instead of dd?: It's cp, if your GNU coreutils is recent enough.

Actual file systems to the rescue

This is not FS-independent and OS-independent, at all! Allocation of blocks to files is a specific of specific file systems, in which they differ. So, it cannot strictly be independent!

Ext4's delayed allocation to the rescue

Ext4 supports allocate-on-flush: Blocks are allocated to files as the data is flushed to disk, not earlier. This means that if your system (and a modern Fedora most definitely does that) uses file system buffers, the allocation sizes of files that are being sequentially extended are very large, so the fragmentation would be very low. You can activate that, and have to do nothing afterwards, but simply use cp or dd (I'd prefer cp, as that not only solves your first, but also your second question!).

In ext4, that's called delayed allocation; just add the delalloc mount option! (See man ext4)

Using cp will (on a modern Fedora) use a copy_file_range call, which then leads to a contiguous allocation when things are flushed to disk.

XFS to the rescue

XFS does delayed allocation by default; see ext4.

same file system: reflink

When you use cp from Fedora >=34, or a GNU coreutils >=9.0 cp, it has support for reflinks, i.e. simply not copying data, but instead simply marking blocks as used by two files, and making only a copy if one changes. That's pretty nice feature, but of course only works if the source and target file are on the same file system.

The effect is that the target file is exactly as (un-)fragmented as the source file, because it is literally the same blocks.

different file system: XFS allocation groups

XFS doesn't manage the free and used space as "one thing" for the whole file system, it has multiple allocation groups. To cite man xfs:

The data section contains all the filesystem metadata (inodes, directories, indirect blocks) as well as the user file data for ordinary (non-realtime) files and the log area if the log is internal to the data section. The data section is divided into a number of allocation groups. The number and size of the allocation groups are chosen by mkfs.xfs(8) so that there is normally a small number of equal-sized groups. The number of allocation groups controls the amount of parallelism available in file and block allocation. It should be increased from the default if there is sufficient memory and a lot of allocation activity. The number of allocation groups should not be set very high, since this can cause large amounts of CPU time to be used by the filesystem, especially when the filesystem is nearly full. More allocation groups are added (of the original size) when xfs_growfs(8) is run.

So, to solve the "concurrent allocation leads to fragmentation" problem, you simply need enough allocation groups! I'd expect a reasonably sized file system to have a couple, but simply increasing the count from the default (I use 5 for a 4-device striped LVM volume, so that seems to perform well) would work. You'd need to reformat or add more storage, however, to be able to increase the number of allocations groups.

Tools to the rescue

dd isn't the tool of choice for overwriting a file with another: cp is; it will, in its current version (9.0) and the version shipped with your Fedora (8.32 with fedora patches) use copy_file_range, which tells the underlying file system how much data will be copied in the end – so that allocation simply works on block.

Improve this answer
5
  • 2
    if cp doesn't use copy_file_range(), it'll likely just open() the target file with O_WRONLY|O_TRUNC, which plausibly could result in the previously allocated blocks being freed and new allocations made. (That's what e.g. cp from coreutils 8.32 on Ubuntu 22.04 did when I tried it, and I read that as being the original issue.) Something like cat srcfile 1<> dstfile && truncate -r srcfile dstfile might do to actually overwrite the file and truncate it to length afterward. Commented Jun 11, 2022 at 17:19
  • @ilkkachu I checked that! It indeed does openat(AT_FDCWD, "targetfile", O_WRONLY|O_CREAT|O_EXCL, 0644) = 4, says my strace, and then does a bunch of read()/write(). Your cat … 1<> … is elegant, as it might lead to a splice Commented Jun 11, 2022 at 17:25
  • 1
    yes, that makes the suggestion to "just use cp" rather less than helpful. Commented Jun 11, 2022 at 17:29
  • well, I didn't say "just use cp", I said "cp in its current version and the version shipped with your fedora", aimed at the question that actually says "Fedora root file system"! But I didn't do that in the TL;DR, so I need to fix that. Commented Jun 11, 2022 at 17:32
  • @terdon thanks for the edit, you're right, this should stand on its own. Commented Jun 11, 2022 at 18:55

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.