Skip to main content
Improved punctuation, capitalization, wording and formatting; added tag.
Source Link

Hardlinks Hard links seem to take several hundred bytes just for the link itself (not file datanot file data)

I was thinking of implementing the rsync / hardlink / snapshot backup strategy and was wondering how much dataspace a hardlink hard link takes up. Like  Like, it has to put an entry for the extra link as a directory entry, etc. I couldn't  I couldn't find any information on this and, and I guess it is file system dependent. The only  The only info I could could find was suggestionsanecdotal: Either  Either they take no space    (probably meaning they take no space for file contents). Or the space, or the space they take is negligible, so they only take only a few few bytes to store the hardlinkhard link.

I took a couple of systems (one a vm VM and one on real hardware) and did and did the following in the root directory, as root:

The usr directory had about 54,000 files files. The  The space used on the hdHD increased by about 34 MB MB. So  So this works out around 600 bytes bytes per hardlink, orhard link – or am I doing something wrong?

The file name size is about 1.5 MB MB altogether    (I got that by doing ls -R and redirecting it to a file).

The rsync with hardlinkshard links works so well I was I was planning on using it for daily backup on on a couple of the work servers. I also  I also thought it would be easy to make incremental backups / snapshots like this for a considerable period of time. However, after ten days 30 MB MB is 300 MB MB and so on. In addition, if there have only been only a few few changes to the actual file data/contents, say, a few hundred KB KB – then storing 30+ MB MB of hardlinkshard links per day seemed excessive, but I take your point about the size of modern disks. It  It was simply that I had not seen this hardlinkhard link size mentioned anywhere that that I thought thought I may may be doing something wrong. Is 600 bytes  Is 600 bytes normal for a hardlinkhard link on a Linux OS?

To calculate the space used, I did a df before and after the cp -al.

I re-ran the tests again to test the answerGilles's answer (given below that) that a directory directory/folder entry takes up 4 KB KB and this this was skewing my numbers, so so this time by placing 20,000 files files in a single directory and and doing the cp -al to another directory. The results  The results were very different, after.  After taking off the length of the filenames, the hardlinks the hard links worked out to about 13 bytes bytes per hardlinkhard link, much much better than 600. For  For completeness, I did the test again, while while working on the answer given below saying that this is due to each entry for a directory directory/folder taking up 4 KB. But  But this time I created thousands of directories and placed one file in each directory. The result after the maths (increased space taken on hd /HD divided by number of files, ignoring directories) was almost exactly 4 KB KB for each file, showing that a hardlink hard link does only take up a few bytes but that an entry for, but an actual directory/folder takes 4 KB KB.

Hardlinks seem to take several hundred bytes just for the link itself (not file data)

I was thinking of implementing the rsync / hardlink / snapshot backup strategy and was wondering how much data a hardlink takes up. Like it has to put an entry for the extra link as a directory entry etc. I couldn't find any information on this and I guess it is file system dependent. The only info I could find was suggestions: Either they take no space  (probably meaning they take no space for file contents). Or the space they take is negligible, so they only take a few bytes to store the hardlink.

I took a couple of systems (one a vm and one on real hardware) and did the following in the root directory as root:

The usr directory had about 54,000 files. The space used on the hd increased by about 34 MB. So this works out around 600 bytes per hardlink, or am I doing something wrong?

The file name size is about 1.5 MB altogether  (I got that by doing ls -R and redirecting it to a file).

The rsync with hardlinks works so well I was planning on using it for daily backup on a couple of the work servers. I also thought it would be easy to make incremental backups / snapshots like this for a considerable period of time. However, after ten days 30 MB is 300 MB and so on. In addition if there have only been a few changes to the actual file data/contents, say a few hundred KB then storing 30+ MB of hardlinks per day seemed excessive, but I take your point about the size of modern disks. It was simply that I had not seen this hardlink size mentioned anywhere that I thought I may be doing something wrong. Is 600 bytes normal for a hardlink on a Linux OS?

To calculate the space used I did a df before and after the cp -al.

I re-ran the tests again to test the answer below that a directory/folder entry takes up 4 KB and this was skewing my numbers, so this time by placing 20,000 files in a single directory and doing the cp -al to another directory. The results were very different, after taking off the length of the filenames, the hardlinks worked out to about 13 bytes per hardlink, much better than 600. For completeness I did the test again, while working on the answer given below that this is due to each entry for a directory/folder taking up 4 KB. But this time I created thousands of directories and placed one file in each directory. The result after the maths (increased space taken on hd / by number of files, ignoring directories) was almost exactly 4 KB for each file, showing that a hardlink does only take up a few bytes but that an entry for an actual directory/folder takes 4 KB.

Hard links seem to take several hundred bytes just for the link itself (not file data)

I was thinking of implementing the rsync / hardlink / snapshot backup strategy and was wondering how much space a hard link takes up.  Like, it has to put an entry for the extra link as a directory entry, etc.  I couldn't find any information on this, and I guess it is file system dependent.  The only info I could find was anecdotal:  Either they take no space  (probably meaning they take no space for file contents), or the space they take is negligible, so they take only a few bytes to store the hard link.

I took a couple of systems (one a VM and one on real hardware) and did the following in the root directory, as root:

The usr directory had about 54,000 files.  The space used on the HD increased by about 34 MB.  So this works out around 600 bytes per hard link – or am I doing something wrong?

The file name size is about 1.5 MB altogether  (I got that by doing ls -R and redirecting it to a file).

The rsync with hard links works so well I was planning on using it for daily backup on a couple of the work servers.  I also thought it would be easy to make incremental backups / snapshots like this for a considerable period of time. However, after ten days 30 MB is 300 MB and so on. In addition, if there have been only a few changes to the actual file data/contents say, a few hundred KB – then storing 30+ MB of hard links per day seemed excessive, but I take your point about the size of modern disks.  It was simply that I had not seen this hard link size mentioned anywhere that I thought I may be doing something wrong.  Is 600 bytes normal for a hard link on a Linux OS?

To calculate the space used, I did a df before and after the cp -al.

I re-ran the tests again to test Gilles's answer (given below) that a directory/folder entry takes up 4 KB and this was skewing my numbers, so this time by placing 20,000 files in a single directory and doing the cp -al to another directory.  The results were very different.  After taking off the length of the filenames, the hard links worked out to about 13 bytes per hard link, much better than 600.  For completeness, I did the test again, while working on the answer saying that this is due to each directory/folder taking up 4 KB.  But this time I created thousands of directories and placed one file in each directory. The result after the maths (increased space taken on HD divided by number of files, ignoring directories) was almost exactly 4 KB for each file, showing that a hard link does only take up a few bytes, but an actual directory/folder takes 4 KB.

hardlinks Hardlinks seem to take several hundred bytes just for the link itself (not file data)

Additional InfoSetup

Firstly, thank you for all the answers.

So I re-ran the tests again to test the answer below that a directory/folder entry takes up 4KB and this was skewing my numbers, so this time by placing 20,000 files in a single directory and doing the cp -al to another directory. The results were very different, after taking off the length of the filenames, the hardlinks worked out to about 13 bytes per hardlink, much better than 600. Ok, so then for completeness working on the answer given below that this is due to each entry for a directory/folder taking up 4KB I did the test again, but this time I created thousands of directories and placed one file in each directory. The result after the maths (increased space taken on hd / by number of files (ignoring directories) was almost exactly 4KB for each file, showing that a hardlink does only take up a few bytes but that an entry for an actual directory/folder takes 4KB.


So I was thinking of implementing the rsync / hardlink /snapshot snapshot backup strategy and was wondering how much data a hardlink tooktakes up, like. Like it has to put an entry for the extra link as a directory entry etc. Anyway I couldn't seem to find any information on this and I guess it is file system dependent. TheThe only info I could find was suggestions: Either they tooktake no space (probably meaning they take no space for file contents), to. Or the space they take is negligible to, so they only take a few bytes to store the hardlink.

So I took a couple of systems (one a vm and one on real hardware) and did the following in the root directory as root:

The usr directory had about 54,000 files. The space used on the hd increased by about 34MB34 MB. So this works out around 600 bytes per hardlink, or am I doing something wrong?


 

Additional Info

I amI'm using LVM on both systems, formatted as ext4.

The file name size is about 1.5MB5 MB altogether (I got that by doing ls -Rls -R and redirecting it to a file).

To be honest, theThe rsync with hardlinks works so well I was planning on using it for daily backup on a couple of the work servers. I also thought it would be easy to make incremental backups / snapshots like this for a considerable period of time. However, after ten days 30mb30 MB is 300mb300 MB and so on. In addition if there have only been a few changes to the actual file data/contents, say a few hundred KB then storing 30+ MB of hardlinks per day seemed excessive, but I take your point about the size of modern disks. ItIt was simply that I had not seen this hardlink size mentioned anywhere that I thought I may be doing something wrong. Is 600 bytes normal for a hardlink on a Linux OS?

To calculate the space used I did a df before and after the cp -al.

I re-ran the tests again to test the answer below that a directory/folder entry takes up 4 KB and this was skewing my numbers, so this time by placing 20,000 files in a single directory and doing the cp -al to another directory. The results were very different, after taking off the length of the filenames, the hardlinks worked out to about 13 bytes per hardlink, much better than 600. For completeness I did the test again, while working on the answer given below that this is due to each entry for a directory/folder taking up 4 KB. But this time I created thousands of directories and placed one file in each directory. The result after the maths (increased space taken on hd / by number of files, ignoring directories) was almost exactly 4 KB for each file, showing that a hardlink does only take up a few bytes but that an entry for an actual directory/folder takes 4 KB.

hardlinks seem to take several hundred bytes just for the link itself (not file data)

Additional Info

Firstly, thank you for all the answers.

So I re-ran the tests again to test the answer below that a directory/folder entry takes up 4KB and this was skewing my numbers, so this time by placing 20,000 files in a single directory and doing the cp -al to another directory. The results were very different, after taking off the length of the filenames, the hardlinks worked out to about 13 bytes per hardlink, much better than 600. Ok, so then for completeness working on the answer given below that this is due to each entry for a directory/folder taking up 4KB I did the test again, but this time I created thousands of directories and placed one file in each directory. The result after the maths (increased space taken on hd / by number of files (ignoring directories) was almost exactly 4KB for each file, showing that a hardlink does only take up a few bytes but that an entry for an actual directory/folder takes 4KB.


So I was thinking of implementing the rsync / hardlink /snapshot backup strategy and was wondering how much data a hardlink took up, like it has to put an entry for the extra link as a directory entry etc. Anyway I couldn't seem to find any information on this and I guess it is file system dependent. The only info I could find was suggestions they took no space (probably meaning they take no space for file contents), to the space they take is negligible to they only take a few bytes to store the hardlink.

So I took a couple of systems (one a vm and one on real hardware) and did the following in the root directory as root:

The usr directory had about 54,000 files. The space used on the hd increased by about 34MB. So this works out around 600 bytes per hardlink, or am I doing something wrong?


 

I am using LVM on both systems, formatted as ext4.

The file name size is about 1.5MB altogether (I got that by doing ls -R and redirecting it to a file).

To be honest, the rsync with hardlinks works so well I was planning on using it for daily backup on a couple of the work servers. I also thought it would be easy to make incremental backups / snapshots like this for a considerable period of time. However, after ten days 30mb is 300mb and so on. In addition if there have only been a few changes to the actual file data/contents, say a few hundred KB then storing 30+ MB of hardlinks per day seemed excessive, but I take your point about the size of modern disks. It was simply that I had not seen this hardlink size mentioned anywhere that I thought I may be doing something wrong. Is 600 bytes normal for a hardlink on a Linux OS?

To calculate the space used I did a df before and after the cp -al.

Hardlinks seem to take several hundred bytes just for the link itself (not file data)

Setup

I was thinking of implementing the rsync / hardlink / snapshot backup strategy and was wondering how much data a hardlink takes up. Like it has to put an entry for the extra link as a directory entry etc. I couldn't find any information on this and I guess it is file system dependent. The only info I could find was suggestions: Either they take no space (probably meaning they take no space for file contents). Or the space they take is negligible, so they only take a few bytes to store the hardlink.

I took a couple of systems (one a vm and one on real hardware) and did the following in the root directory as root:

The usr directory had about 54,000 files. The space used on the hd increased by about 34 MB. So this works out around 600 bytes per hardlink, or am I doing something wrong?

Additional Info

I'm using LVM on both systems, formatted as ext4.

The file name size is about 1.5 MB altogether (I got that by doing ls -R and redirecting it to a file).

The rsync with hardlinks works so well I was planning on using it for daily backup on a couple of the work servers. I also thought it would be easy to make incremental backups / snapshots like this for a considerable period of time. However, after ten days 30 MB is 300 MB and so on. In addition if there have only been a few changes to the actual file data/contents, say a few hundred KB then storing 30+ MB of hardlinks per day seemed excessive, but I take your point about the size of modern disks. It was simply that I had not seen this hardlink size mentioned anywhere that I thought I may be doing something wrong. Is 600 bytes normal for a hardlink on a Linux OS?

To calculate the space used I did a df before and after the cp -al.

I re-ran the tests again to test the answer below that a directory/folder entry takes up 4 KB and this was skewing my numbers, so this time by placing 20,000 files in a single directory and doing the cp -al to another directory. The results were very different, after taking off the length of the filenames, the hardlinks worked out to about 13 bytes per hardlink, much better than 600. For completeness I did the test again, while working on the answer given below that this is due to each entry for a directory/folder taking up 4 KB. But this time I created thousands of directories and placed one file in each directory. The result after the maths (increased space taken on hd / by number of files, ignoring directories) was almost exactly 4 KB for each file, showing that a hardlink does only take up a few bytes but that an entry for an actual directory/folder takes 4 KB.

re-ran tests using lots of files in one dir and then lots of dirs containing just one file
Source Link
jamie
  • 53
  • 3

Additional Info

Firstly, thank you for all the answers.

So I re-ran the tests again to test the answer below that a directory/folder entry takes up 4KB and this was skewing my numbers, so this time by placing 20,000 files in a single directory and doing the cp -al to another directory. The results were very different, after taking off the length of the filenames, the hardlinks worked out to about 13 bytes per hardlink, much better than 600. Ok, so then for completeness working on the answer given below that this is due to each entry for a directory/folder taking up 4KB I did the test again, but this time I created thousands of directories and placed one file in each directory. The result after the maths (increased space taken on hd / by number of files (ignoring directories) was almost exactly 4KB for each file, showing that a hardlink does only take up a few bytes but that an entry for an actual directory/folder takes 4KB.


So I was thinking of implementing the rsync / hardlink /snapshot backup strategy and was wondering how much data a hardlink took up, like it has to put an entry for the extra link as a directory entry etc. Anyway I couldn't seem to find any information on this and I guess it is file system dependent. The only info I could find was suggestions they took no space (probably meaning they take no space for file contents), to the space they take is negligible to they only take a few bytes to store the hardlink.

So I took a couple of systems (one a vm and one on real hardware) and did the following in the root directory as root:

mkdir link cp -al usr link 

The usr directory had about 54,000 files. The space used on the hd increased by about 34MB. So this works out around 600 bytes per hardlink, or am I doing something wrong?


I am using LVM on both systems, formatted as ext4.

The file name size is about 1.5MB altogether (I got that by doing ls -R and redirecting it to a file).

To be honest, the rsync with hardlinks works so well I was planning on using it for daily backup on a couple of the work servers. I also thought it would be easy to make incremental backups / snapshots like this for a considerable period of time. However, after ten days 30mb is 300mb and so on. In addition if there have only been a few changes to the actual file data/contents, say a few hundred KB then storing 30+ MB of hardlinks per day seemed excessive, but I take your point about the size of modern disks. It was simply that I had not seen this hardlink size mentioned anywhere that I thought I may be doing something wrong. Is 600 bytes normal for a hardlink on a Linux OS?

To calculate the space used I did a df before and after the cp -al.

So I was thinking of implementing the rsync / hardlink /snapshot backup strategy and was wondering how much data a hardlink took up, like it has to put an entry for the extra link as a directory entry etc. Anyway I couldn't seem to find any information on this and I guess it is file system dependent. The only info I could find was suggestions they took no space (probably meaning they take no space for file contents), to the space they take is negligible to they only take a few bytes to store the hardlink.

So I took a couple of systems (one a vm and one on real hardware) and did the following in the root directory as root:

mkdir link cp -al usr link 

The usr directory had about 54,000 files. The space used on the hd increased by about 34MB. So this works out around 600 bytes per hardlink, or am I doing something wrong?


I am using LVM on both systems, formatted as ext4.

The file name size is about 1.5MB altogether (I got that by doing ls -R and redirecting it to a file).

To be honest, the rsync with hardlinks works so well I was planning on using it for daily backup on a couple of the work servers. I also thought it would be easy to make incremental backups / snapshots like this for a considerable period of time. However, after ten days 30mb is 300mb and so on. In addition if there have only been a few changes to the actual file data/contents, say a few hundred KB then storing 30+ MB of hardlinks per day seemed excessive, but I take your point about the size of modern disks. It was simply that I had not seen this hardlink size mentioned anywhere that I thought I may be doing something wrong. Is 600 bytes normal for a hardlink on a Linux OS?

To calculate the space used I did a df before and after the cp -al.

Additional Info

Firstly, thank you for all the answers.

So I re-ran the tests again to test the answer below that a directory/folder entry takes up 4KB and this was skewing my numbers, so this time by placing 20,000 files in a single directory and doing the cp -al to another directory. The results were very different, after taking off the length of the filenames, the hardlinks worked out to about 13 bytes per hardlink, much better than 600. Ok, so then for completeness working on the answer given below that this is due to each entry for a directory/folder taking up 4KB I did the test again, but this time I created thousands of directories and placed one file in each directory. The result after the maths (increased space taken on hd / by number of files (ignoring directories) was almost exactly 4KB for each file, showing that a hardlink does only take up a few bytes but that an entry for an actual directory/folder takes 4KB.


So I was thinking of implementing the rsync / hardlink /snapshot backup strategy and was wondering how much data a hardlink took up, like it has to put an entry for the extra link as a directory entry etc. Anyway I couldn't seem to find any information on this and I guess it is file system dependent. The only info I could find was suggestions they took no space (probably meaning they take no space for file contents), to the space they take is negligible to they only take a few bytes to store the hardlink.

So I took a couple of systems (one a vm and one on real hardware) and did the following in the root directory as root:

mkdir link cp -al usr link 

The usr directory had about 54,000 files. The space used on the hd increased by about 34MB. So this works out around 600 bytes per hardlink, or am I doing something wrong?


I am using LVM on both systems, formatted as ext4.

The file name size is about 1.5MB altogether (I got that by doing ls -R and redirecting it to a file).

To be honest, the rsync with hardlinks works so well I was planning on using it for daily backup on a couple of the work servers. I also thought it would be easy to make incremental backups / snapshots like this for a considerable period of time. However, after ten days 30mb is 300mb and so on. In addition if there have only been a few changes to the actual file data/contents, say a few hundred KB then storing 30+ MB of hardlinks per day seemed excessive, but I take your point about the size of modern disks. It was simply that I had not seen this hardlink size mentioned anywhere that I thought I may be doing something wrong. Is 600 bytes normal for a hardlink on a Linux OS?

To calculate the space used I did a df before and after the cp -al.

Tweeted twitter.com/#!/StackUnix/status/620407178910941184
deleted 21 characters in body; edited tags
Source Link
Gilles 'SO- stop being evil'
  • 865.9k
  • 205
  • 1.8k
  • 2.3k
Loading
appended answer 215539 as supplemental
Source Link
Michael Mrozek
  • 95.8k
  • 40
  • 245
  • 236
Loading
grammar
Source Link
ctrl-alt-delor
  • 28.8k
  • 11
  • 66
  • 113
Loading
Source Link
jamie
  • 53
  • 3
Loading