0

How can I get the size of a large directory with lot of small sub-directories on Linux? I just want the total size the parent directory.

I tried du -s mydirectory/ but it is taking a long time to execute and I cannot keep the terminal open for long.

Is there a faster way to get the size of the directory faster or a way to run in the background so that I can close the terminal and get the results later?

Improve this question
4
  • 5
    It's unlikely you can make it much faster, because usually, the bottleneck is simply how fast the data on the storage device can be read, and even command queuing can only do so much. That being said, even large directory trees shouldn't take long to traverse on a modern filesystem. What's the type of file system you're doing this on? And what kind of storage does that file system reside on? Commented Sep 29, 2023 at 14:06
  • @MarcusMüller I may be in an unusual environment but even on NAS shares where the storage system is all NVMe I often see a du taking 2-3 hours. Since the filesystems in questtion may have 1-2PB of occupied space this should be expected. The other parameter is the average size of the files. Traversing a directory tree consisting of 100GB files will be much faster than if the files are all 1MB. Commented Sep 29, 2023 at 17:56
  • @doneal24 yep, I've seen large directory iterations take a while, as well. But that's what is to be investigated here: what's the actual platform, can we identify a bottleneck? I mean, this might be a bunch of independent mounted CIFS shares where parallelization makes sense, or it might just be one humongous ZFS on a bunch of datacenter NVMes and it'd be unlikely (yet not impossible) that it does… I like Joachim's answer with the options to investigate, but he could have possibly written something in-depth, had he the information. Commented Sep 29, 2023 at 17:59
  • 3
    What does "cannot keep the terminal open for long" actually mean? This seems like a bizarre requirement. Commented Sep 30, 2023 at 11:29

4 Answers 4

Reset to default
8

For the nohup solution suggested in Marcus's comment to the screen answer, you find the output in nohup.out. You may want to redirect it to a suitable file, e.g.:

nohup du -s myfolder > size-of-myfolder.txt &
Improve this answer
6

You can use screen in the background:

screen -S du -d -m du -s my_dir/

then to re-attach:

screen -r du
Improve this answer
1
  • 5
    nohup would probably be the less-new-things-to-learn solution :) Commented Sep 29, 2023 at 14:11
2

If you are looking for a solution to being able to track the size of folders quickly on demand, there are some options:

  • The CephFS filesystem shows the total disk usage of the contents of a folder (including subfolders) as the "size" of the folder, rather than showing the size of the storage needed for the entries in the folder. You can simple do ls -Slh to find the biggest files and folders and repeat this in sub-folders. CephFS is meant for clusters with many machines. It is possible to run it on a single machine but is more complicated to set up than a normal filesystem.

  • If not interested in the size of sub-folders, you could use the project quota feature of XFS. Assign each folder of interest to a project to be tracked by project quota. The quota report then shows you how much space each project is using.

  • Similarly, if you can create dedicated user accounts for each folder to track, you could use ordinary user quota to track usage (both size and number of files). This works with a wider range of filesystems than for project quota.

Improve this answer
3
  • 1
    hm, do you have any insights in btrfs / zfs subvolumes, whether they could offer the same sub-tree usage counts as XFS project quotas? Commented Sep 29, 2023 at 18:01
  • oh, btw, the web server linked to in your profile is down. Commented Sep 29, 2023 at 18:01
  • @MarcusMüller (1) No, haven't tried that. A quick look at man btrfs-subvolume and man btrfs-qgroup suggests btrfs subvolume quota can track data usage (and enforce limits) for a subvolme. (2) Thanks, updated http to https in link. Commented Oct 2, 2023 at 14:49
1

Another approach if you use a window manager in POSIX shell in the background:

{ size=$(du -s myfolder/ | cut -d' ' -f1) size=$((size/1024)) zenity --info --text="myfolder/ size is ${size}MiB" } &

See http://mywiki.wooledge.org/ArithmeticExpression

If you put the control operator & at the end of a command, e.g. command &, the shell executes the command in the background in a subshell. The shell does not wait for the command to finish, and the return status is 0. Pid of the last backgrounded command is available via the special variable $!

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.