19

This is a more general question about 'chmoding' recursively.

I have this script which at some point needs to change the permissions recursively in a folder which has a few hundred thousand files. There are new files added in that folder every day, but the ones that are already there have the permissions already set and they don't change.

My question is... when I call 

chmod 775 . -R

does it try to set the permission for the files that already have the right permissions set, or only for the new files that don't have the right permissions?

It seems to always take ages to get past this command in the script, even though the 'new' files are only a few thousand and it should do their permissions fairly quickly.

I've looked at the man page for chmod, but it doesn't seem to mention anything on this case.

If chmod doesn't check for permissions beforehand, should I start looking at combining 'find' with 'chmod'?

Improve this question
14
  • 3
    I wonder if it's really slower to check the permissions and change them if they're not correct than directly set them to the correct value. Commented Jun 18, 2013 at 16:46
  • 1
    if anyone stumbles upon this and wants the find + chmod command, here it is: find . ! -perm 775 -print0 | xargs -0 -I {} chmod 775 {} Commented Jun 18, 2013 at 16:46
  • @lgeorget, so you are saying it is slower to use find|chmod? than just to chmod everything. (sorry, didn't understand from your comment). cheers Commented Jun 18, 2013 at 16:48
  • In my humble opinion, it's probably slower as it needs to lauch two processes and redirect the output of the first one to the second but I'm not sure. It depends on the time it takes to set permissions which may not be that important since they are just 3 bytes to modify in the inode. Commented Jun 18, 2013 at 16:53
  • 1
    @depquid The main performance issue here is reading the data into the disk cache. After the first run everything is in the disk cache (unless there is too little memory) thus you are testing the performance of something which is not the bottleneck in the real situation. Commented Jun 18, 2013 at 17:25

5 Answers 5

Reset to default
12

find / chmod optimization

Both find and chmod have to read

  1. all directory entries
  2. the inodes for all these entries

You probably get a performance improvement by first reading all the entries and then all the inodes (on a rotating disk) because then the disk head does not move between the directory and the inodes). As chmod is stupid (as one of the other answers explains) it should be called through find only. But even then it may help to read all the inodes before the first gets written (assuming you have enough free RAM for the disk cache). I suggest this:

find . -printf "" # reading the file names only find . ! -perm 775 -printf "" # reading all the inodes (file names are cached) find . ! -perm 775 -exec chmod 775 + # writing to the cache without reading from disk

The good solution: ACLs

The good solution may be completely different: If the files are created in this directory (and not moved from somewhere else) then ACLs can do the job on the fly. You just have to set the default ACLs on the parent directory.

Further improvement may be reached by filesystem optimizations. If it is ext3/ext4 then you may run e2fsck -D from time to time. Maybe it helps to put this directory onto a separate volume. You may try different filesystems or filesystem settings (e.g. different inode sizes).

Improve this answer
4
  • ACLs are good as long as you are not working on an NFSv4 mount. Commented May 25, 2016 at 13:22
  • The find solution about doubled my time, chmoding inside a docker container. Commented Dec 12, 2017 at 14:55
  • I really wanted to use ACL's to speed up my docker builds, but unfortunately it appears they have no effect in the Dockerfile. stackoverflow.com/questions/59043049/… I then tried using setfacl --recursive commands on startup, but that takes just as long. It appears that the saving is in automatic setting of permissions on file creation. I'm new to ACL's though so I might have missed something. Commented Mar 10, 2021 at 19:21
  • @Programster I am not very familiar with docker. I guess that the ACL changes are somehow not detected as change thus maybe no new image layer is created (that could be checked). I suggest to modify the setfacl call so that it also changes the basic permissions, e.g.: setfacl -m d:o::rwx,o::rwx /opt Commented Mar 10, 2021 at 23:44
10

Assuming the use of chmod from the GNU coreutils package on Ubuntu 12.10.

chmod 775 . -R executes the fchmodat system call for each file that it finds irrespective of whether the permissions need changing or not. I confirmed this by both inspecting the code and using strace chmod 775 . -R (snippet below) to list the actual behaviour.

newfstatat(4, "d", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 fchmodat(4, "d", 0775) = 0 newfstatat(4, "c", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 fchmodat(4, "c", 0775) = 0 newfstatat(4, "a", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 fchmodat(4, "a", 0775) = 0 newfstatat(4, "b", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 fchmodat(4, "b", 0775) = 0

There are a couple of disadvantages of running fchmodat on each file

  • The extra system call will likely become significant if a large number of files are changed. The find/xargs/chmod method mentioned by others will likely be quicker by only changing files that need changing.
  • The call to fchmodat changes the file status modification (ctime) of each file. This will cause every file/inode to change each time and will likely cause excess disk writes. It might be possible to use mount options to stop these excess writes.

A simple experiment shows the ctime changes happening for straight chmod

auser@duncow:/tmp/blah.test$ ls -lc total 0 -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 a -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 b -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 c -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 d auser@duncow:/tmp/blah.test$ chmod 775 . -R auser@duncow:/tmp/blah.test$ ls -lc total 0 -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 a -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 b -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 c -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 d

But this does not change for find/xargs/chmod a few minutes later

auser@duncow:/tmp/blah.test$ date Tue Jun 18 18:27:27 BST 2013 auser@duncow:/tmp/blah.test$ find . ! -perm 775 -print0 | xargs -0 -I {} chmod 775 {} auser@duncow:/tmp/blah.test$ ls -lc total 0 -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 a -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 b -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 c -rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 d

I would always tend to use the find/xargs/chmod version because find gives more control over selecting things.

Improve this answer
9

chmod might or might not change the permissions of files that are already set to what you want, but if not, it would still need to check them to see what their current permissions are[0]. With hundreds of thousands of files, I don't think it would matter either way; the time is most likely being spent by the tools stating every file.

You can try using find to either check for files newer than the last run or files that need chmod to be run, but I don't think you'll get much speed improvement.

If possible for your script, you might be able to get the new files put into a separate directory first, as a "holding" area. Then you can chmod THAT directory (which only has new files), and mv them in with the rest. That should be substantially faster, but unfortunately won't work for every application.

[0] Even if it does try to set the permission of files that don't need any changes, the underlying filesystem probably won't do anything with the request, because it's unnecessary.

Improve this answer
2
  • Thanks for that. I will try the find | chmod version and see if it makes things faster. If not I will try to modify the script to implement a 'holding' folder as you suggested. Commented Jun 18, 2013 at 16:56
  • The reason you wouldn't get a speed improvement is that the inode has to be read both for the ctime and the access rights. Commented Jun 18, 2013 at 16:57
3

coreutils/src/chmod.c#process_file shows that chmod(1) always tries to set the mode and then checks back again with fstatat(2).

Files are processed via fts(3), which has to stat(2) all traversed file system objects beforehand to build its data tree.

Unixlore features in Speeding Up Bulk File Operations measurements where chmod(1) is timed against an find / xargs approach: the latter wins by magnitudes.

Here the command line adapted to the original question:

find . -print0 | xargs -0 chmod 775

Two reasons:

  1. File system traversal is decoupled from operations on the files via the pipe between the two processes, which might even run on different cores.

  2. fts(3) operation is minimized, because xargs(1) 'flattens' out the directory trees.

So yes: you should definitely use find / xargs for a simple solution.

Other options:

  • Play with the umask and the source code of the process(es) writing the new files.

  • If you are using Linux, chances are your system has enabled the inotify kernel subsystem. In this case, you can create a script with an efficient solution via inotifywait(1).

Sidenote: unless you want to execute permissions on your files, I'd suggest modifying the invocation as so:

find . -type f -print0 | xargs -0 chmod 664 find . -type d -print0 | xargs -0 chmod 775

Comment on priming the disk cache with find . -printf "":

This might speed up the execution of the following chmod operations, however depends on available memory and i/o load. So it might work, or not. Decoupling traversal (find) and chmod operation already provides for caching, so priming the cache might be superfluous.

Improve this answer
1

Have you considered changing the process(es) that create the file to have them created with 0775 mode? Look at the umask value in the environment - 0002 may help.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.