1

Back in 2011 we had a peculiar problem.

We noticed that the processing tool for one of our clients was taking a long time processing ~20000 p/hr. Since we needed to process upwards 3 million files a month possibly in a 3 day span this was a problem for us.

We also noticed that this was not a CPU utilization issue which was around 10 percent. This tool used to write the files to a folder named by the area short code(e.g. DEL for Delhi). A folder could eventually hold anywhere between 5000 to 200000 files once the processing job was over.

We also observed that while files were being written to a folder, if we removed the files that were previously written to that folder(when > ~10000) into a subfolder, that sped up the job considerably.

Eventually the issue was solved when our tool vendor modified the tool to write files to folders named by zipcode within the folder with name of area code(DEL/110012 and so on). Each of these folders now contained to a maximum of approx 2000 files.

Edit 1: The OS used was HP-UX. Edit 2: File size averaged in the range 25-50 KB.

So why does it take longer to add new files to a directory that has a large number of files in it?

Improve this question
8
  • Should we guess what was the file system ? Commented May 1, 2018 at 8:59
  • Sorry not aware of that but the OS was HP-UX. Commented May 1, 2018 at 9:01
  • 1
    Then your question is a bit broad and opinion base, it will depend on how hpux has been setup at this time. Commented May 1, 2018 at 9:13
  • I was wondering if the issue was due to adding the entry corresponding to the file in the directory. As the number of entries increased adding the entry was somehow timetaking(thereby when the files were moved to a subfolder, it became easier to add entries to the directory again) Commented May 1, 2018 at 9:18
  • 1
    @Kiwy, it may lack enough details for people who don't know what would HP-UX use, but that doesn't make it broad. And it certainly is not opinion-based. Commented May 1, 2018 at 13:07

1 Answer 1

Reset to default
0

If there are many files, there is more to search/shuffle around when adding a new file. One way around this is to partition the directory, i.e., create subdirectories for the first character of the name (or thereabouts, git(1) uses the first two hex digits of the object hash which it uses as filename; Fedora package repositories use the first letter in lowercase).

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.