Skip to main content
Unix & Linux

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

4
  • 1
    Thanks so much. I've seen a lot of people use sub-folders, and in fact years ago I had that type of solution on a different setup, but its another layer I was hoping to avoid. It looks like the overhead from doing it that way, however, is going to be far less than finding a fs that just works for this purpose. RE: XFS, its surprising that its so bad at high counts of files since its the knee-jerk answer frequently given. BTRFS, wiki: "directory entries appear as directory items, whose right-hand key values are a CRC32C hash of their filename". isnt that the same problem we have? Commented Aug 12, 2015 at 18:44
  • @Chris.Caldwell: You'd have to check, but I assume BTRFS handles hash collisions by supporting multiple entries in the same hash bucket, rather than ENOSPC. Have you thought about just keeping your stuff in a database, instead of separate files in the filesystem? I haven't ever had to build a system to handle this kind of data. I use XFS, which is great for what I use it for (storing videos, and general purpose Unix source code and stuff.) Commented Aug 13, 2015 at 0:33
  • 1
    The way filesystems are designed, a level of directories is less overhead. Two fast lookups in small tables will be faster than one slow lookup in an overflowing table that's storing more data than it was optimized for. Like I said, you don't have to perfectly distribute your files among directories, so you can just take the first 4 characters of your filenames, and insert a /. Hopefully that won't affect too many places in your code. (You do have to make sure directories get created, if creating a new file fails with ENOENT). Ask on serverfault if there are other filesystems. Commented Aug 13, 2015 at 0:36
  • @Chris.Caldwell: I should really copy this answer to a question where it's relevant. There are a few existing ones. I was curious what one is "supposed" to use for object storage, and found some docs about Swift. Apparently it stores objects as separate files on XFS (but with a separate XFS for each disk, not RAID. It handles redundancy itself). Commented Aug 13, 2015 at 7:40