Return to Answer

If you're using the filesystem as an object store, you might want to look at using a filesystem that specializes in that, possibly to the detriment of other characteristics. A quick Google search found Ceph, which appears to be open source, and can be mounted as a POSIX filesystem, but also accessed with other APIs. IDK if it's worth using on a single host, without taking advantage of replication.

IDK if btrfs is good at this, but I think it may be. I think Facebook employs its lead developer for a reason. :P Some benchmarks I've seen, of stuff like untarring a Linux kernel source, show btrfs does well.

I know reiserfs was optimized for this case, but it's barely if at all maintained anymore. I really can't recommend going with reiser4. It might be interesting to experiment, though. But it's by far the least futureproof choice. I've also seen reports of performance degradation on aged reiserFS, and there's no good defrag tool. (google filesystem millions of small files, and look at some of the existing stackexchange answers.)

If you're using the filesystem as an object store, you might want to look at using a filesystem that specializes in that, possibly to the detriment of other characteristics. A quick Google search found Ceph, which appears to be open source, and can be mounted as a POSIX filesystem, but also accessed with other APIs. I don't know if it's worth using on a single host, without taking advantage of replication.

I don't know if btrfs is good at this, but I think it may be. I think Facebook employs its lead developer for a reason. :P Some benchmarks I've seen, of stuff like untarring a Linux kernel source, show btrfs does well.

I know reiserfs was optimized for this case, but it's barely, if at all, maintained anymore. I really can't recommend going with reiser4. It might be interesting to experiment, though. But it's by far the least future-proof choice. I've also seen reports of performance degradation on aged reiserFS, and there's no good defrag tool. (google filesystem millions of small files, and look at some of the existing stackexchange answers.)

If you're using the filesystem as an object store, you might want to look at using a filesystem that specializes in that, possibly to the detriment of other characteristics. Google found Ceph, which appears to be open source, and can be mounted as a POSIX filesystem, but also accessed with other APIs. IDK if it's worth using on a single host, without taking advantage of replication.

I know reiserfs was optimized for this case, but it's barely if at all maintained anymore. I really can't recommend going with reiser4. It might be interesting to experiment, though. But it's by far the least futureproof choice. I've also seen reports of performance degradation aged reiserFS, and there's no good defrag tool. (google filesystem millions of small files, and look at some of the existing stackexchange answers.)

If you're using the filesystem as an object store, you might want to look at using a filesystem that specializes in that, possibly to the detriment of other characteristics. A quick Google search found Ceph, which appears to be open source, and can be mounted as a POSIX filesystem, but also accessed with other APIs. IDK if it's worth using on a single host, without taking advantage of replication.

I know reiserfs was optimized for this case, but it's barely if at all maintained anymore. I really can't recommend going with reiser4. It might be interesting to experiment, though. But it's by far the least futureproof choice. I've also seen reports of performance degradation on aged reiserFS, and there's no good defrag tool. (google filesystem millions of small files, and look at some of the existing stackexchange answers.)

If you're using the filesystem an object store, you might want to look at using a filesystem that specializes in that, possibly to the detriment of other characteristics. Google found Ceph, which appears to be open source, and can be mounted as a POSIX filesystem, but also accessed with other APIs. IDK if it's worth using on a single host, without taking advantage of replication.

If you already have your software built do use a filesystem directly as an object store, rather than going through something like Swift to handle the replication / load-balancing, then you can at least avoid having all your files in a single directory. (I didn't see Swift docs say how they lay out their files into multiple directories, but I'm certain they do.)

If you're using the filesystem as an object store, you might want to look at using a filesystem that specializes in that, possibly to the detriment of other characteristics. Google found Ceph, which appears to be open source, and can be mounted as a POSIX filesystem, but also accessed with other APIs. IDK if it's worth using on a single host, without taking advantage of replication.

If you already have your software built to use a filesystem directly as an object store, rather than going through something like Swift to handle the replication / load-balancing, then you can at least avoid having all your files in a single directory. (I didn't see Swift docs say how they lay out their files into multiple directories, but I'm certain they do.)