Timeline for How to fix intermittant "No space left on device" errors during mv when device has plenty of space?
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 22, 2018 at 3:09 | history | edited | Jeff Schaller♦ | CC BY-SA 3.0 | expanded the IDK's |
| Dec 16, 2017 at 4:33 | history | edited | Peter Cordes | CC BY-SA 3.0 | Don't make it sound like Google *developed* Ceph |
| Nov 23, 2015 at 11:15 | history | edited | Peter Cordes | CC BY-SA 3.0 | a couple typos |
| Aug 14, 2015 at 8:38 | history | edited | Peter Cordes | CC BY-SA 3.0 | XFS might win for Swift because of xattrs performance. |
| Aug 13, 2015 at 7:48 | history | edited | Peter Cordes | CC BY-SA 3.0 | Info on Swift |
| Aug 13, 2015 at 7:40 | comment | added | Peter Cordes | @Chris.Caldwell: I should really copy this answer to a question where it's relevant. There are a few existing ones. I was curious what one is "supposed" to use for object storage, and found some docs about Swift. Apparently it stores objects as separate files on XFS (but with a separate XFS for each disk, not RAID. It handles redundancy itself). | |
| Aug 13, 2015 at 7:39 | history | edited | Peter Cordes | CC BY-SA 3.0 | Info on Swift |
| Aug 13, 2015 at 0:36 | comment | added | Peter Cordes | The way filesystems are designed, a level of directories is less overhead. Two fast lookups in small tables will be faster than one slow lookup in an overflowing table that's storing more data than it was optimized for. Like I said, you don't have to perfectly distribute your files among directories, so you can just take the first 4 characters of your filenames, and insert a /. Hopefully that won't affect too many places in your code. (You do have to make sure directories get created, if creating a new file fails with ENOENT). Ask on serverfault if there are other filesystems. | |
| Aug 13, 2015 at 0:33 | comment | added | Peter Cordes | @Chris.Caldwell: You'd have to check, but I assume BTRFS handles hash collisions by supporting multiple entries in the same hash bucket, rather than ENOSPC. Have you thought about just keeping your stuff in a database, instead of separate files in the filesystem? I haven't ever had to build a system to handle this kind of data. I use XFS, which is great for what I use it for (storing videos, and general purpose Unix source code and stuff.) | |
| Aug 12, 2015 at 18:44 | comment | added | Chris.Caldwell | Thanks so much. I've seen a lot of people use sub-folders, and in fact years ago I had that type of solution on a different setup, but its another layer I was hoping to avoid. It looks like the overhead from doing it that way, however, is going to be far less than finding a fs that just works for this purpose. RE: XFS, its surprising that its so bad at high counts of files since its the knee-jerk answer frequently given. BTRFS, wiki: "directory entries appear as directory items, whose right-hand key values are a CRC32C hash of their filename". isnt that the same problem we have? | |
| Aug 11, 2015 at 23:27 | history | edited | Peter Cordes | CC BY-SA 3.0 | added 484 characters in body |
| Aug 11, 2015 at 23:21 | history | answered | Peter Cordes | CC BY-SA 3.0 |