I'm working on a python script that will be accessed via the web, so there will be multiple users trying to append to the same file at the same time. My worry is that this might cause a race condition where if multiple users wrote to the same file at the same time and it just might corrupt the file.
For example:
#!/usr/bin/env python g = open("/somepath/somefile.txt", "a") new_entry = "foobar" g.write(new_entry) g.close Will I have to use a lockfile for this as this operation looks risky.
You can use file locking:
import fcntl new_entry = "foobar" with open("/somepath/somefile.txt", "a") as g: fcntl.flock(g, fcntl.LOCK_EX) g.write(new_entry) fcntl.flock(g, fcntl.LOCK_UN) Note that on some systems, locking is not needed if you're only writing small buffers, because appends on these systems are atomic.
man 2 flockflock will not return until it has acquired a lock.
buffering=0 since I was not opening the file in binary mode, but using flush on the file descriptor before releasing the lock resolved the issue.If you are doing this operation on Linux, and the cache size is smaller than 4KB, the write operation is atomic and you should be good.
More to read here: Is file append atomic in UNIX?
You didn't state what platform you use, but here is an module you can use that is cross platform: File locking in Python
Depending on your platform/filesystem location this may not be doable in a safe manner (e.g. NFS). Perhaps you can write to different files and merge the results afterwards?
mkfifomay be an interesting option.mkfifocreates a FIFO special file. Anyone can write to the file at random, then one single process reads out of the FIFO. That way you don't need to use file locking.O_APPEND, the target filesystem is POSIX-compliant, and your writes are all short enough to be accomplished in a single syscall, there will be no corruption in the first place.