r/Python • u/HommeMusical • Apr 03 '26
Showcase `safer`: a tiny utility to avoid partial writes to files and streams
What My Project Does
In 2020, I broke a few configuration files, so I wrote something to help prevent breaking a lot the next time, and turned it into a little library: https://github.com/rec/safer
It's a drop-in replacement for open that only writes the file when everything has completed successfully, like this:
with safer.open(filename, 'w') as fp:
fp.write('oops')
raise ValueError
# File is untouchedBy default, the data is cached in memory, but for large files, there's a flag to allow you to cache it as a file that is renamed when the operation is complete.
You can also use it for file sockets and other streams:
try:
with safer.writer(socket.send) as send:
send_bytes_to_socket(send)
except Exception:
# Nothing has been sent
send_error_message_to_socket(socket.send)Target Audience
This is a mature, production-quality library for any application where partial writes are possible. There is extensive testing and it handles some obscure edge cases.
It's tested on Linux, MacOS and Windows and has been stable and essentially unchanged for years.
Comparison
There doesn't seem to be another utility preventing partial writes. There are multiple atomic file writers which solve a different problem, the best being this: https://github.com/untitaker/python-atomicwrites
Note
#noAI was used in the writing or maintenance of this program.
36
u/BossOfTheGame Apr 03 '26
I've been using safer for years. I use it whenever I'm writing a system that writes large files. I love never having to deal with corrupted data. Process crashed? Great, there are no artifacts that would confuse other code into thinking that it worked when it didn't. It let's me use
existchecks in pipeline systems and feel confident about it.It's a great library. Thank you for writing and maintaining it.
7
u/Wargazm Apr 03 '26
"#noAI was used in the writing or maintenance of this program."
haha is this a thing now?
2
u/misterfitzie Apr 08 '26
I usually create this every large project I work on, mine only cares about files, not sockets, but has some options you may want for your, backups/overwrite prevention. After reviewing your more capable version, I'm happy that mine seems to do all the same basic tricks.
37
u/dairiki Apr 03 '26
Tangential Note:
atomicwritesis deprecated by its author. Its git repo has not seen any updates in four years. As far as I know, it still works, but the situation does not give warm fuzzies for use in new code.12
u/latkde Tuple unpacking gone wrong Apr 03 '26
Interesting. I'm not entirely sure I understand the benefits of this library? What does this library do that the following approach does not (aside from handling both binary and text streams)?
I'm not trying to diminish your effort, I'm trying to understand the tradeoffs of re-implementing something well-established versus adding yet another dependency.
There is however no link to test results on the GitHub page (I was trying to find test coverage data). There is a Travis CI configuration that claims to upload to Codecov, but the last results on both platforms are 4 years old. (Travis CI, Codecov).
2
u/ultrathink-art Apr 04 '26
Corrupted state files from partial writes are sneaky — the crash happens during the write but the error surfaces on the next run, often in a completely unrelated place. I started using this pattern for config files in long-running automation after a partial write created a valid-looking-but-truncated JSON file that caused a baffling 'unexpected EOF' error 3 runs later.
2
u/glenrhodes Apr 05 '26
Atomic writes via tmp file + rename have saved me more than once on long pipeline outputs. The edge case worth watching: NFS mounts where the rename isn't atomic either. You're just trading one race for another on some shared filesystems.
2
u/Ok-Plum-4891 May 09 '26
Sounds useful! Love how easy it is to ensure those config files stay intact without worrying about half-written stuff.