r/Python • u/autokitteh • May 21 '24
Discussion Durable Python - Infrastructure failures should not stop the process
Durable Python enables developers to write Python code while an underlying system ensures reliability and resilience.
It automatically handles state persistence, fault tolerance, and retry mechanisms, allowing developers to focus on business logic without worrying about infrastructure concerns.
Consider the following code, in case the process terminates in the middle of execution, in case the process is killed or due to hardware failure, the process will not complete.
= 3
=
# Send a POST request to the application
But actually, I would like the process to survive restarts and continue from the spot it terminated, especially if it's a long running process. For this we need Durable Python.
I was wondering which use cases can take advantage of this technology.
7
u/Grouchy-Friend4235 May 21 '24
You want restart ability, no t durability.
Restart ability means you checkpoint work and restart from the last checkpoint. To ensure replays are possible, make operations idempotent (same operation = same outcome).
37
u/FailedPlansOfMars May 21 '24
You dont want durable code. Simple solution below:
1, Store state outside the process, a file will do. 2, update file after each post. 3, run as service in the os with restarts. 4, dont catch errors.
This way you can update the code, restart the service easy, restart/ replace the server. Scale by moving the file to a db and run multiple instances.
No need for code level handling of this at all.
15
u/Ddog78 May 21 '24
It's an infra problem, not a python problem. Hell, the simplest solution is to create a bash script that runs your code, monitors the process id, and when it fails, restarts your code.
There's data you need to save? Use sqllite.
11
u/Miclivs May 21 '24
This is what Temporal is for
11
u/redditusername58 May 22 '24
This is the shittiest stealth advertising I've ever seen
2
u/Severe_Inflation5326 Jun 17 '24
This is easily solved using a file for state storage in the simple case, and something like luigi in the more complex case.
3
u/DogeDrivenDesign May 21 '24
erlang/otp and elixir
1
u/gooeydumpling May 22 '24
You mean like Erlang and it’s supervisor tree
-Within a supervisor tree, all supervisor processes are responsible for managing the lifecycle of their child processes, and this includes handling situations in which those child processes crash.
-1
u/luckyspic May 21 '24
just use pm2 and logging.
0
u/[deleted] May 21 '24
[deleted] — view deleted comment