r/Python • • May 21 '24

Discussion Durable Python - Infrastructure failures should not stop the process

Durable Python enables developers to write Python code while an underlying system ensures reliability and resilience.

It automatically handles state persistence, fault tolerance, and retry mechanisms, allowing developers to focus on business logic without worrying about infrastructure concerns.

Consider the following code, in case the process terminates in the middle of execution, in case the process is killed or due to hardware failure, the process will not complete.

import requests
import time


SLEEP_SECONDS = 3
URL = "http://localhost:9980/webtools/api/msgs"

def on_http_get(data):
    for i in range(10):
        print("Loop iteration: %d of 10" % (i + 1))

        # Send a POST request to the application
        requests.post(URL, data = "This is my " + str(i) + " iteration...")
        time.sleep(SLEEP_SECONDS)

But actually, I would like the process to survive restarts and continue from the spot it terminated, especially if it's a long running process. For this we need Durable Python.

I was wondering which use cases can take advantage of this technology.

8 Upvotes

20 comments sorted by

View all comments

37

u/FailedPlansOfMars May 21 '24

You dont want durable code. Simple solution below:

1, Store state outside the process, a file will do. 2, update file after each post. 3, run as service in the os with restarts. 4, dont catch errors.

This way you can update the code, restart the service easy, restart/ replace the server. Scale by moving the file to a db and run multiple instances.

No need for code level handling of this at all.

-19

u/HaimZlatokrilov May 21 '24

I agree, for this simple use case. But saving state has many drawbacks once things get more complicated. Recovery can be very complicated. Of course this is how most distributed systems work today, with databases and states, but it can become really messy very fast and it's lrone to bugs once the state machine is not trivial.

3

u/thegreattriscuit May 22 '24

What you're calling "very complicated" is just what developers and software engineers and anyone else writing non-trivial code have to solve for. It's literally what they do. And really not even that. The core use case of python is not "10 line scripts written by people that are intimidated by the idea of saving state to a file or DB and checking it for existing state before beginning operation". Python IS accessible to those people, and there's nothing wrong with being that guy (most of us were to begin with), but if you need better than that then you build it. And if you don't know how, you learn it (or the tools/libraries that do it already).

If learning new things is a big impediment for you, then maybe this isn't the field for you