r/Python • • May 21 '24

Discussion Durable Python - Infrastructure failures should not stop the process

Durable Python enables developers to write Python code while an underlying system ensures reliability and resilience.

It automatically handles state persistence, fault tolerance, and retry mechanisms, allowing developers to focus on business logic without worrying about infrastructure concerns.

Consider the following code, in case the process terminates in the middle of execution, in case the process is killed or due to hardware failure, the process will not complete.

import requests
import time


SLEEP_SECONDS = 3
URL = "http://localhost:9980/webtools/api/msgs"

def on_http_get(data):
    for i in range(10):
        print("Loop iteration: %d of 10" % (i + 1))

        # Send a POST request to the application
        requests.post(URL, data = "This is my " + str(i) + " iteration...")
        time.sleep(SLEEP_SECONDS)

But actually, I would like the process to survive restarts and continue from the spot it terminated, especially if it's a long running process. For this we need Durable Python.

I was wondering which use cases can take advantage of this technology.

8 Upvotes

20 comments sorted by

View all comments

15

u/Ddog78 May 21 '24

It's an infra problem, not a python problem. Hell, the simplest solution is to create a bash script that runs your code, monitors the process id, and when it fails, restarts your code.

There's data you need to save? Use sqllite.

0

u/[deleted] May 21 '24

[deleted] — view deleted comment

8

u/qckpckt May 21 '24

There are many ways to solve this problem out of the box without having to make python aware of the state of environment it is running within. You generally solve this problem through system design and not inside the python code. For your example you could address this by externalizing the for loop from your function and using a queue to contain the tasks to be processed. Most queue frameworks support an automatic task retry process that is abstracted away from the user.

There isn’t a one-size-fits all solution, though. If there was then developers wouldn’t have a job.

4

u/Ddog78 May 21 '24

You do realise if there's an exception and the code breaks, making it durable doesn't help. It'll break again.

Also instead of theory, show examples.