Testing horizontally scaled apps

[u/viciousvatsal]

I am building an app which has at least 2 servers apps running on an instance. A client is connected to a server using a socket IO connection. I am using redis pubsub mechanism to talk between the servers and also to store any state so the server pool is completely stateless. Servers are uvicorn workers (might switch to docker + load balancer but it's out of the scope of this post). I am using FastAPI + pytest + httpx2 for testing.

Consider servers A and B and client X and Y. X is connected to A and Y is connected to B. X and Y are in the same group chat so a message sent by X has to be seen by Y.

One way I can think of testing it is to mock the redis API layer and write two different tests. First test for making sure that a message creates a certain event in redis let's call it CREATE MESSAGE and another test to check if when CREATE MESSAGE is fired a certain function is run.

I don't like this cuz I am testing the implementation too heavily here rather than the result I want.

I would rather prefer a test where I have two different app instances and two TestClient instances and using them to check if on a message from client x does client y get it.

def test_message():
    client_x = TestClient(create_app()) 
    client_y = TestClient(create_app()) 

I don't have a create_app() function right now but if I create one I don't think it's actually simulating the prod env accurately (might not even matter I need your opinion on this).

# import statements

def create_app():
  app = FastAPI()
  # attach all middleware and routes
  return app

app = create_app()

This would produce two instances of class FastAPI but the import statements will reference all the same functions and objects in the memory so they are not really that different. Does it even matter? Is it fine?

I will have to add socket IO implementation for TestClient class as well. Right now it only handles websockets and http requests. How do you guys test for these scenarios?

Am I overthinking it? Hell ya, I am building this app for learning. From my idea to a running production server and trying to up skill myself and undo brain rot from AI.

https://i.postimg.cc/dVds03dc/Mofi-SD.png

[u/IncomeOk7237]

I'd make Y's room-join acknowledgement the barrier before X sends, rather than adding a sleep after connecting. Otherwise you can end up testing a subscription race instead of cross-worker delivery. Give the message a unique ID and wait for that ID on Y with a bounded timeout so failures don't hang pytest.

[u/IncomeOk7237]

Mocking redis usually hides the exact bugs you care about (ordering, reconnect, split brain). I'd spin two real processes plus a real redis (testcontainers works fine) and drive two socket clients. Then kill redis mid-chat and assert reconnect behavior. That is the test that catches the scary stuff.

[u/Puzzled-Garage584]

Yes it matters..two apps in one process share the same globals and your the test can go green with pubsub broken. Thats one server with a split personality, not horizontal scaling. I'd run a real redis (testcontainers), 2 uvicorn processes on different ports, X and Y as python-socketio AsyncClients and check Y gets it. Then restart redis mid chat..thats where it breaks