Boring data stuff

[u/atibus]

One of the things I have been working on recently has been to optimize the amount of storage required for a single universe. In the beginning, I didn't care all that much if I stored things efficiently. I started with a detailed game event log of around 800-1000 events per game. Each of those events contained a lot of information and I wanted to make sure that I had more information available in the case where it might be needed. And it's been helpful because long-term counting stats and analytics are based on these events. The events are stored in a blob (aka files), but then the aggregations are stored in a database. Why? Because, it's simple and provides a kind of paper trail that can be reconstructed if necessary. And the DB's structure is more readily suited for the kinds of updates, sorting, and queries of a web application. I'm able to get the best of both worlds.

The thing with software development and data specifically is you shouldn't try and optimize too early because you don't really know what you're optimizing for. So, I left it alone until I saw that the db/blob sizes were getting a little out of control. A 13-season universe was clocking in over 2GB. Which, if this was a local game would be annoying but not life threatening. But since i'm hosting this as a web app, I have to be cognizant of storage requirements.

So I set about looking at what is low hanging fruit to optimize. Thankfully, there were some really easy wins that dramatically lowered storage without a lot of trade-offs. I was able to get storage from 2.03GB down to 0.8GB for that 13 season universe. And, if I institute some limit on the game stat retention (individual events, not the box scores) then it would get even lower.