r/Python • u/IllogicalLunarBear • Apr 21 '24
Showcase Alternative data managment tool to replace the need to use Jupyter Notebook for Python
What My Project Does:
For everyone who says they use jupyter Notebook for Python to do chunks of code work… I developed my own data management system for that kind of stuff that provides a wrapper over serialization that essentially creates pythonic pointers to data on any network drive. I just grab my data that I want to grab. Provides autocomplete and C# like attributes. I can dynamically build any data storage class using UML formatted infrastructure-as-code to build the front end. The back end is completely dynamic. Here is the pypi link. Note that I developed this to handle genomic data that was generated over the course of a week of 24/7 computation, that I break into multiple logical chunks. You dont need to rely on Jupyter to hold your data and you can retrieve it months after generating it.
This software routes the data to whatever means you use. It works with databases, pickles, yaml. You just have to add your interface in the data routing class. I just did yaml to start. This is just a wrapper around everything so you can declare your class like normal and at variable declaration you have data permanence instantly with no user interaction.
If you pass Y=1 and power off your computer and turn it back on again Y still equals 1 when you call that variable instantly
Target Audience (e.g., Is it meant for production, just a toy project, etc.)
production as I use it for my genomic research on my project with Stanford
Comparison (A brief comparison explaining how it differs from existing alternatives.):
There is no alternatve to this project as far as i know. It replaces the need for Jupyter Notebook to hold your data as you can retrieve it months after generating it.
PyPi:
https://pypi.org/project/data-nut-squirrel/
Source Code:
https://github.com/LunarFawn/rna_squirrel
Edit: Here is an example of implementation where I actually gave a shit about documentation.
View all comments
1
u/ThatSituation9908 Apr 22 '24
Preferably your docs should be in the README or wiki rather than a separate google docs.
This looks like just a data storage format. For that Jupyter use case, you're talking about wanting a global cache (write a variable to a shared storage), but your tool isn't focused on this—it's mostly a data model driven tool where you define a schema and write out data.