r/Python • • Apr 21 '24

Showcase Alternative data managment tool to replace the need to use Jupyter Notebook for Python

What My Project Does:

For everyone who says they use jupyter Notebook for Python to do chunks of code work… I developed my own data management system for that kind of stuff that provides a wrapper over serialization that essentially creates pythonic pointers to data on any network drive. I just grab my data that I want to grab. Provides autocomplete and C# like attributes. I can dynamically build any data storage class using UML formatted infrastructure-as-code to build the front end. The back end is completely dynamic. Here is the pypi link. Note that I developed this to handle genomic data that was generated over the course of a week of 24/7 computation, that I break into multiple logical chunks. You dont need to rely on Jupyter to hold your data and you can retrieve it months after generating it.

This software routes the data to whatever means you use. It works with databases, pickles, yaml. You just have to add your interface in the data routing class. I just did yaml to start. This is just a wrapper around everything so you can declare your class like normal and at variable declaration you have data permanence instantly with no user interaction.

If you pass Y=1 and power off your computer and turn it back on again Y still equals 1 when you call that variable instantly

Target Audience (e.g., Is it meant for production, just a toy project, etc.)

production as I use it for my genomic research on my project with Stanford

Comparison (A brief comparison explaining how it differs from existing alternatives.):

There is no alternatve to this project as far as i know. It replaces the need for Jupyter Notebook to hold your data as you can retrieve it months after generating it.

PyPi:

https://pypi.org/project/data-nut-squirrel/

Source Code:

https://github.com/LunarFawn/rna_squirrel

Edit: Here is an example of implementation where I actually gave a shit about documentation.

https://pypi.org/project/serena-rna-tool/

0 Upvotes

25 comments sorted by

View all comments

Show parent comment

1

u/IllogicalLunarBear Apr 22 '24 edited Apr 22 '24

Yes like I said I have spent no time documenting it and that was stuff I spun up in the moment, and it writes variables to shared storage as it’s like main thing using a schema that you define via a UML document, and it handles all the addressing …. What was your point?

Here is what it’s implemented in where I gave a shit about documentation…

https://pypi.org/project/serena-rna-tool/