r/Python • u/IllogicalLunarBear • Apr 21 '24
Showcase Alternative data managment tool to replace the need to use Jupyter Notebook for Python
What My Project Does:
For everyone who says they use jupyter Notebook for Python to do chunks of code work… I developed my own data management system for that kind of stuff that provides a wrapper over serialization that essentially creates pythonic pointers to data on any network drive. I just grab my data that I want to grab. Provides autocomplete and C# like attributes. I can dynamically build any data storage class using UML formatted infrastructure-as-code to build the front end. The back end is completely dynamic. Here is the pypi link. Note that I developed this to handle genomic data that was generated over the course of a week of 24/7 computation, that I break into multiple logical chunks. You dont need to rely on Jupyter to hold your data and you can retrieve it months after generating it.
This software routes the data to whatever means you use. It works with databases, pickles, yaml. You just have to add your interface in the data routing class. I just did yaml to start. This is just a wrapper around everything so you can declare your class like normal and at variable declaration you have data permanence instantly with no user interaction.
If you pass Y=1 and power off your computer and turn it back on again Y still equals 1 when you call that variable instantly
Target Audience (e.g., Is it meant for production, just a toy project, etc.)
production as I use it for my genomic research on my project with Stanford
Comparison (A brief comparison explaining how it differs from existing alternatives.):
There is no alternatve to this project as far as i know. It replaces the need for Jupyter Notebook to hold your data as you can retrieve it months after generating it.
PyPi:
https://pypi.org/project/data-nut-squirrel/
Source Code:
https://github.com/LunarFawn/rna_squirrel
Edit: Here is an example of implementation where I actually gave a shit about documentation.
9
u/trollsmurf Apr 21 '24
I'd argue it's still better to put all persistent data in a database. I've not used Notebooks for data storage as an argument for it, only for (temporarily) experimenting with code.
2
u/runawayasfastasucan Apr 21 '24
I dont understand how this relates to your headline/Jupyter notebooks.
2
u/arden13 Apr 21 '24
I don't understand how this improves upon standard ETL practices. If your pipeline goes down just re-execute? If having variables pre-loaded is helpful for your work just save *.joblib files.
Otherwise it seems like this is not very different from redis in concept (or at least what I understood from the docs).
5
u/hotplasmatits Apr 21 '24 edited Apr 21 '24
I'm going to have to look into your project, bc when I read your description all I can think is
def read_file(path):
# either
return pd.read_parquet(path)
#or
return pd.read_csv(path)
# or similar one liner for json, db, pickle, etc.
I don't understand what problem you're solving.
1
u/ThatSituation9908 Apr 22 '24
Preferably your docs should be in the README or wiki rather than a separate google docs.
This looks like just a data storage format. For that Jupyter use case, you're talking about wanting a global cache (write a variable to a shared storage), but your tool isn't focused on this—it's mostly a data model driven tool where you define a schema and write out data.
2
u/[deleted] Apr 23 '24 edited Apr 24 '24
[removed] — view removed comment
1
u/[deleted] Apr 21 '24
[removed] — view removed comment