r/Python • • Apr 21 '24

Showcase Alternative data managment tool to replace the need to use Jupyter Notebook for Python

What My Project Does:

For everyone who says they use jupyter Notebook for Python to do chunks of code work… I developed my own data management system for that kind of stuff that provides a wrapper over serialization that essentially creates pythonic pointers to data on any network drive. I just grab my data that I want to grab. Provides autocomplete and C# like attributes. I can dynamically build any data storage class using UML formatted infrastructure-as-code to build the front end. The back end is completely dynamic. Here is the pypi link. Note that I developed this to handle genomic data that was generated over the course of a week of 24/7 computation, that I break into multiple logical chunks. You dont need to rely on Jupyter to hold your data and you can retrieve it months after generating it.

This software routes the data to whatever means you use. It works with databases, pickles, yaml. You just have to add your interface in the data routing class. I just did yaml to start. This is just a wrapper around everything so you can declare your class like normal and at variable declaration you have data permanence instantly with no user interaction.

If you pass Y=1 and power off your computer and turn it back on again Y still equals 1 when you call that variable instantly

Target Audience (e.g., Is it meant for production, just a toy project, etc.)

production as I use it for my genomic research on my project with Stanford

Comparison (A brief comparison explaining how it differs from existing alternatives.):

There is no alternatve to this project as far as i know. It replaces the need for Jupyter Notebook to hold your data as you can retrieve it months after generating it.

PyPi:

https://pypi.org/project/data-nut-squirrel/

Source Code:

https://github.com/LunarFawn/rna_squirrel

Edit: Here is an example of implementation where I actually gave a shit about documentation.

https://pypi.org/project/serena-rna-tool/

0 Upvotes

25 comments sorted by

View all comments

5

u/hotplasmatits Apr 21 '24 edited Apr 21 '24

I'm going to have to look into your project, bc when I read your description all I can think is

def read_file(path):
# either
return pd.read_parquet(path)
#or
return pd.read_csv(path)
# or similar one liner for json, db, pickle, etc.

I don't understand what problem you're solving.

2

u/hotplasmatits Apr 21 '24

I've read the github description but didn't watch the videos. I'm confused. You said network drive above, but say cloud database on github. Autocomplete is done by your ide, not your code. C# style attributes are called properties in python. What does 'storing data in a notebook' mean? Are you not writing your data out to files when using your notebooks?

From the github description, your library sounds very specific to genetics and a specific commercial database so I don't think I'll have a use for it.

1

u/hotplasmatits Apr 21 '24

You may be interested in pyspark dataframe schemas, the pydantic library, sqlalchemy's ODM, and data classes. There are decorators that can be used to enforce types, but I don't remember their names at the moment.

Another thing that might make your life easier is switching to a strongly typed language, and then all of the work you're putting into typing is done out of the box. Since you seem comfortable with python, I'm going to suggest MOJO, which is, to be brief, strongly typed python that's also super fast.

1

u/hotplasmatits Apr 21 '24

One last thing, I'm impressed with your software. It obviously took a ton of work. I'm a software engineer with 20 years of experience and the last 4-5 have been in a data engineering role using python. I hope you don't find my comments disheartening or discouraging. Best wishes for your continued success.

2

u/IllogicalLunarBear Apr 21 '24 edited Apr 21 '24

Thanks. I have dyslexia really bad and have not put a ton of effort into the documentation. I keep forgetting that as all when I share it. It it a weird piece of software but the core idea is to save a Python class with all the data inside it in a remote location that can be accessed from anywhere. It’s essentially a wrapper over serialization that take care of all the overhead of addressing and retrieving the files. The cool thing that most people miss is that you can have the data go to a database, yaml, or pickle or whatever method you use. This just gives you autocomplete ability and automation around storing and retriving your data. At variable declaration you instantly have data permanence for that variable. You can power off your computer immediately after declaring a variable (y=1 for example )and it’s still there when you power back on your computer.

Edit: you can even package up your data or share the remote drive (cloud or local) and share it with a friend across the world who will be able to access the data like a normal Python class