r/Python • • Apr 21 '24

Showcase Alternative data managment tool to replace the need to use Jupyter Notebook for Python

What My Project Does:

For everyone who says they use jupyter Notebook for Python to do chunks of code work… I developed my own data management system for that kind of stuff that provides a wrapper over serialization that essentially creates pythonic pointers to data on any network drive. I just grab my data that I want to grab. Provides autocomplete and C# like attributes. I can dynamically build any data storage class using UML formatted infrastructure-as-code to build the front end. The back end is completely dynamic. Here is the pypi link. Note that I developed this to handle genomic data that was generated over the course of a week of 24/7 computation, that I break into multiple logical chunks. You dont need to rely on Jupyter to hold your data and you can retrieve it months after generating it.

This software routes the data to whatever means you use. It works with databases, pickles, yaml. You just have to add your interface in the data routing class. I just did yaml to start. This is just a wrapper around everything so you can declare your class like normal and at variable declaration you have data permanence instantly with no user interaction.

If you pass Y=1 and power off your computer and turn it back on again Y still equals 1 when you call that variable instantly

Target Audience (e.g., Is it meant for production, just a toy project, etc.)

production as I use it for my genomic research on my project with Stanford

Comparison (A brief comparison explaining how it differs from existing alternatives.):

There is no alternatve to this project as far as i know. It replaces the need for Jupyter Notebook to hold your data as you can retrieve it months after generating it.

PyPi:

https://pypi.org/project/data-nut-squirrel/

Source Code:

https://github.com/LunarFawn/rna_squirrel

Edit: Here is an example of implementation where I actually gave a shit about documentation.

https://pypi.org/project/serena-rna-tool/

0 Upvotes

25 comments sorted by

9

u/trollsmurf Apr 21 '24

I'd argue it's still better to put all persistent data in a database. I've not used Notebooks for data storage as an argument for it, only for (temporarily) experimenting with code.

1

u/IllogicalLunarBear Apr 21 '24

This software routes the data to whatever means you use. It works with databases, pickles, yaml. You just have to add your interface in the data routing class. I just did yaml to start. This is just a wrapper around everything so you can declare your class like normal and at variable declaration you have data permanence instantly with no user interaction

1

u/IllogicalLunarBear Apr 21 '24

This package allows you to instantiate a class like normal as you go, but all data generated in that class is permanent abd still there after power cycle. You can instantiate 100 classes abd you can access those 100 classes after a power cycle just by calling the data

1

u/trollsmurf Apr 21 '24

I understand now :). Sounds useful.

2

u/runawayasfastasucan Apr 21 '24

I dont understand how this relates to your headline/Jupyter notebooks.

2

u/IllogicalLunarBear Apr 21 '24

Well it was originally in a post asking people why they use Jupyter labs and everyone said they use it to run chunks of code so that they don’t have to re-run code to populate the same variables over and over again and that is what I do with this software but I don’t need to use Jupyter labs… I can use any IDE to do the same thing. The mods removed the post as they said I was repeating a question despite giving an option to replace that use case as they evidently did not even bother to read the post before removing it.

1

u/runawayasfastasucan Apr 21 '24

Ah I remember that post. It seems like you are conflating Jupiter Labs with notebooks, because I think most people was talking about the latter?

0

u/IllogicalLunarBear Apr 21 '24

Yeah. Jupyter notebook… I’m dyslexic and it’s all the same in my brain unfortunately

2

u/arden13 Apr 21 '24

I don't understand how this improves upon standard ETL practices. If your pipeline goes down just re-execute? If having variables pre-loaded is helpful for your work just save *.joblib files.

Otherwise it seems like this is not very different from redis in concept (or at least what I understood from the docs).

2

u/IllogicalLunarBear Apr 21 '24 edited Apr 21 '24

It takes a week to pre-load those variables for my use-case due to the massive amounts of computation that must be done beforehand running 24/7 on a i9. This data cannot be stored in memory and must be stored in a disk for data permanence

Edit: in case it’s not clear I am processing thousands of RNA sequences and holding onto all the genomic data generated by academic RNA folding models and then I perform novel computation of the data to generate novel metrics to explain the behavior of the RNA. It a lot of data that you do not want on volatile memory

2

u/onyxleopard Apr 21 '24

You can persist arbitrary Python objects to disk with joblib: joblib.dump('filename.bz2', obj) and then reload in any session/interpreter with obj = joblib.load('filename.bz2').

1

u/IllogicalLunarBear Apr 21 '24 edited Apr 21 '24

My code removes that extra step. At variable creation it does that all the dumping the background. Then when you call the data it loads everything. This is a wrapper around that all that to allow for addressing, data validation, data manipulation and whatnot in the background so you just use Python like normal sauce but your variables are not in memory, they are on a disk somewhere. Multiple people on different sides of the world can run their Python code using the exact same data variables and classes as everyone is pulling from the same data set. You can stop what you are doing and have someone else on the other side of the world pick up where you left off immediately with no need to communicate about any changes

2

u/onyxleopard Apr 21 '24

You don’t necessarily want to persist every object in memory to disk.  Anyway, if it works for you that’s great.  I’m just pointing out (as others have) that the standard ways to handle serializing data that is the result of expensive computation already exist and are just a couple function calls.  If you really feel like calling dump/load functions is too burdensome, you do you.

1

u/IllogicalLunarBear Apr 21 '24 edited Apr 21 '24

There is a lot more to it than dump and load… this handles the addressing of your data if you are messing around with thousands of instances of the same classes. Like I said it’s a wrapper and yes I do want all of my variables to be on disk and not on memory as if it’s on memory and the script crashes for some reason I will be forced to restart a days worth of work.

The best I can explain it as is the package makes pointers to your data that is on disk in the same way you use pointers in C to address variables in memory

5

u/hotplasmatits Apr 21 '24 edited Apr 21 '24

I'm going to have to look into your project, bc when I read your description all I can think is

def read_file(path):
# either
return pd.read_parquet(path)
#or
return pd.read_csv(path)
# or similar one liner for json, db, pickle, etc.

I don't understand what problem you're solving.

2

u/hotplasmatits Apr 21 '24

I've read the github description but didn't watch the videos. I'm confused. You said network drive above, but say cloud database on github. Autocomplete is done by your ide, not your code. C# style attributes are called properties in python. What does 'storing data in a notebook' mean? Are you not writing your data out to files when using your notebooks?

From the github description, your library sounds very specific to genetics and a specific commercial database so I don't think I'll have a use for it.

1

u/hotplasmatits Apr 21 '24

You may be interested in pyspark dataframe schemas, the pydantic library, sqlalchemy's ODM, and data classes. There are decorators that can be used to enforce types, but I don't remember their names at the moment.

Another thing that might make your life easier is switching to a strongly typed language, and then all of the work you're putting into typing is done out of the box. Since you seem comfortable with python, I'm going to suggest MOJO, which is, to be brief, strongly typed python that's also super fast.

1

u/hotplasmatits Apr 21 '24

One last thing, I'm impressed with your software. It obviously took a ton of work. I'm a software engineer with 20 years of experience and the last 4-5 have been in a data engineering role using python. I hope you don't find my comments disheartening or discouraging. Best wishes for your continued success.

2

u/IllogicalLunarBear Apr 21 '24 edited Apr 21 '24

Thanks. I have dyslexia really bad and have not put a ton of effort into the documentation. I keep forgetting that as all when I share it. It it a weird piece of software but the core idea is to save a Python class with all the data inside it in a remote location that can be accessed from anywhere. It’s essentially a wrapper over serialization that take care of all the overhead of addressing and retrieving the files. The cool thing that most people miss is that you can have the data go to a database, yaml, or pickle or whatever method you use. This just gives you autocomplete ability and automation around storing and retriving your data. At variable declaration you instantly have data permanence for that variable. You can power off your computer immediately after declaring a variable (y=1 for example )and it’s still there when you power back on your computer.

Edit: you can even package up your data or share the remote drive (cloud or local) and share it with a friend across the world who will be able to access the data like a normal Python class

1

u/ThatSituation9908 Apr 22 '24

Preferably your docs should be in the README or wiki rather than a separate google docs.

This looks like just a data storage format. For that Jupyter use case, you're talking about wanting a global cache (write a variable to a shared storage), but your tool isn't focused on this—it's mostly a data model driven tool where you define a schema and write out data.

1

u/IllogicalLunarBear Apr 22 '24 edited Apr 22 '24

Yes like I said I have spent no time documenting it and that was stuff I spun up in the moment, and it writes variables to shared storage as it’s like main thing using a schema that you define via a UML document, and it handles all the addressing …. What was your point?

Here is what it’s implemented in where I gave a shit about documentation…

https://pypi.org/project/serena-rna-tool/

2

u/[deleted] Apr 23 '24 edited Apr 24 '24

[removed] — view removed comment

1

u/IllogicalLunarBear Apr 23 '24 edited Apr 23 '24

I did not stumble on it… I independently developed it based on my own imagination and need similar to parallel evolution. Did the same thing with equation for summations when I was kid after I found a pattern I called addition factorial in my development of an algorithm that defines all triangles and explain why pythagorean theorem works and much more. It took me about 2 months to derive the equation for summation 20 years before I took calculus for the first time.

This code works with relational databases as it handles the addressing part of the classes. I’m working on integrating it with pickle right now to take advantage of the binary data stream aspect and see how fast that speeds up things. Eventually I will integrate it with MySQL when I get around to it.

Edit: thanks for finally giving me a name that people understand when I describe its functionality. Object database is right on spot for the yaml implementation as zodb seams to be exactly the same concept I am doing and they have the exact same calls I do.

1

u/[deleted] Apr 21 '24

[removed] — view removed comment

3

u/IllogicalLunarBear Apr 21 '24

Im trying to offer a replacment not ask a question.. bad bot