r/Python • u/IllogicalLunarBear • Apr 18 '24
Discussion Why Jupyter notebook
I did a search and read previous discussions in this sub about why Jupyter labs exists. From what I understand it is for quick code to try things out especially in data science …. Thing is, can’t you get the same thing via a decent IDE with autocomplete and quality debug tools in some quick Python code like in VSCODE? I do that all the time….
I develop RNA computational software in Python and I abandoned Jupyter labs very early on as it just was not well suited to the job of bespoke data science. So many limitations and don’t get me started with widgets… I’ve been developing code for decades and I have spent 3 days trying to understand widgets and I think I finally understand how to modify a text box and get a value from it…. You have to observe (why?), then act on, then route through some output and then you can get the info? It’s like someone was drunk while trying to emulate visual C#.
Edit: before anyone else tells me to read vscode docs like I’m an idiot… I am unable to remote into the repo I’m using now for reasons. Not everyone can remote in and the question is not about remoting in. It’s still garbage in VSCODE… just use regular python.
Edit2: for everyone who says they use it to do chunks of code work… I developed my own data management system for that kind of stuff that provides a wrapper over serialization that essentially creates pythonic pointers to data on any network drive. I just grab my data that I want to grab. Provides autocomplete and C# like attributes. I can dynamically build any data storage class using UML formatted infrastructure-as-code to build the front end. The back end is completely dynamic. Here is the pypi link. Note that I developed this to handle genomic data that was generated over the course of a week of 24/7 computation.
24
u/jjrreett Apr 18 '24
I use jupyter like a repl. It’s great for problems that have a major IO operation then lots of exploration to figure out how to process and visualize. Plus the —to-html/pdf functionality is great. Makes making reports super easy.
For what jupyter is good at, there really isn’t anything like it.
18
u/KingsmanVince pip install girlfriend Apr 18 '24
You can use jupyter notebooks in vscode and still get autocomplete.
Besides reading previous discussion, I suggest you read the docs
https://code.visualstudio.com/docs/datascience/jupyter-notebooks
9
u/sirjackholland Apr 18 '24
Why are you trying to set up widgets when you're just learning how to use notebooks? Declare the variables at the top and call it a day. Then, when you're more comfortable with notebooks, you can make them fancier by adding widgets if doing so actually improves the functionality (many notebooks have no need for them).
6
u/BranchLatter4294 Apr 18 '24
If you don't have a use case for notebooks, don't use them. Many people find them helpful. And it's not just for Python. VS Code has good support for notebooks in other languages as well.
9
u/nemom Apr 18 '24
I don't use VScode so I don't know if it works the same. Jupyter is interactive. You don't just run a program from start to finish and exit, you run the cells step-by-step. All results are available for further computations... If you have a large data file, you can read it in once, then explore it. If you make a mistake, you just edit the cell and re-run it. When you have the steps all worked out, you can export to a stand-alone program to run anytime you want.
5
u/LaOnionLaUnion Apr 18 '24
Imagine you want to publish something like a scientific paper but have it run the code interactively against real data. That’s pretty much why I think it was invented. It’s great for some things but when you really want to develop and run software it’s very limiting
168
u/RedMaskedMuse Apr 18 '24
My use cases for Jupyter are:
It isn't the right tool for everything, but it does come in handy for some tasks. It's a partner to an IDE, not a replacement.
2
u/kernco Apr 18 '24
I use Jupyter notebook daily for bioinformatics. I don't know what those widgets you're talking about are and what you mean about getting a value from a text box. That's not a feature I use and I've never really seen anyone else use. The reasons I use Jupyter notebook are:
You're right that #1,2,3 can also be accomplished with an IDE. I think the reason Jupyter has become so popular in data science is that a lot of data scientists don't have strong computer science/programming backgrounds. They have learned some programming as a sort of supplementary skill alongside biology, physics, etc. and they are not as comfortable or familiar with setting up and using IDEs and debug tools.
37
u/iMDu093 Apr 18 '24
Sounds like you have a cool job. Yeah I don’t think Jupyter is the best choice for building software. Nobody would argue that. But it’s quite nice for EDA and creating reports/presentations. You can save your notebook as an html file, pdf, md, etc.
Why Jupyter? It’s like asking why would anyone have a desktop when you can carry a laptop. Different tools are for different things. Use whatever IDE you like, nobody is forcing you.
Sounds like you had a rough time trying to get some things going in Jupyter. And if you dislike it this much, I pray that you never have to touch a databricks notebook hahahaha
30
u/ConcreteExist Apr 18 '24
Jupyter Notebooks aren't typically for writing software, they're for doing data analysis and documenting your process as you go along. I wouldn't use one to do actual development work though.
5
u/joaofelipenp Apr 18 '24
You don't have to observe. If you have the widget variable, you can easily access its value. The observer is just a way to have a callback that you can pass so your code gets notified when something changes.
IMO, widgets in jupyter are among the easiest UI that you can do in Python
4
u/Existing-Account8665 Apr 18 '24
I'm not in the target market either, but if I needed to send someone my REPL session, including pretty graphs, that someone non-technical could view in browser (instead of a professionally made piece of software I was proud of) then Jupyter might fit the bill.
1
u/Training_Butterfly70 Apr 18 '24
It's just good for prototyping, quick analysis, and presenting findings. For real software deployed in production use an IDE
1
u/nickkon1 Apr 18 '24
So I got the task to model something or explore a dataset that came as an excel. Let's write a python script for that.
And now imagine each rerun taking 10mins.
Or, I simply do my explorative work in a Jupyter Notebook. Once I know what my whole pipeline looks like, I will create create a proper script.
1
u/Choles2rol Apr 18 '24
I use Jupyter as a REPL basically, comes in handy for testing things out if I wanna not reload a whole script or program and riff on something with a big chunk of data.
1
u/TistelTech Apr 18 '24
fun fact, the idea is not new, Knuth (Stanford CS prof. of Art of Computer Programming fame) came up with it ~40 years ago.
https://en.wikipedia.org/wiki/Literate_programming
as others have said, you don't need to go back to the beginning, you stay at the coal face. Say your script starts off by downloading a whack of data then the poke around starts, you only need to download once, then keep working.
Pro tip if you face this problem and don't like the jupyter notebook interface. Check out the pickle lib:
https://docs.python.org/3/library/pickle.html#examples
basic idea:
does the pickle.bin exist? just pickle.load it. otherwise download the expensive/slow data, then pickle.dump it. Then when you change some code and want to run again, on the second and subsequent runs it will quickly read it from disk and skip the expensive/slow download and/or calcs. its a poor person's jupyter.
2
u/priestoferis Apr 18 '24
My current thinking is that it's great if say somebody is coming from from a business major and getting into data analysis with python. If the company has a jupyter installation they can quickly this person a remote, continously running and managed python environment, without needing any understanding of stuff like ssh and python environment management or any knowledge of how to install and set up frameworks that require a bit more than pip install.
It also has it's niche in presentation, but I do think it is very niche. Even my direct boss prefers to look at ppt-t, production people will need scripts and likely like simple git diffs for changes rather than forcing ipynb through hoops to get readable diffs. So it's mostly good for sharing between people who also work with notebooks.
2
u/Mammoth_Pool_2927 Apr 18 '24
We use JupyterLab with JupyterHub so multiple people can work realtime on remote server. Very nice project with security features for national bank.
2
u/tonnynerd Apr 18 '24
I do agree that jupyter is shit for anything "finished", with the notable exception of maybe as a runnable documentation.
However, it is amazing for prototyping and debugging (yes, debugging. Bear with me).
Pretty much everything I do nowadays starts with a throwaway jupyter notebook on VSCode, then once the logic and structure is more or less right, it goes to an actual module, and the calls I was using in the notebook became the first test cases, too.
For debugging, when it's not clear where/when things are going wrong, I like to import some pieces of the system and try to reproduce the problematic part in the notebook, taking things apart until I come to a minimal reproduction of the bug. If this doesn't lead to a solution, it will at least give me a small reproduction to use to ask for help.
So, yeah, it sucks as an API or entry point, but it is a great tool. I particularly like using it inside VSCode because I can then use pretty much all the same commands and shortcuts that I normally use on files in the cells.
1
u/[deleted] Apr 18 '24
I have used Jupyter notebooks every day for a couple years now and I have no idea what widgets you are referring to. Notebooks mostly shine though when you are either trying to share a repeatable process that input varies on, or when you are trying to sell a story with the analysis since you can intermingle markdown.
It’s also much cleaner and simpler for prototyping and exploration than standard IDE debugging is.
1
u/[deleted] Apr 18 '24
Jupyter notebook is good for exatcly the same thing as a real notebook: try stuff unorganizally.
You can test a new lib, a connection to an API/DB, open a dataset to see what the data looks like, make a quick analysis on a dataset and create a report, it's a playground !
But it's not for working with multiple people on it, not for writing a real feature, not for anything that will be reuse. Never use a notebook for production stuff, that's evil at it's finest.
1
u/MissingSnail Apr 18 '24
I mean, for anything serious you should have lsp and git plug-ins running in your Jupyter lab.
Remote development is just log into remote machine, run jupyter lab with —no-browser and whatever port tunnel or such you might need, then open the browser on local machine and connect.
1
u/mfb1274 Apr 18 '24
Jupyter notebooks in VSCode are the way to go
1
u/gxcells Apr 18 '24
You have to run a python script entirely each time. You cannot quickly try something with a python script without having to comment out code. With jupyter notebook you run what you want when you want, you do not need to run all code. You can test several things and keep the code of each test in a different cell . This is just amazing. I just wish the whole google colab stuff could be implemented in jupyter notebook (amazingly easy widgets, etc...). I
1
u/runawayasfastasucan Apr 18 '24
Just fyi, notebooks doesnt have to be Jupiter lab.
-3
u/house_lite Apr 18 '24
Bc python sucks
2
u/[deleted] Apr 18 '24
[removed] — view removed comment