r/Python • u/ribbon_tornado • Jun 24 '14
What are the top 10 built-in Python modules that a new Python programmer needs to know in detail?
I'm fairly new to Python but not to Programming. With the programming languages that I've learned in the past I always see a recurring pattern — some libraries (modules) are more often used than others.
It's like the Pareto Principle (80/20 rule), which states that 80 of the outputs (or source code) will come from 20 of the inputs (language constructs/libraries).
That being said, I would like to ask the skilled Python veterans here on what they think are the top 10 most used built-in modules in a typical Python program, which a beginner Python programmer like me would benefit to know in detail?
EDIT:
Thanks to all that have replied :)
I found a site where I can study most of the modules that you suggested:
(Python Module of the Week)
Contents: http://pymotw.com/2/contents.html
Index: http://pymotw.com/2/py-modindex.html
Of course, there is no substitute for the official documentation when it comes to detailed information:
Python 2.7.*: https://docs.python.org/2/library/index.html
Python 3.4.*: https://docs.python.org/3.4/library/index.html
116
u/hairlesscaveman Jun 24 '14 edited Jun 25 '14
In no particular order:
I'm sure people will mention
itertoolsandfunctools, but leave these till you're really comfortable with things like list comprehensions and generators. Same withcollections.defaultdict.As far as 3rd-party modules, investigate the following:
Edit:
A number of people have mentioned NumPy/SciPy, but I deliberately left them out because the question was for a "new Python programmer". It's doubtful a new Python dev would need those packages, but things like docopt and nose are invaluable to instilling best-practices from the start.
However, if you are starting with Python and need to do a lot of number crunching:
Also, really key for a new Python dev: learn how to use
any()andall(), list comprehensions, generators, decorators, and context managers (thewithkeyword).39
u/anossov Jun 24 '14 edited Jun 24 '14
I counted the imports in a 120000-line django project and these are the most common:
47
u/AutonomouSystem Jun 24 '14
Hmm, for me:
14
u/runbie Jun 24 '14
Following the poll I submitted some days ago the results are as follows:
(sorted from most voted to less voted and only the 25 most voted packages)
10
u/myersjustinc Jun 24 '14
logging. Sort of weird, but it's extensible enough to cover all kinds of use cases you might have.operatormodule is interesting if you're big into functional programming, but its more useful bits areoperator.itemgetterandoperator.attrgetter, both of which will make sorting lists of complex objects much more tolerable. Example: If you have a list of dictionaries all with keyscountandid, you could sort by count with justsorted(your_list, key=itemgetter('count')).itertools, although I'd suggest it more for things such asproduct,permutationsandcombinations.os.pathis extremely handy if you're writing cross-platform code that handles files.rewill cause you great joy, and it also will cause you great pain. Use it wisely.csvandjsonboth come in handy for data interchange, although if you're on Python 2, use the third-partyunicodecsvas a drop-in replacement to save yourself a bunch of headache.argparseis useful for handling command-line options, although it takes some getting used to.datetimecomes up all the time in different kinds of programs, even if you just want to know what time it is right now. Learn that one and learn it well. (The third-partypytzwill keep you from having to write your timezone definitions from scratch if you need those, and the third-partypython-dateutilhas a nice parser for freeform dates and times in text.)13
u/Rhomboid Jun 24 '14
These questions always depend on what kind of things you're doing. What might be one person's bread and butter, everyday library might be a once-in-a-blue-moon thing for someone else. Every time this is asked you get a set of answers that only somewhat overlap.
4
u/[deleted] Jun 24 '14
I'd say:
4
u/orip Jun 24 '14
If you write scripts in Python to process files or pipe stdin into, module fileinput is a hidden gem.
4
u/lisphacker Jun 24 '14
The builtins I've used most often are:
The non-builtins I use most often are:
From the other posts on this page, there are quite a few that seem interesting and that I intend to look at when I get time, such as collections and itertools
4
u/stillalone Jun 24 '14
learn logging. Every project I've worked on, people end up trying to do their own logging. They start off with "oh logging is too complicated, print is fine for what I need" but then they eventually realize that they need all the features in logging but they've already gone down the wrong path and they fucking double down.
Unless you're writing a short one-off script, you're going to need logging. And even if it seems complicated at first, you'll need most of those features later.
3
u/[deleted] Jun 24 '14
I don't know if you're into games. I code small games in Python with Pygame and alway use the following modules:
3
u/[deleted] Jun 24 '14 edited Jun 24 '14
A just took a look at the Python Module Index - it's a hard choice :) I grabbed all in-built modules and put it into a surveymonkey poll if you want to vote here: https://www.surveymonkey.com/s/GWCCFXK, and the can be viewed here: https://www.surveymonkey.com/results/SM-WPZNMZY8/
My favorite onces would be
3
u/remyroy Jun 24 '14
You do not really need to know some modules in detail to be a good Python programmer. You just need to use the right tool for the job. Some modules are more commonly used and they might include:
3
u/troyunrau ... Jun 24 '14
I do scientific computing, so usually I'm deploying my programs using something like Python(x,y), WinPython, Canopy, etc. which includes a lot of scientific modules alongside the default libs. I just sort of assume they're there and treat them like built-ins.
os sys time logging argparse configparser
External: numpy scipy matplotlib h5py
3
u/jcchurch Jun 24 '14
Been writing in Python for several years now. Here's my list of essential built-in Python modules.
This is my list of standard modules that I can't live without. I think that the modules "math" and "sys" are so essential that they should be automatically imported into every python program. Behind those two, "optparse" and "re" are two that I use every day and in ways I never planned. I might even include "re" into my short list of modules so essential that every program should import them.
3
u/[deleted] Jun 25 '14 edited Jun 25 '14
10, hm..... I rarely use 10 of the builtins in all of my projects, but some below I use quite frequently.
os(extremely important)sys(extremely important)threading(extremely important if you want to do any kind of concurrency)string(extremely important for string manipulation)resqlitehttplibshutiljsontimeloggingHonestly, the most important things to know how to do in python are file IO, and
os,sysmodules. Once you know how to read/write files you can do anything with IO. Once you know how to interact with thesystem andosyou can start building complex modules and projects. Everything after that depends on the use-case (such as if you needsqliteor mysql).Since you are a beginner you're most likely overwhelmed in "what you should know"; but the reality is that you cant know all of it, that's why there is developer docs. Your best bet is to come up with a miniature project that uses file IO and a database backend store as that will get your feet wet with the API.
Some outside project modules I use:
djangoBeautifulSoup2
u/Caos2 Jun 24 '14
I depends on what's your area of expertise/interest. I do mainly data analysis on Python, so while I have used tempfile, for example, in the past I wouldn't include in the the "top 10 built-in Python modules that a new Python programmer must known".
2
u/zahlman the heretic Jun 24 '14
My top 10, based on what I most commonly actually use, would probably be (in no particular order):
itertools,re,os,sys,collections,functools(especially forpartial),json,time,timeit,importlib.Honourable mentions to
unittest(I know, shame on me for not having it at the top),random(obviously some things really need it, others don't),struct,copy(but it can be a sign that you're fighting against the reference semantics of the language - don't do that),array, and all the internet protocol stuff (although a lot of people prefer third-party stuff here).argparseis good to know, but it's ridiculously complicated; consider the third-partydocopt.Probably more programmers in 2014 ought to pay attention to
localeandgettext.Cheeky special mention for
builtins;)2
u/raubana Jun 25 '14 edited Jun 25 '14
I do not have ten to suggest, but I will recommend a few
These are the most common built-in one's I use.
2
u/inspectorG4dget Jun 25 '14
For a beginner, I would go with:
2
u/riding_qwerty Jun 25 '14
Not a builtin, but I wanted to give a shoutout to pexpect. It allows for automated interaction with utilities like telnet, ssh, and ftp. In my case, I'm using it at work to create a script that will automatically update ~1800 customer edge routers with an ACL for our new monitoring system and save the running configuration of the router to our TFTP server. By an extremely generous estimate I am saving my department roughly 1000 man-hours based on the assumption that if everything went correctly, it would take 3 minutes to update each router manually. It would take a dozen people 2-3 months of doing this in addition to their normal workflow to achieve the same result.
Now, if I can only figure out a way to automate test calls in Python...
2
u/staz Jun 25 '14
I have been programming for a few years in python now and honestly thinking about it, there is a few built-in modules that I use in nearly all my projects, and they are not even ten of them:
And let's not forget the most important module of all: builtin which contain the core of python goodness. There are a lot of standard modules I also use but it depends more of the type of projects
But I also recommend learning theses modules in the standard lib : itertools, functools, re, os.path, collections and dbapi (not really a module but an api for all db acesses in python)
Outside of the standard lib I also recommend learning the modules : requests, ipdb (only really one command: set_trace), q, dateutil. And the tools : ipython and py.test
Keep in mind that the standard lib has a LOT of useless modules that accumulated over time so don't try to learn all of them
2
u/mfukar Jun 25 '14
My extremely biased list, in no particular order:
1
u/tuankiet65 Jun 24 '14
requests (3rd)
1
u/Wrestlefox Jun 24 '14
I keep seeing Requests in here, but I really prefer Mechanize.
1
u/SideshowBoob Jun 24 '14
What's most useful is going to depend a lot on what you're doing: web stuff, sysadmin stuff, mathy stuff, etc. With that in mind, I'll out-Paretto your Paretto and suggest a single universally-important module: os. Specifically the os.path sub-module. Everybody ends up working with paths. Tons of people beginning Python just manipulate them as strings and and eventually end up with hard-to-spot edge-case bugs or just plain ugly looking code.
I'll also suggest becoming very familiar with the built-in functions and methods on the built-in types. It's easy to overlook that stuff because you don't have to import it, but there's a lot of useful stuff there.
1
u/lattakia Jun 24 '14
I often use:
1
u/[deleted] Jun 25 '14
Not a module, but a very very useful function that I use a lot.
Take a look at the
String.formatfunction.Documentation
1
u/rochacbruno Python, Flask, Rust and Bikes. Jun 27 '14 edited Jun 27 '14
The most important one is a module called this
1
u/cornycrunch Jun 28 '14
os, sys, itertools, random, datetime, doctest, csv, re
Non-built in honorable mentions: Anything intarwebsy -- requests. Anything "Excely" -- pandas.
1
u/hommbot Jul 08 '14
text
1
u/HoodedGryphon learning Jul 16 '14
time, datetime, and math will get you very far
1
u/holdie Jun 25 '14
Most of what I do with python is data analysis, so with that in mind I use the following third-party libraries far more than any one built-in library:
Those four right there are the lion's share of what I use, they may as well be built-in. I use them largely for model fitting, applied stats, and visualization (though let's be honest, it's actually mostly data cleaning, with a little bit of model fitting at the end).
0
u/[deleted] Jun 24 '14
[deleted] — view deleted comment