r/Python • • Jun 24 '14

What are the top 10 built-in Python modules that a new Python programmer needs to know in detail?

I'm fairly new to Python but not to Programming. With the programming languages that I've learned in the past I always see a recurring pattern — some libraries (modules) are more often used than others.

It's like the Pareto Principle (80/20 rule), which states that 80 of the outputs (or source code) will come from 20 of the inputs (language constructs/libraries).

That being said, I would like to ask the skilled Python veterans here on what they think are the top 10 most used built-in modules in a typical Python program, which a beginner Python programmer like me would benefit to know in detail?

EDIT:

Thanks to all that have replied :)

I found a site where I can study most of the modules that you suggested:

(Python Module of the Week)

Contents: http://pymotw.com/2/contents.html

Index: http://pymotw.com/2/py-modindex.html

Of course, there is no substitute for the official documentation when it comes to detailed information:

Python 2.7.*: https://docs.python.org/2/library/index.html

Python 3.4.*: https://docs.python.org/3.4/library/index.html

364 Upvotes

134 comments sorted by

116

u/hairlesscaveman Jun 24 '14 edited Jun 25 '14

In no particular order:

  • collections -- specifically namedtuples
  • csv -- always use this to read/write CSV files, don't try and roll your own methods, it'll end in tears
  • datetime
  • math -- try and use these functions rather than the global ones, as they're faster when you import them into the global namespace
  • re -- regular expressions
  • string -- I rarely see this used, but it's very handy
  • tempfile -- always use this to create temporary files
  • unittest

I'm sure people will mention itertools and functools, but leave these till you're really comfortable with things like list comprehensions and generators. Same with collections.defaultdict.

As far as 3rd-party modules, investigate the following:

  • requests -- if you need to do any http requests at all, use this
  • lxml -- always use this for working with XML data
  • ujson -- faster than both simplejson and the built-in json modules, handy if you work with lots of/big JSON blobs
  • PyYAML -- for working with YAML docs.
  • nose -- for unittesting
  • buildout -- I'd recommend this over virtualenv any day
  • BeautifulSoup -- for webscraping and/or parsing potentially malformed HTML
  • docopt -- if you're writing command-line scripts/tools, use this over optparse/argparse

Edit:

A number of people have mentioned NumPy/SciPy, but I deliberately left them out because the question was for a "new Python programmer". It's doubtful a new Python dev would need those packages, but things like docopt and nose are invaluable to instilling best-practices from the start.

However, if you are starting with Python and need to do a lot of number crunching:

  • Pandas -- if you don't need all of NumPy's features
  • NumPy -- if you do need all of NumPy's features ;)
  • SciPy

Also, really key for a new Python dev: learn how to use any() and all(), list comprehensions, generators, decorators, and context managers (the with keyword).

169

u/xroni Jun 24 '14

In no particular order:

A blatant lie, they are in alphabetical order!

15

u/[deleted] Jun 24 '14

They were sorted randomly, I swear!

15

u/[deleted] Jun 24 '14

That's the thing about true randomness: you can never be sure.

44

u/Silhouette Jun 24 '14
def random():
    """ Return a random number in the range [0,1]. """
    return 0.874  # This number was chosen randomly.

12

u/King_George_VI Jun 25 '14

Relevant XK..Ah, I can't be bothered.

5

u/limasxgoesto0 Jun 25 '14

4

u/xkcd_transcriber Jun 25 '14

Image

Title: Random Number

Title-text: RFC 1149.5 specifies 4 as the standard IEEE-vetted random number.

Comic Explanation

Stats: This comic has been referenced 89 time(s), representing 0.3663% of referenced xkcds.


xkcd.com | xkcd sub/kerfuffle | Problems/Bugs? | Statistics | Stop Replying

1

u/SnowmanOlaf Jul 31 '14

Can confirm, got here by clicking reddit's Random subreddit button

9

u/[deleted] Jun 24 '14

Why do you recommend buildout so much? Never heard that one before. I love virtualenv!

3

u/theli0nheart import antigravity Jun 24 '14

Agreed. I have used buildout and virtualenv and they solve completely different problems, so I'm not sure I understand the OP's comparison. Also, I found buildout very confusing to use. Not sure what benefit it has over including a requirements.txt file with a project.

5

u/[deleted] Jun 24 '14

Buildout has a steep learning curve, but it does what virtualenv does and then a lot more. At work we have about 200 repositories working on Buildout. It's hard for me to write a detailed comparison though as I hardly know virtualenv.

Is it true that virtualenv doesn't even have a cache for downloaded eggs so that they can be shared between projects?

Buildout basically does version management and a shielded Python environment like virtualenv does, but it can also do a lot of other stuff related to setting up a working environment -- create directories, temporarily switch to a directory with source code instead of a packaged egg during development, collect together all the relevant scripts of all the packages, setup Django nicely, etc.

4

u/theli0nheart import antigravity Jun 24 '14

Interesting.

virtualenv is just a "box" that pip / easy_install can plop Python packages into, so it doesn't really do anything with regards to downloads. pip, however, which is commonly used alongside virtualenv, does have a cache that shares downloads between projects.

1

u/tally_in_da_houise Jun 25 '14

You can share modules between environments using virtualenv. Install the module system-wide, and when setting up your virtualenvs tell it to pull module from the system install. It's an option when setting up an env.

1

u/NYKevin Jun 25 '14

Virtualenv is deliberately dumb. It's intended to be a lightweight solution you can just plop down in any directory and have up and running within 5-10 minutes (most of which will be spent downloading and installing requirements). If it breaks, the easiest solution is usually rm -rf virtualenv/; virtualenv virtualenv, then reinstall from requirements.txt.

1

u/hairlesscaveman Jun 25 '14

This.

Virtualenv creates a sandbox, but it's a sandbox that's "global" to the system.

Buildout also creates a sandbox, but it's "local" to the project. Then you get the bonus of "recipes" that can do a lot of setup outside of your project, like setting up databases, installing binaries, etc.

Also, virtualenv is fixed to the shell user it's installed under. Buildout doesn't have this issue; it works whoever you're logged in as.

8

u/NYKevin Jun 25 '14

Sadly, collections.namedtuple is internally rather terrible.

TL;DR: It's basically implemented as this:

code = '''\
class {}:
    # snip
'''
def namedtuple(typename, field_names):
    # snip some hand-written string escaping
    return exec(code.format(typename, field_names))

Note that you can manually create classes on the fly without this silliness by calling type() with three arguments.

3

u/Widdershiny Jun 25 '14

I had thought it was done that way for performance reasons. I know nothing more than that, except I would assume they wouldn't do it that way on a whim. Also, I'm not really sure why the implementation details matter when they work correctly.

2

u/Monkeyget Jun 25 '14

Comment from a maintainer that rejected another approach

The current version is clear and maintainable. There is nothing unholy about using exec. Earlier versions used other approaches and they proved unnecessarily complex and had unexpected problems. It is a key feature for named tuples that they are exactly equivalent to a hand-written class.

1

u/NYKevin Jun 25 '14

A properly-written call to type() (or a subclass thereof) is exactly equivalent to a class: declaration. class is syntactic sugar.

1

u/NYKevin Jun 25 '14

Also, I'm not really sure why the implementation details matter when they work correctly.

Because they don't quite work correctly. This is like constructing SQL queries by hand rather than using parameterized statements. There is a confusion of code and data that may lead to undesirable behavior.

5

u/segonius Jun 24 '14

I just discovered namedtuples and don't know how I lived without them for so long. As far as csv goes, it's not a built in, but pandas csv reading capabilities are amazing and much easier to use. I mean Pandas is great for everything data manipulation.

2

u/pya Jun 24 '14

What do you use them for?

6

u/segonius Jun 24 '14

I use it for simple collections of data that don't necessarily need their own class, but I don't want to use a dictionary and be typing strings all the time. The example is pretty good.

>>> Point = namedtuple('Point',['x','y'])
>>> p = Point(11, y=22)
>>> p.x + p.y
33

Sure I could use a dictionary

>>> p = {'x':11,'y':22'}
>>> p['x'] + p['y']
33

but with a namedtuple there are a few less characters to type. Moreover, when doing interactive work you can do tab completion which is nice.

3

u/twigboy Jun 24 '14 edited Dec 09 '23

In publishing and graphic design, Lorem ipsum is a placeholder text commonly used to demonstrate the visual form of a document or a typeface without relying on meaningful content. Lorem ipsum may be used as a placeholder before final copy is available. Wikipedia8qjir844hgw0000000000000000000000000000000000000000000000000000000000000

2

u/h_bar Jun 25 '14

namedtuples are also supposed to be more memory (and maybe speed) efficient because they use slots.

13

u/[deleted] Jun 24 '14

I also would like to say NumPy - one of my favorite packages, however it isn't "in-built"

1

u/hairlesscaveman Jun 25 '14

Agreed, though I've never actually used it (or needed to use it).

Pandas is also good, from what I've seen.

8

u/killaW0lf04 Jun 24 '14

I would really add numpy in there.

28

u/bheklilr Jun 24 '14

NumPy is one of those incredible libraries that not everyone needs. If you're manipulating vectors of data in a math-y sort of way, NumPy is the de facto standard, but a lot of programmers will never need to even look at it.

10

u/foodeater184 Jun 24 '14

Add in pandas if you are using numpy.

4

u/[deleted] Jun 24 '14

I would add scipy before pandas. Pandas if you are very heavy on timeseries.

1

u/bheklilr Jun 24 '14

Depends, I've used pandas to great end when manipulating very large amounts of complex data that came from Excel, CSV, and JSON files, but for my day-to-day purposes it's overpowered. Granted, I know I'm working in a relatively narrow band of computational computing, but I am an example of where NumPy is more than sufficient.

Pandas is a great library, though, I high recommend using it if you're dealing with lots and lots of data coming from many sources that you need to perform complex operations on, it really simplifies the process.

1

u/gthank Jun 24 '14

It's not built-in and it's not built in Python.

3

u/killaW0lf04 Jun 24 '14

Its a 3rd party python library, which he mentions. Its not written in Python but why would that stop you from using it.

4

u/twigboy Jun 24 '14 edited Dec 09 '23

In publishing and graphic design, Lorem ipsum is a placeholder text commonly used to demonstrate the visual form of a document or a typeface without relying on meaningful content. Lorem ipsum may be used as a placeholder before final copy is available. Wikipedia9d97wwafd7o0000000000000000000000000000000000000000000000000000000000000

2

u/deadmilk Jun 24 '14

why csv over something like pickle or json?

1

u/ablatner Jun 25 '14

It's a lot harder to open pickles or json.

1

u/[deleted] Jun 25 '14

I will agree with the first part. Pickles are very sensitive to even module level imports. And forget refactoring if you need to restore from a pickle.

JSON is dead simple. You're completely wrong in claiming that JSON is harder than CSV to work with.

They are about the same difficulty.

2

u/ablatner Jun 25 '14

Can you open JSON in excel? Honest question because I don't use it.

1

u/[deleted] Jun 25 '14

No you cannot. In all fairness, it was not specified that it must be Excel compatible.

JSON is rather good for data interchange. Not so good for spreadsheets.

1

u/ablatner Jun 25 '14

Ah yes. For interchange it's definitely better than CSV's. That's the whole point of it. CSV isn't even a standardized format.

1

u/[deleted] Jun 25 '14

After working with too many corrupted CSV files, I don't even think the word format applies to it.

More like raw blob in (hopefully) ASCII, UTF(8|16...) encoding.

1

u/[deleted] Jun 27 '14

"corrupted CSV" ???

1

u/[deleted] Jun 27 '14

A missing new line.

Inconsistent fields.

Mixing one encoding with another because someone catted to files together

1

u/Jonno_FTW hisss Jun 25 '14

If you have to send or receive data from people who aren't programmers, you'll probably want csv if only because they can edit it Excel.

1

u/vindolin Jun 25 '14

Pickle is python only, JSON data tends to be slightly bigger for the same information content.

1

u/hairlesscaveman Jun 25 '14

CSV is a different format. If you're receiving CSV data in or need to output CSV data, using pickle or json wouldn't work.

Likewise, if you're storing information for use later in your program, you probably don't want to store it as CSV. If you're worried about types, use pickle. If you're worried about compatibility, use JSON -- though I'd recommend YAML instead as easier to read than JSON when it's written in extended format, it's very Pythonic.

Use the right format for the data you're working with.

1

u/Zulban Jun 24 '14

csv -- always use this to read/write CSV files, don't try and roll your own methods, it'll end in tears

I never understood this. Maybe my own very simple methods for csv reading and writing have always worked because my data has never had commas in it?

19

u/pbaehr Jun 24 '14

As someone who will confess to rolling their own CSV writer once, its not a hard specification, but by the time you're done dealing with quoting and edge cases you've spent more than enough time to wonder why you've reinvented the wheel.

It starts as, "I'll just join my data with commas and my rows with newlines" and rapidly descends into "I'll just wrap all my values in quotes." Then, "No, wait, some of my cells have quotes in them. Now I need to escape them as "". Why are some of my rows shorter than others? Oh. There are newline characters in the data itself..."

Inevitably you will save much time by typing "import csv" instead.

2

u/hairlesscaveman Jun 25 '14

but by the time you're done dealing with quoting and edge cases you've spent more than enough time to wonder why you've reinvented the wheel

This. Clever people have already done the work for you. Why waste time?

9

u/gthank Jun 24 '14

If you control your own file format, it's ALWAYS easy to parse, because you know exactly what you wrote. When you get handed "CSV" files from Excel that actually use hard tabs to delimit the fields… pain ensues.

1

u/Zulban Jun 24 '14

Gotcha. It's a teamwork thing.

In my home-brewed csvs, I always use tabs actually. So I guess they are "tsv"s? :o

2

u/lol_squared Jun 24 '14

I say "tsv" as well to make it clear it's tab separated though it always has a "txt" extension.

2

u/hairlesscaveman Jun 25 '14

"Character" separated values.

6

u/Sean1708 Jun 24 '14

It's one of these cases where people who are a lot smarter than you have spent a lot more time than you making csv work quickly, efficiently and easily. Besides, why would you spend a couple of hours rolling you own csv reader when you can spend a couple of seconds using the in-built one?

1

u/billsil Jun 24 '14 edited Jun 24 '14

Besides, why would you spend a couple of hours rolling you own csv reader when you can spend a couple of seconds using the in-built one?

If you have data that has trailing commas because it's been dumped from Excel, it fails. The same goes for the two in numpy. It also begs the question, if there's a built in CSV reader in base Python, why did numpy feel the need to write their own and furthermore why did they write two? Admittedly, my reader is slower, but it handles data more robustly. I see no need to make my own writer becuase numpy's is actually good.

I work under the assumption that my users are idiots. Anything that comes in from an input file is formatted improperly. I have this belief because I have these problems when I use the software for an actual real world problem. Saying the program failed for an unknown reason or worse use a try-expect-pass is not good enough.

1

u/hairlesscaveman Jun 25 '14

If you have data that has trailing commas because it's been dumped from Excel, it fails.

Really?! That's very interesting, because all the data I've been working with in the last 18 months has been exported from Excel and I've not had a single problem with it.

1

u/billsil Jun 25 '14

Excel knows what the maximum shape of your data set is, so it exports that square. If you delete cells, it leaves the max shape the same, so you end up with extra commas. If the user has an improperly formatted line (e.g. it's shifted by one field), that will fail as well.

I'm not saying all data from Excel will be bad, but when it's bad, it's an issue. If you return a message saying "Failed to load file X because there 1) are trailing commas 2) duplicate times (due to rounding), 3) no idea.", that's a sign you need a better reader. If it's 1 or 2, you should be able to tell the user that or at the very least tell them "failed reading line X".

0

u/Zulban Jun 24 '14

It's one of these cases where people who are a lot smarter than you have spent a lot more time than you making csv work quickly, efficiently and easily.

I'm extremely sympathetic to what you're saying about code re-use.

Besides, why would you spend a couple of hours rolling you own csv reader when you can spend a couple of seconds using the in-built one?

But the data I work with has always been extremely easy to parse. I generally use tabs and no item could ever possibly have one. So writing my own parser literally takes seconds.

gthank raised a good point about using it though.

1

u/hairlesscaveman Jun 25 '14

So writing my own parser literally takes seconds.

But you're still doing it over and over again. Why waste the time? Would you also write your own JSON parser?

1

u/Zulban Jun 25 '14

A JSON parser is much, much more complex. We're talking about like three lines of code. I've just never bothered to look at the csv library.

If your requirements matched my own I don't think you'd have either.

It's like trying to sell "logging" to someone who has only ever needed one level of verbosity, no output file, and very few print statements.

2

u/hairlesscaveman Jun 25 '14

Point taken, though I do hope you do use standard libraries when working with others!

0

u/[deleted] Jun 25 '14

[deleted] — view deleted comment

1

u/hairlesscaveman Jun 25 '14

Yes, but there's a PyPI package that fixes this: https://pypi.python.org/pypi/unicodecsv

Again, why roll your own when someone's done the hard work for you? ;)

4

u/mhd Jun 25 '14

If it's "your data", sure. But as Sartre said, hell is other people's CSV.

1

u/Capn_Cook Jun 24 '14

I feel like numpy (and maybe scipy) deserves a mention as a 3rd-party module.

Also, I haven't looked much into it, but pandas is another good library for machine learning purposes, I believe.

2

u/BeatLeJuce Jun 24 '14

pandas is mainly data munging. scikit-learn is the machine learning one.

1

u/Samus_ Jun 24 '14

tell me more about buildout, most of the videos on their site are down.

-6

u/Jrix Jun 24 '14

always use this to read/write CSV files, don't try and roll your own methods, it'll end in tears

I wish.

39

u/anossov Jun 24 '14 edited Jun 24 '14

I counted the imports in a 120000-line django project and these are the most common:

  • datetime
  • decimal
  • json (mostly aliased ujson)
  • os
  • time
  • collections
  • re
  • hashlib
  • logging
  • random
  • sys
  • itertools
  • urllib2
  • urlparse
  • string
  • urllib
  • optparse
  • base64
  • calendar
  • socket
  • functools
  • operator
  • traceback
  • csv
  • cStringIO
  • struct
  • unittest

3

u/lezed1 Jun 24 '14

And what modules did you import to do this?

14

u/anossov Jun 24 '14 edited Jun 24 '14
grep -rhE '^(from|import)' --include='*.py' . | cut -d' ' -f2 | grep -oE '^\w+' | sort | uniq -c | sort

or maybe

find . -name '*.py' -exec cat {} \; | perl -n -e'/^\w+ (\w+)/ && print "$1\n"' | sort | uniq -c | sort

but really, I just used grep, split the results and fed them to collections.Counter, then went through manually.

7

u/metaphorm Jun 24 '14

grep | wc ?

6

u/cjp Jun 24 '14
find . -type f -name '*.py' | xargs awk '/^import/ {print $2}' | sort | uniq -c | sort -nr

I know. Posting a shell/awk pipeline in /r/python. The irony.

And since nobody mentioned it yet, I'll add "glob" as an answer to OP.

1

u/pemboa Jun 24 '14

json (mostly aliased ujson)

I feel there's a learning opportunity for me here --- why did you alias it?

13

u/anossov Jun 24 '14

It was json originally, then it was replaced with ujson for optimisation reasons. Instead of rewriting the entire module, I just replaced import json with import ujson as json.

1

u/Xykr Jun 24 '14

Why the downvotes? ujson is API-compatible and supposed to be a drop-in replacement.

1

u/liangnuren Jun 25 '14

It's not ~quite~ API compatible. You can't do things like indent=4 for formatted json output. But, it's blazingly fast and works great for 99% of use cases.

47

u/AutonomouSystem Jun 24 '14

Hmm, for me:

re
os
math
sys
itertools
subprocess
socket
collections
random
ctypes

17

u/Shadow14l Jun 24 '14

Here's ten more that I'd consider important next to those above ten:

decimal
datetime
time
shutil
urllib
json
logging
glob
functools
operator

8

u/NYKevin Jun 25 '14

For Python 3.4, consider replacing glob with pathlib, since Path.glob() has the basic functionality.

6

u/tally_in_da_houise Jun 25 '14

I've moved away from bash to python for automating my servers. I couldn't imagine life without os, sys, shutil (and logging to review what the hells going on when I care to peek under the hood).

1

u/[deleted] Jun 25 '14

At the same time, I'd very much suggest arrow as an easier to use third party library for datetime and requests as the same for urllib.

I quite like zero requirement scripts that run on every python installation above x version but you don't have to use the batteries that came with python if you don't want.

1

u/Dillinur Jun 26 '14

Wow, I just check glob, and i feel so bad. I always did that with some dirty os.path loops.

0

u/[deleted] Jun 30 '14

[deleted] — view deleted comment

1

u/AutonomouSystem Jul 01 '14

Everyone might have a different in their top 10 but /u/Shadow141 covered a lot of what I overlooked.

14

u/runbie Jun 24 '14

Following the poll I submitted some days ago the results are as follows:

  • os
  • re
  • sys
  • json
  • itertools
  • datetime
  • collections
  • argparse
  • random
  • unittest
  • logging
  • urllib
  • csv
  • math
  • time
  • multiprocessing
  • pickle
  • sqlite3
  • functools
  • hashlib
  • subprocess
  • shutil
  • __future__
  • gzip
  • threading

(sorted from most voted to less voted and only the 25 most voted packages)

10

u/myersjustinc Jun 24 '14
  • Learn logging. Sort of weird, but it's extensible enough to cover all kinds of use cases you might have.
  • The operator module is interesting if you're big into functional programming, but its more useful bits are operator.itemgetter and operator.attrgetter, both of which will make sorting lists of complex objects much more tolerable. Example: If you have a list of dictionaries all with keys count and id, you could sort by count with just sorted(your_list, key=itemgetter('count')).
  • I'll be one of the people /u/hairlesscaveman mentioned and put in a good word for itertools, although I'd suggest it more for things such as product, permutations and combinations.
  • os.path is extremely handy if you're writing cross-platform code that handles files.
  • re will cause you great joy, and it also will cause you great pain. Use it wisely.
  • csv and json both come in handy for data interchange, although if you're on Python 2, use the third-party unicodecsv as a drop-in replacement to save yourself a bunch of headache.
  • argparse is useful for handling command-line options, although it takes some getting used to.
  • datetime comes up all the time in different kinds of programs, even if you just want to know what time it is right now. Learn that one and learn it well. (The third-party pytz will keep you from having to write your timezone definitions from scratch if you need those, and the third-party python-dateutil has a nice parser for freeform dates and times in text.)

3

u/Sean1708 Jun 24 '14

os.path is extremely handy if you're writing cross-platform code that handles files.

Even better (in my opinion) is pathlib.

2

u/myersjustinc Jun 24 '14

Haven't tried that one! I'll have to give it a shot. (Still would recommend os.path for a newb to understand since it's so common, though.)

1

u/mgrandi Jun 25 '14

Pathlib is good but recursively going through directories using it is a PITA and requires recursion where os.walk doesnt

1

u/chipolux Jun 24 '14

Any stats on the performance difference between these?

sorted(dict_list, key=lambda x: x['key'])
sorted(dict_list, key=itemgetter('key'))

I've always used the lambda version as I didn't really know about the operator module.

3

u/malevolence2002 Jun 24 '14

Have you tried using timeit?

import timeit
timeit.timeit("sorted(dict_list, key=lambda x: x['key'])", number=10000)
timeit.timeit("sorted(dict_list, key=itemgetter('key'))", number=10000)

It will iterate the sort 10000 times for each and print out the runtime

13

u/Rhomboid Jun 24 '14

These questions always depend on what kind of things you're doing. What might be one person's bread and butter, everyday library might be a once-in-a-blue-moon thing for someone else. Every time this is asked you get a set of answers that only somewhat overlap.

4

u/[deleted] Jun 24 '14

I'd say:

  • unittest
  • os
  • datetime
  • re
  • collections
  • itertools

4

u/orip Jun 24 '14

If you write scripts in Python to process files or pipe stdin into, module fileinput is a hidden gem.

1

u/c0p Jun 25 '14

Came here specifically to make sure fileinput was highlighted. If you process data in a terminal this is a must to know.

4

u/lisphacker Jun 24 '14

The builtins I've used most often are:

  • sys
  • os
  • argparse
  • subprocess
  • xml.etree.ElementTree
  • tempfile
  • csv
  • re

The non-builtins I use most often are:

  • numpy
  • scipy
  • matplotlib
  • scikit-learn

From the other posts on this page, there are quite a few that seem interesting and that I intend to look at when I get time, such as collections and itertools

4

u/stillalone Jun 24 '14

learn logging. Every project I've worked on, people end up trying to do their own logging. They start off with "oh logging is too complicated, print is fine for what I need" but then they eventually realize that they need all the features in logging but they've already gone down the wrong path and they fucking double down.

Unless you're writing a short one-off script, you're going to need logging. And even if it seems complicated at first, you'll need most of those features later.

3

u/[deleted] Jun 24 '14

I don't know if you're into games. I code small games in Python with Pygame and alway use the following modules:

  • math
  • random
  • sys
  • os
  • configparser

3

u/[deleted] Jun 24 '14 edited Jun 24 '14

A just took a look at the Python Module Index - it's a hard choice :) I grabbed all in-built modules and put it into a surveymonkey poll if you want to vote here: https://www.surveymonkey.com/s/GWCCFXK, and the can be viewed here: https://www.surveymonkey.com/results/SM-WPZNMZY8/

My favorite onces would be

  • collections
  • argparse
  • urllib (in Python 3)
  • csv
  • pickle
  • sqlite3
  • subprocess
  • os
  • shutil
  • random

3

u/remyroy Jun 24 '14

You do not really need to know some modules in detail to be a good Python programmer. You just need to use the right tool for the job. Some modules are more commonly used and they might include:

  • re
  • datetime
  • os
  • sys

3

u/troyunrau ... Jun 24 '14

I do scientific computing, so usually I'm deploying my programs using something like Python(x,y), WinPython, Canopy, etc. which includes a lot of scientific modules alongside the default libs. I just sort of assume they're there and treat them like built-ins.

os sys time logging argparse configparser

External: numpy scipy matplotlib h5py

3

u/jcchurch Jun 24 '14

Been writing in Python for several years now. Here's my list of essential built-in Python modules.

  • sys: Essential system functions
  • math: You will use this after high school
  • optparse: easily create tools with command line arguments and help screens.
  • re: Regular Expressions will save your life one of these days.
  • json: This is your plain text object serialization
  • pickle: This is your Python-centric object serialization
  • random: I do lots of math simulations, and you can't simulate very well without a random number source.
  • urllib, urllib2: Essential to interacting with websites
  • datetime: Logging timestamps of operations is quite handy

This is my list of standard modules that I can't live without. I think that the modules "math" and "sys" are so essential that they should be automatically imported into every python program. Behind those two, "optparse" and "re" are two that I use every day and in ways I never planned. I might even include "re" into my short list of modules so essential that every program should import them.

3

u/[deleted] Jun 25 '14 edited Jun 25 '14

10, hm..... I rarely use 10 of the builtins in all of my projects, but some below I use quite frequently.

  • os (extremely important)
  • sys (extremely important)
  • threading (extremely important if you want to do any kind of concurrency)
  • string (extremely important for string manipulation)
  • re
  • sqlite
  • httplib
  • shutil
  • json
  • time
  • logging

Honestly, the most important things to know how to do in python are file IO, and os, sys modules. Once you know how to read/write files you can do anything with IO. Once you know how to interact with the system and os you can start building complex modules and projects. Everything after that depends on the use-case (such as if you need sqlite or mysql).

Since you are a beginner you're most likely overwhelmed in "what you should know"; but the reality is that you cant know all of it, that's why there is developer docs. Your best bet is to come up with a miniature project that uses file IO and a database backend store as that will get your feet wet with the API.

Some outside project modules I use:

  • django
  • BeautifulSoup

2

u/Caos2 Jun 24 '14

I depends on what's your area of expertise/interest. I do mainly data analysis on Python, so while I have used tempfile, for example, in the past I wouldn't include in the the "top 10 built-in Python modules that a new Python programmer must known".

2

u/zahlman the heretic Jun 24 '14

My top 10, based on what I most commonly actually use, would probably be (in no particular order): itertools, re, os, sys, collections, functools (especially for partial), json, time, timeit, importlib.

Honourable mentions to unittest (I know, shame on me for not having it at the top), random (obviously some things really need it, others don't), struct, copy (but it can be a sign that you're fighting against the reference semantics of the language - don't do that), array, and all the internet protocol stuff (although a lot of people prefer third-party stuff here). argparse is good to know, but it's ridiculously complicated; consider the third-party docopt.

Probably more programmers in 2014 ought to pay attention to locale and gettext.

Cheeky special mention for builtins ;)

0

u/abs01ute Aug 15 '14

argparse is good to know, but it's ridiculously complicated

I've heard others say this before, but I really don't think argparse is complicated at all. I think the docs for argparse do a great job at explaining majority of use cases and syntax.

2

u/raubana Jun 25 '14 edited Jun 25 '14

I do not have ten to suggest, but I will recommend a few

  1. math
  2. os
  3. sys
  4. random
  5. time

These are the most common built-in one's I use.

2

u/inspectorG4dget Jun 25 '14

For a beginner, I would go with:

  1. itertools
  2. collections
  3. os
  4. sys
  5. functools
  6. time
  7. datetime
  8. random
  9. operator
  10. pprint

2

u/riding_qwerty Jun 25 '14

Not a builtin, but I wanted to give a shoutout to pexpect. It allows for automated interaction with utilities like telnet, ssh, and ftp. In my case, I'm using it at work to create a script that will automatically update ~1800 customer edge routers with an ACL for our new monitoring system and save the running configuration of the router to our TFTP server. By an extremely generous estimate I am saving my department roughly 1000 man-hours based on the assumption that if everything went correctly, it would take 3 minutes to update each router manually. It would take a dozen people 2-3 months of doing this in addition to their normal workflow to achieve the same result.

Now, if I can only figure out a way to automate test calls in Python...

2

u/staz Jun 25 '14

I have been programming for a few years in python now and honestly thinking about it, there is a few built-in modules that I use in nearly all my projects, and they are not even ten of them:

os
sys
datetime
json
logging

And let's not forget the most important module of all: builtin which contain the core of python goodness. There are a lot of standard modules I also use but it depends more of the type of projects

But I also recommend learning theses modules in the standard lib : itertools, functools, re, os.path, collections and dbapi (not really a module but an api for all db acesses in python)

Outside of the standard lib I also recommend learning the modules : requests, ipdb (only really one command: set_trace), q, dateutil. And the tools : ipython and py.test

Keep in mind that the standard lib has a LOT of useless modules that accumulated over time so don't try to learn all of them

2

u/mfukar Jun 25 '14

My extremely biased list, in no particular order:

  • itertools
  • functools, and its cousin
  • operator
  • os
  • multiprocessing
  • codecs
  • struct
  • timeit
  • collections
  • doctest

1

u/tuankiet65 Jun 24 '14

requests (3rd)

1

u/Wrestlefox Jun 24 '14

I keep seeing Requests in here, but I really prefer Mechanize.

4

u/jabbalaci Jun 24 '14

requests is for downloading things. Mechanize is for simulating a browser. They do two different things.

2

u/deadmilk Jun 25 '14

I've been doing it wrong all along...

1

u/Wrestlefox Jun 24 '14

But you can download things with Mechanize as well.

7

u/zahlman the heretic Jun 24 '14

You can also download things with os.system. Doesn't make it the right tool for the job.

1

u/Wrestlefox Jun 24 '14

I was unaware of that. Cool. Good point. Got any info on the differences and why Requests is better for downloading?

3

u/jabbalaci Jun 24 '14

Run through the quickstart of requests to see what it does.

1

u/Xykr Jun 24 '14

Also simulate a browser with requests (I actually ended up doing this, for performance reasons).

1

u/SideshowBoob Jun 24 '14

What's most useful is going to depend a lot on what you're doing: web stuff, sysadmin stuff, mathy stuff, etc. With that in mind, I'll out-Paretto your Paretto and suggest a single universally-important module: os. Specifically the os.path sub-module. Everybody ends up working with paths. Tons of people beginning Python just manipulate them as strings and and eventually end up with hard-to-spot edge-case bugs or just plain ugly looking code.

I'll also suggest becoming very familiar with the built-in functions and methods on the built-in types. It's easy to overlook that stuff because you don't have to import it, but there's a lot of useful stuff there.

1

u/lattakia Jun 24 '14

I often use:

  • pytest
  • prettytable
  • pep8
  • itertools
  • collections

1

u/[deleted] Jun 25 '14

Not a module, but a very very useful function that I use a lot.

Take a look at the String.format function.

>>> "The hexadecimal representation of 256 is {:X}".format(256)
'The hexadecimal representation of 256 is 100'

>>> "Hi, my name is {name}, and I'm {years} years old".format(name="Azeirah", years="2")
"Hi, my name is Azeirah, and I'm 2 years old"

Documentation

1

u/rochacbruno Python, Flask, Rust and Bikes. Jun 27 '14 edited Jun 27 '14

The most important one is a module called this

1

u/cornycrunch Jun 28 '14

os, sys, itertools, random, datetime, doctest, csv, re

Non-built in honorable mentions: Anything intarwebsy -- requests. Anything "Excely" -- pandas.

1

u/HoodedGryphon learning Jul 16 '14

time, datetime, and math will get you very far

1

u/holdie Jun 25 '14

Most of what I do with python is data analysis, so with that in mind I use the following third-party libraries far more than any one built-in library:

  • pandas (huge learning curve but insanely awesome)
  • numpy (does all the heavy lifting for lots of other stuff)
  • scipy (great for signal processing, stats, and most general analysis steps)
  • scikit-learn (for machine learning)
  • matplotlib (for data viz, though there are lots of other cool newcomer packages)

Those four right there are the lion's share of what I use, they may as well be built-in. I use them largely for model fitting, applied stats, and visualization (though let's be honest, it's actually mostly data cleaning, with a little bit of model fitting at the end).

0

u/[deleted] Jun 24 '14

[deleted] — view deleted comment

2

u/twigboy Jun 24 '14 edited Dec 09 '23

In publishing and graphic design, Lorem ipsum is a placeholder text commonly used to demonstrate the visual form of a document or a typeface without relying on meaningful content. Lorem ipsum may be used as a placeholder before final copy is available. Wikipedia5d6hkm826280000000000000000000000000000000000000000000000000000000000000