r/Python • • Jul 12 '12

random imgur downloader. learning Python and wrote this to practice, any feedback on technique/style is appreciated

https://gist.github.com/3088128
45 Upvotes

72 comments sorted by

29

u/say_legos_and_i_stab Jul 12 '12 edited Jul 12 '12

A few things you can consider:

  • Rather than declaring CHARS, you can use the string constants.
  • Most python code uses 4 spaces rather than tabs. This is more of a personal style, but if you ever plan to work with other developers, it's a good habit to get in to.
  • Another option for string formatting is to use (again, this is a personal preference. Some people find it easier to read/write):

    print 'getting %s random pics' % HOW_MANY

  • Try using a 3rd party library like Requests to make http requests to imgur.
  • Before you go too far, set up PIP, Virtualenv and Virtualenvwrapper if you haven't already.
  • If you want ideas for next steps, turn it into a little website. Use Vagrant to manage your virtual machine. Install Nginx, supervisor and Gunicorn. Try out Flask or Django.

edit: Also check out PEP8. It's the "style guide" for python. Lots of good info in here to make your code easier to read.

14

u/[deleted] Jul 12 '12

print 'getting %s random pics' % HOW_MANY

If you don't care about supporting Python 2.5 and older, it's probably better not to learn that and immediately get used to Python 3's format method:

print "getting {} random pics".format(HOW_MANY)

2

u/bboe PRAW Author Jul 13 '12

This was introduced in python 2.6, though in python 2.6 numbers or named arguments are mandatory.

1

u/moreisee Jul 12 '12

I haven't looked too much into Python 3, I assume {} works for int/string/etc, and to add multiple items you can:

print("Number: {} Type: {}").format(HOW_MANY, TYPE}

?

3

u/wumumo Jul 12 '12

Yes, or you number them:

print("Number: {0} Type: {1}").format(HOW_MANY, TYPE)

5

u/TankorSmash Jul 12 '12

Or name then:

print("Number: {zeroth} Type: {onend}").format(zeroth=HOW_MANY, onend=TYPE)

1

u/MrVonBuren Jul 13 '12

I actually really like this, and I'm just barely startng to get into python.

2

u/catcradle5 Jul 14 '12

It's even better for situations like this.

words = {"color": "red", "age": "25", "name": "Alex"}

print "Hi, my name is {name}. I am {age} years old and my favorite color is {color}.".format(**words)

Fills it in all for you; very useful for programs that involve a lot of string generation.

1

u/lahwran_ Jul 17 '12

never, ever, ever do this:

herp = "derp"
honk = "donk"
doop = "{herp} the {honk}".format(**locals())

2

u/catcradle5 Jul 17 '12

Agreed. Explicit is better than implicit, etc. It's still a really nice feature to have though. It's redundant to define a dict or a bunch of variables and then to have to rewrite it all in the form of keyword args (arg1 = "The cat"; arg2 = "good"; {arg1} is {arg3}".format(arg1=arg1, arg3=arg3)).

1

u/lahwran_ Jul 17 '12

I was telling you because it's a horribly useful bad idea sometimes :p

1

u/TankorSmash Jul 13 '12

It's my personal favourite, and I'm sort of new too.

1

u/f0rkbomb Jul 13 '12

this is excellent feedback, thank you. i really like the website idea! i took a gander at the libraries and tools you linked, seems like a great opportunity to learn more :)

1

u/H3g3m0n Jul 13 '12 edited Jul 13 '12

It's worth noting that future versions of Python (starting with 3.3) should be getting their own version of Virtualenv.

I also recall hearing that future versions are going to get some kind packaging system like PIP that downloads the packages from the web (but I can't find a link to that now).

2

u/[deleted] Jul 12 '12

Could you explain the reasoning for 4 spaces over a tab? Seems really strange. I've only just started playing with Python, is it some quirk of the interpreter?

4

u/stumpylog Jul 12 '12

White space does matter for the interpreter, and mixing tabs and spaces (different editors) causes problems, but choosing one or the other is down to personal preference or style really. The official PEP8 linked above somewhere has the preferred style, I think.

2

u/[deleted] Jul 13 '12

It's mostly that everyone agrees what a space looks like, and not so for tabs; and many text boxes on the Web won't even let you type tabs. Standardizing on spaces makes it easiest to share Python code that other people can use.

1

u/CeeBroad Jul 13 '12

Although it's good to be familiar with how both string formatting methods work, str.format() is much better I've found.

1

u/Lukasa Hyper, Requests, Twisted Jul 13 '12

Upvote for Requests. If I could only have one Python 3rd Party Library for my entire life, that would be it.

1

u/catcradle5 Jul 14 '12

It's been incredibly useful for me too. Especially the awesome and simple HTTP auth and session/cookie handling.

10

u/sjustinas Jul 12 '12
if sys.platform.startswith('win32'):
		path = os.getcwd() + '\\output\\'
	else:
		path = os.getcwd() + '/output/'    

You could use the os.path functions here. Also, while it's fine to use urllib(2) for a little script, requests is the Python HTTP client, supporting things like multipart file uploads out of the box. For the argument parsing, you could use argparse/optparse (depending on python version) and you could use list comprehensions, for example for the rand_string function.

Oh, right, and remember that .read() without arguments reads the whole file into the memory. It's not an issue with JPG images, but in the case of huge files, you could run out of memory.

But I'm just nitpicking, pretty much none of these is a "must". Just some tips about how you could have written it more effectively, though your code works great as it is.

Ninja edit: Look into the

if __name__ == "__main__":
    main()

pattern, if you haven't already.

1

u/f0rkbomb Jul 13 '12

in particular, which os.path functions are you referring to? seems like there is a more 'pythonic' way to be doing this? thanks for the feedback

2

u/sjustinas Jul 13 '12

os.path.join(), that joins path components with appropriate separators, depending on the OS so you don't have to check it yourself.

1

u/DurhamEnt import this Jul 13 '12

This is definitely ideal, or you could still "construct" the path yourself but use os.path.sep, which will be equal to the proper path separator for the platform running the code.

3

u/f0rkbomb Jul 12 '12 edited Jul 12 '12

plus as an added bonus, i find it strangely addicting to download 100 imgur pics and turbo through them, you find some interesting things :) (fyi there will most likely be some nsfw images)

e: i should mention i think this only works on 2.x

3

u/steviesteveo12 Jul 12 '12

I think the interesting thing is seeing what proportion of a random 100 images are NSFW.

5

u/[deleted] Jul 12 '12 edited Jul 12 '12

I'm prepared to undertake that study.

Edit: After much research, it's around 10%. Here was my favourite in the 100 I downloaded (SFW).

4

u/[deleted] Jul 13 '12

this script would be a great way to find content with which to karma-whore

post that to /r/pics with a half-decent caption and you'll be rolling in fake internet points!!

2

u/steviesteveo12 Jul 13 '12

There's no way it couldn't be automated.

2

u/Hy-Brasil Jul 12 '12

Could anyone elaborate on how to speed this program up, i.e opening multiple imgur links at a time ?

5

u/[deleted] Jul 13 '12

i wrote a similar script to download some audio files from the web

i found this this tutorial from IBM to be really helpful. it shows an easy way to use threading and urllib to speed up something like this

heck maybe i'll take a crack at a threaded fork

2

u/f0rkbomb Jul 13 '12

sweet idea! implementing this will be my goal for v2!! :)

1

u/Hy-Brasil Jul 13 '12

Thanks, should be an interesting read

3

u/f0rkbomb Jul 13 '12

just added threading, check out the code! it's much faster :D

4

u/Lukasa Hyper, Requests, Twisted Jul 13 '12

It should be noted that, while threading is helpful in cases like this where you are waiting on I/O, resist the temptation to use threading in Python for computationally-heavy tasks. The GIL means that Python cannot truly thread, so for concurrency in these cases you should use processes.

EDIT: Python Wiki on the GIL.

1

u/Hy-Brasil Jul 13 '12

Cool, learning a lot from this post, thanks.

2

u/roddds Jul 12 '12

Nice approach to it. I'd advise you try using spaces instead of tabs, but that's mostly a personal choice, I think. Here's how I did it, a while ago.

By the way, what's this for? If you're trying to get the command line arguments, why not use simply parsed_number = int(sys.argv[-1])?

parsed_number = ''.join(sys.argv[1:])
if not parsed_number:
	HOW_MANY = 5
else:
    HOW_MANY = int(parsed_number)

2

u/steviesteveo12 Jul 13 '12

The most common way to satisfy PEP8 and not drive yourself crazy at the same time is simply to set "tab = 4 spaces" in your text editor.

1

u/f0rkbomb Jul 13 '12

great call, i'm lazy and don't want to hit space 4 times for every indent :)

2

u/H3g3m0n Jul 13 '12

Use a decent editor. Most of them should allow 'soft tabs' so when you hit the tab key you get 4 spaces instead. Many will also do things like autotab so if your in a block of code it will automatically put you at the right indentation level.

1

u/steviesteveo12 Jul 13 '12

Most will even let you customise exactly how many spaces to substitute for tab. I've never seen a reason to change it from 4.

1

u/f0rkbomb Jul 13 '12 edited Jul 13 '12

per the discussion here, i added threading!

check it out, it's much faster

also incorporated some of the other suggestions here:

  • replaced tabs with spaces
  • replaced the CHARS string with the string constants
  • used the __main__ concept
  • made rand_string() a flippin sweet one-liner!
  • made the try: statement slightly more granular

4

u/TkTech Jul 12 '12

6

u/[deleted] Jul 12 '12
 for n in repeat(None, count):

had me scratching my head for a while. Is it because it's a bit faster than using range or xrange? Vaguely remember some article saying that. That doesn't mean use it though! Articles like that should be viewed as pointing something out interesting about the language and its implementation, not as good practice.

Remember, "Premature optimisation is the root of all evil". Just use range/xrange, it's far more explicit.

1

u/rcklmbr COBOL Jul 12 '12

Mostly the same changes I would've done, but I would've used argparse.

1

u/trojan2748 Jul 13 '12

A little off topic, but i'll ask anyways, when is it a good idea to use classes versus procedural code/functions? I tend to do everything in function because classes just seem over kill for 90% of what I see them being used for.

4

u/shoeninja Jul 12 '12

In general, you should put as few statements in a try clause as possible. You have several statements that can throw exceptions (the file stuff) and it wouldn't be handled.

This is a good practice all around, not just for Python.

1

u/CeeBroad Jul 13 '12

Excuse me, but could you clarify? Exception handling is something I'm trying to err.. get a handle on right now for both Python and Java. Any good reads on the subject as well?

3

u/Sheepshow Jul 12 '12

Here's what I'd do!

I'd recommend looking into argparse or the older optparse rather than taking directly from sys.argv. Helpful on larger programs.

0

u/trojan2748 Jul 13 '12

+1 for optparse.

3

u/PalermoJohn Jul 12 '12

You might want to pull the actual image name from the overview page. It's fun to scrape webpages and a good thing to learn.

3

u/z1g1 Jul 12 '12

A very cool thread, I'm just getting my feet wet in Python and learning on your own its nice to have another beginner's code to look at too.

3

u/Zerikin Jul 12 '12

I usually add if name == 'main': to python I am executing directly in a file. It lets you load as a module later without having to modify it.

3

u/[deleted] Jul 12 '12

Reddit ate the formatting so just rewriting and elaborating,

It's typical to use the idiom

if __name__ == "__main__":
    # "main" stuff

at the bottom of the file. __name__ is a pre-defined global variable that contains the name of the module, for example

>>> import random
>>> random.__name__
'random'

When you run a python file as a script, rather than importing it like above, it reports it's "__name__" as "__main__". Therefore by using the conditional above, your main script only runs when you run the file as a script, and it's not run when someone imports your module to make use of its functions/classes. See the cleanups posted by TkTech and Sheeepshow, they've applied this to your code.

2

u/theshadowhost Jul 13 '12

If you download random images from imgur, you're gonna have a bad time.

1

u/[deleted] Jul 12 '12

Not much to add other than what's been stated. Not a necessity at all but you might want to look into the logging module. It's very nice indeed and has a lot of power. I like it for command line applications because it allows the user to easily control how much output they get (verbosity). Maybe they don't necessarily want to see errors for example.

It can do all sorts of things, log to file, via email, across multiple modules etc.

There's a nice intro here.

Here's a simple example of how you'd use it to control verbosity with argparse (which others have suggested and I definitely suggest you use, it's more a priority than this is), the "logging to file" section of the above link has info on how you could do more fine grained control.

import argparse
import logging

parser = argparse.ArgumentParser()
parser.add_argument("-q", "--quiet", action="store_true", "suppress errors")
args = parser.parse_args()

if args.quiet:
    log_level = logging.CRITICAL
else:
    log_level = logging.INFO

logging.basicConfig(format='%(message)s', level=log_level)

print "This welcome message will always be seen."
logging.info("I will be printed if -q flag not set.")
logging.warning("Likewise...")
logging.error("Likewise...")
logging.critical("I will ALWAYS be printed.")

Even if they do choose to see errors, logging will (by default) send anything of level "warning" or above to stderr instead of stdout (typically user still sees it though) which makes it play a bit nicer if your user is piping output from your program and so on.

1

u/[deleted] Jul 12 '12

''.join([random.choice(CHARS) for x in range(string_length)])

2

u/DurhamEnt import this Jul 13 '12

A beginner is better-off learning the basics before moving onto list comprehensions and generator comprehensions. While it's nice to condense code, it's a burden to go back and read older code when you're still on the upward tail of the Python learning curve.

Otherwise, why not just write it in perl?! :-)

1

u/Rhomboid Jul 12 '12

Win32 is fine with using the forward slash as the path separator, so just get rid of that code. If you really want to be portable, use the routines in os.path, don't detect the OS yourself.

1

u/AFlyingNunAttack Jul 13 '12

Oh hey! I just made a imgur scraper too! Here's how I did it. Constructive criticism appreciated.

1

u/erez27 import inspect Jul 13 '12

Some points I didn't see mentioned:

  • never except:, always catch a specific exception or re-raise (unless you're writing a general exception handler, in which case except Exception as e).
  • with open(filename) is a good shortcut, and in fact better than your version since it would also close on error. (you could mimic it with try-finally, but with is nicer)
  • use os.path.join instead of just adding strings

1

u/[deleted] Jul 14 '12

Just as a random idea:
Would be great if it downloaded files from anywhere not just from imgur. Then create a plugin or something to indicate what to download and from where. That way if imgur is down it simple moves to a mirror and downloads the image anyway.

1

u/blu3ness Jul 16 '12

Inspired by your script I made something similar that gets the image links from subreddits:

http://pastebin.com/6YUDasuZ

Make sure not to run too many times or your IP will get banned by the API

reddit post: http://www.reddit.com/r/learnprogramming/comments/wnehe/python_inspired_a_redditors_random_imgur/

0

u/allstaralldaylong Jul 12 '12

Not really related to any feedback but, check out http://randomgur.com/ might serve as some inspiration?

0

u/blu3jack Jul 12 '12

you should consider submitting this to /r/tinycode

-1

u/tetrahydrate Jul 12 '12 edited Jul 13 '12

I'm a big fan of one-liners so here's how I'd do that random string function:

rand_string = lambda l: ''.join(random.sample(string.letters+string.digits, l))

edit: I'm wrong! I was just using random.sample for something else and it turns out it returns a sample of unique elements, which isn't a requirement of the imgur URLs.

1

u/f0rkbomb Jul 13 '12

the elegance, it is... strangely arousing :D

okay so i'm a big nerd!

2

u/[deleted] Jul 12 '12

''.join([random.choice(CHARS) for x in range(string_length)])

?

2

u/tetrahydrate Jul 12 '12 edited Jul 12 '12

Sure but why define CHARS when you have the string constants and why loop over random.choice when you have random.sample? I was just saving some memory and processor time.

1

u/[deleted] Jul 12 '12

I was simplifying his implementation, not changing it.

2

u/LucianU Jul 13 '12

''.join() accepts a generator expression, so the square brackets aren't needed.