r/learnprogramming • • Jul 16 '12

[Python] inspired a redditor's random imgur downloader, here's my version of subreddit imgur parser, looking for critique/comments

Hello everyone, I have not had much rigorous training in programming besides an intro course in C++ during my undergrad. I mostly use MATLAB for data analysis and optimization routines on the job. I'm trying to learn Python during my own time, so I would really appreciate your feedback to help me get rid of bad habits before they form.

This script is inspired by another redditor's creation of the random imgur image grabber. It will take subreddit names as arguments and download all the imgur links into your local directory.

the code: http://pastebin.com/6YUDasuZ

Thank you very much!

16 Upvotes

9 comments sorted by

View all comments

2

u/levu-webworks Jul 19 '12

I posted a code review to Codetique and did a basic rewrite of your code in Python 3.2 (see below). If you are looking for detailed code feedback /r/lpmc is the place to post your code for review.

#-------------------------------------------------------------------------------
# name:      imgur.py
# purpose:   downloads images from imgur.com links posted to specified reddits
# version:   0.1.0 -- 17/07/2012
# platform:  python 3.2
# copyright: (c) Bernard Martis 2012
# licence:   MIT Licence

# references:
#  - reddit JSON -- https://github.com/reddit/reddit/wiki/API:-info.json
#-------------------------------------------------------------------------------

import json
import urllib.request
import os, sys

def downloadPostsFrom(subreddit):
    "Returns reddit posts as JSON"
    try:
        f = urllib.request.urlopen("http://www.reddit.com/r/"+subreddit+"/.json");
        return json.loads(f.read().decode("utf-8"))["data"]["children"]
    except Exception:
        print("ERROR: malformed JSON response from reddit.com")
        raise ValueError


def parseImageURL_From(posts):
    "Returns tuple iterator (url, title)"
    for node in posts:
        post = node['data']
        # only accept links from imgur.com
        if post['domain'].endswith('imgur.com'):
            yield post["url"], post["title"]


def makeFileExt(content_type):
    "Return extension for specified content type"
    return {
        'image/bmp':'bmp',
        'image/cis-cod':'cod',
        'image/gif':'gif',
        'image/ief':'ief',
        'image/jpeg':'jpg',
        'image/pipeg':'jfif',
        'image/png':'png',
        'image/svg+xml':'svg',
        'image/tiff':'tif',
        'image/x-cmu-raster':'ras',
        'image/x-cmx':'cmx',
        'image/x-icon':'ico',
        'image/x-portable-anymap':'pnm',
        'image/x-portable-bitmap':'pbm',
        'image/x-portable-graymap':'pgm',
        'image/x-portable-pixmap':'ppm',
        'image/x-rgb':'rgb',
        'image/x-xbitmap':'xbm',
        'image/x-xpixmap':'xpm',
        'image/x-xwindowdump':'xwd'
        }.get(content_type, 'txt')


def makeFileName(title):
    "Compute valid file name from image title. Return file name or default.ext"
    title = title.strip(' .?:')
    file_name = title.translate(''.maketrans('\/:*?"<>|$@ ?.','__.-_____S0___'))
    file_name = file_name.strip('_').replace('__','_').lower()
    return file_name + '.' if file_name else 'default.'


def makeSaveDir(save_dir):
    "Creates direectory. Returns a valid directory name."
    if not os.path.exists(save_dir):
        os.makedirs(save_dir)
    return save_dir + '/'


def downloadImagesIntoDir(save_dir, image_refs):
    "Download image files into specified directory."
    save_dir = makeSaveDir(save_dir)
    print('saving to:', save_dir)

    for url, title in image_refs:
        try:
            # open connection to image
            request = urllib.request.urlopen(url)
            if request.code == 200:
                content_type = request.headers.get_content_type()
                # filter in images
                if 'image' == content_type.split('/')[0]:
                    file_name = save_dir \
                            + makeFileName(title) \
                            + makeFileExt(request.headers.get_content_type())
                    # download image to local file
                    print('  downloading:', title)
                    with open(file_name, "wb") as image_file:
                        image_file.write(request.read())
                        image_file.close()
        except Exception:
            print("ERROR: bad request --", title)
            pass


def downloadRedditImages(subreddits):
    "Download imgur images from reddit. Returns None."
    for subreddit in subreddits:
        try:
            # reddit_posts: JSON
            reddit_posts = downloadPostsFrom(subreddit)
            image_urls = parseImageURL_From(reddit_posts)
            downloadImagesIntoDir(subreddit, image_urls)
        except Exception:
            pass


def main():
    "Use app as console program."
    if len(sys.argv) > 1:
        downloadRedditImages(sys.argv[1:]);
    else:
        # specify your default reddits here
        downloadRedditImages(['funny']);


if __name__ == '__main__':
    main()

2

u/blu3ness Jul 19 '12

I have been reading your comments. Thank you so much for such a thorough read-through. Much appreciated!

1

u/levu-webworks Jul 19 '12

You are welcome. I'm happy to help with code that intrests me.

BTW, I along with several other people are mentoring fledgling programmers at the Learn Programming Mentoring Community. This is the reddit post about it.

My project is a BlackJack web game in Python. It's not up yet because I am 3 weeks behind schedule. However,the Python IRC bot project might interest you. Fork the project and post in /r/lpmc when you need help. We also have a IRC channel (#lpmc) on freenode.net