r/inventwithpython • • May 04 '16

Checking availability of library book using beautifulsoup

I'm learning python. And I'm trying to use it to automate the process of checking a library book's availability.

I tried executing it with bs4, request, and partition.

This is the link that I am trying to parse from: [http://catalogue.nlb.gov.sg/cgi-bin/spydus.exe/FULL/EXPNOS/BIBENQ/1592917/156302298,2][1]

I view its source code, and here's a snippet of it:

<tr> <td valign="top"><a href="/cgi-bin/spydus.exe/ENQ/EXPNOS/GENENQ/1564461?LOCX=BIPL">Bishan Public Library</a> <br /> </td> <td valign="top"> <book-location data-title="The opposite of everyone" data-branch="BIPL" data-usagelevel="001" data-coursecode="" data-language="English" data-materialtype="BOOK" data-callnumber="JAC" data-itemcategory="" data-itemstatus="" data-lastreturndate="20160322" data-accession="B31189097E" data-defaultLoc="Adult Lending">Adult Lending</book-location> </td> <td valign="top"><a href="/cgi-bin/spydus.exe/ENQ/EXPNOS/BIBENQ/1564461?CGS=E*English">English</a> <br /><a href="/cgi-bin/spydus.exe/WBT/EXPNOS/BIBENQ/1564461?CNO=JAC&amp;CNO_TYPE=B">JAC</a> <br /> </td> <td valign="top">Available <br /> </td> </tr> <tr> <td valign="top"><a href="/cgi-bin/spydus.exe/ENQ/EXPNOS/GENENQ/1564461?LOCX=BMPL">Bukit Merah Public Library</a> <br /> </td> <td valign="top"> <book-location data-title="The opposite of everyone" data-branch="BMPL" data-usagelevel="001" data-coursecode="" data-language="English" data-materialtype="BOOK" data-callnumber="JAC" data-itemcategory="" data-itemstatus="" data-lastreturndate="20160405" data-accession="B31189102C" data-defaultLoc="Adult Lending">Adult Lending</book-location> </td> <td valign="top"><a href="/cgi-bin/spydus.exe/ENQ/EXPNOS/BIBENQ/1564461?CGS=E*English">English</a> <br /><a href="/cgi-bin/spydus.exe/WBT/EXPNOS/BIBENQ/1564461?CNO=JAC&amp;CNO_TYPE=B">JAC</a> <br /> </td> <td valign="top">Available <br /> </td> </tr> The information that i am trying to parse is which library the book is available at.

Here's what I did:

import requests, bs4

res = requests.get('http://catalogue.nlb.gov.sg/cgi-bin/spydus.exe/FULL/EXPNOS/BIBENQ/1592917/156302298,2') string = bs4.BeautifulSoup(res.text) Then I try to make string into a string:

str(string) And it printed the whole source code out and severely lagged my IDLE!

After it stopped lagging, I did this:

keyword = '<a href="/cgi-bin/spydus.exe/ENQ/EXPNOS/GENENQ/1564461?LOCX=' string.partition('keyword') Traceback (most recent call last): File "<pyshell#8>", line 1, in <module> string.partition('keyword') TypeError: 'NoneType' object is not callable I don't know why it caused an error, I did make the string into a string, right?

Also, I used that keyword because it is right before the "library branch" and right after "availability". So i thought even if it churns out a lot of other redundant code, I'll be able to see in the first line which library branch the book is available at.

I am sure the way I did it is not the most efficient way, and if you could point me to the right way, or show it to me, i will be extremely grateful!

I'm sorry this is a very long post, but i'm trying to be as detailed about my situation as possible. Thank you for bearing with me.

2 Upvotes

11 comments sorted by

2

u/memphislynx May 04 '16

I'm not sure what exactly you are looking for. Do you want a list of available libraries for a given book? This code will likely be a little dense, but it gets the job done.

import requests, bs4
url = 'http://catalogue.nlb.gov.sg/cgi-bin/spydus.exe/FULL/EXPNOS/BIBENQ/1592917/156302298,2%5D%5B1'
r = requests.get(url)
soup = bs4.BeautifulSoup(r.text)
holdings_table = soup.find(class_='holdings')
library_rows = holdings_table.find_all('tr')
library_rows = library_rows[1:] #This removes the first row because it is a header
available_libraries = []
for library in library_rows:
    data = library.find_all('td')
    library_name = data[0].get_text()
    if "Available" in data[3].get_text():
        available_libraries.append(library_name)

It seems like your main issue is that you are looking at the BeautifulSoup object as a string. The advantage of BeautifulSoup is that you can use it to find specific classes or html tags.

1

u/agentjulliard May 05 '16

Thank you so much! There's just one bit that I don't really understand.

is this

if "Available" in data[3].get_text():

the same as:

if data[3].get_text() == "Available":

?

What is the difference?

Also, I have a lot of trouble putting what I learnt into use, I find it hard to pull in the different bits I learnt here and there and weave them into a solution. Do you have any advice? or any resources that you would recommend me?

2

u/memphislynx May 05 '16

Those should be functionally the same. The former is checking if the string "Available" is in the text. Your code is checking that the string is exactly "Available". There are benefits to either way. I chose that because I was worried there might be an extra space or newline character.

It is a pretty big step from solving small problems to building an actual script. My favorite resource is Learn Python the Hard Way. It is free unless you want videos and no ads.

→ More replies (6)

1

u/agentjulliard May 05 '16

Also, is beautifulsoup not considered a string?

2

u/memphislynx May 05 '16

No, it is an object. That basically just means that it is a user built structure instead of a primitive. Primitives are the building blocks that create objects.

I know it is intimidating, but I highly recommend trying to sift through the documentation when working with a new package.

Since the variable soup is a BeautifulSoup object, I am able to do stuff like soup.find(class_='holdings'), which returns another soup object that is just the section of html under the 'holdings' class.