Web Scraping - It's Your Civic Duty - Practical Business Python

Introduction

More and more information from local, state and federal governments is being placed on the web. However, a lot of the data is not presented in a way that is easy to download and manipulate. I think it is an important civic duty for us all to be aware of how government money is spent. Having the data in a more accessible format is a first step in that process.

In this article, I’ll use BeautifulSoup to scrape some data from the Minnesota 2014 Capital Budget. Then I’ll load the data into a pandas DataFrame and create a simple plot showing where the money is going.

My purpose in writing this is not to make any specific political statements about this data set. I chose this data because:

    I personally found it interesting
    I think it translates well across other states and across the world
    It highlights several useful python tools both in and outside of the stdlib
    The process is applicable to other domains outside of government data
    It is a manageable size so you can understand it using basic tools

The Data

I live in MN so thought I would take a look at what sort of budget information is available to us via the various state websites. To be honest, there is a lot of information but it seems like the vast majority is stored in a PDF or on an HTML page.

I applaud the state for making the data available but it is not easy to analyze the data in the way it is currently presented. As I looked through the Minnesota government website, I found this 2014 Capital Budget page which is actually pretty straightforward to understand.

The first part of the document contains a high level summary of all the projects receiving capital dollars as well as how the capital budget will be funded.

The second part of the document has a lot of detail on each of the summary items. For the purpose of this exercise, I am only going to scrape the summary section but the same basic principle can be applied to the detailed line items.

One final note, I realize that this data set is not that large and that you could easily type it all into Excel. However, if we were to scale this to pull in more data, you quickly get to the point where hand typing the data just does not make sense. The principles I walk through will scale to much larger sets. I hope it has the added bonus that you will learn something as well. I know I enjoyed working on this little project.

The Tools

For this particular task, I am going to use 2 very common python tools for scraping the site:

    BeautifulSoup to parse the data
    Requests to get the data from the website.

Strictly speaking, Requests is not being used for much in this case but I think it makes sense to start using it. If/when you start getting more complicated situations, you’ll be happy you are already using it.

Scrapy is another powerful tool for doing web scraping but for my needs BeautifulSoup was perfect so that’s what I’m sticking with for this article. Maybe I’ll look at it for a future article.

Once I scrape the data, I’ll convert it to a pandas DataFrame so that I can analyze and plot the data.

One final note, I’m trying to use idiomatic python as much as possible. My current environment is python 2.7 but I’ll use the print_function to make the python 3 conversion much easier. Also, I’m going to use the defaultdict to streamline the processing of the data. This was first introduced in python 2.5 and is pretty handy when working with dictionaries where the values are lists.

Now we need to initialize the variables. I’m going to use two dictionaries. One will store all of the expense items and the other will include the funding source. Note, I am not going to store the total. We can calculate it so we’ll skip that piece of data. I am using the defaultdict to make it easy to append the values I scrape:

Use requests to get the data and pass it to BeautifulSoup. In my final script, I’m going to store the HTML to disk so that I don’t need to hit the website every time I run it. I won’t show it in this section in order to keep the code short.

Understand Your HTML

The key to understanding any scraping is looking at the HTML and understanding how you want to pull your data out.

In this case, I downloaded the HTML into an editor and collapsed some of the data. It is very helpful that there is a div that wraps the data I need:

Within that div, there are mutliple tables which ultimately contain the info we need:

In the example above, we want to parse out two pieces of data - the description (Universty of Minnesota) and the amount (119,367,000). Another item to note is that the number comes through with commas as well as parenthesis for negative values so we are going to need to clean it up a little. I also found that I pulled in a lot of extra white space in the process, so using string.strip is a good idea.

Here is the clean up function we’ll use:
Now that we know how to get to our tables, use BeautifulSoup’s powerful API to get at our data.
Parse each row in the table and add to the appropriate dictionary depending on whether it is a funding line or expense line

Convert the Data

Our dictionaries contain the data we need, let’s add them to a pandas DataFrame using DataFrame.from_dict() :

It looks like everything was processed correctly. Now, we can analyze the data any way we want.

Plot The Data

In this specific case, I am going to generate a simple horizontal bar graph so that it is easy to see where the biggest expenditures are.

First, I’ll sort both sets of data:

Regardless of the format, I think you’ll agree that viewing the capital budget in this plot yields a lot more insight than the raw HTML data.

Final Thoughts

This little project has been useful for me and I hope it provides a starting point for you to understand how to use various python tools to scrape the web. In this case, I learned a little bit that I think could be applicable to lots of other projects. I also am curious about this little slice of data and intend to look into it some more and see what insight I can glean.

For reference, here is the complete code for this example. This version will download the data to a file and use that locally instead of hitting the site each time.

Source: http://pbpython.com/web-scraping-mn-budget.html

Beginner’s guide to Web Scraping in Python (using Beautiful Soup)

Introduction

The need and importance of extracting data from the web is becoming increasingly loud and clear. Every few weeks, I find myself in a situation where we need to extract data from the web. For example, last week we were thinking of creating an index of hotness and sentiment about various data science courses available on the internet. This would not only require finding out new courses, but also scrape the web for their reviews and then summarizing them in a few metrics! This is one of the problems / products, whose efficacy depends more on web scrapping and information extraction (data collection) than the techniques used to summarize the data.

Ways to extract information from web

There are several ways to extract information from the web. Use of APIs being probably the best way to extract data from a website. Almost all large websites like Twitter, Facebook, Google, Twitter, StackOverflow provide APIs to access their data in a more structured manner. If you can get what you need through an API, it is almost always preferred approach over web scrapping. This is because if you are getting access to structured data from the provider, why would you want to create an engine to extract the same information.

Sadly, not all websites provide an API. Some do it because they do not want the readers to extract huge information in structured way, while others don’t provide APIs due to lack of technical knowledge. What do you do in these cases? Well, we need to scrape the website to fetch the information.

There might be a few other ways like RSS feeds, but they are limited in their use and hence I am not including them in the discussion here.

What is Web Scraping?

Web scraping is a computer software technique of extracting information from websites. This technique mostly focuses on the transformation of unstructured data (HTML format) on the web into structured data (database or spreadsheet).

You can perform web scrapping in various ways, including use of Google Docs to almost every programming language. I would resort to Python because of its ease and rich eocsystem. It has a library known as ‘Beautiful Soup’ which assists this task. In this article, I’ll show you the easiest way to learn web scraping using python programming.

For those of you, who need a non-programming way to extract information out of web pages, you can also look at import.io . It provides a GUI driven interface to perform all basic web scraping operations. The hackers can continue to read this article!

Libraries required for web scraping

As we know, python is a open source programming language. You may find many libraries to perform one function. Hence, it is necessary to find the best to use library. I prefer Beautiful Soup (python library), since it is easy and intuitive to work on. Precisely, I’ll use two Python modules for scraping data:

Urllib2: It is a Python module which can be used for fetching URLs. It defines functions and classes to help with URL actions (basic and digest authentication, redirections, cookies, etc). For more detail refer to the documentation page.

Beautiful Soup: It is an incredible tool for pulling out information from a webpage. You can use it to extract tables, lists, paragraph and you can also put filters to extract information from web pages. In this article, we will use latest version Beautiful Soup 4. You can look at the installation instruction in its documentation page.

Beautiful Soup does not fetch the web page for us. That’s why, I use urllib2 in combination with the BeautifulSoup library.

Python has several other options for HTML scraping in addition to Beatiful Soup. Here are some others:

    -mechanize
    -scrapemark
    -scrapy

Basics – Get familiar with HTML (Tags)

While performing web scarping, we deal with html tags. Thus, we must have good understanding of them.                      
 you already know basics of HTML, you can skip this section. Below is the basic syntax of HTML:
  This syntax has various tags as elaborated below:

    <!DOCTYPE html> : HTML documents must start with a type declaration
      HTML document is contained between <html> and </html>
      The visible part of the HTML document is between <body> and </body>
       HTML headings are defined with the <h1> to <h6> tags
       HTML paragraphs are defined with the <

Scrapping a web Page using Beautiful Soup

Here, I am scraping data from a Wikipedia page. Our final goal is to extract list of state, union territory capitals in India. And some basic detail like establishment, former capital and others form this wikipedia page. Let’s learn with doing this project step wise step:

Import necessary libraries:

#import the library used to query a website
import urllib2
#specify the url
wiki = "https://en.wikipedia.org/wiki/List_of_state_and_union_territory_capitals_in_India"
#Query the website and return the html to the variable 'page'
page = urllib2.urlopen(wiki)
#import the Beautiful soup functions to parse the data returned from the website
from bs4 import Beautiful Soup
#Parse the html in the 'page' variable, and store it in Beautiful Soup format
soup = Beautiful Soup(page)

Use function “prettify” to look at nested structure of HTML page

Above, you can see that structure of the HTML tags. This will help you to know about different available tags and how can you play with these to extract information.

Work with HTML tags

    soup.<tag>: Return content between opening and closing tag including tag.
    In[30]:soup.title
    Out[30]:<title>List of state and union territory capitals in India - Wikipedia, the free encyclopedia</title>
    soup.<tag>.string: Return string within given tag
    In [38]:soup.title.string
    Out[38]:u'List of state and union territory capitals in India - Wikipedia, the free encyclopedia'

Find all the links within page’s <a> tags::  We know that, we can tag a link using tag “<a>”. So, we should go with option soup.a and it should return the links available in the web page. Let’s do it.

    In [40]:soup.a
    Out[40]:<a id="top"></a>

Above, you can see that, we have only one output. Now to extract all the links within <a>, we will use

Above, it is showing all links including titles, links and other information.  Now to show only links, we need to iterate over each a tag and then return the link using attribute “href” with get.

Find the right table: As we are seeking a table to extract information about state capitals, we should identify the right table first. Let’s write the command to extract information within all table tags.

all_tables=soup.find_all('table')

Now to identify the right table, we will use attribute “class” of table and use it to filter the right table. In chrome, you can check the class name by right click on the required table of web page –> Inspect element –> Copy the class name OR go through the output of above command find the class name of right table.

right_table=soup.find('table', class_='wikitable sortable plainrowheaders')

\right_table

Extract the information to DataFrame: Here, we need to iterate through each row (tr) and then assign each element of tr (td) to a variable and append it to a list. Let’s first look at the HTML structure of the table (I am not going to extract information for table heading <th>)
Above, you can notice that second element of <tr> is within tag <th> not <td> so we need to take care for this. Now to access value of each element, we will use “find(text=True)” option with each element.  Let’s look at the code

#Generate lists

A=[]
B=[]
C=[]
D=[]
E=[]
F=[]
G=[]
for row in right_table.findAll("tr"):

    cells = row.findAll('td')
    states=row.findAll('th') #To store second column data
    if len(cells)==6: #Only extract table body not heading
        A.append(cells[0].find(text=True))
        B.append(states[0].find(text=True))
        C.append(cells[1].find(text=True))
        D.append(cells[2].find(text=True))
        E.append(cells[3].find(text=True))
        F.append(cells[4].find(text=True))
        G.append(cells[5].find(text=True))

#import pandas to convert list to data frame

import pandas as pd
df=pd.DataFrame(A,columns=['Number'])
df['State/UT']=B
df['Admin_Capital']=C
df['Legislative_Capital']=D
df['Judiciary_Capital']=E
df['Year_Capital']=F
df['Former_Capital']=G
df

Similarly, you can perform various other types of web scraping using “Beautiful Soup“. This will reduce your manual efforts to collect data from web pages. You can also look at the other attributes like .parent, .contents, .descendants and .next_sibling, .prev_sibling and various attributes to navigate using tag name. These will help you to scrap the web pages effectively.-

But, why can’t I just use Regular Expressions?

Now, if you know regular expressions, you might be thinking that you can write code using regular expression which can do the same thing for you. I definitely had this question. In my experience with Beautiful Soup and Regular expressions to do same thing I found out:

Code written in Beautiful Soup is usually more robust than the one written using regular expressions. Codes written with regular expressions need to be altered with any changes in pages. Even Beautiful Soup needs that in some cases, it is just that Beautiful Soup is relatively better.

Regular expressions are much faster than Beautiful Soup, usually by a factor of 100 in giving the same outcome.

So, it boils down to speed vs. robustness of the code and there is no universal winner here. If the information you are looking for can be extracted with simple regex statements, you should go ahead and use them. For almost any complex work, I usually recommend BeautifulSoup more than regex.

End Note

In this article, we looked at web scraping methods using “Beautiful Soup” and “urllib2” in Python. We also looked at the basics of HTML and perform the web scraping step by step while solving a challenge. I’d recommend you to practice this and use it for collecting data from web pages.


 Source : http://www.analyticsvidhya.com/blog/2015/10/beginner-guide-web-scraping-beautiful-soup-python/