Web Scraping a JavaScript Heavy Website: Keeping Things Simple

One of the most common difficulties with web scraping is pulling information from sites that do a lot of rendering on the client side. When faced with scraping a site like this, many programmers reach for very heavy-handed solutions like headless browsers or frameworks like Selenium. Fortunately, there's usually a much simpler way to get the information you need.

But before we dive into that, let's first take a step back and talk about how browsers work so we know where we're headed. When you navigate to a site that does a lot of rendering in the browser -- like Twitter or Forecast.io -- what really happens?

First, your browser makes a single request for an HTML document. That document contains enough information to bootstrap the loading of the rest of the page. It loads some basic markup, potentially some inline CSS and Javascript, and probably a few <script> and <link> elements that point to other resources that the browser must then download in order to finish rendering the page.

Before the days of heavy JavaScript usage, the original HTML document contained all the content on the page. Any external calls to load CSS of Javascript were merely to enhance the presentation or behavior of the page, not change the actual content.

But on sites that rely on the client to do most of the page rendering, the original HTML document is essentially a blank slate, waiting to be filled in asynchronously. In the words of Jamie Edberg -- first paid employee at Reddit and currently a Reliability Architect at Netflix -- when the page first loads, you often "get a rectangle with a lot of divs, and API calls are made to fill out all the divs."

To see exactly what this "rectangle with a lot of divs" looks like, try navigating to sites like Twitter or Forecast.io with Javascript turned off in your browser. This will prevent any client-side rendering from happening and allow you to see what the original page looks like before content is added asynchronously.

Once you've seen the content that comes with the original HTML document, you'll start to realize how much of the content is actually being pulled in asynchronously. But rather than wait for the page to load... and then for some Javascript to load... and then for some data to come back from the asynchronous Javascript requests, why not just skip to the final step?

If you examine the network traffic in your browser as the page is loading, you should be able to see what endpoints the page is hitting to load the data. Flip over to the XHR filter inside the "Network" tab in the Chrome web inspector. These are essentially undocumented API endpoints that the web page is using to pull data. You can use them too!

The endpoints are probably returning JSON-encoded information so that the client-side rendering code can parse it an add it to the DOM. This means it's usually straightforward to call those endpoints directly from your application and parse the response. Now you have the data you need without having to execute Javascript or wait for the page to render or any of that nonsense. Just go right to the source of the data!

Let's take a look at how we might do this on Twitter's homepage. When a logged-in user navigates to twitter.com, Tweets are added to a user's timeline with calls to this endpoint. Pull that up in your browser and you'll see a JSON object that contains a big blob of HTML that's injected into the page. Make a call to this endpoint and then parse your info from the response, rather than waiting for the entire page to load.

It's a similar situation when we look at Forecast.io. The HTML document that's returned from the server provides the skeleton for the page, but all of the forecast information is loaded asynchronously. If you pull up your web inspector, refresh the page and then look for the XHR requests in the "Network" tab, you'll see a call to this endpoint that pulls in all the forecast data for your location.

scraping-forecast-io

Now you don't need to load the entire page and wait for the DOM to be ready in order to scrape the information you're looking for. You can go directly to the source to make your application much faster and save yourself a bunch of hassle.

Wanna learn more? I've written a book on web scraping that tons of people have already downloaded. Check it out!

Source: http://tubes.io/blog/2013/08/28/web-scraping-javascript-heavy-website-keeping-things-simple/

Street scraping begins tonight

Finally, city plows will attack residental neighbourhoods tonight to hack down the road ruts that have jolted drivers, damaged the suspension of vehicles and caused accidents.

Only one question: What took them so long?

Last week's dump of snow caused the city to deploy more than 300 pieces of equipment to clear the main streets and collector routes -- even back alleys and sidewalks.

The city had mulled since last Thursday whether to order the plow. But it wasn't decided until Monday afternoon to begin plowing the hundreds of kilometres of residential streets starting today at 7 p.m.

CAA Manitoba wanted faster action. The organization representing drivers believes the hard-and-high ruts made it dangerous to drive city streets.

That is, calling on the city to plow residential streets when ruts -- like the ones left from last week's blizzard -- cause dangerous driving conditions.

"When the roads are in an unsafe condition, then money can't be the first factor," CAA spokeswoman Liz Peters said on Monday.

"Safety has to be the main factor. We think it is time to plow."

Peters said they have reports from its members the ruts are currently so deep, cars can't get out of them to change lanes.

As well, Peters said because there is ice at the bottom of the ruts, vehicles are sliding into other vehicles.

She said one motorist told CAA when they tried to drive out of the ruts they lost control and began spinning.

Forecast snow delayed plow: public works

North Kildonan's Jeff Browaty, a member of Mayor Sam Katz's powerful executive policy committee (EPC), said he doesn't understand why it took so long to clear residential streets.

"I'm surprised and frankly a bit disappointed it took them this long to initiate a full residential plowing operation," Browaty said. "In my own experiences, I've found navigating residential streets in North Kildonan, even at very slow speeds, to be challenging due to the ruts and slick nature of the compacted snow."

Tonight's full residential street plowing brings an accompanying ban on residential street parking.

Coun. Russ Wyatt said council delegated the decision on when to plow to the public works director, Brad Sacher.

Sacher said the city could have tackled residential streets last week but held off until now because more snow was in the forecast.

"We didn't want to start a plow if more snowfall was coming," Sacher said, adding the city believed most streets were passable and the decision was to use the weekend to assess the situation.

With nothing but sunny skies in the forecast, Sacher said residential streets will now be done.

'We must all slow down': Wyatt

Wyatt, finance chairman and also a member of EPC, said he found residential streets were rutted but driveable.

"We must all slow down a bit when driving," Wyatt (Transcona) said. "We live in Winnipeg -- not Waco, Texas.

"Most any other city hit with last Thursday's storm would have been shut down for a week."

A survey of city councillors found driving conditions varied across the city.

Brian Mayes (St. Vital) said parts of his ward were clear but he did receive complaints from residents living near St. George School and River Pointe.

John Orlikow (River Heights-Fort Garry) said he believed driving conditions were "dangerous" in parts of his ward.

Grant Nordman (St. Charles), another member of EPC, said motorists need to relax.

"Was driving good? No; Was it passable? Yes," Nordman said. "It's Winnipeg, it's winter -- leave earlier, take your time, relax."

Wyatt said the residential plow will exhaust city's 2013 snow clearing budget, adding he's concerned another snow storm this month will push the city into a deficit for this year.

"I think we should have waited," Wyatt said.

"The streets were passable."

What’s the worst road you’ve driven on since last week’s storm?

Join the conversation in the comments below.

WORD ON THE STREET

MECHANICS: Vehicles don't like it rough

If you feel jarred by ruts that have carved up Winnipeg streets, your vehicle feels your pain.

The ruts cause problems in the front end of a vehicle, specifically in the alignment, steering components and suspension, Rudy Epp owner of Rudy's Auto Service said.

If tire pressure is low, the ruts can damage rims and tires above and beyond what may happen to the suspension, said Myron Naumik, body shop manager of Macdonald Auto Body.

"Just turn up the radio and take it slow, and make sure you have the proper tire pressure."

POLICE: A question of traction

Winnipeg does not put snow tires on police cruisers. "It's something we would like to see, but ultimately it's the service's responsibility. Our members are well-trained drivers," said George Van Mackelbergh, vice-president of the Winnipeg Police Association.

"The city is talking about cutbacks to the police budget. If they cut back services they won't be paying money for snow tires.

"But it's a difficult time to respond to calls because the other drivers are getting reacquainted with the dangerous conditions."

FIREFIGHTERS: Heavy trucks don't need snow tires

The large trucks firefighters drive probably don't need snow tires, said Alex Forrest, president of the United Firefighters of Winnipeg.

Forrest said firefighters respond to 93,000 fire and paramedic calls per year and many of those are in winter driving conditions.

"It's part of being a firefighter here. We're a winter city and we go in the worst conditions."

Source:http://www.winnipegfreepress.com/local/street-scraping-begins-tonight-235188231.html