Webscraping with Rvest

Rvest works fine with tabular data. If, however, you are working with data outside of Wikipedia, you will find that website data is very rarely available in a <table> and is instead part of a hierarchical tree, which is a pain to process/clean in R.

In such cases, working with Python/BeautifulSoup4 and importing the clean and normalized data into R will save frustration over time, even offsetting the overhead of using two languages.

The reason why so many people were mixing Python code with R was specifically for these sort of task. Web scraping in R has really caused me to not touch another tool outside of R for a few years now and it is great.

Well done Hadley Wickham being inspired by libraries like Beautiful Soup and bringing a great tool to R.

It really looks as easy as it can get. The good part of R is that many R packages are designed in a similar way (highly specialized methods, doing a good job). Combining that with %>% makes you really efficient.

This seems really an intuitive way of getting the tables. What would be the most similar library in python for those cases where R isn't available in the system (with the permissions in some labs machines, unfortunately it takes weeks-forever to get R installed)?

I've written a few blog posts where I used Rvest to get data and R's great visualization tools to visualize it. R has a ton of issues as a platform and language but this is a fantastic package and it has a great ecosystem for small data (the majority of data).

all of these web scraping frameworks, doesn't it tell you that the web needs more wide spread semantic markup?

<jobs-list> <job> <employer> YCombinator </employer> <position> ... </position> </job> </jobs-list>

Something like that?

I know what everyone will say, it is so terse and convoluted, but maybe something like

<ul semantic-markup="jobs"> <li semantic-markup="job"> <p semantic-markup="job-employer"> YCombinator </p> <p semantic-markup="job-position"> ... </p> </li> </ul>

Seems like a lot of work though...maybe I take that back.

I've had great success with Ruby/Mechanize for regular html scraping and phantomjs for dynamic page scraping.

Rvest is for webscrapping newbs. A more seasoned R person would still use PhantomJS and RSelenium as it actually collects all the page's information but Rvest only collections a portion of it. Try it on washingtonpost.com and you will see.