# rvest

rvest is a new R package to make it easy to scrape information from web pages. In this example, we show a simple scraping task using pipeR's Pipe() together with side effects to indicate scraping process.

In this example, we scrape the description of CRAN packages and list the most popular keywords.

First, we load the libaries we need.

library(rvest) # devtools::install_github("hadley/rvest")
library(rlist) # devtools::install_github("rlist","renkun-ken")
library(pipeR)


Then we build a pipeline to scrape the texts in the description column, split the texts into words, create a table in which the most popular keywords are listed. To monitor the process, we add some side effects using message() to indicate the working progress.

url <- "http://cran.r-project.org/web/packages/available_packages_by_date.html"
Pipe(url)$.(~ message(Sys.time(),": downloading"))$
html()$html_nodes(xpath = "//tr//td[3]")$
.(~ message("number of packages: ", length(.)))$html_text(trim = TRUE)$
.(~ message(Sys.time(),": text extracted"))$list.map(Pipe(.)$
strsplit("[^a-zA-Z]")$unlist(use.names = FALSE)$
tolower()$list.filter(nchar(.) > 3L)$
value)$# put everything in a large character vector unlist()$
# create a table of word count
table()$# sort the table descending sort(decreasing = TRUE)$
# take out the first 100 elements

# 2015-03-29 09:02:30: downloading
# number of packages: 6457
# 2015-03-29 09:02:39: text extracted

# <Pipe: array>
# .
#           data       analysis         models           with      functions
#            978            822            542            451            398
#     regression     estimation        package          model          using
#            340            311            301            289            273
#          tools          based           from       bayesian         linear
#            264            255            219            197            197
#        methods           time      interface   multivariate    statistical
#            189            184            182            147            137
#    generalized           test     clustering         series          tests
#            132            121            117            117            117
#      inference   distribution     statistics      selection         random
#            111            109            109            107            103
#      algorithm        spatial       multiple       modeling     simulation
#             99             99             95             94             91
#          mixed  distributions     likelihood         method      modelling
#             89             83             83             80             79
#        network         sparse         robust classification           sets
#             79             79             77             76             73
#        mixture       survival       sampling        effects           high
#             71             69             68             67             67


As we have pointed out, the side effects use special syntax so it is easy to distinguish mainstream pipeline and side effect steps.