Tools to Work with the 'Splash' JavaScript Rendering Service in R
選択できるのは25トピックまでです。 トピックは、先頭が英数字で、英数字とダッシュ('-')を使用した35文字以内のものにしてください。
boB Rudis 5c2b43b154 initial commit 7年前
R initial commit 7年前
img initial commit 7年前
man initial commit 7年前
tests initial commit 7年前
.Rbuildignore initial commit 7年前
.gitignore initial commit 7年前
.travis.yml initial commit 7年前
DESCRIPTION initial commit 7年前
NAMESPACE initial commit 7年前
NEWS.md initial commit 7年前
README.Rmd initial commit 7年前
README.md initial commit 7年前
splashr.Rproj initial commit 7年前

README.md

splashr : Tools to Work with the 'Splash' JavaScript Rendering Service

Ridicuously basic functionality working at the moment. More coming soon

TL;DR: This package works with Splash rendering servers which are really just a REST API & lua scripting interface to a QT browser. It's an alternative to the Selenium ecosystem and does not do everything Selenium can, but if you're just trying to get a page back that needs javascript rendering, this is a nice alternative.

You can also get it running with two commands:

sudo docker pull scrapinghub/splash
sudo docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash

(Do whatever you Windows ppl do with Docker on your systems to make ^^ work.)

All you need for this package to work is a running Splash instance. You provide the host/port for it and it's scrape-tastic from there.

About Splash

'Splash' https://github.com/scrapinghub/splash is a javascript rendering service. It’s a lightweight web browser with an 'HTTP' API, implemented in Python using 'Twisted'and 'QT' and provides some of the core functionality of the 'RSelenium' or 'seleniumPipes'R pacakges but with a Java-free footprint. The (twisted) 'QT' reactor is used to make the sever fully asynchronous allowing to take advantage of 'webkit' concurrency via QT main loop. Some of Splash features include the ability to process multiple webpages in parallel; retrieving HTML results and/or take screenshots; disabling images or use Adblock Plus rules to make rendering faster; executing custom JavaScript in page context; getting detailed rendering info in HAR format.

The following functions are implemented:

  • render_html: Return the HTML of the javascript-rendered page.
  • render_jpeg: Return a image (in JPEG format) of the javascript-rendered page.
  • render_png: Return a image (in PNG format) of the javascript-rendered page.
  • splash: Configure parameters for connecting to a Splash server
  • splashr: Tools to Work with the 'Splash' JavaScript Rendering Service

Installation

devtools::install_github("hrbrmstr/splashr")
options(width=120)

Usage

library(splashr)
library(magick)
library(rvest)

# current verison
packageVersion("splashr")
## [1] '0.1.0'
splash("splash", 8050L) %>%
  splash_active()
## Status of splash instance on [http://splash:8050]: ok. Max RSS: 349298688
splash("splash", 8050L) %>%
  splash_debug()
## List of 7
##  $ active  : list()
##  $ argcache: int 0
##  $ fds     : int 18
##  $ leaks   :List of 4
##   ..$ Deferred  : int 50
##   ..$ LuaRuntime: int 1
##   ..$ QTimer    : int 1
##   ..$ Request   : int 1
##  $ maxrss  : int 341112
##  $ qsize   : int 0
##  $ url     : chr "http://splash:8050"
##  - attr(*, "class")= chr [1:2] "splash_debug" "list"
## NULL

Notice the difference between a rendered HTML scrape and a non-rendered one:

splash("splash", 8050L) %>%
  render_html("http://marvel.com/universe/Captain_America_(Steve_Rogers)")
## {xml_document}
## <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr">
## [1] <head>\n<script src="http://widget-cdn.rpxnow.com/manifest/login?version=1.114.1_widgets_244" type="text/javascri ...
## [2] <body id="index-index" class="index-index" onload="findLinks('myLink');">\n\n\t<div id="page_frame" style="overfl ...
read_html("http://marvel.com/universe/Captain_America_(Steve_Rogers)")
## {xml_document}
## <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr">
## [1] <head>\n<meta http-equiv="X-UA-Compatible" content="IE=Edge">\n<link href="https://plus.google.com/10852333737344 ...
## [2] <body id="index-index" class="index-index" onload="findLinks('myLink');">\n\n\t<div id="page_frame" style="overfl ...

Web page snapshots are easy-peasy too:

splash("splash", 8050L) %>%
  render_png("http://marvel.com/universe/Captain_America_(Steve_Rogers)")

splash("splash", 8050L) %>%
  render_jpeg("http://marvel.com/universe/Captain_America_(Steve_Rogers)") 

Test Results

library(splashr)
library(testthat)

date()
## [1] "Fri Feb  3 14:58:40 2017"
test_dir("tests/")
## testthat results ========================================================================================================
## OK: 0 SKIPPED: 0 FAILED: 0
## 
## DONE ===================================================================================================================