Tools to Work with the 'Splash' JavaScript Rendering Service in R
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

138 lines
5.3 KiB

4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
4 years ago
  1. ---
  2. output: rmarkdown::github_document
  3. ---
  4. `splashr` : Tools to Work with the 'Splash' JavaScript Rendering Service
  5. TL;DR: This package works with Splash rendering servers which are really just a REST API & `lua` scripting interface to a QT browser. It's an alternative to the Selenium ecosystem which was really engineered for application testing & validation.
  6. Sometimes, all you need is a page scrape after javascript has been allowed to roam wild and free over your meticulously crafted HTML tags. So, this package does not do _everything_ Selenium can, but if you're just trying to get a page back that needs javascript rendering, this is a nice alternative.
  7. It's also an alternative to `phantomjs` (which you can use in R within or without a Selenium context as it's it's own webdriver) and it may be useful to compare renderings between this package & `phantomjs`.
  8. You can also get it running with two commands:
  9. sudo docker pull scrapinghub/splash
  10. sudo docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 scrapinghub/splash
  11. (Do whatever you Windows ppl do with Docker on your systems to make ^^ work.)
  12. You can run Selenium in Docker, so this is not unique to Splash. But, a Docker context makes it so that you don't have to run or maintain icky Python stuff directly on your system. Leave it in the abandoned warehouse district where it belongs.
  13. All you need for this package to work is a running Splash instance. You provide the host/port for it and it's scrape-tastic fun from there!
  14. ### About Splash
  15. >'Splash' <https://github.com/scrapinghub/splash> is a javascript rendering service. It’s a lightweight web browser with an 'HTTP' API, implemented in Python using 'Twisted'and 'QT' and provides some of the core functionality of the 'RSelenium' or 'seleniumPipes' R packages but with a Java-free footprint. The (twisted) 'QT' reactor is used to make the sever fully asynchronous allowing to take advantage of 'webkit' concurrency via QT main loop. Some of Splash features include the ability to process multiple webpages in parallel; retrieving HTML results and/or take screenshots; disabling images or use Adblock Plus rules to make rendering faster; executing custom JavaScript in page context; getting detailed rendering info in HAR format.
  16. The following functions are implemented:
  17. - `render_html`: Return the HTML of the javascript-rendered page.
  18. - `render_har`: Return information about Splash interaction with a website in [HAR](http://www.softwareishard.com/blog/har-12-spec/) format.
  19. - `render_jpeg`: Return a image (in JPEG format) of the javascript-rendered page.
  20. - `render_png`: Return a image (in PNG format) of the javascript-rendered page.
  21. - `splash`: Configure parameters for connecting to a Splash server
  22. Some functions from `HARtools` are imported/exported and `%>%` is imported/exported.
  23. ### TODO
  24. Suggest more in a feature req!
  25. - <strike>Implement `render.json`</strike>
  26. - Implement `execute` (you can script Splash!)
  27. - <strike>Add integration with [`HARtools`](https://github.com/johndharrison/HARtools)</strike>
  28. - _Possibly_ writing R function wrappers to start Splash which would also support enabling javascript profiles, request filters and proxy profiles from with R directly, possibly using [`harbor`](https://github.com/wch/harbor)
  29. - Testing results with all combinations of parameters
  30. ### Installation
  31. ```{r eval=FALSE}
  32. devtools::install_github("hrbrmstr/splashr")
  33. ```
  34. ```{r message=FALSE, warning=FALSE, error=FALSE}
  35. options(width=120)
  36. ```
  37. ### Usage
  38. ```{r message=FALSE, warning=FALSE, error=FALSE}
  39. library(splashr)
  40. library(magick)
  41. library(rvest)
  42. library(anytime)
  43. library(hrbrmisc) # github
  44. library(tidyverse)
  45. # current verison
  46. packageVersion("splashr")
  47. splash("splash", 8050L) %>%
  48. splash_active()
  49. splash("splash", 8050L) %>%
  50. splash_debug()
  51. ```
  52. Notice the difference between a rendered HTML scrape and a non-rendered one:
  53. ```{r}
  54. splash("splash", 8050L) %>%
  55. render_html("http://marvel.com/universe/Captain_America_(Steve_Rogers)")
  56. read_html("http://marvel.com/universe/Captain_America_(Steve_Rogers)")
  57. ```
  58. You can also profile pages:
  59. ```{r}
  60. splash("splash", 8050L) %>%
  61. render_har("http://www.poynter.org/") -> har
  62. print(har)
  63. ```
  64. You can use [`HARtools::HARviewer`](https://github.com/johndharrison/HARtools/blob/master/R/HARviewer.R) — which this pkg import/exports — to get view the HAR in an interactive HTML widget.
  65. Full web page snapshots are easy-peasy too:
  66. ```{r eval=FALSE}
  67. splash("splash", 8050L) %>%
  68. render_png("http://marvel.com/universe/Captain_America_(Steve_Rogers)")
  69. ```
  70. ```{r eval=TRUE, include=FALSE}
  71. splash("splash", 8050L) %>%
  72. render_png("http://marvel.com/universe/Captain_America_(Steve_Rogers)") %>%
  73. image_write("img/cap.png")
  74. ```
  75. ![](img/cap.png)
  76. ```{r eval=FALSE}
  77. splash("splash", 8050L) %>%
  78. render_jpeg("http://marvel.com/universe/Captain_America_(Steve_Rogers)")
  79. ```
  80. ```{r eval=TRUE, include=FALSE}
  81. splash("splash", 8050L) %>%
  82. render_jpeg("http://marvel.com/universe/Captain_America_(Steve_Rogers)") %>%
  83. image_write("img/cap.jpg")
  84. ```
  85. ![](img/cap.jpg)
  86. ### Test Results
  87. ```{r message=FALSE, warning=FALSE, error=FALSE}
  88. library(splashr)
  89. library(testthat)
  90. date()
  91. test_dir("tests/")
  92. ```
  93. ### Code of Conduct
  94. Please note that this project is released with a [Contributor Code of Conduct](CONDUCT.md). By participating in this project you agree to abide by its terms.