@ -15,7 +15,14 @@ You can also get it running with two commands:
sudo docker pull hrbrmstr/splashttpd
sudo docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 hrbrmstr/splashttpd
(Do whatever you Windows ppl do with Docker on your systems to make ^^ work.)
Do whatever you Windows ppl do with Docker on your systems to make ^^ work.
Folks super-new to Docker on Unix-ish platforms should [make sure to do](https://github.com/hrbrmstr/splashr/issues/3#issuecomment-280686494):
sudo groupadd docker
sudo usermod -aG docker $USER
(`$USER` is your username and shld be defined for you in the environment)
If using the [`harbor`](https://github.com/wch/harbor) package you can use the convience wrappers in this pacakge:
@ -26,7 +33,7 @@ and then run:
stop_splash(splash_container)
when done. All of that happens on your localhost so use `localhost` as the Splash server parameter.
when done. All of that happens on your localhost and you will not need to specify `splash_obj` to many of the `splashr` functions if you're running Splash in this default configuration as long as you use named parameters. You can also use the pre-defined `splash_local` object if you want to use positional parameters.
You can run Selenium in Docker, so this is not unique to Splash. But, a Docker context makes it so that you don't have to run or maintain icky Python stuff directly on your system. Leave it in the abandoned warehouse district where it belongs.
@ -111,6 +118,8 @@ options(width=120)
### Usage
NOTE: ALL of these examples assume Splash is running in the default configuraiton on `localhost` (i.e. started with `start_splash()` or the docker example commands) unless otherwise noted.
```{r message=FALSE, warning=FALSE, error=FALSE}
library(splashr)
library(magick)
@ -124,27 +133,23 @@ library(tidyverse)
# current verison
packageVersion("splashr")
splash("splash", 8050L) %>%
splash_active()
splash("splash", 8050L) %>%
splash_debug()
```
Notice the difference between a rendered HTML scrape and a non-rendered one:
@ -12,7 +12,14 @@ You can also get it running with two commands:
sudo docker pull hrbrmstr/splashttpd
sudo docker run -p 5023:5023 -p 8050:8050 -p 8051:8051 hrbrmstr/splashttpd
(Do whatever you Windows ppl do with Docker on your systems to make ^^ work.)
Do whatever you Windows ppl do with Docker on your systems to make ^^ work.
Folks super-new to Docker on Unix-ish platforms should [make sure to do](https://github.com/hrbrmstr/splashr/issues/3#issuecomment-280686494):
sudo groupadd docker
sudo usermod -aG docker $USER
(`$USER` is your username and shld be defined for you in the environment)
If using the [`harbor`](https://github.com/wch/harbor) package you can use the convience wrappers in this pacakge:
@ -23,7 +30,7 @@ and then run:
stop_splash(splash_container)
when done. All of that happens on your localhost so use `localhost` as the Splash server parameter.
when done. All of that happens on your localhost and you will not need to specify `splash_obj` to many of the `splashr` functions if you're running Splash in this default configuration as long as you use named parameters. You can also use the pre-defined `splash_local` object if you want to use positional parameters.
You can run Selenium in Docker, so this is not unique to Splash. But, a Docker context makes it so that you don't have to run or maintain icky Python stuff directly on your system. Leave it in the abandoned warehouse district where it belongs.
@ -31,7 +38,7 @@ All you need for this package to work is a running Splash instance. You provide
### About Splash
> 'Splash' <https://github.com/scrapinghub/splash> is a javascript rendering service. It’s a lightweight web browser with an 'HTTP' API, implemented in Python using 'Twisted'and 'QT' [and provides some of the core functionality of the 'RSelenium' or 'seleniumPipes' R packages but with a Java-free footprint]. The (twisted) 'QT' reactor is used to make the sever fully asynchronous allowing to take advantage of 'webkit' concurrency via QT main loop. Some of Splash features include the ability to process multiple webpages in parallel; retrieving HTML results and/or take screenshots; disabling images or use Adblock Plus rules to make rendering faster; executing custom JavaScript in page context; getting detailed rendering info in HAR format.
> 'Splash' <https://github.com/scrapinghub/splash> is a javascript rendering service. It’s a lightweight web browser with an 'HTTP' API, implemented in Python using 'Twisted'and 'QT' \[and provides some of the core functionality of the 'RSelenium' or 'seleniumPipes' R packages but with a Java-free footprint\]. The (twisted) 'QT' reactor is used to make the sever fully asynchronous allowing to take advantage of 'webkit' concurrency via QT main loop. Some of Splash features include the ability to process multiple webpages in parallel; retrieving HTML results and/or take screenshots; disabling images or use Adblock Plus rules to make rendering faster; executing custom JavaScript in page context; getting detailed rendering info in HAR format.
The following functions are implemented:
@ -56,7 +63,7 @@ Mini-DSL (domain-specific language). These can be used to create a "script" with
- `splash_har`: Return information about Splash interaction with a website in HAR format.
- `splash_html`: Return a HTML snapshot of a current page.
- `splash_png`: Return a screenshot of a current page in PNG format.
- `splash_user_agent:Overwrite the User-Agent header for all further requests. NOTE: There are many "helper" user agent strings to go with`splash_user_agent`. Look for objects in`splashr`starting with`ua_`.
- `splash_user_agent:Overwrite the User-Agent header for all further requests. NOTE: There are many "helper" user agent strings to go with`splash\_user\_agent`. Look for objects in`splashr`starting with`ua\_\`.
`httr` helpers. These help turn various bits of `splashr` objects into `httr`-ish things:
@ -108,6 +115,8 @@ options(width=120)
### Usage
NOTE: ALL of these examples assume Splash is running in the default configuraiton on `localhost` (i.e. started with `start_splash()` or the docker example commands) unless otherwise noted.
You can use [`HARtools::HARviewer`](https://github.com/johndharrison/HARtools/blob/master/R/HARviewer.R) — which this pkg import/exports — to get view the HAR in an interactive HTML widget.