|
|
|
---
|
|
|
|
output:
|
|
|
|
rmarkdown::github_document
|
|
|
|
editor_options:
|
|
|
|
chunk_output_type: console
|
|
|
|
---
|
|
|
|
```{r pkg-knitr-opts, include=FALSE}
|
|
|
|
hrbrpkghelpr::global_opts()
|
|
|
|
```
|
|
|
|
|
|
|
|
```{r badges, results='asis', echo=FALSE, cache=FALSE}
|
|
|
|
hrbrpkghelpr::stinking_badges()
|
|
|
|
```
|
|
|
|
|
|
|
|
```{r description, results='asis', echo=FALSE, cache=FALSE}
|
|
|
|
hrbrpkghelpr::yank_title_and_description()
|
|
|
|
```
|
|
|
|
|
|
|
|
## What's Inside The Tin
|
|
|
|
|
|
|
|
The following functions are implemented:
|
|
|
|
|
|
|
|
### DSL
|
|
|
|
|
|
|
|
- `web_client`/`webclient`: Create a new HtmlUnit WebClient instance<br/><br/>
|
|
|
|
|
|
|
|
- `wc_go`: Visit a URL<br/>
|
|
|
|
|
|
|
|
- `wc_html_nodes`: Select nodes from web client active page html content
|
|
|
|
- `wc_html_text`: Extract attributes, text and tag name from webclient page html content<br/><br/>
|
|
|
|
- `wc_html_attr`: Extract attributes, text and tag name from webclient page html content
|
|
|
|
- `wc_html_name`: Extract attributes, text and tag name from webclient page html content
|
|
|
|
|
|
|
|
- `wc_headers`: Return response headers of the last web request for current page
|
|
|
|
- `wc_browser_info`: Retreive information about the browser used to create the 'webclient'
|
|
|
|
- `wc_content_length`: Return content length of the last web request for current page
|
|
|
|
- `wc_content_type`: Return content type of web request for current page<br/><br/>
|
|
|
|
|
|
|
|
- `wc_render`: Retrieve current page contents<br/><br/>
|
|
|
|
|
|
|
|
- `wc_css`: Enable/Disable CSS support
|
|
|
|
- `wc_dnt`: Enable/Disable Do-Not-Track
|
|
|
|
- `wc_geo`: Enable/Disable Geolocation
|
|
|
|
- `wc_img_dl`: Enable/Disable Image Downloading
|
|
|
|
- `wc_load_time`: Return load time of the last web request for current page
|
|
|
|
- `wc_resize`: Resize the virtual browser window
|
|
|
|
- `wc_status`: Return status code of web request for current page
|
|
|
|
- `wc_timeout`: Change default request timeout
|
|
|
|
- `wc_title`: Return page title for current page
|
|
|
|
- `wc_url`: Return load time of the last web request for current page
|
|
|
|
- `wc_use_insecure_ssl`: Enable/Disable Ignoring SSL Validation Issues
|
|
|
|
- `wc_wait`: Block HtlUnit final rendering blocks until all background JavaScript tasks have finished executing
|
|
|
|
|
|
|
|
### Just the Content (pls)
|
|
|
|
|
|
|
|
- `hu_read_html`: Read HTML from a URL with Browser Emulation & in a JavaScript Context
|
|
|
|
|
|
|
|
### Content++
|
|
|
|
|
|
|
|
- `wc_inspect`: Perform a "Developer Tools"-like Network Inspection of a URL
|
|
|
|
|
|
|
|
## Installation
|
|
|
|
|
|
|
|
```{r install-ex, results='asis', echo=FALSE, cache=FALSE}
|
|
|
|
hrbrpkghelpr::install_block()
|
|
|
|
```
|
|
|
|
|
|
|
|
## Usage
|
|
|
|
|
|
|
|
```{r cache=FALSE}
|
|
|
|
library(htmlunit)
|
|
|
|
library(tidyverse) # for some data ops; not req'd for pkg
|
|
|
|
|
|
|
|
# current verison
|
|
|
|
packageVersion("htmlunit")
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
Something `xml2::read_html()` cannot do, read the table from <https://hrbrmstr.github.io/htmlunitjars/index.html>:
|
|
|
|
|
|
|
|
![](man/figures/test-url-table.png)
|
|
|
|
|
|
|
|
```{r ex1}
|
|
|
|
test_url <- "https://hrbrmstr.github.io/htmlunitjars/index.html"
|
|
|
|
|
|
|
|
pg <- xml2::read_html(test_url)
|
|
|
|
|
|
|
|
html_table(pg)
|
|
|
|
```
|
|
|
|
|
|
|
|
☹️
|
|
|
|
|
|
|
|
But, `hu_read_html()` can!
|
|
|
|
|
|
|
|
```{r ex2}
|
|
|
|
pg <- hu_read_html(test_url)
|
|
|
|
|
|
|
|
html_table(pg)
|
|
|
|
```
|
|
|
|
|
|
|
|
All without needing a separate Selenium or Splash server instance.
|
|
|
|
|
|
|
|
### Content++
|
|
|
|
|
|
|
|
We can also get a HAR-like content + metadata dump:
|
|
|
|
|
|
|
|
```{r ex3}
|
|
|
|
xdf <- wc_inspect("https://rstudio.com")
|
|
|
|
|
|
|
|
colnames(xdf)
|
|
|
|
|
|
|
|
select(xdf, method, url, status_code, content_length, load_time)
|
|
|
|
|
|
|
|
group_by(xdf, content_type) %>%
|
|
|
|
summarise(
|
|
|
|
total_size = sum(content_length),
|
|
|
|
total_load_time = sum(load_time)/1000
|
|
|
|
)
|
|
|
|
```
|
|
|
|
|
|
|
|
### DSL
|
|
|
|
|
|
|
|
```{r ex4}
|
|
|
|
wc <- web_client(emulate = "chrome")
|
|
|
|
|
|
|
|
wc %>% wc_browser_info()
|
|
|
|
|
|
|
|
wc <- web_client()
|
|
|
|
|
|
|
|
wc %>% wc_go("https://usa.gov/")
|
|
|
|
|
|
|
|
# if you want to use purrr::map_ functions the result of wc_html_nodes() needs to be passed to as.list()
|
|
|
|
|
|
|
|
wc %>%
|
|
|
|
wc_html_nodes("a") %>%
|
|
|
|
sapply(wc_html_text, trim = TRUE) %>%
|
|
|
|
head(10)
|
|
|
|
|
|
|
|
wc %>%
|
|
|
|
wc_html_nodes(xpath=".//a") %>%
|
|
|
|
sapply(wc_html_text, trim = TRUE) %>%
|
|
|
|
head(10)
|
|
|
|
|
|
|
|
wc %>%
|
|
|
|
wc_html_nodes(xpath=".//a") %>%
|
|
|
|
sapply(wc_html_attr, "href") %>%
|
|
|
|
head(10)
|
|
|
|
```
|
|
|
|
|
|
|
|
Handy function to get rendered plain text for text mining:
|
|
|
|
|
|
|
|
```{r ex5}
|
|
|
|
wc %>%
|
|
|
|
wc_render("text") %>%
|
|
|
|
substr(1, 300) %>%
|
|
|
|
cat()
|
|
|
|
```
|
|
|
|
|
|
|
|
### htmlunit Metrics
|
|
|
|
|
|
|
|
```{r echo=FALSE}
|
|
|
|
cloc::cloc_pkg_md()
|
|
|
|
```
|
|
|
|
|
|
|
|
## Code of Conduct
|
|
|
|
|
|
|
|
Please note that this project is released with a Contributor Code of Conduct.
|
|
|
|
By participating in this project you agree to abide by its terms.
|