htmltidy

3.6 KiB

Raw Blame History

---
output: rmarkdown::github_document
editor_options: 
  chunk_output_type: console
---
```{r pkg-knitr-opts, include=FALSE}
hrbrpkghelpr::global_opts()
```

```{r badges, results='asis', echo=FALSE, cache=FALSE}
hrbrpkghelpr::stinking_badges()
```

```{r description, results='asis', echo=FALSE, cache=FALSE}
hrbrpkghelpr::yank_title_and_description()
```

Partly inspired by [this SO question](http://stackoverflow.com/questions/37061873/identify-a-weblink-in-bold-in-r) and because there's a great deal of cruddy HTML out there that needs fixing to use properly when scraping data.

It relies on a locally included version of [`libtidy`](http://www.html-tidy.org/) and works on macOS, Linux & Windows.

It also incorporates an `htmlwidget` to view and test XPath queries on HTML/XML content and another widget to view an XML document in a collapseable tree view.

## What's Inside The Tin

```{r ingredients, results='asis', echo=FALSE, cache=FALSE}
hrbrpkghelpr::describe_ingredients()
```

## Installation

```{r install-ex, results='asis', echo=FALSE, cache=FALSE}
hrbrpkghelpr::install_block()
```

## Usage

```{r usage}
library(htmltidy)

# current verison
packageVersion("htmltidy")

library(XML)
library(xml2)
library(httr)
library(purrr)
```

This is really "un-tidy" content:

```{r untidy-01}
res <- GET("https://rud.is/test/untidy.html")
cat(content(res, as="text"))
```

Let's see what `tidy_html()` does to it.

It can handle the `response` object directly:

```{r tidy-01}
cat(tidy_html(res, list(TidyDocType="html5", TidyWrapLen=200)))
```

But, you'll probably mostly use it on HTML you've identified as gnarly and already have that HTML text content handy:

```{r options-01}
cat(tidy_html(content(res, as="text"), list(TidyDocType="html5", TidyWrapLen=200)))
```

NOTE: you could also just have done:

```{r options-02}
cat(tidy_html(url("https://rud.is/test/untidy.html"), 
              list(TidyDocType="html5", TidyWrapLen=200)))
```

You'll see that this differs substantially from the mangling `libxml2` does (via `read_html()`):

```{r options-03}
pg <- read_html("https://rud.is/test/untidy.html")
cat(toString(pg))
```

It can also deal with "raw" and parsed objects:

```{r raw-01}
tidy_html(content(res, as="raw"))

tidy_html(content(res, as="text", encoding="UTF-8"))

tidy_html(content(res, as="parsed", encoding="UTF-8"))

tidy_html(suppressWarnings(htmlParse("https://rud.is/test/untidy.html")))
```

And, show the markup errors:

```{r errors-01}
invisible(tidy_html(url("https://rud.is/test/untidy.html"), verbose=TRUE))
```

## Testing Options

```{r more-options-01}
opts <- list(TidyDocType="html5",
             TidyMakeClean=TRUE,
             TidyHideComments=TRUE,
             TidyIndentContent=FALSE,
             TidyWrapLen=200)

txt <- "<html>
<head>
      <style>
        p { color: red; }
      </style>
    <body>
          <!-- ===== body ====== -->
         <p>Test</p>

    </body>
        <!--Default Zone
        -->
        <!--Default Zone End-->
</html>"

cat(tidy_html(txt, option=opts))
```

But, you're probably better off running it on plain HTML source.

Since it's C/C++-backed, it's pretty fast:

```{r speed-01}
book <- readLines("http://singlepageappbook.com/single-page.html")
sum(map_int(book, nchar))
system.time(tidy_book <- tidy_html(book))
```

(It's usually between 20 & 25 milliseconds to process those 202 kilobytes of HTML.) Not too shabby.

## htmltidy Metrics

```{r cloc, echo=FALSE}
cloc::cloc_pkg_md()
```

## Code of Conduct

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.
3.6 KiB Raw Blame History

3.6 KiB

Raw Blame History
