[![Travis-CI Build Status](https://travis-ci.org/hrbrmstr/htmltidy.svg?branch=master)](https://travis-ci.org/hrbrmstr/htmltidy) `htmltidy` — Clean up gnarly HTML/XHTML Inspired by [this SO question](http://stackoverflow.com/questions/37061873/identify-a-weblink-in-bold-in-r) and because there's a great deal of cruddy HTML out there that needs fixing to use properly when scraping data. It relies on a locally included version of [`libtidy`](http://www.html-tidy.org/) and works on macOS, Linux & Windows. The following functions are implemented: - `tidy_html` : Tidy or "Pretty Print" HTML/XHTML Documents ### Installation ``` r devtools::install_github("hrbrmstr/htmltidy") ``` ### Usage ``` r library(htmltidy) # current verison packageVersion("htmltidy") ## [1] '0.2.0' library(XML) library(xml2) library(httr) library(purrr) ``` This is really "un-tidy" content: ``` r res <- GET("http://rud.is/test/untidy.html") cat(content(res, as="text")) ## ## ## ## ## This is some really poorly formatted HTML ## ## as is this portion
``` Let's see what `tidy_html()` does to it: ``` r cat(tidy_html(content(res, as="text"), list(TidyDocType="html5", TidyWrapLen=200))) ## ## ## ## ## ## ## ## ## This is some really poorly formatted HTML as is this portion ##
## ## ``` NOTE: you could also just have done: ``` r cat(tidy_html(url("http://rud.is/test/untidy.html"), list(TidyDocType="html5", TidyWrapLen=200))) ## ## ## ## ## ## ## ## ## This is some really poorly formatted HTMLas is this portion ##
## ## ``` You'll see that this differs substantially from the mangling `libxml2` does (via `read_html()`): ``` r pg <- read_html("http://rud.is/test/untidy.html") cat(toString(pg)) ## ## ## ## This is some really poorly formatted HTML ## ## as is this portion
``` It can also deal with "raw" and parsed objects: ``` r tidy_html(content(res, as="raw")) ## [1] 3c 21 44 4f 43 54 59 50 45 20 68 74 6d 6c 3e 0a 3c 68 74 6d 6c 20 78 6d 6c 6e 73 3d 22 68 74 74 70 3a 2f 2f 77 77 ## [39] 77 2e 77 33 2e 6f 72 67 2f 31 39 39 39 2f 78 68 74 6d 6c 22 3e 0a 3c 68 65 61 64 3e 0a 3c 6d 65 74 61 20 6e 61 6d ## [77] 65 3d 22 67 65 6e 65 72 61 74 6f 72 22 20 63 6f 6e 74 65 6e 74 3d 0a 22 48 54 4d 4c 20 54 69 64 79 20 66 6f 72 20 ## [115] 48 54 4d 4c 35 20 66 6f 72 20 52 20 76 65 72 73 69 6f 6e 20 35 2e 30 2e 30 22 20 2f 3e 0a 3c 74 69 74 6c 65 3e 3c ## [153] 2f 74 69 74 6c 65 3e 0a 3c 2f 68 65 61 64 3e 0a 3c 62 6f 64 79 3e 0a 3c 2f 62 6f 64 79 3e 0a 3c 2f 68 74 6d 6c 3e ## [191] 0a tidy_html(content(res, as="text", encoding="UTF-8")) ## [1] "\n\n\n\n\n\n\n\nThis is some really poorly formatted HTML as is this\nportion\n
\n\n\n" tidy_html(content(res, as="parsed", encoding="UTF-8")) ## {xml_document} ## ## [1] \n \n \nThis is some really poorly formatted HTML as is this\nportion\n tidy_html(htmlParse("http://rud.is/test/untidy.html")) ## ## ## ## ## ## ## ## ## This is some really poorly formatted HTML as is this ## portion ##
## ## ## ``` ### Testing Options ``` r opts <- list(TidyDocType="html5", TidyMakeClean=TRUE, TidyHideComments=TRUE, TidyIndentContent=FALSE, TidyWrapLen=200) txt <- "

Test

" cat(tidy_html(txt, option=opts)) ## ## ## ## ## ## ## ## ##

Test

## ## ``` But, you're probably better off running it on plain HTML source. Since it's C/C++-backed, it's pretty fast: ``` r book <- readLines("http://singlepageappbook.com/single-page.html") sum(map_int(book, nchar)) ## [1] 207501 system.time(tidy_book <- tidy_html(book)) ## user system elapsed ## 0.021 0.001 0.023 ``` (It's usually between 20 & 25 milliseconds to process those 202 kilobytes of HTML.) Not too shabby. ### Code of Conduct Please note that this project is released with a [Contributor Code of Conduct](CONDUCT.md). By participating in this project you agree to abide by its terms.