You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
15 lines
598 B
15 lines
598 B
% Generated by roxygen2: do not edit by hand
|
|
% Please edit documentation in R/spiderbar-package.R
|
|
\docType{package}
|
|
\name{spiderbar}
|
|
\alias{spiderbar}
|
|
\title{Parse and Test Robots Exclusion Protocol Files and Rules}
|
|
\description{
|
|
The 'Robots Exclusion Protocol' (\url{https://www.robotstxt.org/orig.html}) documents a set
|
|
of standards for allowing or excluding robot/spider crawling of different areas of
|
|
site content. Tools are provided which wrap The \code{rep-cpp} \url{https://github.com/seomoz/rep-cpp}
|
|
C++ library for processing these `robots.txt`` files.
|
|
}
|
|
\author{
|
|
Bob Rudis (bob@rud.is)
|
|
}
|
|
|