You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
15 lines
602 B
15 lines
602 B
% Generated by roxygen2: do not edit by hand
|
|
% Please edit documentation in R/rep-package.R
|
|
\docType{package}
|
|
\name{rep}
|
|
\alias{rep}
|
|
\alias{rep-package}
|
|
\title{Tools to Parse and Test Robots Exclusion Protocol Files and Rules}
|
|
\description{
|
|
The 'Robots Exclusion Protocol' (\url{http://www.robotstxt.org/orig.html}) documents a set
|
|
of standards for allowing or excluding robot/spider crawling of different areas of
|
|
site content. Tools are provided which wrap The 'rep-cpp\code{<https://github.com/seomoz/rep-cpp> C++ library for processing these}robots.txt`` files.
|
|
}
|
|
\author{
|
|
Bob Rudis (bob@rud.is)
|
|
}
|
|
|