You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
14 lines
549 B
14 lines
549 B
#' Tools to Parse and Test Robots Exclusion Protocol Files and Rules
|
|
#'
|
|
#' The 'Robots Exclusion Protocol' <http://www.robotstxt.org/orig.html> documents a set
|
|
#' of standards for allowing or excluding robot/spider crawling of different areas of
|
|
#' site content. Tools are provided which wrap The 'rep-cpp` <https://github.com/seomoz/rep-cpp>
|
|
#' C++ library for processing these 'robots.txt' files.
|
|
#'
|
|
#' @md
|
|
#' @name rep
|
|
#' @docType package
|
|
#' @author Bob Rudis (bob@@rud.is)
|
|
#' @useDynLib rep, .registration=TRUE
|
|
#' @importFrom Rcpp sourceCpp
|
|
NULL
|