Sophie

Sophie

distrib > Momonga > 6 > i686 > by-pkgid > 500d3a04cd9a8648c6228d7e65fca015

perl-WWW-RobotRules-Parser-0.04001-3m.mo6.noarch.rpm

Description:

WWW::RobotRules::Parser allows you to simply parse robots.txt files as
described in http://www.robotstxt.org/wc/norobots.html. Unlike
WWW::RobotRules (which is very cool), this module does not take into
consideration your user agent name when parsing. It just parses the
structure and returns a hash containing the whole set of rules. You can
then use this to do whatever you like with it.

Sources packages:

Other version of this rpm: