gopherproxy/robots.txt, branch master gopherproxy https://git.wyvern.rip/gopherproxy/atom/robots.txt?h=master 2018-04-03T21:32:43Z Add support for robots.txt user agent control from web crawlers (#10) 2018-04-03T21:32:43Z James Mills prologic@shortcircuit.net.au 2018-04-03T21:32:43Z urn:sha1:fe18a79dd2f7115b30d440a025a7f7f72486dd09