Nov 25, 2006

converacrawler/0.9d 7-9745.san2.attens.net bot

converacrawler/0.9d (+http://www.authoritativeweb.com/crawl)
63.241.61.7 7-9745.san2.attens.net

This bot came in and refused to take no for a answer it tried to load every page I had. So its clear that it didn't spider my site to get those links it got them from google or somewhere else.

converacrawler was orginaly banned as a email harvestor but it now looks like a real search site at www.govmine.com.

They claim you can add this to robots.txt.

User-agent: ConveraCrawler
Disallow: /

No comments: