Nov 6, 2006

38.113.234.180 crawl1.cosmixcorp.com

cosmixcorp.com Is a Health search sytem

cfetch/1.0
38.113.234.180 crawl1.cosmixcorp.com

voyager/1.0
38.113.234.180 crawl1.cosmixcorp.com

They have started changing useragents lately. The site says its using the Voyager useragent.
What is your crawler's HTTP user-agent string?

voyager/1.0


Thats really strange since it keeps using cfetch/1.0 most of the time.
I had been banning them by ip but will try the robots file again.


Add this to robots.txt
User-agent: voyager
Disallow: /

2 comments:

Anonymous said...

Beware of anything coming from Performance Systems Internation in Washington DC. I have blocked all of ^38. I have not found anything good coming out of the 38. range in years!

Jeff Geerling said...

Added to my page on bad IP addresses - I was just attacked (tons of page-not-found errors) by this bot.

http://www.lifeisaprayer.com/web-design/2010/bad-annoying-ip-addresses