Details for the CosmixCrawler Internet robot. Details for this robot include owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
| Owner |
|
| Country |
|
| Description |
The IP address given is part of a block assigned to Abovenet Communications, Inc. (www.above.net) |
| IP Addresses |
209.133.117.19 |
| HTTP User Agent |
CosmixCrawler/0.1 |
| Exclusion |
It is not known whether this robot supports the robots.txt exclusion standard, which is described at http://www.robotstxt.org/wc/exclusion.html#robotstxt. To stop the robot from visiting the /cgi-bin directory:
User-agent:
Disallow: /cgi-bin |
| Further Info. |
|
NAT August 2005 (Revised)
| |
This article viewed: 1630 times |
Back |
Copyright © 2004-2009 Janet Systems Ltd.