Identification of the MSNBot Internet robot. These include owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.
| Owner |
MSN Search |
| Country |
USA |
| Description |
MSNBot is a prototype web-crawling robot. It is used to collect documents from the web to build a searchable index. Currently MSN still use Inktomi for their search content. |
| IP Addresses |
65.54.188.143 65.54.188.146 207.46.98.42 |
| HTTP User Agent |
msnbot/0.11 (+http://search.msn.com/msnbot.htm) msnbot/0.3 (+http://search.msn.com/msnbot.htm) msnbot/1.0 (+http://search.msn.com/msnbot.htm) |
| Exclusion |
This robot supports the robots.txt exclusion standard, which is described at http://www.robotstxt.org/wc/exclusion.html#robotstxt. To stop the robot from visiting the /cgi-bin directory:
User-agent: MSNBot Disallow: /cgi-bin/
Note, it does not look at the robots.txt file at each visit, but rather downloads the details every day. For this reason it may take a while for the robot to conform to the robots.txt file if changes have been made. |
| Further Info. |
www.msnsearch.com/msnbot.htm |
NAT August 2005 (Revised)
| |
This article viewed: 2447 times |
Back |
Copyright © 2004-2009 Janet Systems Ltd.