spacer
Janet Systems DotNetNuke Websites, Hosting, Custom Servers
Tag Words
spacer
Web Robot Articles

Identification of the Wget Internet robot. These include owner, description, HTTP user agent and whether this robot adheres to the robot exclusion standard.

Owner  
Country User dependant
Description Wget is a computer program which is used to retrieve files from the Internet. It is able to work non-interactively and may be run from a script or scheduled cron job. the original work was linux/unix based however, windows binaries are available. The program in itself is not malicious and may be a valuable tool.
IP Addresses User dependant
HTTP User Agent Wget/1.9+cvs-stable (Red Hat modified)
Wget/1.8.2
Exclusion This robot supports the robots.txt exclusion standard, which is described at http://www.robotstxt.org/wc/exclusion.html#robotstxt. To stop the robot from visiting the /cgi-bin directory:

 

User-agent: wget
Disallow: /cgi-bin/

Further Info. www.gnu.org/software/wget/wget.html

NAT August 2005 (Revised)

  This article viewed: 2794 times Back

Copyright © 2004-2009 Janet Systems Ltd.

    spacer
    Print