Robot Exclusion - GNU Wget 1.13.4 Manual

Robot Exclusion - GNU Wget 1.13.4 Manual: "If you know what you are doing and really really wish to turn off the robot exclusion, set the robots variable to ‘off’ in your .wgetrc. You can achieve the same effect from the command line using the -e switch, e.g. ‘wget -e robots=off url...’."