Quote:
Originally Posted by George
Robots.txt does a good job of keeping the legit search engines out of /cgi-bin
|
It does but it doesn't stop them from crawling and reading content which in our case adds up to 5 - 6GB of wasted bandwidth every month.
Webmasters Tool says that reported errors are not real errors and that we report them as errors because we found files which we can't use.
But when amount of files happened to be over 120,000, it makes you wonder, what the hell is wrong with G. and why not to go to sitemap instead and suck on information we want little bitch to see and crawl.
mimika