Up Yours, HotBot
Well, I was looking at our website stats and we’re still doing pretty well, roughly 10,000 unique visitors per month, and reasonable numbers of hits. The not so good news is that about 25% of the total bandwidth was being eaten by Yahoo’s search robot. I’ve added a robots.txt file to the public_html directory and disallowed that robot indexing any file that might be kind of large (.pdfs and common web graphics files). Let me know if this does anything wonky.
Also, we continue to get searches for fake cocks and Rebecca Newell vagina. We know less about the latter than we might like, to be honest.
Wow, Rebecca Newell is sexy. I must be becoming a lesbian.
Just did a Google and yahoo search comparison for “Oregon Commentator hate issue”… no PDF’s come up for either, which probably is a good thing. Web crawlers (and metacrawlers) can potentially drain performance of a website, which is why the option to disallow crawler indexing is available. As far as I know, google doesn’t employ crawlers, which some developers see as “greedy.” Besides, Google really is the top dog of search engines….Who the hell uses Yahoo anymore anyways?