Robot Abuse

What the hell happened to netiquette in the creation of web crawling robots? I seem to remember 30 seconds being the absolute minimum interval between requests to the same domain for a well-behaved robot, and several minutes being even better. Jesus, the crawler I wrote for a project in grad school in 1996 had a [...]

Popularity: 14% [?]

Tags: