Boothroyd's deleted wiki bioQuestion: Why doesn't the
Wikipedia robots.txt disallow all indexing by the Wayback Machine? Someone apparently considered disallowing the indexing of just User pages at one time, but even this is currently commented out:
QUOTE
# Don't allow the wayback-maschine to index user-pages
#User-agent: ia_archiver
#Disallow: /wiki/User
#Disallow: /wiki/Benutzer
Why would Wikipedia be more interested in disallowing User pages, and not at all interested in disallowing biographical articles that were deleted from en.wikipedia.org? When a bio is deleted and the associated history and talk pages are zapped also, there is presumably a reason for this. Why allow the Wayback Machine to show material that Wikipedia has decided not to show?
Answer: It's done to increase the
drama of the whole thing. Either that, or the entire Wikipedia enterprise is utterly incompetent.
For
archive.org the disallow would be instant and retroactive. Unlike other bots, they look at the robots.txt in real time, whenever anyone requests a URL from Wikipedia.
This post has been edited by bambi: