QUOTE(anthony @ Fri 2nd April 2010, 8:09pm)
The English Wikipedia database, uncompressed: 5.34 terabytes
The English Wikipedia database, compressed: 32 gigabytes
Weird. 167 to 1.
The weirdness is that the best English text compression is roughly 9 to 1 (without any external dictionary for common words like "the").
So what the heck compresses down 167 times? I would assume the database is mostly English text, no? It's too small to include any images. So what's the rest of the crud on WP that takes up so much space but is just.... space? (IMG:
smilys0b23ax56/default/blink.gif)
And no, this is not a snarky comment on content. Even crappy content and vandalism should take up the same space as the best and finest prose. Provided it's not run-on letter vandalism ala zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz.
Wikipedia and Information Theory
anthony Wikipedia and Information Theory 
