Wikipedia and Information Theory Options

[anthony]

The English Wikipedia database, uncompressed: 5.34 terabytes
The English Wikipedia database, compressed: 32 gigabytes

[MZMcBride]

anthony is almost certainly referring to pages-meta-history.xml.7z, which is "all pages with complete edit history" and weighs in at 31.9 GB. 7-Zip is pretty nifty compression, so much so that it's been seriously suggested lately to drop the bzip2 dumps altogether.

[Kelly Martin]

anthony is almost certainly referring to pages-meta-history.xml.7z, which is "all pages with complete edit history" and weighs in at 31.9 GB. 7-Zip is pretty nifty compression, so much so that it's been seriously suggested lately to drop the bzip2 dumps altogether.

If that's what's being discussed, then yeah, a good deal of the predictability of the dump's character stream is the inherent redundancy in English text, the extremely low SNR of the XML markup (XML is typically horrid in that regard), and the high degree of repetitiveness in the content because each revision is displayed in the dump in full and thus is likely substantially similar to text previously included in the dump. I'm not familiar with 7zip's specific compression approach, but most general-purpose stream compressors are likely to achieve significant compression when asked to compress a sequence of data which repeats itself either exactly or nearly exactly, as long as the repetitions are not too far separated from one another. Since the dumps are hierarchical, with all of the revisions of a given article appearing in the dump in sequential order without interleaving revisions from other articles, that condition will tend to hold for most frequently-edited articles.

The dump does not reflect how the revisions are stored in the running database; there they use a variety of compression methods in the actual running database to reduce storage needs. The wheel has been reinvented many times, in MediaWiki land, and they're still working on rounding off the corners.