FORUM WARNING [2] Division by zero (Line: 2933 of /srcsgcaop/boardclass.php)
Wikipedia and Information Theory -
     
 
The Wikipedia Review: A forum for discussion and criticism of Wikipedia
Wikipedia Review Op-Ed Pages

Welcome, Guest! ( Log In | Register )

> Wikipedia and Information Theory
anthony
post
Post #1


Postmaster
*******

Group: Regulars
Posts: 2,034
Joined:
Member No.: 2,132
The English Wikipedia database, uncompressed: 5.34 terabytes
The English Wikipedia database, compressed: 32 gigabytes
User is offlineProfile CardPM
Go to the top of the page
+Quote Post
 
 Reply to this topicStart new topic
Replies
MZMcBride
post
Post #2


Ãœber Member
*****

Group: Regulars
Posts: 671
Joined:
Member No.: 10,962
anthony is almost certainly referring to pages-meta-history.xml.7z, which is "all pages with complete edit history" and weighs in at 31.9 GB. 7-Zip is pretty nifty compression, so much so that it's been seriously suggested lately to drop the bzip2 dumps altogether.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post
anthony
post
Post #3


Postmaster
*******

Group: Regulars
Posts: 2,034
Joined:
Member No.: 2,132
QUOTE(MZMcBride @ Sat 3rd April 2010, 6:07am) *

anthony is almost certainly referring to pages-meta-history.xml.7z, which is "all pages with complete edit history" and weighs in at 31.9 GB.


Yes, vs. the uncompressed version of that same file.

QUOTE(Milton Roe @ Sat 3rd April 2010, 5:25pm) *

It still boggles my mind that all of WP, all past pages and non-oversighted history included, minus images, can be compressed and stored on one flash drive.


Yep. Mine too. And with a non-customized, off-the-shelf compression algorithm at that. I could geek on about it for hours. But I shouldn't.
User is offlineProfile CardPM
Go to the top of the page
+Quote Post

Posts in this topic
anthony   Wikipedia and Information Theory  
Moulton   The English Wikipedia database, uncompressed: 5.34...  
Milton Roe   The English Wikipedia database, uncompressed: 5.3...  
Moulton   So what the heck compresses down 167 times? 167 re...  
Milton Roe   [quote name='Milton Roe' post='229941' date='Sat ...  
CharlotteWebb   They don't do that, so their high compression...  
Milton Roe   Not the "good" revisions necessarily, j...  
Somey   So what the heck compresses down 167 times? I wou...  
Kelly Martin   The full dump contains full-text versions of each ...  
Milton Roe   anthony is almost certainly referring to pages-me...  
Milton Roe   [quote name='Milton Roe' post='229982' date='Sat ...  
anthony   Most people with a smart phone would pay $5 ...  
Kelly Martin   anthony is almost certainly referring to pages-met...  

 

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 

-   Lo-Fi Version Time is now:
 
     
FORUM WARNING [2] Cannot modify header information - headers already sent by (output started at /home2/wikipede/public_html/int042kj398.php:242) (Line: 0 of Unknown)