QUOTE(EricBarbour @ Thu 2nd July 2009, 4:55pm)
You're starting to see the true horror of keeping a major multi-colocated database website up and going. It's like running LexisNexis, but on open-source software and with no money and no hot backup. I'm amazed they don't have major outages every day.
Indeed, it's amazing that they stay up as much as they do, given the number of single points of failure there are in their architecture. To be fair, many of them are forced by their choice to use mysql as a backend database, instead of a more robust product like, say, Oracle.
The router downtime that was mentioned above should have been avoidable if they had a proper hot-spare environment. But that costs a bit more money, and spending money on that would cut down on the size of Jimmy's castle fund.
QUOTE(EricBarbour @ Thu 2nd July 2009, 5:34pm)
They use
Squid. A lot. There's probably no practical way in hell to figure out where a chunk of data is in a system like this at any given moment.
I think something like two-thirds of their servers are just Squid engines. MediaWiki is
way too slow to generate all the pages that Wikimedia serves across all the projects in real time.
It constantly amuses me to hear PHP culties go on about how well PHP scales, citing Wikipedia as proof, when the reality is that PHP scales like crap and the WMF has to throw tons of hardware at their half-assed content engine just to keep up.
When Wikipedia goes down
thekohser When Wikipedia goes down 
