Brittanica Nature response, A request for info from Andrew Orlowski Options

Hi all.

I just got an e-mail from Andrew Orlowski asking me if I have any material to add for a report he wishes to publish in relation to a Brittanica response to the Nature article about Wikipedia. Rather than just think something up by myself, I thought that I would give this to the forum to see what people think.

Anyway, here is the e-mail that I received:

http://corporate.britannica.com/britannica...re_response.pdf

Comments on this reponse by EB are very welcome.

I'll be following up my follow-up, if you'd like to provide any material on the record.

a

Obviously, it'd be easier if Andrew would register an account and post here himself, but I am happy to be a go-between in this instance.

Does anyone have anything to say on this matter?

wikipedia fan boys have something to say

http://en.wikipedia.org/wiki/User_talk:Jimbo_Wales#Nature

Well aren't they stupid.

http://www.theregister.co.uk/2006/03/23/br...a_nature_study/

This guy is really obsessed!

Darn it. We must have been too slow providing backup info. There I was thinking we had like a week or so.

In summary, he is saying that Nature were paid by Wikipedia to provide good press for them. Or if not paid, then encouraged somehow...

At first glance, this seems to be a fairly scathing attack on Nature, which is odd considering their real target is Wikipedia.

Some of EB's claims here seem to be valid however (for example, minor inaccuracies were given the same weight as major ones, although some people would say an inaccuracy is an inaccuracy is an inaccuracy), and as such a number of Wikipedians (myself included, this has been discussed on Wikien-l) would welcome another study into the relative accuracy of WP and EB. The more information and analysis the better, because that provides more opportunity for WP to improve.

On the other hand, in many cases EB has simply made comments to the effect of "we don't agree that this is wrong", and have left it at that. WP corrected all the errors identified by the Nature reviewers: Wikipedia:External peer review/Nature December 2005/Errors.

In more detail:

• EB asserts that some of the comparisons used text that did not appear in the main EB, rather in the Britannica yearbooks and the student version of EB. However they don't say which entries they are referring to.
• EB notes that they have no separate entry for "punctuated equilibrium", and that the EB text compared with WP's article was drawn from a longer entry on the theory of evolution. As such, they claim that it is unfair to criticise the EB entry for having critical omissions. But surely that's their problem for dealing with the topic too briefly?
• EB have quibbled with the qualifications of the reviewers Nature engaged, saying specifically that the reviewers "were scientists, not encyclopedists" - so who does EB get to peer review their articles? Surely for science articles, they must get scientists to do peer review, or do they leave it exclusively to "editors who have extensive experience in publishing an encyclopedia"?

Finally, in the appendices, EB addresses all of the errors and omissions identified by the Nature reviewers. They're quite long, so I'll let you read them yourself, but as a sample of the general character, EB's first three responses start with "We do not accept this criticism, which only reflects the point of view of the
reviewer", "We do not accept the validity of this review", and "We do not accept these criticisms." This is the most interesting part for me, when it is compared with the way WP editors responded to the review: Wikipedia:External peer review/Nature December 2005/Errors.

In summary, another review would be great. As I said on the mailing list:

...we should invite EB to organise another study, to be conducted by a journal of their choice. Perhaps they can even contribute to the method (multiple reviewers for each comparison would be a good inclusion), on the condition that the results of the study are published at the same time as the list of errors. Then we can fix them within days, just like with the Nature review, and put that fact out in a press release.

For reference, here is the original list of errors and omissions identified by the Nature reviewers, have a look yourself: supplementary_information.pdf

From the article:

The idea became notorious two years ago when DARPA, under retired Admiral Poindexter, invested in an online "terror casino" to predict world events such as assassinations. The public didn't quite share the sunny view of this utopian experiment, and Poindexter was invited to resign.

What do these seemingly disparate projects have in common? The idea that you can vote for the truth.

I don't think Orlowski remotely understands the notion of futures markets. The "terror market" and Wikipedia have zero in common.

So how could a respected science publication make such a grave series of errors?

I don't think he thinks they're that respected if they're this easily swayed, so this sentence comes across as a sarcastic jab.

"Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopedia Britannica."

That's your problem, EB. Make an article!

In one case, for example. Nature's peer reviewer was sent only the 350 word introduction to a 6,000 word Britannica article on lipids - which was criticized for containing omissions.

Now that is damning, and Nature should answer this accusation. And that's not the only damning thing in all this. The PDF is quite good, Britannica seems to be examing each error individually and setting the record straight. I'd love to see Nature's response to this. And there could be some useful information pulled from this PDF to improve Wikipedia.

WP corrected all the errors identified by the Nature reviewers: Wikipedia:External peer review/Nature December 2005/Errors.

what if Nature was wrong and you "fixed" wikipedia articles with false information provided by nature?


for example

Reviewer comment: Books: Principles of Stellar Dynamics (1943).

Britannica response: We do not accept this. The Library of Congress, our source for publication
dates, cites 1942 as the year of publication for this book.

Britannica has the Library of Congress backing its claim, what does Nature have? According to Britannica, Nature was not even willing to disclose the raw data they used to perform the study.

Well, its not totally unbelievable for Wikipedia to have paid Nature a sum of money to make them look good.

Darn it. We must have been too slow providing backup info. There I was thinking we had like a week or so.

I'll be following up my follow-up,

I think that this article is simply one of a series, and he was asking for feedback on a future article which he intends to write. Of course, I could be absolutely wrong, but that's what it looks like to me.

Well, its not totally unbelievable for Wikipedia to have paid Nature a sum of money to make them look good.

I think Andrew Orlowski's theory is more likely

Perhaps the clue lies not in the news report, but in the evangelism of the accompanying editorial. Nature's news and features editor Jim Giles, who was responsible for the Wikipedia story, has a fondness for "collective intelligence", one critical website suggets.

"As long as enough scientists with relevant knowledge played the market, the price should reflect the latest developments in climate research," Giles concluded of one market experiment in 2002.

The idea became notorious two years ago when DARPA, under retired Admiral Poindexter, invested in an online "terror casino" to predict world events such as assassinations. The public didn't quite share the sunny view of this utopian experiment, and Poindexter was invited to resign.

Jim Giles is probably just another Wikipedia fan boy.

Well, its not totally unbelievable for Wikipedia to have paid Nature a sum of money to make them look good.

Ok.

Nature's selection of the aritcles to be reviewed didn't do much to expose Wikipedia's weaknesses. The articles were generally from subject areas in which Wikipedia is most reliable. Wikipedia's main weakness is that it prattles on and on when it has nothing to say. A review of articles about political or social science topics could have demonstrated the vast vacume. The Nature article omitted a review of this wasteland of vague, innacurate stubs and articles about legitimate albeit obscure topics written primarily as a carrier for an editor's view of the topic. Nature would have done well to compare treatment of controversial topics.

Let's talk to EB about sponsoring another review then, or really any other journal, institute, publisher or any other body that wants to do a review. It's all better for WP, we can just use the results to improve the articles.

Very good point, Hushthis, and it is probably what Andrew Orlowski is saying. However, in reality, I think its equally likely that they were actually paid hard cash to do it. Wikipedia certainly needed it, and were on the way down beforehand. Of course, whether Wikipedia would have gone way down we don't know. But this certainly helped them out. Note that of the what 10 different studies of Wikipedia articles and their quality, this was the first one that put Wikipedia in any kind of a good light. Interesting that this would happen amidst Seigenthaler scandals and the like.

Of course, it might just be that Nature happened to do their survey and not check things out well, or that they happened to select an area which Wikipedia was good at, or that Wikipedia tried to portray what was overall pretty bad for them as if it was a lot better than it was.

Anyway, I think that if you automatically assume based on this tiny thing that Wikipedia is better than Brittanica (when the survey clearly stated the opposite) then you are naive.

Note that of the what 10 different studies of Wikipedia articles and their quality, this was the first one that put Wikipedia in any kind of a good light. Interesting that this would happen amidst Seigenthaler scandals and the like.

WP:EPR lists the external peer reviews that have been conducted on WP. Note that the Nature review is by far the most comprehensive of all conducted so far, and there is only one other review that was conducted in a formal process (the other evaluates the Featured Article process and concludes that it "is not ideal, but it does seem relatively rigorous").

Most of the other ones (the informal ones) were conducted by newspapers. You can see the results and responses for yourself with the link I gave above. They're all pretty interesting, if people haven't seen them already.

Yes, well, I wonder if those that gave Wikipedia very low marks (e.g. "0" on articles) would have changed their mind had they given a more "exhaustive" review. All of them bar Nature put Wikipedia in a very, very bad light.

Yet Wikipedia now says that Nature is the only one that matters. Interesting that.