FAC Reviewed Options

[Ottava]

A review by David Lindsay (a Georgetown student, booooo!) reviewed FAC and Wiki's FAs. Surprise, surprise, Literature was not reviewed in the process. Some of the results might amuse you.

I emailed the dear chap to point out two things: 1) he should have done the core topics (science, history, literature, philosophy/religion, etc) that have comparable entries in Britannica or the such and 2) that not all FAs are the same.

But yes, I am sure that people here will have a lot of lovely things to say.

[thekohser]

I read about this days ago on the CPOV list. I didn't want to bring it to WR's attention, since I suck so bad.

[Milton Roe]

A review by David Lindsay (a Georgetown student, booooo!) reviewed FAC and Wiki's FAs. Surprise, surprise, Literature was not reviewed in the process. Some of the results might amuse you.

I emailed the dear chap to point out two things: 1) he should have done the core topics (science, history, literature, philosophy/religion, etc) that have comparable entries in Britannica or the such and 2) that not all FAs are the same.

But yes, I am sure that people here will have a lot of lovely things to say.

Too bad he didn't collect enough responses to be able to estimate an inter-rater alpha score. So we dunno what the expert score means, since we have no way of knowing if they are stable, and reproducable between raters who can't see each other's results.

So what if professor Jones gives some article a score of 9/10, if professor Smith only gives the same article a 5? That just leads to natural questions about the validity of your method. Which (of course!) we don't want.

Illustrative true story about raters: I once sent a grant proposal to the DoD for some military funding, in response to a public grant proposals request (it looked like something I'd been working on anyway-- you have no idea how many litterboxes there are for cats on military bases). They had no idea what to make of it, thinking it might be a 3 out of 5 score, so they sent it out to two outside reviewers. One reviewer thought it was the screwiest idea he'd ever heard of, and he also had never heard of me, so he gave it a 1 out of 5. The other thought it was the best idea since sliced bread, and gave it a 5/5, despite the fact he hadn't heard of me, either. The DoD had no idea what to make of tho such disparite scores, so they averaged them to a 3. No money.

Now, another organization with curiosity would have looked at those scores, and done some additional investigation to find out what the hell was going on, with such wild variation. (IMG:smilys0b23ax56/default/hmmm.gif) But this is Your Government. They weren't that curious. Score: 3. (IMG:smilys0b23ax56/default/ermm.gif)

[NuclearWarfare]

The FAC regulars had a blast with this review: http://en.wikipedia.org/wiki/Wikipedia_tal...s_FAC_a_failure

[Ottava]

Now, another organization with curiosity would have looked at those scores, and done some additional investigation to find out what the hell was going on, with such wild variation.

Well, you did verify that the DoD did a better job finding a second person at least instead of going off of just one. (IMG:smilys0b23ax56/default/smile.gif)

How about three or four reviewers as a minimum? Would make sense.

[taiwopanfob]

The FAC regulars had a blast with this review: http://en.wikipedia.org/wiki/Wikipedia_tal...s_FAC_a_failure

It comes as no surprise that to a man they all believe that the Sacred Policies of the project are trumping centuries of scholarship and the like. All those experts are "useless" because they have "no idea" about NPOV. SlimVirgin 'knowingly' rolling her eyes when one expert dismisses a position as false simply because some other expert says the opposite.

To the wikipediots, it's all a conspiracy: these experts have turf to defend, careers to make, and so forth. More mundane explanations like an unwillingness to educate nitwits who think scholarship is "useless", or are unwilling to use their heads instead of following Holy Writ, no matter how stupid the outcome may be.

The central lesson of last N centuries of progress, the one driven into every expert worthy the title is "you might be wrong", followed by careful training on how to tell one way or another. Wikipediots are simply unwilling to question their "pillars", unable to see their own turf defending, their own (worthless) wiki-careers ... the irony is thus lost completely on them.

[Casliber]

Yeah, it was a shame the review couldn't have been a bit more comprehensive or embellished a bit (didn't mention FAR, nor did it distinguish between articles' current state and that which might have been when it passed FAC , and I agree about more than one expert. Many many experts have views which do not lie in the centre of their fields of expertise.) Anyway, its a start I guess...

[Kelly Martin]

The FAC regulars had a blast with this review: http://en.wikipedia.org/wiki/Wikipedia_tal...s_FAC_a_failure

It's interesting that they failed to recognize the purpose of the study, which was to determine if it is reasonable, when doing research about Wikipedia, to use the fact that a Wikipedia article is "featured" as an indication that the article is of high quality, and the conclusion that such a rule is potentially unwarranted. The Wikipedians chose to read far more into the study than the study's author was willing to say, and then criticize the study roundly for drawing conclusions that it did not actually draw.

[EricBarbour]

To the wikipediots, it's all a conspiracy: these experts have turf to defend, careers to make, and so forth. More mundane explanations like an unwillingness to educate nitwits who think scholarship is "useless", or are unwilling to use their heads instead of following Holy Writ, no matter how stupid the outcome may be.

As always, Wikipedia is infallible, and its critics are heretics.

(Odd, how much the FAC's behaviour resembles that of the Catholic Church, circa, oh, around 1209 or so.....)

This post has been edited by EricBarbour:

[Theanima]

I never really participated in FAC, as it tends to be a tight clique of editors and any outsider view is frowned upon. I think there are better ways of creating top quality articles; for example, instead of seeking the shiny badge and a position on the WBFAN, editors could just be satisfied they produced a decent article without seeking recognition.

One person's definition of "high quality" differs from another. Likewise, some FACs get a huge amount of participation, and others get just three people.

Just my thoughts on FAC.

[Malleus]

I think there are better ways of creating top quality articles; for example, instead of seeking the shiny badge and a position on the WBFAN, editors could just be satisfied they produced a decent article without seeking recognition.

I expect that there are better ways, but how do you know your article is "decent" if nobody except you ever assesses it?

[Ottava]

I expect that there are better ways, but how do you know your article is "decent" if nobody except you ever assesses it?

I actually think this is decent even though it was never measured beyond the previous version before I expanded. It has some gaps and is far from finished, but yeah. There needs to be some grammar and language polishing. But you did say "decent" and not "good". (IMG:smilys0b23ax56/default/tongue.gif)

[Theanima]

I think there are better ways of creating top quality articles; for example, instead of seeking the shiny badge and a position on the WBFAN, editors could just be satisfied they produced a decent article without seeking recognition.

I expect that there are better ways, but how do you know your article is "decent" if nobody except you ever assesses it?

As with RFA, most FACs are approved by a tiny bunch of people. Just because there was so-called consensus does not mean the candidate will be a good admin. Same with FAC. If I wrote something I believed was good enough, that's all that would matter for me. I don't care what the crowd at FAC think.

[Ottava]

As with RFA, most FACs are approved by a tiny bunch of people. Just because there was so-called consensus does not mean the candidate will be a good admin. Same with FAC. If I wrote something I believed was good enough, that's all that would matter for me. I don't care what the crowd at FAC think.

Can you link to anything you wrote, by chance? I mean, everyone knows the pages that I, Malleus, Iri, etc. have written.

[Milton Roe]

It comes as no surprise that to a man they all believe that the Sacred Policies of the project are trumping centuries of scholarship and the like. All those experts are "useless" because they have "no idea" about NPOV. SlimVirgin 'knowingly' rolling her eyes when one expert dismisses a position as false simply because some other expert says the opposite.

To the wikipediots, it's all a conspiracy: these experts have turf to defend, careers to make, and so forth. More mundane explanations like an unwillingness to educate nitwits who think scholarship is "useless", or are unwilling to use their heads instead of following Holy Writ, no matter how stupid the outcome may be.

Slimvirgin is actually on record as saying that WP should not use the world "pseudoscience" because it's (A) pejorative, and (B) meaningless. The second because what's scientific and what is not scientific, comes down merely to a question of WP:ILIKEIT vs WP:IDONTLIKEIT among outside writers. There's nothing objective about it.

I think basically she's a "social constructionist" when it comes to science. Theories of science are no more real to her than theories of politics or ethics or aesthetics or any other branch of philosophy. (IMG:smilys0b23ax56/default/wacko.gif) The success of modern engineering to her must be a frigging miracle.

However, her view would explain much. Why not then use New York Times articles as sources on scientific fact? They have as much right to their opinion as anybody, and (actually) even more, since they're a "newspaper of record." The same for big government agencies and what their minions think. If they say it, it's verifiable and comes from a "reliable source " (the government by definition is a RS). Nevermind if it's "true" since not only doesn't WP care about that, but it's meaningless when it comes to real world science debates anyway.

[Ottava]

Slimvirgin is actually on record as saying that WP should not use the world "pseudoscience" because it's (A) pejorative, and (B) meaningless. The second because what's scientific and what is not scientific, comes down merely to a question of WP:ILIKEIT vs WP:IDONTLIKEIT among outside writers. There's nothing objective about it.

I think the problem is not "scientific" vs "non-scientific" but should be "science" vs "non-science". There are many approaches to literature that are rigorous and academic, but they should not be treated as science. However, there are approaches in linguistics and philology that should be (as it deals with the processes of sounds, letters, physiological interpretation, and the rest as opposed to philosophy and meaning).

Of course, Religion is not science. The Bible's pages should be treated different from, say, the pages on Toads. But then there are gray areas where there are things that are treated "Scientifically" but aren't science. Ghosts, for instance. Sure, they have scientific approaches but there is no actual science there. They have no actually captured ghosts, dissected them, and had dozens of studies analysing various parts. Instead, ghosts end up being more of a sociological discipline (how people respond to them, social customs surrounding, them, etc).

[A Horse With No Name]

I never really participated in FAC, as it tends to be a tight clique of editors and any outsider view is frowned upon. I think there are better ways of creating top quality articles; for example, instead of seeking the shiny badge and a position on the WBFAN, editors could just be satisfied they produced a decent article without seeking recognition.

One person's definition of "high quality" differs from another. Likewise, some FACs get a huge amount of participation, and others get just three people.

Just my thoughts on FAC.

Being a "featured article" on Wikipedia is like being the tallest office building in Casper, Wyoming -- really, no one cares at all. Emphasis on no one. (IMG:smilys0b23ax56/default/dry.gif)

[GlassBeadGame]

I never really participated in FAC, as it tends to be a tight clique of editors and any outsider view is frowned upon. I think there are better ways of creating top quality articles; for example, instead of seeking the shiny badge and a position on the WBFAN, editors could just be satisfied they produced a decent article without seeking recognition.

One person's definition of "high quality" differs from another. Likewise, some FACs get a huge amount of participation, and others get just three people.

Just my thoughts on FAC.

Being a "featured article" on Wikipedia is like being the tallest office building in Casper, Wyoming -- really, no one cares at all. Emphasis on no one. (IMG:smilys0b23ax56/default/dry.gif)

A bunch of nobodies seem to care.

[GlassBeadGame]

I never really participated in FAC, as it tends to be a tight clique of editors and any outsider view is frowned upon. I think there are better ways of creating top quality articles; for example, instead of seeking the shiny badge and a position on the WBFAN, editors could just be satisfied they produced a decent article without seeking recognition.

One person's definition of "high quality" differs from another. Likewise, some FACs get a huge amount of participation, and others get just three people.

Just my thoughts on FAC.

Being a "featured article" on Wikipedia is like being the tallest office building in Casper, Wyoming -- really, no one cares at all. Emphasis on no one. (IMG:smilys0b23ax56/default/dry.gif)

A bunch of nobodies seem to care.

I think you'd have to include yourself in that "bunch of nobodies". Nobody I know cares about the BFAN crap, for instance. Except you.

I'm assuming you're talking to the other nobody and are just too lazy to quote correctly. I had to Google "WBFAN" to figure out what it was.

I just noticed this:

The article is certainly interesting, but a piece of properly peer-reviewed scientific research it most definitely is not. Malleus Fatuorum 22:10, 15 April 2010 (UTC)

Is First Monday faking its peer review process? Or is that just your way to impress them down on the WP notice board? I notice that august body of "peers" didn't call you on it.

[Ottava]

Easily passed FAC but then exposed as a major failure. Surprisingly, one person who did mention sourcing before it was restarted as seen here was Orlady.

The diff of passing and now doesn't reveal much. Much of the changes were from YellowMonkey prepping the FAR by removing links that don't match the content cited. That wouldn't explain the rest of the problems, such as citations not matching sources, things out of date being used for current stuff, etc.

I guess this goes to show that people are not doing enough to check sources vs content and probably taking a recognizable user and just accepting what they put forth. The same thing happened with Gary King and the Samus Aran page (I found quite a lot of plagiarism which people should have found before).

This post has been edited by Ottava:

[Kelly Martin]

I guess this goes to show that people are not doing enough to check sources vs content and probably taking a recognizable user and just accepting what they put forth. The same thing happened with Gary King and the Samus Aran page (I found quite a lot of plagiarism which people should have found before).

Well, duh. Wikipedia's FA process has been far more about politics than about content for a long time now. At least half of the objections are fundamentally specious and are only raised in order to make the proposer owe the objector a favor for withdrawing the objection. Very rarely are articles from well-connected individuals (with lots of backed-up favors) scrutinized closely, while articles being fostered by relatively unknown people are run over the coals for no reason at all.

[Ottava]

I guess this goes to show that people are not doing enough to check sources vs content and probably taking a recognizable user and just accepting what they put forth. The same thing happened with Gary King and the Samus Aran page (I found quite a lot of plagiarism which people should have found before).

Well, duh. Wikipedia's FA process has been far more about politics than about content for a long time now. At least half of the objections are fundamentally specious and are only raised in order to make the proposer owe the objector a favor for withdrawing the objection. Very rarely are articles from well-connected individuals (with lots of backed-up favors) scrutinized closely, while articles being fostered by relatively unknown people are run over the coals for no reason at all.

Well, isn't that what a rather well publicized forum should scrutinize when such a pattern becomes blatant? I would hope that some people would take some time and expose some of the worse offenders, put up some FARs, and embarrass the people who went about causing some of the mess. For instance, this is the second article by Nichalp being FAR'd for sources not matching. I see five more potential FARs.

I have a suspicion that national related FAs are prone to the gaming.

[Cock-up-over-conspiracy]

As always, Wikipedia is infallible, and its critics are heretics.

The cult-like similarities are even greater and more specific ...

Ex-Wikipedia cult adherents are easily caste in the role of being 'the unreliable apostate'.

To help understanding, Wikipedia criticism is years behind the sociology of New Religious Movements. It is sadly, tiresomely re-treading debate and re-playing dynamics that the NRM world has already covered.

Wikipedia criticism ought to be largely moved over into that context and brought up to day quickly ... as the criticism of a New Religious Movement with Jimbo as its Messiah.

For example, to quote Massimo Introvigne, thought to be a bit of a cult apologists or defender.

Massimo Introvigne in his 'Defectors, Ordinary Leavetakers and Apostates' defines three types of narratives constructed by apostates of new religious movements:

Type I narratives characterize the exit process as defection, in which the organization and the former member negotiate an exiting process aimed at minimizing the damage for both parties.

Type II narratives involve a minimal degree of negotiation between the exiting member, the organization they intend to leave, and the environment or society at large, implying that the ordinary apostate holds no strong feelings concerning his past experience in the group.

Type III narratives are characterized by the ex-member dramatically reversing their loyalties and becoming a professional enemy of the organization they have left.

These apostates often join an oppositional coalition fighting the organization, often claiming victimization.

Historically, the Type III argument was then taken and inverted to then discredit highly valid cult criticism, and the accurate evidence of abuse by ex-members, in both academia and legal actions defending cults.

Only recently, has the pendulum managed to be swung back again to allow individuals to admit that ex-adherents can and do actually provide the most reliable witness to what was going on in a cult.

In the case of cults, it was shown that highly influential academics such as Dr. Eileen Barker were doing what under normal circumstances would be called conspiring to alter public and academic opinion about ex-adherents in an attempt to normalise cult activities.

At that time, such action may have been understandable. The cult bashers and born again Christians etc on the opposite side had gone too far. But it remains a strategy the cults still use to discredit closer inspection.

And that is what Wikipedia Foundation and Wiki-cultists do. The concept of there being a "Anti-cult movement", rather than just a loose collection of reasonably concerned members of the public, is one of their ... and the cults ... creation. Ditto, we see that too regards the Wikipedia. Any critic is an instant member of the Anti-Wikipedian.

The only thing that is a worst waste of time than engaging with a Wiki-cultists is engage is a religious cultists who is also a Wiki-cultists and skilled in playing their game.

If there are any sympathetic anti-cultist admins out there, please let me know.

[A Horse With No Name]

As with any award-generating culture, excellence runs a distant second to cronyism and politics.

[Theanima]

I guess this goes to show that people are not doing enough to check sources vs content and probably taking a recognizable user and just accepting what they put forth. The same thing happened with Gary King and the Samus Aran page (I found quite a lot of plagiarism which people should have found before).

Well, duh. Wikipedia's FA process has been far more about politics than about content for a long time now. At least half of the objections are fundamentally specious and are only raised in order to make the proposer owe the objector a favor for withdrawing the objection. Very rarely are articles from well-connected individuals (with lots of backed-up favors) scrutinized closely, while articles being fostered by relatively unknown people are run over the coals for no reason at all.

As I said before; the concentration is on whether it has suitable alt text, or if the references are formatted consistently, or if an image is in an appropriate position. Hardly anyone cares about checking references to see if they back up what is claimed. Hardly anyone checks for copyvios. Hardly anyone checks for the source's reliability - often, they can't as it's from a book no-one but the author has access to.

This demonstrates well the problems with FAC.

[dogbiscuit]

As I said before; the concentration is on whether it has suitable alt text, or if the references are formatted consistently, or if an image is in an appropriate position. Hardly anyone cares about checking references to see if they back up what is claimed. Hardly anyone checks for copyvios. Hardly anyone checks for the source's reliability - often, they can't as it's from a book no-one but the author has access to.

This demonstrates well the problems with FAC.

In other words, what is needed is for people who understand what the articles should be about reviewing them.

In the review, one comment made was that the references were unusual, so simply checking references to confirm that what is paraphrased matches the references is not sufficient: you need someone with a good sense of what is a reasonable reference, which you get from having a good knowledge of the subject.

For example, I might write an article on Churchill, referencing lots of fine articles in the Times and Sunday Times (which might be perfectly adequate, though clearly targeting a different market than a reference work). While a Wikipedian FA reviewer would probably rate those sources as quite desirable, an expert on Churchill would be aghast.

[Ottava]

For example, I might write an article on Churchill, referencing lots of fine articles in the Times and Sunday Times (which might be perfectly adequate, though clearly targeting a different market than a reference work). While a Wikipedian FA reviewer would probably rate those sources as quite desirable, an expert on Churchill would be aghast.

Clearly you've never taken an article to FAC if you really believe that claptrap.

A reviewer of the Johnson FAC was aghast that I used the major, world renown biography of Walter Jackson Bate over the one he favored, John Wain's. I added a few citations to Wain to make him happy, but I made sure to cite all of the important material to the better biography. I also put Bate's, who has been vetted for a long time, over those published post 2005 simply because the information is more certain and had time for any corrections to be made.

[chrisoff]

Why is there such a falloff of FAC reviewers? Less than ever are now reviewing. No one want to do the reviewing any more.

[Milton Roe]

Why is there such a falloff of FAC reviewers? Less than ever are now reviewing. No one want to do the reviewing any more.

Our paychecks are late. Could you look into that, please?

[Kelly Martin]

Why is there such a falloff of FAC reviewers? Less than ever are now reviewing. No one want to do the reviewing any more.

What does one gain from reviewing an FAC? When you have figured out the answer to that question, you will have the answer to your question as well.

[Abd]

Why is there such a falloff of FAC reviewers? Less than ever are now reviewing. No one want to do the reviewing any more.

I'm noticing a lack of volunteers all over the project. It's one of the signs of the end game, a possible impending collapse which would, in theory, accelerate as it becomes more and more frustrating for fewer and fewer editors to maintain the project, and thus more drop out. This is an inevitable result of placing no value on general editor time, but only on the time of a core, and even there, not handling process for true efficiency.

When I've seen lip service paid to the value of time, it has been to prohibit editors from voluntarily engaging in activities considered, by the "community," to be wastes of time, like WP:Esperanza. Basically, more bluntly, the message was "Stop chit-chatting, wasting time, back to the salt mines!"

[Eva Destruction]

When I've seen lip service paid to the value of time, it has been to prohibit editors from voluntarily engaging in activities considered, by the "community," to be wastes of time, like WP:Esperanza. Basically, more bluntly, the message was "Stop chit-chatting, wasting time, back to the salt mines!"

With all due respect, you couldn't have misread that particular situation more if you were doing it deliberately. Esperanza wasn't shut down because it was "a waste of time", it was shut down because they deliberately set themselves up as an "above the law" cabal, whose self-appointed "Advisory Council" would hold discussions on IRC and issue binding directives to their members; you can read the whole wretched saga here. I'd have thought you, of all people, would be totally opposed to everything they stood for.

[Kelly Martin]

When I've seen lip service paid to the value of time, it has been to prohibit editors from voluntarily engaging in activities considered, by the "community," to be wastes of time, like WP:Esperanza. Basically, more bluntly, the message was "Stop chit-chatting, wasting time, back to the salt mines!"

Esperanza was shut down because it represented a competing power center, not because it was a "waste of time".

[GlassBeadGame]

Anyone recall the Noir Emo Kids of Wikipedia? I don't remember whether it was exactly Esperanza or not but they worked, made thousands of "edits," sending out birthday greetings to Wikipedians. They were 13, 14, 15 years old. It was a Kathy Lee sweatshop kind of operation. Well a little clique of these kids got all disillusioned at about the same time. (2007?) I think it was precipitated by some kind of medical melt down by one of their members. They made some magnificent user page rants about all the work they were conned into doing and how they felt exploited. One even said something perceptive like "I have nothing to offer to any real encyclopedia." I suspect much of this material has since disappeared for a combination of evil and right thinking motives. But if any of the old-schoolers helped dump the bodies a safe distance from the sweatshop maybe they could tell?

[Eva Destruction]

Anyone recall the Noir Emo Kids of Wikipedia? I don't remember whether it was exactly Esperanza or not but they worked, made thousands of "edits," sending out birthday greetings to Wikipedians. They were 13, 14, 15 years old. It was a Kathy Lee sweatshop kind of operation. Well a little clique of these kids got all disillusioned at about the same time. (2007?) I think it was precipitated by some kind of medical melt down by one of their members. They made some magnificent user page rants about all the work they were conned into doing and how they felt exploited. One even said something perspective like "I have nothing to offer to any real encyclopedia." I suspect much of this material has since disappeared for a combination of evil and right thinking motives. But if any of the old-schoolers helped dump the bodies a safe distance from the sweatshop maybe they could tell?

The list is preserved for posterity. I think some of them may still be at it.

Actually, if you ever think you've seen Wikipedia's collective semiconscious at its most dimwitted, visit WP:Department of Fun. It's the closest I've yet seen to a written equivalent of a cat being dragged across a chalkboard by its tail.

[John Limey]

A review by David Lindsay (a Georgetown student, booooo!) reviewed FAC and Wiki's FAs. Surprise, surprise, Literature was not reviewed in the process. Some of the results might amuse you.

The scant literature on featured articles is discussed.

I emailed the dear chap to point out two things: 1) he should have done the core topics (science, history, literature, philosophy/religion, etc) that have comparable entries in Britannica or the such

Why?

and 2) that not all FAs are the same.

Hmmm. I think that was actually the whole point of the article. Some are good. Some are bad.

The FAC regulars had a blast with this review: http://en.wikipedia.org/wiki/Wikipedia_tal...s_FAC_a_failure

It's interesting that they failed to recognize the purpose of the study, which was to determine if it is reasonable, when doing research about Wikipedia, to use the fact that a Wikipedia article is "featured" as an indication that the article is of high quality, and the conclusion that such a rule is potentially unwarranted. The Wikipedians chose to read far more into the study than the study's author was willing to say, and then criticize the study roundly for drawing conclusions that it did not actually draw.

As a personal acquaintance of the author and someone who saw this article long ago, I am happy to see that at least a few people understand what David meant to say, and he has taken as many opportunities as available to point out that he was trying to say just what is said above and that other conclusions, while they might be drawn, are not necessarily his own.

[chrisoff]

(Odd, how much the FAC's behaviour resembles that of the Catholic Church, circa, oh, around 1209 or so.....)

I think the reviewer was put down for not talking in FAC-speak and understanding the arcane processes that go on behind the scenes, so his observations were dismissed as off the wall.

This post has been edited by chrisoff:

[Ottava]

Why is there such a falloff of FAC reviewers? Less than ever are now reviewing. No one want to do the reviewing any more.

Of the ones who did a lot of the reviewing, one was banned and another was harassed until he retired(ish?).

A review by David Lindsay (a Georgetown student, booooo!) reviewed FAC and Wiki's FAs. Surprise, surprise, Literature was not reviewed in the process. Some of the results might amuse you.

The scant literature on featured articles is discussed.

Literature. Capital "L". It means great works of writing. Not "previous studies of whatever I'm working on". >.<

This post has been edited by Ottava:

[John Limey]

Why is there such a falloff of FAC reviewers? Less than ever are now reviewing. No one want to do the reviewing any more.

Of the ones who did a lot of the reviewing, one was banned and another was harassed until he retired(ish?).

A review by David Lindsay (a Georgetown student, booooo!) reviewed FAC and Wiki's FAs. Surprise, surprise, Literature was not reviewed in the process. Some of the results might amuse you.

The scant literature on featured articles is discussed.

Literature. Capital "L". It means great works of writing. Not "previous studies of whatever I'm working on". >.<

Ahh, my mistake, when I see the words literature and review close together when discussing an academic paper I assume that we are talking about a literature review, which is not always a correct assumption.

Though literature was omitted, quite a lot of similar topics seem to have been relatively over-reviewed: three paintings, a piece of music, and a composer so that the Arts are covered well enough. Perhaps he decided to structure in general groups and thus got a lot of visual and musical arts but no literature.

[Ottava]

Ahh, my mistake, when I see the words literature and review close together when discussing an academic paper I assume that we are talking about a literature review, which is not always a correct assumption.

Though literature was omitted, quite a lot of similar topics seem to have been relatively over-reviewed: three paintings, a piece of music, and a composer so that the Arts are covered well enough. Perhaps he decided to structure in general groups and thus got a lot of visual and musical arts but no literature.

That's the problem. The "arts" are all different. Literary criticism is more akin to history, political science, psychology, and the rest than it is to studies in other "arts". Why? Because there are so many people that analyze historical, political, and psychological aspects over the "technique" or "aesthetics" than in the other fields.


Anyway, here is David Lindsay's response to me (it should elaborate on the background of the study-reproduced per permission):

The initial sampling was random, and thus certain clusters of articles in one area or another (and the lack of certain areas) is largely a function of experts declining to participate in the study. I am sorry that literature was not covered.

Second, while I am myself in no position to evaluate your articles, I have no reason to believe that they are not quite excellent. There was certainly nothing in my study to indicate that all, or even, most featured articles are "bad". My results merely show that a certain portion are not very good, which has implications for future research on quality.

Third, the point of the study was precisely that some featured articles (yours may well be among them) are much better than others, and I have noted this in the article. Thus, as I say, the problem is not one of quality (there are many excellent articles in Wikipedia) but one of quality control - articles that are featured may be truly world-class or they may be, as one of my experts remarked, equivalent to the work of high school students. Nearly the entire point of my article (and I bold this because it seems to have escaped the attention of so many people) is that not all featured articles are of high-quality, and thus that future researchers should not develop models (as in Blumenstock, Poderi, Wilkinson, Rahm, etc.) that rely on Wikipedia's internal quality assessments. I am well aware that the study has been billed elsewhere as showing that this or that is a failure, but this is not an entirely straightforward conclusion from my work. There are, and I fear that this must be repeated, truly excellent featured articles on Wikipedia but there are also fairly abysmal ones and there are also excellent non-featured articles. Thus, again, the point of my study is that quality research must use a more sophisticated approach than accepting quality assessments such as featured articles (much less A-Class,GA,B-Class, and so on). Also, given that there are some very bad featured articles, I suggested that the featured article process (a term which I use loosely to encompass everything from FAC to FARC) might be (and ought to be) improved if it is to serve as effective quality control. I'm sorry to go on so long, but it seems that people are reading many things in between the lines of my paper which were never said nor intended.

[chrisoff]

There are, and I fear that this must be repeated, truly excellent featured articles on Wikipedia but there are also fairly abysmal ones and there are also excellent non-featured articles. Thus, again, the point of my study is that quality research must use a more sophisticated approach than accepting quality assessments such as featured articles (much less A-Class,GA,B-Class, and so on). Also, given that there are some very bad featured articles, I suggested that the featured article process (a term which I use loosely to encompass everything from FAC to FARC) might be (and ought to be) improved if it is to serve as effective quality control.

Can anyone really disagree with this conclusion?

[Ottava]

There are, and I fear that this must be repeated, truly excellent featured articles on Wikipedia but there are also fairly abysmal ones and there are also excellent non-featured articles. Thus, again, the point of my study is that quality research must use a more sophisticated approach than accepting quality assessments such as featured articles (much less A-Class,GA,B-Class, and so on). Also, given that there are some very bad featured articles, I suggested that the featured article process (a term which I use loosely to encompass everything from FAC to FARC) might be (and ought to be) improved if it is to serve as effective quality control.

Can anyone really disagree with this conclusion?

Well, I'm sure YellowMonkey would be happy if people helped out with FARs.

[GlassBeadGame]

Anyway, here is David Lindsay's response to me (it should elaborate on the background of the study-reproduced per permission):

The initial sampling was random, and thus certain clusters of articles in one area or another (and the lack of certain areas) is largely a function of experts declining to participate in the study. I am sorry that literature was not covered.

Second, while I am myself in no position to evaluate your articles, I have no reason to believe that they are not quite excellent. There was certainly nothing in my study to indicate that all, or even, most featured articles are "bad". My results merely show that a certain portion are not very good, which has implications for future research on quality.

Third, the point of the study was precisely that some featured articles (yours may well be among them) are much better than others, and I have noted this in the article. Thus, as I say, the problem is not one of quality (there are many excellent articles in Wikipedia) but one of quality control - articles that are featured may be truly world-class or they may be, as one of my experts remarked, equivalent to the work of high school students. Nearly the entire point of my article (and I bold this because it seems to have escaped the attention of so many people) is that not all featured articles are of high-quality, and thus that future researchers should not develop models (as in Blumenstock, Poderi, Wilkinson, Rahm, etc.) that rely on Wikipedia's internal quality assessments. I am well aware that the study has been billed elsewhere as showing that this or that is a failure, but this is not an entirely straightforward conclusion from my work. There are, and I fear that this must be repeated, truly excellent featured articles on Wikipedia but there are also fairly abysmal ones and there are also excellent non-featured articles. Thus, again, the point of my study is that quality research must use a more sophisticated approach than accepting quality assessments such as featured articles (much less A-Class,GA,B-Class, and so on). Also, given that there are some very bad featured articles, I suggested that the featured article process (a term which I use loosely to encompass everything from FAC to FARC) might be (and ought to be) improved if it is to serve as effective quality control. I'm sorry to go on so long, but it seems that people are reading many things in between the lines of my paper which were never said nor intended.

Poor Mr. Lindsay. I bet he wishes he studied the relative intelligence of dogs and cats. Their owners would still pester him but the animals at least wouldn't email him. He has a management problem with the fanatic Wikipedians. His response above is not part of the learned discussion. Nor is the posting on WP. It is to get the crazies off his case. That's why he goes out of his way to say there are many excellent articles on WP (which he believes) and many could very well be those of his new constant correspondent (which he has no idea.) Unfortunately I work across his purposes to point this out which only makes managing the fanatics more difficult.

BTW, Ottava do you beleive that Lindsay's article is properly peer reviewed?

[Ottava]

BTW, Ottava do you beleive that Lindsay's article is properly peer reviewed?

I don't really know how to "beleive" anything. At least use a spell checker when being snide, especially after you went rabid on Malleus earlier for a screwed up link.

[dogbiscuit]

For example, I might write an article on Churchill, referencing lots of fine articles in the Times and Sunday Times (which might be perfectly adequate, though clearly targeting a different market than a reference work). While a Wikipedian FA reviewer would probably rate those sources as quite desirable, an expert on Churchill would be aghast.

Clearly you've never taken an article to FAC if you really believe that claptrap.

Clearly, you are an twit if you cannot see that the point I was making is that it is not enough simply to check the sources, but for a review process to be useful, the reviewers need to have knowledge of the subject, and may not have any idea as to what is the real deal on the subject matter.

Unfortunately, Wikipedia is founded on the principle that a million idiots can do the work of one intelligent person, and it is assumed that the FAC review process is gifted with the same magical abilities with far fewer idiots.

If you think it's clear that I'm a "twit", then I think it's equally clear that you're a fucking idiot.

You just did the classic Wikipedian NPA workaround thing which is to be rude about someone without using direct words, so I just made it obvious in return - you've already implicitly called me a fucking idiot in your original response so you didn't need the extra reply.

The point I was making was simple, plain and obvious to anyone who understands the concerns about non-experts concocting articles based on a pot-pouri of sources. Just to be clear, if you know nothing about the detail of a subject, then you could:

a) See that there are some sources referenced and tick a box.

or with a bit more effort

b) See that the words appear to match the sources.

or with more effort

c) Google to see if there are other articles or sources that suggest the sourcing is not representative (but you really need access to more specialist sources than Yahoo! Answers if you are going to validate academic subjects)

or with more effort

d) Read through the sources more completely to get a sense of whether the balance of the source has been kept

or with more effort

e) pop down the library and read up on the subject and research your own sources.

By the time you are down at c and beyond, you really need some subject expertise to find your way around, not the typical computer geek knowledge that is over-represented on Wikipedia.

So unless you are really a subject matter expert, then the amount of effort required to validate the quality of an article in terms of content rather than superficial measures of grammar and syntax and reference counting is going to submerge any volunteer effort, even at 42 articles a month.

Alternatively, you can seek to build a review panel of subject matter experts who can quickly say whether the article is on the mark or not. Alternatively again, you could see what other processes you could come up with to provide a system that can review content rather than style, leaving the debates about commas and capitals to those who have the time and enthusiasm for that grunt work (and having written a book or two myself, there is plenty of work to be done in managing style to keep the game players content).

What is not an acceptable answer is simply to say that "this is what the process is and it is good enough for us editors" when even in a small, flawed research article it can be shown that the process does not produce reliably good articles.

What would be really interesting would be to get the newly appointed WMF Readers' Representative take on what FA status means to readers - or should mean. It is a simple test of that role, and the general point about how Wikipedia can self-certify the quality of its output (or not) would seem to be important to the long term respectability of the project.

I fail to see why all this should be such an anathema to Wikipedia, after all it was some process along these lines that was envisaged when Wikipedia was first evolved. Then some twit thought that crowd-sourcing was the complete solution, failing to recognise that other open source models actually do rely on subject matter experts. Take any quality open source software project, and the people who do the gate keeping are subject matter experts.

So, do you care to contemplate that point, or do you just want to fuck around? I'm quite happy to poll the mods for a ban if you prefer the latter.

[Ottava]

The point I was making was simple, plain and obvious to anyone who understands the concerns about non-experts concocting articles based on a pot-pouri of sources. Just to be clear, if you know nothing about the detail of a subject, then you could:

a) See that there are some sources referenced and tick a box.

or with a bit more effort

b) See that the words appear to match the sources.

or with more effort

c) Google to see if there are other articles or sources that suggest the sourcing is not representative (but you really need access to more specialist sources than Yahoo! Answers if you are going to validate academic subjects)

or with more effort

d) Read through the sources more completely to get a sense of whether the balance of the source has been kept

or with more effort

e) pop down the library and read up on the subject and research your own sources.

By the time you are down at c and beyond, you really need some subject expertise to find your way around, not the typical computer geek knowledge that is over-represented on Wikipedia.

I agree with what you say for the most part. Three things:

1. There are a lot of bad, bad reviewers.

2. I have wasted hours looking through references (offline and the rest) verifying. 50% of the time people thanked me for pointing out problems (Theleftorium, GaryKing, TonyTheTiger). The rest of the time people loathed me for revealing that they were scamming the FAC system. The first group most likely did it passively and didn't know better. The second group most likely knew what they were doing and didn't like being exposed.

3. For Awadewit's Proserpine article, I pointed out how there were a lot of sources missing. She huffed and puffed and got the article through even though it wasn't even close to being 50% complete. I have many books on Percy Bysshe Shelley's addition to the works, his use of mythology, etc, that were essential to understanding the play. All she cared about was feminism stuff and ignored most of the actual criticism on the subject. I even gave her excerpts from Bush's work on Romantic poets and mythology that had many pages devoted to the topic. She never added. Most of her FAs have serious flaws with them.

[GlassBeadGame]

The point I was making was simple, plain and obvious to anyone who understands the concerns about non-experts concocting articles based on a pot-pouri of sources. Just to be clear, if you know nothing about the detail of a subject, then you could:

a) See that there are some sources referenced and tick a box.

or with a bit more effort

b) See that the words appear to match the sources.

or with more effort

c) Google to see if there are other articles or sources that suggest the sourcing is not representative (but you really need access to more specialist sources than Yahoo! Answers if you are going to validate academic subjects)

or with more effort

d) Read through the sources more completely to get a sense of whether the balance of the source has been kept

or with more effort

e) pop down the library and read up on the subject and research your own sources.

By the time you are down at c and beyond, you really need some subject expertise to find your way around, not the typical computer geek knowledge that is over-represented on Wikipedia.

I agree with what you say for the most part. Three things:

1. There are a lot of bad, bad reviewers.

2. I have wasted hours looking through references (offline and the rest) verifying. 50% of the time people thanked me for pointing out problems (Theleftorium, GaryKing, TonyTheTiger). The rest of the time people loathed me for revealing that they were scamming the FAC system. The first group most likely did it passively and didn't know better. The second group most likely knew what they were doing and didn't like being exposed.

3. For Awadewit's Proserpine article, I pointed out how there were a lot of sources missing. She huffed and puffed and got the article through even though it wasn't even close to being 50% complete. I have many books on Percy Bysshe Shelley's addition to the works, his use of mythology, etc, that were essential to understanding the play. All she cared about was feminism stuff and ignored most of the actual criticism on the subject. I even gave her excerpts from Bush's work on Romantic poets and mythology that had many pages devoted to the topic. She never added. Most of her FAs have serious flaws with them.

Again, you so awesome.

[dogbiscuit]

The point I was making was simple, plain and obvious to anyone who understands the concerns about non-experts concocting articles based on a pot-pouri of sources. Just to be clear, if you know nothing about the detail of a subject, then you could:

a) See that there are some sources referenced and tick a box.

or with a bit more effort

b) See that the words appear to match the sources.

or with more effort

c) Google to see if there are other articles or sources that suggest the sourcing is not representative (but you really need access to more specialist sources than Yahoo! Answers if you are going to validate academic subjects)

or with more effort

d) Read through the sources more completely to get a sense of whether the balance of the source has been kept

or with more effort

e) pop down the library and read up on the subject and research your own sources.

By the time you are down at c and beyond, you really need some subject expertise to find your way around, not the typical computer geek knowledge that is over-represented on Wikipedia.

I agree with what you say for the most part. Three things:

1. There are a lot of bad, bad reviewers.

2. I have wasted hours looking through references (offline and the rest) verifying. 50% of the time people thanked me for pointing out problems (Theleftorium, GaryKing, TonyTheTiger). The rest of the time people loathed me for revealing that they were scamming the FAC system. The first group most likely did it passively and didn't know better. The second group most likely knew what they were doing and didn't like being exposed.

3. For Awadewit's Proserpine article, I pointed out how there were a lot of sources missing. She huffed and puffed and got the article through even though it wasn't even close to being 50% complete. I have many books on Percy Bysshe Shelley's addition to the works, his use of mythology, etc, that were essential to understanding the play. All she cared about was feminism stuff and ignored most of the actual criticism on the subject. I even gave her excerpts from Bush's work on Romantic poets and mythology that had many pages devoted to the topic. She never added. Most of her FAs have serious flaws with them.

I was trying to avoid the issue of the reviewers themselves being in some way corrupt, incompetent or being bullied by Those Who Must Be Obeyed, but clearly that is an additional problem to be counteracted in designing processes, including the selection of those who operate the processes.

Wikipedia doesn't do "obvious" though, does it?

[GlassBeadGame]

The one clear lesson of Lindsay's study (remember, the topic of this thread?) is that Wikipedia is not capable of making accurate evaluations of the quality of its own articles. A lecture on how Mr. Smartypants goes about making such an evaluation seems pointless as there is no reason to believe that he is any more successful at this task than any other Wikipedian. A more productive approach would be to consider how outside resources could be used to make better quality evaluations. But Wikipedians don't like to ponder that.

[Ottava]

I was trying to avoid the issue of the reviewers themselves being in some way corrupt, incompetent or being bullied by Those Who Must Be Obeyed, but clearly that is an additional problem to be counteracted in designing processes, including the selection of those who operate the processes.

Well, human failure is always a factor in process failure. By the way, there are some FACs that pass with a handful of supports with some that don't seem to go that in-depth. Then there are instances like this which dragged on forever and had a lot of supports but wasn't passed until after two months because of two opposes. It prompted one of my favorite quotes from Malleus: "Question. Is this FAC being held open to allow time for every single one of wikipedia's 7,989,534 editors to express an opinion on it?"

[dogbiscuit]

A more productive approach would be to consider how outside resources could be used to make better quality evaluations. But Wikipedians don't like to ponder that.

In principle it does not have to be outside resources - any sane organisation trying to write an encyclopedia would welcome subject masters within the project, but Wikitwits see expertise as a threat to their ability to be the most important contributors.

In reality, the Wikipedians who are concerned about expertise are really wanting to pretend that they know as much as any expert, and having real experts around would expose that myth.

Three or four years ago, I would have assumed that it was simply the immaturity of the project that meant that the issue had not yet been resolved, but now I understand just how corrupt (in the broken sense mainly) the management of the project is that makes it impossible to fix this. It is one of those aspects of the project that has a morbid fascination for me, seeing just how insane the rationalisations become of doing things really wrong rather than trying to evolve ways to fix things.

[dogbiscuit]

So, if my recollection is right, a better question is: why is this thread here instead of the more respectable Articles forum?

I asked for it to be moved, but Obesity either decided not to, or doesn't know how.

It needs splitting and a move. I had a moment where I thought I was an involved admin, but then I recovered...

OK, hacked out most of the bitch-fest.

[Moulton]

Is this (or is this not) a Featured Thread?

OK, hacked out most of the bitch-fest.

OK. So this one is the Featured Thread, and the other one is the Fetid Thread?

[dogbiscuit]

Is this (or is this not) a Featured Thread?

OK, hacked out most of the bitch-fest.

OK. So this one is the Featured Thread, and the other one is the Fetid Thread?

I know it is hard to tell at times, but this is the rest.

[Ottava]

I wonder if the experts checking the pages actually went and looked up the sources to see if the text matched. I'd like to know if they simply disagreed with the sources, if they disagreed with the interpretation of them, or what really.

[Cla68]

The FAC reviewers, like Tony1 (I don't know if he is still involved or not), were/are good at pointing out bad English, grammar, flow issues, and Manual of Style violations. Sometimes, they even make good points about aspects of the topic that have been missed or sources that should have been used but weren't.

Of course, because the reviewers aren't usually very knowledgeable about the subject area itself, they often have trouble providing constructive criticism as far as actual topic coverage goes. In this recent example, I gave an FAC reviewer an earful because he objected to the use of one source for so much of the information on the article. Knowing a little of the subject, I knew that that source was the only one readily available in English. Instead of automatically opposing the article, the reviewer should have asked first if there were any other sources available. Or, he/she could have gone to one of the few off-wiki web forums where fans of that topic gather (and there are only a few of them on subjects that arcane) and asked if there were any other English sources on that particular aspect of the topic.

I think I've said this before, out of the 30 or so times that I've nominated an article for FAC, only once did a reviewer go to his local (in this case a college) library and checked my sources and looked to see if there any that I wasn't using that I should have been. As this study shows, this can affect the quality of the resulting article. I have asked off-wiki experts to informally review my articles, and I have taken their feedback seriously.

This post has been edited by Cla68:

[BelovedFox]

The FAC reviewers, like Tony1 (I don't know if he is still involved or not), were/are good at pointing out bad English, grammar, flow issues, and Manual of Style violations. Sometimes, they even make good points about aspects of the topic that have been missed or sources that should have been used but weren't.

Of course, because the reviewers aren't usually very knowledgeable about the subject area itself, they often have trouble providing constructive criticism as far as actual topic coverage goes. In this recent example, I gave an FAC reviewer an earful because he objected to the use of one source for so much of the information on the article. Knowing a little of the subject, I knew that that source was the only one readily available in English. Instead of automatically opposing the article, the reviewer should have asked first if there were any other sources available. Or, he/she could have gone to one of the few off-wiki web forums where fans of that topic gather (and there are only a few of them on subjects that arcane) and asked if there were any other English sources on that particular aspect of the topic.

I think I've said this before, out of the 30 or so times that I've nominated an article for FAC, only once did a reviewer go to his local (in this case a college) library and checked my sources and looked to see if there any that I wasn't using that I should have been. As this study shows, this can affect the quality of the resulting article. I have asked off-wiki experts to informally review my articles, and I have taken their feedback seriously.

The only expert I ever contacted about an article (to which I would expect an expert's guidance would be helpful) declined to go more in depth than "it's pretty good", because he felt that the Wiki system was useless and that the article overrepresented a view he found erroneous.

While I'm hardly an anti-elitist, this is the exact case where the experts' own involvement is a liability. The alternate view has the backing of Ivy League scientists and is a defended view, not fringe. But the scientist in question would have been all for hacking it out entirely.

[Ottava]

The only expert I ever contacted about an article (to which I would expect an expert's guidance would be helpful) declined to go more in depth than "it's pretty good", because he felt that the Wiki system was useless and that the article overrepresented a view he found erroneous.

While I'm hardly an anti-elitist, this is the exact case where the experts' own involvement is a liability. The alternate view has the backing of Ivy League scientists and is a defended view, not fringe. But the scientist in question would have been all for hacking it out entirely.

What was the subject area?

It would probably be best to start lumping things based on the article "type". In literature you can have multiple perspectives that are coexistent but competitive where something like linguistics tends to have rival schools that tend to dislike each other. I also wonder how academic POVs are related to Wiki POVs (especially in terms of "weight").

[BelovedFox]

The only expert I ever contacted about an article (to which I would expect an expert's guidance would be helpful) declined to go more in depth than "it's pretty good", because he felt that the Wiki system was useless and that the article overrepresented a view he found erroneous.

While I'm hardly an anti-elitist, this is the exact case where the experts' own involvement is a liability. The alternate view has the backing of Ivy League scientists and is a defended view, not fringe. But the scientist in question would have been all for hacking it out entirely.

What was the subject area?

It would probably be best to start lumping things based on the article "type". In literature you can have multiple perspectives that are coexistent but competitive where something like linguistics tends to have rival schools that tend to dislike each other. I also wonder how academic POVs are related to Wiki POVs (especially in terms of "weight").

This was geology (IMG:smilys0b23ax56/default/smile.gif)

[Ottava]

This was geology (IMG:smilys0b23ax56/default/smile.gif)

Seriously? What is the competing theory? That the world was flat? ;/

This post has been edited by Ottava: