Content contributors, statistical analysis Options

[radek]

The problem I have with the proposed statistical "rules" that have been presented in this discussion is that they're all ad hoc, rather than empirical. That is, instead of taking a sample of edits or editors, categorizing them by inspection, and then doing an analysis of variance (or some other regression analysis) to identify metrics that are correlated with the already-determined categorizations, you instead identify metrics that you have a priori decided ought to correspond with categorizations. That's methodologically bankrupt; a decisional rule that uses a metric as proxy to categorize members of a population has to be empirically justified, and not just backdoored in by handwaving.

And it's not enough to generate a statistic and then look to see if the extremes fit your hypothesis (e.g. Peter's post giving the "top scorers" on some metric which I think is edits per page); such an analysis is vulnerable to confirmation bias. You need to look at a broad sample from the entire population, not just the three-sigma tail, if you want an actual predictive rule.

From where I sit the "statistical" evidence I've seen posted in this thread ranges from inadequate to farcical. Let's take radek's four-way categorization. Not hard to test it: Take a sample of about 50 editors, categorize them by inspection (not of their statistics, but of their apparent behavior based on examining their edits) into the categories provided. Then generate the statistics radek proposes, and run the numbers to see if the metrics really do predict the categorization, and with what degree of certainty. Until you actually do this, you're just pissing into the wind.

I'm actually sort of doing this. There are two difficulties however. First, is how to sample these 50 editors. I can just pull people off the top of my head or what have you but I'm wary of some kind of bias - basically, I'm not sure how to randomly select these 50 people (this isn't a problem - at least to first approx - with articles, since we have the Random Article feature). The second part, as already mentioned is that for the low epp editors there's no way to distinguish "Posting a Lot at AN/I" from "Running Featured Article Reviews" because soxred counts both as edits to WP. The only way I can think of separating it out is by manually looking at the last 1000 or so edits of a particular editor and counting up the proportion of times they posted to ANI (or similar). This is doable but time consuming.

Overall though, I'm not sure if even then I'd call this "scientific" - it's more like those "Political compass" tests if anything. One thing which WOULD BE interesting is if somehow I could get this data on ALL editors (say, with more than 1000) edits and see which "cell" (or corner of the scatter plot) is "saturated" which one is "over saturated" and which one comes up empty.

(and if you look at that scatterplot above, the 4-way categorization does correctly predict for the 5 people I labeled on there. Malleus is regarded as "content creator". Dr.Blofeld is a "wiki gnome" (under this definition of gnome). Etc. But that's still a small and non-random sample so while encouraging it's not serious evidence at this point)

This post has been edited by radek: Mon 31st October 2011, 8:59pm

[Peter Damian]

The problem I have with the proposed statistical "rules" that have been presented in this discussion is that they're all ad hoc, rather than empirical.

I read no further than that sentence, as it is clear you have no idea what you are talking about. Radek clearly does.

[Kelly Martin]

The problem I have with the proposed statistical "rules" that have been presented in this discussion is that they're all ad hoc, rather than empirical.

I read no further than that sentence, as it is clear you have no idea what you are talking about. Radek clearly does.

Until I see a t-test or F-test score, I'm going to have to assume that you have no idea what you're talking about. Radek is at least making an effort. I'd like to see the actual analysis run; I rather doubt that the two axes in his proposal are truly orthogonal, for example.

I suppose I should actually break down and look at the data set and see what, if anything, can be sucked out of it. In any case, I've been around long enough to distrust sloppy stats. It's amazing how often people's intuitions are wrong about statistical measures, especially in populations that exhibit markedly unbalanced distributions.

[Peter Damian]

What Kelly said.

Peter I was not having a go at you at all. I'm a blunt person, trying here to influence your methology, which I think is at the moment skewed. We can talk, it doesnt have to be all or nothing.

As I said, if you think an argument is wrong, you have to say what is wrong with it. You have to be clear what my argument is, which Kelly isn't, because she clearly hasn't even read the original post, and you have to say what is wrong with it.

The first point is that from a defined population (currently active admins) there is a wide range of epp values. That is an objectively measurable fact.

The second point is that there is no logical connection between high epp and high content. The limiting case is someone who edits an article 100 times by adding the numbers 1-99, then deleting them. That editors epp is a very high 100, but zero content. Conversely an editor who creates an entire article offline then adds to Wikipedia in a single edit has the lowest possible epp, but is creating a lot of content.

The third point is empirical: from the given, precisely defined sample, there is an empircal connection. Those with low epps tend to have mechanical repetitive editing patterns - they are always doing the same kind of thing. By contrast, those with high epps tend (note the word 'tend') to be 'content contributors'.

The fourth point is a behavioural observation. Those who flit from article to article will find it difficult to make insightful and meaningful contributions, which requires careful (and long) study of the whole article.

These were my conclusions. The point that the sample was chosen in the way it was is stupid and irrelevant. If I want to study whether Conservatives drive expensive cars, obviously I have to select just Conservatives. And in any case, to avoid accusations of selection bias, I deliberately ran the study over all 720 admins, without exception.

especially in populations that exhibit markedly unbalanced distributions.

What do you mean by an 'unbalanced distribution'?

[Ceoil]

I'm not a hallowed logician like you are, sitting on a cloud and only receptive to perfectly formed refutals with equasitions and things, but can spot specious argument when I see it. Dear god man, how unattractive was that post. Get a grip.

This post has been edited by Ceoil: Mon 31st October 2011, 9:22pm

[Kelly Martin]

I'm actually sort of doing this. There are two difficulties however. First, is how to sample these 50 editors. I can just pull people off the top of my head or what have you but I'm wary of some kind of bias - basically, I'm not sure how to randomly select these 50 people (this isn't a problem - at least to first approx - with articles, since we have the Random Article feature). The second part, as already mentioned is that for the low epp editors there's no way to distinguish "Posting a Lot at AN/I" from "Running Featured Article Reviews" because soxred counts both as edits to WP. The only way I can think of separating it out is by manually looking at the last 1000 or so edits of a particular editor and counting up the proportion of times they posted to ANI (or similar). This is doable but time consuming.

Overall though, I'm not sure if even then I'd call this "scientific" - it's more like those "Political compass" tests if anything. One thing which WOULD BE interesting is if somehow I could get this data on ALL editors (say, with more than 1000) edits and see which "cell" (or corner of the scatter plot) is "saturated" which one is "over saturated" and which one comes up empty.

(and if you look at that scatterplot above, the 4-way categorization does correctly predict for the 5 people I labeled on there. Malleus is regarded as "content creator". Dr.Blofeld is a "wiki gnome" (under this definition of gnome). Etc. But that's still a small and non-random sample so while encouraging it's not serious evidence at this point)

It looks like the soxred data generates around two dozen metrics per user, some of which are obviously interdependent (as the percentages necessarily add to 100%, so there's at least one degree of freedom eaten there). We can get random users by sampling the "All Users" list, but the problem with that is that most of them will be extremely low (that is, zero) edit count users; not terribly useful. However, the filtering process could be automated using the exposed API (http://en.wikipedia.org/w/api.php), and that API could also be used to automate gathering the "how much does this editor post to ANI" statistics you showed an interest in (although the API throttle might make that a slow process). So it's not unattainable, not in the least.

[radek]

The problem I have with the proposed statistical "rules" that have been presented in this discussion is that they're all ad hoc, rather than empirical.

I read no further than that sentence, as it is clear you have no idea what you are talking about. Radek clearly does.

Until I see a t-test or F-test score, I'm going to have to assume that you have no idea what you're talking about. Radek is at least making an effort. I'd like to see the actual analysis run; I rather doubt that the two axes in his proposal are truly orthogonal, for example.

I suppose I should actually break down and look at the data set and see what, if anything, can be sucked out of it. In any case, I've been around long enough to distrust sloppy stats. It's amazing how often people's intuitions are wrong about statistical measures, especially in populations that exhibit markedly unbalanced distributions.

I think that what Kelly is talking about above is something like External Validity (there seems to be some other criticisms mixed in as well).

One way to do it is to somehow generate a list of randomly selected editors, then pass this list out to people familiar with Wikipedia and ask them to categorize these people according to the criteria above. Then see to what extent the subjective categorizations match up with categorizations based on epp and % articles. This wouldn't be totally ideal as people can have quite skewed and biased notions of themselves and others, in additions to having widely different definitions (a clear example is that guy calling Dr. Blofeld a "content creator" in that thread)

Another way would be to first define what "gnomish" edit is, what a "content creating" edit is, what a "drama queen" post is etc. Then with these pre-set definition in hand go out and get that list of randomly selected editors and again, see if it matches up. This would be way too much work.

(and in fact I'm somewhat ok with just DEFINING high % low epp editors as "Wiki gnomes" and high % high epp editors as "Content creators". Most of the trouble is with the low % folks)

Hmmm, so you want a t stat or an F test. One thing I could do is to see if epp or % articles predict admin status (the logit or probit regression I mentioned before). Two problems would be the lack of randomness I mentioned above, and also that ideally we'd want to have the epp and % articles BEFORE a person became an admin, so that we get the causality right. But I don't think there's data on that though I might email soxred and axe him 'bout it.

I'm actually sort of doing this. There are two difficulties however. First, is how to sample these 50 editors. I can just pull people off the top of my head or what have you but I'm wary of some kind of bias - basically, I'm not sure how to randomly select these 50 people (this isn't a problem - at least to first approx - with articles, since we have the Random Article feature). The second part, as already mentioned is that for the low epp editors there's no way to distinguish "Posting a Lot at AN/I" from "Running Featured Article Reviews" because soxred counts both as edits to WP. The only way I can think of separating it out is by manually looking at the last 1000 or so edits of a particular editor and counting up the proportion of times they posted to ANI (or similar). This is doable but time consuming.

Overall though, I'm not sure if even then I'd call this "scientific" - it's more like those "Political compass" tests if anything. One thing which WOULD BE interesting is if somehow I could get this data on ALL editors (say, with more than 1000) edits and see which "cell" (or corner of the scatter plot) is "saturated" which one is "over saturated" and which one comes up empty.

(and if you look at that scatterplot above, the 4-way categorization does correctly predict for the 5 people I labeled on there. Malleus is regarded as "content creator". Dr.Blofeld is a "wiki gnome" (under this definition of gnome). Etc. But that's still a small and non-random sample so while encouraging it's not serious evidence at this point)

It looks like the soxred data generates around two dozen metrics per user, some of which are obviously interdependent (as the percentages necessarily add to 100%, so there's at least one degree of freedom eaten there). We can get random users by sampling the "All Users" list, but the problem with that is that most of them will be extremely low (that is, zero) edit count users; not terribly useful. However, the filtering process could be automated using the exposed API (http://en.wikipedia.org/w/api.php), and that API could also be used to automate gathering the "how much does this editor post to ANI" statistics you showed an interest in (although the API throttle might make that a slow process). So it's not unattainable, not in the least.

I have no idea on how to do any of that.

[Ottava]

Hi Peter. I'd like to engage Eric, he is often astute but ruined by a perception of bitterness on the part of the faithful. I think he his criticism would be more valuable if he dropped the veneer. God knows the project is lacking introspection, and tends to shoot messengers. <four tides: Ceoil>

Hey, how are you doing?


I do have a bit of nitpicking to do regarding your defense of Amanda after she totally trashed the To Autumn page with OR, plagiarism, etc. You defended her. Those are exactly the kind of person who Wikipedia needs to boot. You also defended Fowler who was someone who did that same thing quite regularly. I dont know if you still feel the same way about such people, but you were putting personal views of a person above what they were actually doing regarding content.

At least the true MySpacers and Drama Queens stay out of articles. The people who are dangerous are those who affect articles in a very negative manner.

[Peter Damian]

I'm not a hallowed logician like you are, sitting on a cloud and only receptive to perfectly formed refutals with equasitions and things, but can spot specious argument when I see it. Dear god man, how unattractive was that post. Get a grip.

Where is the specious argument?

[timbo]

Thinking out loud here...

Each edit changes article size.

Content Creators, whether then write by editing a page 50 times in a row or by writing offline and then adding everything at once, tend to MARKEDLY INCREASE article size in mainspace.

Administrative Gnomes, adding a link here or correcting a spelling there, tend to impact article size in mainspace very little.

Soap Opera Sallies tend to skew their editing away from mainspace.

I don't think the Facebook Faction is a real category. The real fourth group are the Temporary Tramps that come in, write a page about their uncle or their favorite Transformer character, and then leave the project. These can be quantified or eliminated from the equation on the basis of lifetime total edits.

Edits per page is probably highly correlated to these types, but it seems to me that change in article size is more important than edits per page as an identifying metric.


tim

[Peter Damian]

(and in fact I'm somewhat ok with just DEFINING high % low epp editors as "Wiki gnomes" and high % high epp editors as "Content creators". Most of the trouble is with the low % folks)

That would actually be perfect. We start with the behavioural assumption first. Anyone who is editing 3 times a minute or more on different articles is unlikely to be contributing what we call 'content'. That's behind our whole idea of what 'content' is. Namely, stuff you have to study the whole article carefully in order to add.

I think the harder one is the high epp. E.g. FT2 famously spends a huge amount of time editing and re-editing the same sentence, sometimes 100 edits just for one paragraph. But it still reads like the long-winding verbose nonsense that it was in the first place.

But even there, does it matter? Let's just define 'content' as what is added by high epp'ers. Then we have the logical deduction that a very high proportion of admins are not content-producers.

What is actually much more difficult is choosing another population to compare with. Non-admins are too large. Anything else risks selection bias.

[Kelly Martin]

Another way would be to first define what "gnomish" edit is, what a "content creating" edit is, what a "drama queen" post is etc. Then with these pre-set definition in hand go out and get that list of randomly selected editors and again, see if it matches up. This would be way too much work.

This, fundamentally, is the problem with an objective, quantitative analysis of Wikipedia editors and editing. It is, as you say, "way too much work" to code enough editors or edits to do any meaningful analysis. I've only seen a few studies that did, in fact, do such coding, and interestingly enough all of the studies I've seen that did do so ended up contradicting conventional wisdom in at least some ways.

So instead of doing it right, because doing it right is too much work, y'all settle for doing it wrong, and hoping nobody notices. Which, to be fair, nobody usually does. Back when I was in grad school, I knew a guy who (for a master's thesis, I believe) reviewed a couple hundred peer-reviewed papers in the social sciences; he reported that less than 10% of the papers he reviewed were free of serious methodological flaws in their use of statistical method, and nearly half stated conclusions that could not be supported from the data. And every one of these had been passed on in peer review. My conclusion is that social scientists, in general, do not understand statistics.

[Peter Damian]

Also, for the record, here are the first 27 of editors on http://en.wikipedia.org/wiki/Wikipedia:WBFAN

The distribution is quite different from that of the sample admins. You may argue what "Wikipedia:WBFAN" signifies. I reply: it is an objectively verifiable fact that the one precisely defined population has an entirely different distribution from the other. I really don't understand Kelly's problem. As long as you define the criterion by which you select your population, there shouldn't be a problem.


YellowMonkey 3.69
Casliber 6.57
Hurricanehink 7.16
Wehwalt 20.51
Lord_Emsworth 3.47
Brianboulton 15.01
Ealdgyth 6.87
Ucucha 2.58
David_Fuchs 6.99
Mike_Christie 7.18
Malleus_Fatuorum 12.46
Sasata 3.25
Juliancolton 2.09
Awadewit 7.55
Cla68 6.15
Jimfbleak 4.13
DrKiernan 3.49
Iridescent 1.53
Serendipodous 12.34
Parrot_of_Doom 14.48
Ruhrfisch 4.18
Mav 3.1
Ian_Rose 8.57
Parsecboy 4.71
Piotrus 3.75
Acdixon 4.19
Johnleemk 2.41
Karanacs 3.92

[radek]

Thinking out loud here...

Each edit changes article size.

Content Creators, whether then write by editing a page 50 times in a row or by writing offline and then adding everything at once, tend to MARKEDLY INCREASE article size in mainspace.

Administrative Gnomes, adding a link here or correcting a spelling there, tend to impact article size in mainspace very little.

Soap Opera Sallies tend to skew their editing away from mainspace.

I don't think the Facebook Faction is a real category. The real fourth group are the Temporary Tramps that come in, write a page about their uncle or their favorite Transformer character, and then leave the project. These can be quantified or eliminated from the equation on the basis of lifetime total edits.

Edits per page is probably highly correlated to these types, but it seems to me that change in article size is more important than edits per page as an identifying metric.


tim

The thing about change in article size is right but again, I don't know what an efficient way of collecting such data would be.

The Facebook faction appears to contain some long term editors like GWH, so it's not just Temporary Tramps. TTs would look like "content creators" according to the categorization since they'd have very high epp (that one page on their uncle) and and very high % in articles, for the most part (if they get dragged through ani for creating these uncle pages it might go down a bit).

Again, there's no presumption in any of the above that just cuz someone is in the "Content creator" category they're "good" - as Ottava points out many of these could be "bad". I can't think of such a distinction for the FFs but it might be there - I dunno, somebody who just makes everyone feel welcome or something.

[Peter Damian]

Another way would be to first define what "gnomish" edit is, what a "content creating" edit is, what a "drama queen" post is etc. Then with these pre-set definition in hand go out and get that list of randomly selected editors and again, see if it matches up. This would be way too much work.

This, fundamentally, is the problem with an objective, quantitative analysis of Wikipedia editors and editing. It is, as you say, "way too much work" to code enough editors or edits to do any meaningful analysis. I've only seen a few studies that did, in fact, do such coding, and interestingly enough all of the studies I've seen that did do so ended up contradicting conventional wisdom in at least some ways.

So instead of doing it right, because doing it right is too much work, y'all settle for doing it wrong, and hoping nobody notices. Which, to be fair, nobody usually does. Back when I was in grad school, I knew a guy who (for a master's thesis, I believe) reviewed a couple hundred peer-reviewed papers in the social sciences; he reported that less than 10% of the papers he reviewed were free of serious methodological flaws in their use of statistical method, and nearly half stated conclusions that could not be supported from the data. And every one of these had been passed on in peer review. My conclusion is that social scientists, in general, do not understand statistics.

What is your qualification in statistics, Kelly?

This post has been edited by Peter Damian: Mon 31st October 2011, 9:45pm

[radek]

Also, for the record, here are the first 27 of editors on http://en.wikipedia.org/wiki/Wikipedia:WBFAN

The distribution is quite different from that of the sample admins. You may argue what "Wikipedia:WBFAN" signifies. I reply: it is an objectively verifiable fact that the one precisely defined population has an entirely different distribution from the other. I really don't understand Kelly's problem. As long as you define the criterion by which you select your population, there shouldn't be a problem.


YellowMonkey 3.69
Casliber 6.57
Hurricanehink 7.16
Wehwalt 20.51
Lord_Emsworth 3.47
Brianboulton 15.01
Ealdgyth 6.87
Ucucha 2.58
David_Fuchs 6.99
Mike_Christie 7.18
Malleus_Fatuorum 12.46
Sasata 3.25
Juliancolton 2.09
Awadewit 7.55
Cla68 6.15
Jimfbleak 4.13
DrKiernan 3.49
Iridescent 1.53
Serendipodous 12.34
Parrot_of_Doom 14.48
Ruhrfisch 4.18
Mav 3.1
Ian_Rose 8.57
Parsecboy 4.71
Piotrus 3.75
Acdixon 4.19
Johnleemk 2.41
Karanacs 3.92

Quick comment on this list - while this probably isn't a problem for these guys above, once you start going down a further bit you have a bunch people who didn't actually WRITE the FAs, just NOMINATED them.

[Kelly Martin]

What I want is a test. That is, I want a decisional rule: something like "if editor's epp < 3.0, then editor is a content creator, with p=0.8". There are rigorous methods for adducing such decisional rules from appropriate sample data. But the proposed rules that have been offered so far are not derived using those methods; they are instead just generated ad hoc. This is appropriate for the investigatory phase of the analysis, but you can't just stop there.

And none of the hypotheses I've seen thrown out have been rigorously tested, even though in most cases I think they can be, in some cases fairly easily. Why is this?

[radek]

Another way would be to first define what "gnomish" edit is, what a "content creating" edit is, what a "drama queen" post is etc. Then with these pre-set definition in hand go out and get that list of randomly selected editors and again, see if it matches up. This would be way too much work.

This, fundamentally, is the problem with an objective, quantitative analysis of Wikipedia editors and editing. It is, as you say, "way too much work" to code enough editors or edits to do any meaningful analysis. I've only seen a few studies that did, in fact, do such coding, and interestingly enough all of the studies I've seen that did do so ended up contradicting conventional wisdom in at least some ways.

So instead of doing it right, because doing it right is too much work, y'all settle for doing it wrong, and hoping nobody notices. Which, to be fair, nobody usually does. Back when I was in grad school, I knew a guy who (for a master's thesis, I believe) reviewed a couple hundred peer-reviewed papers in the social sciences; he reported that less than 10% of the papers he reviewed were free of serious methodological flaws in their use of statistical method, and nearly half stated conclusions that could not be supported from the data. And every one of these had been passed on in peer review. My conclusion is that social scientists, in general, do not understand statistics.

Well, I'm not going to send off my four-color chart off to an academic journal or anything. Like I keep saying, to me, at this point, this is more like that Political Compass quiz - potentially informative but flawed. I'm interested in making this more rigorous (cuz I got that itch) but there's also a limited amount of time I'm willing to devote to it. If you can help with any of the suggestions I mentioned in a concrete way, I'd appreciate it.

[Kelly Martin]

What is your qualification in statistics, Kelly?

What does that matter?

[Peter Damian]

What is your qualification in statistics, Kelly?

What does that matter?

It matters a lot.