Content contributors, statistical analysis Options

[Peter Damian]

My blog post for today http://ocham.blogspot.com/2011/10/repetiti...-wikipedia.html on whether there are statistically measurable properties that distinguish 'content contributors' from wiki-gnomes. Conclusion: the statistical difference is strongly indicative of a real difference, discussed in detail on the blog.

Remaining questions: why do content contributors remain on the project, given that they have a lower status than those who perform repetitive and tedious work?

Easily-learned repetitive labour is nearly always paid less in real life than labour which requires either specialised learning, or some innate but scarce skill. The simple reason for this is supply and demand. Rare or difficult-to-acquire skills are by definition in short supply, and will attract a higher price than common, easily acquired skills (at least, to my simple mind - I don't know any economics).

So why is the situation apparently reversed on Wikipedia? The statistics suggest that the majority of administrators use these low-value skills like vandal reversion, template adding, linking to the Estonian Wikipedia etc. Yet their status on Wikipedia is high, whereas that of 'content contributors' is low.

This post has been edited by Peter Damian:

[Kelly Martin]

The problem I have with the proposed statistical "rules" that have been presented in this discussion is that they're all ad hoc, rather than empirical. That is, instead of taking a sample of edits or editors, categorizing them by inspection, and then doing an analysis of variance (or some other regression analysis) to identify metrics that are correlated with the already-determined categorizations, you instead identify metrics that you have a priori decided ought to correspond with categorizations. That's methodologically bankrupt; a decisional rule that uses a metric as proxy to categorize members of a population has to be empirically justified, and not just backdoored in by handwaving.

And it's not enough to generate a statistic and then look to see if the extremes fit your hypothesis (e.g. Peter's post giving the "top scorers" on some metric which I think is edits per page); such an analysis is vulnerable to confirmation bias. You need to look at a broad sample from the entire population, not just the three-sigma tail, if you want an actual predictive rule.

From where I sit the "statistical" evidence I've seen posted in this thread ranges from inadequate to farcical. Let's take radek's four-way categorization. Not hard to test it: Take a sample of about 50 editors, categorize them by inspection (not of their statistics, but of their apparent behavior based on examining their edits) into the categories provided. Then generate the statistics radek proposes, and run the numbers to see if the metrics really do predict the categorization, and with what degree of certainty. Until you actually do this, you're just pissing into the wind.

(This is on my mind at the moment because I've been reading some of the materials related to the dual-polarization radar that the NWS just put up here in Chicago. They've done a lot of research to try to come up with rules to translate the various additional metrics the new radar offers into actionable information such as "hail", "snow", "freezing rain", and "graupel". While they have some concepts of what they think will happen, the actual methodology used in the field is based on collecting the metrics and cross-correlating it with "ground truth" reports of what is actually going on in the field. They're doing it right, which is why they can predict the weather, and you can't.)

[Peter Damian]

The problem I have with the proposed statistical "rules" that have been presented in this discussion is that they're all ad hoc, rather than empirical.

I read no further than that sentence, as it is clear you have no idea what you are talking about. Radek clearly does.

[Kelly Martin]

The problem I have with the proposed statistical "rules" that have been presented in this discussion is that they're all ad hoc, rather than empirical.

I read no further than that sentence, as it is clear you have no idea what you are talking about. Radek clearly does.

Until I see a t-test or F-test score, I'm going to have to assume that you have no idea what you're talking about. Radek is at least making an effort. I'd like to see the actual analysis run; I rather doubt that the two axes in his proposal are truly orthogonal, for example.

I suppose I should actually break down and look at the data set and see what, if anything, can be sucked out of it. In any case, I've been around long enough to distrust sloppy stats. It's amazing how often people's intuitions are wrong about statistical measures, especially in populations that exhibit markedly unbalanced distributions.

[radek]

The problem I have with the proposed statistical "rules" that have been presented in this discussion is that they're all ad hoc, rather than empirical.

I read no further than that sentence, as it is clear you have no idea what you are talking about. Radek clearly does.

Until I see a t-test or F-test score, I'm going to have to assume that you have no idea what you're talking about. Radek is at least making an effort. I'd like to see the actual analysis run; I rather doubt that the two axes in his proposal are truly orthogonal, for example.

I suppose I should actually break down and look at the data set and see what, if anything, can be sucked out of it. In any case, I've been around long enough to distrust sloppy stats. It's amazing how often people's intuitions are wrong about statistical measures, especially in populations that exhibit markedly unbalanced distributions.

I think that what Kelly is talking about above is something like External Validity (there seems to be some other criticisms mixed in as well).

One way to do it is to somehow generate a list of randomly selected editors, then pass this list out to people familiar with Wikipedia and ask them to categorize these people according to the criteria above. Then see to what extent the subjective categorizations match up with categorizations based on epp and % articles. This wouldn't be totally ideal as people can have quite skewed and biased notions of themselves and others, in additions to having widely different definitions (a clear example is that guy calling Dr. Blofeld a "content creator" in that thread)

Another way would be to first define what "gnomish" edit is, what a "content creating" edit is, what a "drama queen" post is etc. Then with these pre-set definition in hand go out and get that list of randomly selected editors and again, see if it matches up. This would be way too much work.

(and in fact I'm somewhat ok with just DEFINING high % low epp editors as "Wiki gnomes" and high % high epp editors as "Content creators". Most of the trouble is with the low % folks)

Hmmm, so you want a t stat or an F test. One thing I could do is to see if epp or % articles predict admin status (the logit or probit regression I mentioned before). Two problems would be the lack of randomness I mentioned above, and also that ideally we'd want to have the epp and % articles BEFORE a person became an admin, so that we get the causality right. But I don't think there's data on that though I might email soxred and axe him 'bout it.

I'm actually sort of doing this. There are two difficulties however. First, is how to sample these 50 editors. I can just pull people off the top of my head or what have you but I'm wary of some kind of bias - basically, I'm not sure how to randomly select these 50 people (this isn't a problem - at least to first approx - with articles, since we have the Random Article feature). The second part, as already mentioned is that for the low epp editors there's no way to distinguish "Posting a Lot at AN/I" from "Running Featured Article Reviews" because soxred counts both as edits to WP. The only way I can think of separating it out is by manually looking at the last 1000 or so edits of a particular editor and counting up the proportion of times they posted to ANI (or similar). This is doable but time consuming.

Overall though, I'm not sure if even then I'd call this "scientific" - it's more like those "Political compass" tests if anything. One thing which WOULD BE interesting is if somehow I could get this data on ALL editors (say, with more than 1000) edits and see which "cell" (or corner of the scatter plot) is "saturated" which one is "over saturated" and which one comes up empty.

(and if you look at that scatterplot above, the 4-way categorization does correctly predict for the 5 people I labeled on there. Malleus is regarded as "content creator". Dr.Blofeld is a "wiki gnome" (under this definition of gnome). Etc. But that's still a small and non-random sample so while encouraging it's not serious evidence at this point)

It looks like the soxred data generates around two dozen metrics per user, some of which are obviously interdependent (as the percentages necessarily add to 100%, so there's at least one degree of freedom eaten there). We can get random users by sampling the "All Users" list, but the problem with that is that most of them will be extremely low (that is, zero) edit count users; not terribly useful. However, the filtering process could be automated using the exposed API (http://en.wikipedia.org/w/api.php), and that API could also be used to automate gathering the "how much does this editor post to ANI" statistics you showed an interest in (although the API throttle might make that a slow process). So it's not unattainable, not in the least.

I have no idea on how to do any of that.

[Peter Damian]

(and in fact I'm somewhat ok with just DEFINING high % low epp editors as "Wiki gnomes" and high % high epp editors as "Content creators". Most of the trouble is with the low % folks)

That would actually be perfect. We start with the behavioural assumption first. Anyone who is editing 3 times a minute or more on different articles is unlikely to be contributing what we call 'content'. That's behind our whole idea of what 'content' is. Namely, stuff you have to study the whole article carefully in order to add.

I think the harder one is the high epp. E.g. FT2 famously spends a huge amount of time editing and re-editing the same sentence, sometimes 100 edits just for one paragraph. But it still reads like the long-winding verbose nonsense that it was in the first place.

But even there, does it matter? Let's just define 'content' as what is added by high epp'ers. Then we have the logical deduction that a very high proportion of admins are not content-producers.

What is actually much more difficult is choosing another population to compare with. Non-admins are too large. Anything else risks selection bias.

[Malleus]

(and in fact I'm somewhat ok with just DEFINING high % low epp editors as "Wiki gnomes" and high % high epp editors as "Content creators". Most of the trouble is with the low % folks)

That would actually be perfect. We start with the behavioural assumption first. Anyone who is editing 3 times a minute or more on different articles is unlikely to be contributing what we call 'content'. That's behind our whole idea of what 'content' is. Namely, stuff you have to study the whole article carefully in order to add.

I think the harder one is the high epp. E.g. FT2 famously spends a huge amount of time editing and re-editing the same sentence, sometimes 100 edits just for one paragraph. But it still reads like the long-winding verbose nonsense that it was in the first place.

But even there, does it matter? Let's just define 'content' as what is added by high epp'ers. Then we have the logical deduction that a very high proportion of admins are not content-producers.

What is actually much more difficult is choosing another population to compare with. Non-admins are too large. Anything else risks selection bias.

What about those users like me who failed at RfA?

[Ceoil]

(and in fact I'm somewhat ok with just DEFINING high % low epp editors as "Wiki gnomes" and high % high epp editors as "Content creators". Most of the trouble is with the low % folks)

That would actually be perfect. We start with the behavioural assumption first. Anyone who is editing 3 times a minute or more on different articles is unlikely to be contributing what we call 'content'. That's behind our whole idea of what 'content' is. Namely, stuff you have to study the whole article carefully in order to add.

I think the harder one is the high epp. E.g. FT2 famously spends a huge amount of time editing and re-editing the same sentence, sometimes 100 edits just for one paragraph. But it still reads like the long-winding verbose nonsense that it was in the first place.

But even there, does it matter? Let's just define 'content' as what is added by high epp'ers. Then we have the logical deduction that a very high proportion of admins are not content-producers.

What is actually much more difficult is choosing another population to compare with. Non-admins are too large. Anything else risks selection bias.

What about those users like me who failed at RfA?

Your Rfa was not so much a failure as an assassination. I'm sure your savy enough to realise the orchestration.

This post has been edited by Ceoil:

[Ceoil]

A point Peter should make is that its a hard and at times mindbending slog pushing against boring stupid people. Most of the onsite angst comes from this, and it tips otherwise good people over the edge.

The Joy: I'm fairly randy but even I draw the line at the HWNN, he's endearing and funny, and I like him, but jesus not my type. Also, and this is important, yes I'm a hot broad. 26, busty and a doctor, very lonely.

This post has been edited by Ceoil: