NYB on The Volokh Conspiracy, Some First Thoughts on Wikipedia Options

[tarantino]

http://www.volokh.com/posts/1242098183.shtml

Ira requests:

I make only one request: that regular Wikipedians who are looking over my shoulder, as well as Wikipedia critics from Wikipedia Review and elsewhere, bear in mind that this is a general-interest audience. Please don't hijack the comment threads with our own internal disputes and debates. No one here wants to read who is a sockpuppet of whom or whether so-and-so's block was fair or not. We have ANI and Wikipedia Review to hash those things out later.

[Daniel Brandt]

NYB, when are you going to put your name on your user page? Cool_Hand_Luke should do it too.

[Newyorkbrad]

NYB, when are you going to put your name on your user page? Cool_Hand_Luke should do it too.

I actually had planned to do it this week, as I mentioned in this thread last week. I might well have done it when I returned to editing last summer, but I didn't particularly appreciate the feeling of being coerced into it.

[Daniel Brandt]

I actually had planned to do it this week, as I mentioned in this thread last week. I might well have done it when I returned to editing last summer, but I didn't particularly appreciate the feeling of being coerced into it.

Maybe the move would be easier if the User pages has a "noindex" just like the User_talk pages do. I think there is more work to be done on the "noindex" front. It's true that your user page would most likely be in the top ten results in a search for your real name, because your real name is unusual and Google is infatuated with everything from Wikipedia.

That's all the more reason to level the field by adopting a "noindex" policy for all User pages. It might even take the wind out of some of those pesky teenage editors if they were unable to get a hormone kick from their Google juice.

[Lar]

I actually had planned to do it this week, as I mentioned in this thread last week. I might well have done it when I returned to editing last summer, but I didn't particularly appreciate the feeling of being coerced into it.

Maybe the move would be easier if the User pages has a "noindex" just like the User_talk pages do. I think there is more work to be done on the "noindex" front. It's true that your user page would most likely be in the top ten results in a search for your real name, because your real name is unusual and Google is infatuated with everything from Wikipedia.

That's all the more reason to level the field by adopting a "noindex" policy for all User pages. It might even take the wind out of some of those pesky teenage editors if they were unable to get a hormone kick from their Google juice.

Strikes me that the default for all namespaces ought to be noindex, and then evaluate namespace by namespace whether to be non default, (so article would be indexed) and then evaluate page by page (where necessary) whether to be non default from the non default (so that the article called "Daniel Brandt is a big weenie" is noindex even though articles generally are...)

I could swear there was a proposal to actually do that at some point.

But specifically, I think user ought to be noindex just like user talk.

[gomi]

Strikes me that the default for all namespaces ought to be noindex, .... I could swear there was a proposal to actually do that at some point.

If it is supposed to be an encyclopedia, what possible argument is there for indexing anything other than article pages? Even then, it would seem that #REDIRECT pages and most fancruft pages should be NOINDEX. Of course, that presumes Wikipedia's real intention is to be an encyclopedia. As we know, nothing could be further from the truth. The Wikipidiots desperately want their drama and ego-centric masturbation to be google-able.

[Milton Roe]

Strikes me that the default for all namespaces ought to be noindex, .... I could swear there was a proposal to actually do that at some point.

Yuh, it came from ME, among others (actually I suggested only BLPs be automatically noindexed, but it really should be all BLPs, userspace, and maybe even article TALK space). The rest of the encyclopedia articles deserve to be google-findable.

http://wikipediareview.com/index.php?s=&sh...ndpost&p=157102

Let's see you find anyone who suggested it sooner.

Milt

(Born to have my ideas ignored, then percolate up through people's unconscienceness' eventally)

[Somey]

Let's see you find anyone who suggested it sooner.

I was suggesting that they do something like that back in 2006 - this post, for example.

Not to take credit or anything...

[Milton Roe]

Let's see you find anyone who suggested it sooner.

I was suggesting that they do something like that back in 2006 - this post, for example.

Not to take credit or anything...

Meanwhile, the very least they could do would be to alter their robots.txt file to prevent unwanted biographies from being indexed by Google, but they can't even be bothered to do that, can they? Nooooo.

Ohhhhh, very good! Did you know the actual HTML(?) tag for that, at the time? AFAIK, we didn't discover NOINDEX until some time later. And were astounded at how easy this was, and yet WP still refuses to alter their software to allow it (yes, you can't add {{NOINDEX}} to a mainspace page and have it do anything, even if you want to).

[Somey]

Ohhhhh, very good! Did you know the actual HTML(?) tag for that, at the time? AFAIK, we didn't discover NOINDEX until some time later. And were astounded at how easy this was, and yet WP still refuses to alter their software to allow it (yes, you can't add {{NOINDEX}} to a mainspace page and have it do anything, even if you want to).

To be fair (not that they deserve fairness under the circumstances), it wasn't quite that simple, originally. The searchbots only care about the URL (i.e., they don't know what a "namespace" is), so in theory the Wikipedia folks could have simply moved any questionable BLP or other article into some sort of "quasi-namespace" and added the quasi-namespace to robots.txt.

IOW, they could have moved "Daniel Brandt" to "BLP/Daniel Brandt," added "http://en.wikipedia.org/wiki/BLP/*" to robots.txt as an exception, and that article (and others with "BLP/" in front of their titles) would have been de-indexed. However, that would have broken external links to the article, and since redirects are essentially the same as the target article as far as Google is concerned, they would either have had to accept the breakage of those links (no big deal IMO, if used only in specific cases), or needed some means of preventing the Redirect from working specifically for searchbots. WR's board software can do things like that, but MediaWiki either isn't that sophisticated, or nobody wants to develop/enable an extension to make it that sophisticated.

"NOINDEX" tagging is actually harder to implement, but since theoretically it could be added to any article by any editor (if not disabled by the developers, as they've done), it's a better solution (or would be if it were fully enabled).

Anyway, the reason it's harder is that MediaWiki produces and outputs its HTML page headers before it ever reads, much less parses, the article text in the database. That "noindex" has to be in the headers as a META tag, but "pre-parsing" the article text would add to the processing overhead and affect performance... So, I'd assume (I haven't actually looked, to be honest) that parsing for the __NOINDEX__ "magic word" takes place at the point of storing the text - i.e., "post-parsing" - and the presence of the magic word sets a flag in a separate field, which wouldn't have to be parsed.

One of these days I'll take a look at it and see how they actually did that. Regardless though, a really good MediaWiki developer shouldn't have needed more than a couple of days to figure out the technical aspects - a couple of hours, more likely. Certainly not two years, in any event - the fact is, they just didn't want to do it, and wouldn't have if people like Newyorkbrad hadn't gotten behind the idea. Which is why I believe Mr. Brad should be highly commended for taking a stand on it.