Checkuser data retention Options

[gomi]

here:

Tim Starling tstarling at wikimedia.org
Thu Sep 11 03:11:52 UTC 2008

Jon wrote:
> I could not find this in the privacy policy... however, what is
> Wikimedia's current data retention policy? That is to ask, how long do
> projects keep data for use in tools such as checkuser?

CheckUser data used to be kept for 3 months, but Aaron recently increased
it to 5 months. I'm not sure why or on whose authority.

<http://svn.wikimedia.org/viewvc/mediawiki/trunk/extensions/CheckUser/CheckUser.php?r1=39734&r2=40620>

-- Tim Starling

[Rootology]

Kelly said recently that the Checkuser data was moved to it's own separate table that hadn't been cleared since November 2007. Whats the discrepancy here?

[Kelly Martin]

Kelly said recently that the Checkuser data was moved to it's own separate table that hadn't been cleared since November 2007. Whats the discrepancy here?

Probably my misunderstanding of the changes to the checkuser code. Those changes meant that checkuser data could potentially be kept indefinitely, which I misinterpreted to mean that they were being kept indefinitely. My mistake.

[gomi]

I should hasten to add that there are probably site-specific overrides possible, and it is well within the realm of possibility that Wikipedia uses one. That would explain why Starling was so cavalier about giving out the above information.

More on the subject, from the same thread:

Tim Starling tstarling at wikimedia.org
Thu Sep 11 04:26:16 UTC 2008

Gregory Maxwell wrote:
> I think Jon was inquiring about more than just checkuser (notice the
> "such as"). I would assume that anyone asking about data retention in
> general is not overly concerned with the specific modes of retention,
> but is more concerned with the maximum retention time (across all
> modes) of any particular type of private data.

The other logs are not automatically rotated, and need to be manually
purged. The retention time is thus not consistent. Typically we have kept
around 6 months of data. There are error logs, and logs for various kinds
of special requests. They are not used for sockpuppet investigation.

I've said in the past that I think 6 months would be a reasonable horizon
for all private data -- it would give us plenty of data for operations,
and would be a far shorter period than that used by the large commercial
websites.

-- Tim Starling

[C H]

Note, Tim Starling has since reverted the change to the CheckUser data maximum age, with the comment "If you want such a policy change, have an open discussion about it, don't get together with some troll-hunting mates on a private mailing list and make your own rules."

That makes me curious as to what he's talking about. Who is this developer Aaron and who are his "troll-hunting mates?" Is the private mailing list being referred to the global checkuser list or another one? I notice that Aaron is the one who added the email logging to checkuser, which Brion reverted, and then added a "toned-down version" of the email logging, which Brion also reverted.

[Rootology]

Links to the mail reversions?

[Rootology]

The curious question here is: does each project actually use the standard Mediawiki software we're watching developers revert war over, here? Is this the EXACT trunk and SQL schema that English Wikipedia uses, and that Commons users, and Meta, and WikiQuote?

IF it is, someone with enough familiarity with all of this and the public records may be able to ferret out all sorts of valuable information.

[C H]

Links to the mail reversions?

Just scroll down that same page.

Here and here.

Note, it looks like the email logging was re-enabled here.

This post has been edited by C H: Mon 15th September 2008, 9:19pm

[Random832]

The curious question here is: does each project actually use the standard Mediawiki software we're watching developers revert war over, here? Is this the EXACT trunk and SQL schema that English Wikipedia uses, and that Commons users, and Meta, and WikiQuote?

IF it is, someone with enough familiarity with all of this and the public records may be able to ferret out all sorts of valuable information.

There's no history of what versions go live when (and not every version goes live at all), but - otherwise, yeah, that's the software - there's some local configuration variables though.

Some of these are at http://noc.wikimedia.org/conf/

I can probably decipher some of this if there's anything in particular.

This post has been edited by Random832: Tue 16th September 2008, 12:36am

[Kelly Martin]

Links to the mail reversions?

Just scroll down that same page.

Here and here.

Note, it looks like the email logging was re-enabled here.

The email logging was reverted originally because they conflicted with SUL; that was a technical decision, not a policy one; it was fully intended that they'd be brought back once the conflict was resolved.

[anthony]

Who is this developer Aaron

Aaron Schulz a.k.a. User:Voice of All?

Is there an up-to-date list of all CVS commiters?

[jch]

Who is this developer Aaron

Aaron Schulz a.k.a. User:Voice of All?

Is there an up-to-date list of all CVS commiters?

http://svn.wikimedia.org/users.php

Also, it's SVN, not CVS. I think CVS is a drugstore...

[Lar]

Who is this developer Aaron

Aaron Schulz a.k.a. User:Voice of All?

Is there an up-to-date list of all CVS commiters?

http://svn.wikimedia.org/users.php

Also, it's SVN, not CVS. I think CVS is a drugstore...

SVN aims to be a replacement for CVS, which in turn aimed to be a replacement for RCS... These are all "Open" version control systems as opposed to say, PVCS or VSS, which are "Closed" (with respect to version control this refers to whether a module is locked or whether multiple can work on it at once and then you merge the changes, not Open/Closed in the free software sense)

[Rootology]

Last night I poked around all the listed materials out of curiosity, and darned if I can find where wgCUDMaxAge and the retention for CU data is actually stored on the separate table. I have a feeling it's out of view. Since this is the principle "privacy" matter (especially after Poetgate) that everyone is always worried about, I'm honestly wondering if the benefits of hiding this information from being displayed in a clear fashion outweigh the possible harm.

This post has been edited by Rootology: Fri 19th September 2008, 5:58pm

[jch]

Who is this developer Aaron

Aaron Schulz a.k.a. User:Voice of All?

Is there an up-to-date list of all CVS commiters?

http://svn.wikimedia.org/users.php

Also, it's SVN, not CVS. I think CVS is a drugstore...

SVN aims to be a replacement for CVS, which in turn aimed to be a replacement for RCS... These are all "Open" version control systems as opposed to say, PVCS or VSS, which are "Closed" (with respect to version control this refers to whether a module is locked or whether multiple can work on it at once and then you merge the changes, not Open/Closed in the free software sense)

That was what we Internets users call a "joke" Lar. I think you're spending too much time around your OTRS-using humorless sweetie.

[Kelly Martin]

Last night I poked around all the listed materials out of curiosity, and darned if I can find where wgCUDMaxAge and the retention for CU data is actually stored on the separate table. I have a feeling it's out of view. Since this is the principle "privacy" matter (especially after Poetgate) that everyone is always worried about, I'm honestly wondering if the benefits of hiding this information from being displayed in a clear fashion outweigh the possible harm.

This would be configured in the LocalSettings.php file, which cannot be directly published as it contains passwords for the database engine and other information that would be Bad to let get out. There is a redacted LocalSettings.php file somewhere on Wikimedia's site, but it is rather out of date, as I recall.

[Krimpet]

Last night I poked around all the listed materials out of curiosity, and darned if I can find where wgCUDMaxAge and the retention for CU data is actually stored on the separate table. I have a feeling it's out of view. Since this is the principle "privacy" matter (especially after Poetgate) that everyone is always worried about, I'm honestly wondering if the benefits of hiding this information from being displayed in a clear fashion outweigh the possible harm.

From CheckUser.php:

CODE

    # Every 100th edit, prune the checkuser changes table.
    wfSeedRandom();
    if( 0 == mt_rand( 0, 99 ) ) {
        # Periodically flush old entries from the recentchanges table.
        global $wgCUDMaxAge;
        $cutoff = $dbw->timestamp( time() - $wgCUDMaxAge );
        $recentchanges = $dbw->tableName( 'cu_changes' );
        $sql = "DELETE FROM $recentchanges WHERE cuc_timestamp < '{$cutoff}'";
        $dbw->query( $sql );
    }

Also, while Wikimedia's LocalSettings does require() a file called "PrivateSettings.php" that's not world-viewable that contains database passwords and the like, it's loaded before the CheckUser extension is, meaning any attempts to hide CU settings in there would be overridden by the defaults when CheckUser is loaded.

I think it can be confidently said, then, that CU data on Wikimedia is deleted after the default 90 days.

[tarantino]

I think it can be confidently said, then, that CU data on Wikimedia is deleted after the default 90 days.

This is confirmed by Tim Starling -

It's the same everywhere, it's three months. Neither the Board nor the
executive have expressed any desire to make that decision, but they are
free to weigh in if they want to. We chose the three month figure as a
compromise between privacy advocates and troll hunters.