Wikipedia:Pending changes/Metrics/Preliminary Analysis#Additions

Note: Per Ocaasi's request, here is some basic analysis on the Pending Changes data. This analysis is meant to serve as a starting point to help focus the data for community discussion. Since individual reviewers have the deepest experience with the feature and how it works with specific types of articles, I've kept most of the commentary on the quantitative aspects. The exception here is the discussion of most frequently edited articles, where I pointed out some obvious commonalities. Hopefully the cuts of data in this analysis will be useful. If not, there are the full tables which people may use to look at the data in detail.

Throughout this analysis, I've focused on metrics that we've frequently used to evaluate the success of Pending Changes:

• Participation: approximated by the number of anonymous edits an article under Pending Changes receives
• Quality: the quality of the anonymous edits. This is approximated by the % of anonymous edits that are not reverted
• Work: the amount of work required from reviewers to manage the incoming anonymous edits. This is approximated by the number of anonymous edits an article receives, the idea being that anonymous edits require review, which requires work from reviewers.

Howief (talk) 23:52, 20 August 2010 (UTC)

First, I made a simplifying assumption. The amount of time that an article is under pending changes for varies and this variation results in outliers later in the analysis. For example, slag was under pending changes for only 38 seconds. I decided to exclude articles that have been under pending changes for less than one week (152 out of 1647, or 9.2% of total). The total number of articles that have been under pending changes for at least one week is 1495.

Comparing articles by the total number of edits would not take into account the fact that some articles have been under pending changes longer than others. So normalizing by the number of days under pending changes can help us compare articles that have been under Pending Changes for differing amounts of time. Here is a histogram of articles by the number of days the articles were under Pending Changes:

class="wikitable sort" | align="center" style="background:#f0f0f0;"|Ave Edits/Day | align="center" style="background:#f0f0f0;"|# of Articles | align="center" style="background:#f0f0f0;"|% of total
>10	3	0.2%
5-10	14	0.9%
2-5	70	4.7%
1-2	150	10%
.5-1	206	14%
.25-.5	238	16%
.1-.25	226	15%
0-.1	303	20%
0	285	19%
Total	1,495	100%

One of the goals of Pending Changes is to open up semi-protected articles to anonymous editors. Yet from the data, we see that 84% of the articles under pending changes receive, on average, less that one edit per day. Much of this may be due to the natural edit velocity of the articles that have been placed under Pending Changes. Note that this is not a judgement on the value of the edits as a single edit may prove very valuable to the quality of an article. All this analysis tells us is that the frequency of anonymous edits is relatively low for a large percentage of the articles that were under Pending Changes.

Of articles that were under Pending Changes for at least one week, here are the 10 articles that received the most number of edits per day from anonymous users:

class="wikitable sort" | align="center" style="background:#f0f0f0;"|Page title | align="center" style="background:#f0f0f0;"|Time under Pending (days) | align="center" style="background:#f0f0f0;"|Total anon edits while under Pending | align="center" style="background:#f0f0f0;"| Anon edits/day | align="center" style="background:#f0f0f0;"|% nonreverted/total anon edits
Total Drama World Tour	20.77	493	23.7	71.8%
Spain national football team	8.00	158	19.8	65.8%
Toy Story 3	12.47	146	11.7	51.4%
Australia's Next Top Model, Cycle 6	14.21	140	9.9	65.7%
Juan Pablo Pino	14.00	132	9.4	17.4%
Wayne Rooney	8.05	70	8.7	12.9%
SummerSlam (2010)	14.33	114	8.0	27.2%
The Twilight Saga: Eclipse	30.00	224	7.5	42.0%
Jason Leopold	7.63	56	7.3	17.9%
Manchester City F.C.	10.81	77	7.1	26.0%

It's interesting to note that almost all of these articles are about current events. Toy Story 3 and The Twilight Saga: Eclipse were released around the time of the trial. Total Drama World Tour and Australia's Next Top Model, Cycle 6 are TV shows that were active during the time of the trial. The World Cup gives rise to Spain national football team, Juan Pablo Pino, Wayne Rooney, and Manchester City F.C.. And SummerSlam (2010) is a WWE Wrestling event that took place on Aug. 15, 2010.

Here are examples of articles that received between 1-2 edits per day:

class="wikitable sort" | align="center" style="background:#f0f0f0;"|Page title | align="center" style="background:#f0f0f0;"|Time under Pending (days) | align="center" style="background:#f0f0f0;"|Total anon edits while under Pending | align="center" style="background:#f0f0f0;"| Anon edits/day | align="center" style="background:#f0f0f0;"|% nonreverted/total anon edits
Thomas Edison	7.16	12	1.7	0.0%
Club Penguin	42.20	70	1.7	14.3%
The Simpsons	33.35	55	1.6	12.7%
Sum 41	44.67	73	1.6	49.3%
2010 Central Canada earthquake	42.30	69	1.6	34.8%
Nicolas Mahut	41.54	67	1.6	67.2%
Republic of Macedonia	27.28	44	1.6	15.9%
Oksana Grigorieva	22.40	36	1.6	50.0%
Baseball	44.28	71	1.6	4.2%
World War II	10.64	17	1.6	0.0%

And the following are examples of articles that received no edits during the time they were under pending changes:

class = "wikitable sort" | align="center" style="background:#f0f0f0;"|Page title | align="center" style="background:#f0f0f0;"|Time under Pending (days) | align="center" style="background:#f0f0f0;"|Total anon edits while under Pending | align="center" style="background:#f0f0f0;"| Anon edits/day | align="center" style="background:#f0f0f0;"|% nonreverted/total anon edits
Sonny Fai	45.29	0	-
Richard Goldberg	45.29	0	-
Wildcliff	45.29	0	-
Steven Geller	45.29	0	-
Robert Garside	45.28	0	-
Tran Van Huong	45.28	0	-
William B. Black (Illinois politician)	45.28	0	-
Suryoyo Sat	45.23	0	-
Tig Notaro	45.23	0	-
Steve Hoffman	45.23	0	-

If we take the articles that have at least one anonymous edit per day while under pending changes (237 total articles), we can then look at the quality of the edits. Articles which have at least 50% of anonymous edits unreverted:

class="wikitable" | align="center" style="background:#f0f0f0;"|Page title | align="center" style="background:#f0f0f0;"|Time under Pending (days) | align="center" style="background:#f0f0f0;"|Total anon edits while under Pending | align="center" style="background:#f0f0f0;"| Anon edits/day | align="center" style="background:#f0f0f0;"|% nonreverted/total anon edits
Alvin Greene	17.00	25	1.5	88.0%
Uruguay national football team	33.14	111	3.3	82.0%
Total Drama World Tour	20.77	493	23.7	71.8%
France	34.43	150	4.4	68.0%
Nicolas Mahut	41.54	67	1.6	67.2%
Sonisphere Festival	46.39	66	1.4	66.7%
Angel Locsin	17.65	18	1.0	66.7%
Spain national football team	8.00	158	19.8	65.8%
Australia's Next Top Model, Cycle 6	14.21	140	9.9	65.7%
John Isner	41.54	89	2.1	65.2%
Australia	44.28	90	2.0	64.4%
The Hunchback of Notre Dame (1996 film)	45.23	51	1.1	62.7%
Gary Glitter	43.30	45	1.0	62.2%
Madonna (entertainer)	42.23	118	2.8	60.2%
This Is It (Michael Jackson concerts)	45.44	62	1.4	58.1%
Big Brother 2010 (UK)	14.00	56	4.0	55.4%
2010–11 Arsenal F.C. season	28.61	34	1.2	52.9%
2010 Formula One season	25.29	55	2.2	52.7%
Toy Story 3	12.47	146	11.7	51.4%
Oksana Grigorieva	22.40	36	1.6	50.0%
List of programs broadcast by Cartoon Network	21.39	30	1.4	50.0%

On the opposite end of the spectrum, the following articles that have at least one anonymous edit per day, none of which were unreverted:

class="wikitable sort" | align="center" style="background:#f0f0f0;"|Page title | align="center" style="background:#f0f0f0;"|Time under Pending (days) | align="center" style="background:#f0f0f0;"|Total anon edits while under Pending | align="center" style="background:#f0f0f0;"| Anon edits/day | align="center" style="background:#f0f0f0;"|% nonreverted/total anon edits
List of serial killers by country	9.20	32	3.5	0.0%
Osama bin Laden	9.16	31	3.4	0.0%
David Henrie	7.26	19	2.6	0.0%
Batman	11.21	25	2.2	0.0%
Chicken	32.90	64	1.9	0.0%
Emile Heskey	15.32	29	1.9	0.0%
Robert Kotick	8.58	15	1.7	0.0%
Brad Pitt	10.92	19	1.7	0.0%
Thomas Edison	7.16	12	1.7	0.0%
World War II	10.64	17	1.6	0.0%
Westboro Baptist Church	34.14	49	1.4	0.0%
Dinosaur	43.55	60	1.4	0.0%
Akshay Kumar	12.33	16	1.3	0.0%
Apple	34.38	42	1.2	0.0%
Danielle Staub	18.91	23	1.2	0.0%
South Slavic languages	16.54	20	1.2	0.0%
Christopher Columbus	29.12	34	1.2	0.0%
Penguin	11.48	13	1.1	0.0%
Victoria of the United Kingdom	13.58	15	1.1	0.0%
Turtle	19.36	21	1.1	0.0%
Moon	20.08	21	1.0	0.0%
Amy Winehouse	11.48	12	1.0	0.0%
Cheetah	11.62	12	1.0	0.0%
Obesity	45.79	46	1.0	0.0%

Another way of cutting the data is by anonymous edit count (rather than average anonymous edits per day). The following is a histogram based on the total number of edits during the trial:

class="wikitable sort" | align="center" style="background:#f0f0f0;"|Total Anon Edits | align="center" style="background:#f0f0f0;"|# of Articles | align="center" style="background:#f0f0f0;"|# of Total
>200	3	0.2%
100-199	32	2.1%
50-99	108	7.2%
25-49	190	13%
10-24	302	20%
5-9	192	13%
1-4	383	26%
0	285	19%
Total	1495	100%

Of articles that received at least 50 edits during the period, the following articles had over 50% of the anonymous edits unreverted:

class="wikitable sort" | align="center" style="background:#f0f0f0;"|Page title | align="center" style="background:#f0f0f0;"|Time under Pending (days) | align="center" style="background:#f0f0f0;"|Total anon edits while under Pending | align="center" style="background:#f0f0f0;"| Anon edits/day | align="center" style="background:#f0f0f0;"|% nonreverted/total anon edits
Uruguay national football team	33.14	111	3.3	82.0%
Total Drama World Tour	20.77	493	23.7	71.8%
France	34.43	150	4.4	68.0%
Nicolas Mahut	41.54	67	1.6	67.2%
Sonisphere Festival	46.39	66	1.4	66.7%
Spain national football team	8.00	158	19.8	65.8%
Australia's Next Top Model, Cycle 6	14.21	140	9.9	65.7%
John Isner	41.54	89	2.1	65.2%
Australia	44.28	90	2.0	64.4%
The Hunchback of Notre Dame (1996 film)	45.23	51	1.1	62.7%
Madonna (entertainer)	42.23	118	2.8	60.2%
This Is It (Michael Jackson concerts)	45.44	62	1.4	58.1%
Big Brother 2010 (UK)	14.00	56	4.0	55.4%
2010 Formula One season	25.29	55	2.2	52.7%
Toy Story 3	12.47	146	11.7	51.4%

And of the articles that received at least 50 edits during the trial, the 20 articles with the lowest percentage of unreverted anonymous edits:

class="wikitable sort" | align="center" style="background:#f0f0f0;"|Page title | align="center" style="background:#f0f0f0;"|Time under Pending (days) | align="center" style="background:#f0f0f0;"|Total anon edits while under Pending | align="center" style="background:#f0f0f0;"| Anon edits/day | align="center" style="background:#f0f0f0;"|% nonreverted/total anon edits
Will Smith	18.81	58	3.1	6.9%
Frank Lampard	46.46	58	1.2	6.9%
Star Wars	46.27	59	1.3	6.8%
Cat	46.43	153	3.3	6.5%
John Terry	46.44	51	1.1	5.9%
Ranbir Kapoor	49.86	53	1.1	5.7%
The Beatles	44.26	129	2.9	4.7%
Miloš Krasić	46.46	67	1.4	4.5%
Baseball	44.28	71	1.6	4.2%
World of Warcraft	39.65	98	2.5	4.1%
Jessica Alba	43.32	50	1.2	4.0%
Down syndrome	43.55	53	1.2	3.8%
Pig	43.33	58	1.3	3.4%
RuneScape	37.03	121	3.3	3.3%
Salman Khan	46.40	93	2.0	2.2%
Atheism	39.64	51	1.3	2.0%
Wonders of the World	45.78	51	1.1	2.0%
William Shakespeare	22.89	64	2.8	1.6%
Chicken	32.90	64	1.9	0.0%
Dinosaur	43.55	60	1.4	0.0%

Hopefully these cuts of data will help focus the discussion of Pending Changes. Please feel free to add additional commentary. Howief (talk) 23:52, 20 August 2010 (UTC)

I believe that the main goal of Pending Changes is that destructive edits of IPs are not going live. So that the frustrated IPs will do less vandalism. I do not understand why an IP does vandalism, if it does not get visible after all. The articles on the table were current events, the IP behaviour changes too much over time :( Of course, IPs could have tried to falsify the results of the Pending Changes trial to dump the feature :( --Chris.urs-o (talk) 09:57, 13 September 2010 (UTC)

class="wikitable sort" | align="center" style="background:#f0f0f0;"|Page title | align="center" style="background:#f0f0f0;"|Pending Changes after | align="center" style="background:#f0f0f0;"|Total IP edits before Pending | align="center" style="background:#f0f0f0;"| Rev of these IP edits | align="center" style="background:#f0f0f0;"|nonreverted/total IP edits | align="center" style="background:#f0f0f0;"|Pending Changes: nonreverted/total IP edits
Spain national football team	"21:16, 11 July 2010"	143 (10 days)	15	89.5%	65.8%
Toy Story 3	14:49, 23 July 2010	156 (10 days)	30	80.8%	51.4%
Australia's Next Top Model, Cycle 6	21:01, 21 July 2010	200 (10 days)	68	66.0%	65.7%
SummerSlam (2010)	18:11, 21 July 2010	167 (10 days)	45	73.1%	27.2%
The Twilight Saga: Eclipse	21:24, 25 April 2010	32 (10 days)	17	46.9%	42.0%
Jason Leopold	10:57, 28 July 2010	7 (10 days)	2	71.4%	17.9%

• 2010; nonreverted/total IP edits
• Apple:
• Pending Changes trial after 16:47, 1 July 2010
• 0/0 (Jan), 0/0 (Feb), 0/0 (March), 0/0 (April), 0/0 (May), 0/0 (June)
• Down syndrome:
• Pending Changes trial after 12:50, 22 June 2010
• 0/0 (Jan), 0/0 (Feb), 0/0 (March), 0/0 (April), 0/0 (May), 0/5 (June)
• The Hunchback of Notre Dame (1996 film):
• Pending Changes trial after 20:32, 20 June 2010
• 0/0 (Jan), 0/0 (Feb), 0/0 (March), 0/0 (April), 0/0 (May), 8/13 (June)
• --Chris.urs-o (talk) 18:13, 13 September 2010 (UTC)