User:Monkbot/task 18: cosmetic cs1 template cleanup#convert language names to codes

It is to be expected that some significant portion of cs1|2 transclusions will hold some number of superfluous parameters. One purpose of this task to remove those superfluous parameters. Empty parameters in cs1|2 templates occur for various reasons: the parameter is no longer required or not required in 'this' citation; a skeleton template was copied from a template documentation page and only partially filled in; deprecated error messages were 'fixed' by removing the parameter's value; the template was reworked; the template was placed by automated or semiautomated tools that inserted empty parameters; there are, no doubt, other explanations. For cs1|2, empty parameters serve no purpose.

Editors commonly complain that inline references make it more difficult to read wikitext. Empty parameters in cs1|2 templates occupy space for no meaningful purpose and, as a consequence, contribute to the wikitext readability problem.

The table lists all of the cs1|2 templates. The second column states the approximate number of mainspace articles using the template where at least one parameter was empty. The counts were last updated on the date listed at the bottom of the table.

cs1|2 does not support positional (unnamed) parameters. A positional parameter is one that does not contain an assignment operator (=). Positional parameters that contain nothing or only whitespace, have two forms:

:||

:|}}

Task 18 deletes the first pipe and any whitespace that exists between it and the succeeding pipe or brace.

Task 18 ignores positional parameters that have content other than whitespace because cs1|2 emits a reader-facing text ignored error message for those sorts of positional parameters.

To cs1|2, parameters that are hidden with html comment markup ({{tag|!--|content={{var|content}}}}) can appear to be empty positional parameters because MediaWiki removes html comments and their content before handing the template to Module:Citation/CS1. cs1|2 templates entirely within html markup are ignored.

Empty parameters that have this form are deleted along with the html comment markup:

:|<!-- param= -->

Similarly, empty parameters inside html comment markup are deleted; parameters with assigned values are retained:

:<!-- |param={{var|value}} |param= --> → <!-- |param={{var|value}} -->

To cs1|2, parameters that are hidden with html comment markup ({{tag|!--|content={{var|content}}}}) can appear to be empty positional parameters because MediaWiki removes html comments and their content before handing the template to Module:Citation/CS1. cs1|2 templates entirely within html markup are ignored.

Parameters that have this form are repaired by moving the preceding pipe into the html comment markup:

:|<!-- param={{var|value}} --> → <!-- |param={{var|value}} -->

A similar configuration is commonly found. This form may or may not look like an empty positional parameter to cs1|2 depending on the surrounding wikimarkup. This form is repaired for the purposes of consistency:

:|<!-- param={{var|value}} |--> → <!-- |param={{var|value}} --> |

Comments that do not have an assignment operator (=) but which occupy all of the space between a pair of pipes or a pipe and the template's closing brace are repaired by moving the leading pipe inside the html comment:

:|<!-- plain text html comment --> | → <!-- |plain text html comment --> |

There are several parameters and specific parameter/value pairs that, under certain conditions, to not contribute the rendering of the template. Task 18 deletes these:

• {{anchor|deadurl}}{{para|deadurl}} – is routinely added (with the assigned value y) to cs1|2 templates created by the unmaintained tool ReFill. Because {{para|deadurl}} is no-longer supported by cs1|2, when it is present in cs1|2 templates, cs1|2 emits the error message: {{error-small|Unknown parameter {{para|deadurl|plain=yes}} ignored ({{para|url-status|plain=yes}} suggested)}} and adds the article to {{cl|Pages with citations using unsupported parameters}} which gnomes keep mostly empty. When task 18 encounters {{para|deadurl|y}} (or the similar {{para|deadurl|yes}} and {{para|deadurl|true}} and their hyphenated forms), the parameter shall be deleted. Replacement is not necessary because {{para|deadurl|y}} (when it was supported) was redundant to the cs1|2 default state. See {{para|url-status}} documentation.
• {{anchor|mode}}{{para|mode}} – this parameter accepts one of two keywords: cs1 or cs2. The purpose is to direct the template to render in the specified style. For example, {{tlx|citation}} is a native cs2 template. {{tnull|citation}} will render as if it were a cs1 template when given {{para|mode|cs1}}; similarly, cs1 templates render in cs2 style when given {{para|mode|cs2}}. Setting {{para|mode}} to the template's native style does not change the rendering. Task 18 deletes {{para|mode}} parameters when they match the template's native style. See {{para|mode}} documentation.
• {{anchor|nlsytle}}{{para|name-list-style}} – controls how name lists are rendered. When assigned one of the values amp, ampersand, and, &, or serial, cs1|2 inserts an ampersand or the word 'and' between the last two names of a name list. When none of the name lists have two or more names, the parameter does not change the rendering. When assigned the value vanc, name lists are rendered in Vancouver style. {{para|name-list-style|vanc}} requires {{para|last}} and {{para|first}} (or aliases); without {{para|first}} aliases, the parameter does not change the rendering. See {{para|name-list-style}} documentation.
• {{anchor|nllink}}{{para|<{{var|name-list}}>-link}}, {{anchor|nlmask}}{{para|<{{var|name-list}}>-mask}}{{anchor|fn_a_blink}}^[a] – these wikilink or mask a <{{var|name}}> in an associated <{{var|name-list}}>. When there is no associated {{para|<{{var|name}}>}} (for example, {{para|author-mask14|2}} but no {{para|author14}}), these parameters do not change the rendering.
• {{anchor|nopp}}{{para|no-pp}} – suppresses p. and pp. page annotation when either of {{para|page}} or {{para|pages}} has a value; when there is no {{para|page}} or {{para|pages}} parameter, this parameter does not change the rendering. When found in {{tlx|citation}} templates where {{para|journal}} has an assigned value or when found in {{tlx|cite journal}}, {{para|no-pp}} has no value because pagination in journal citations does not use the P. and pp. annotation.
• {{anchor|origyear}}{{para|orig-year}} – a free-form parameter that accompanies {{para|year}}, {{para|date}} or when neither of those are present, {{para|publication-date}}. When none of those are present, {{para|orig-year}} is not rendered.
• {{anchor|postscript}}{{para|postscript}} – specifies the rendered template's terminal punctuation. Default terminal punctuation for cs1 is a dot; for cs2, terminal punctuation is omitted. {{para|postscript|.}} in cs1 templates and {{para|postscript|none}} in cs2 templates does not change the rendering. The form {{para|postscript|<!--none-->}} is ignored by cs1|2 templates because the html comments make {{para|postscript}} an empty parameter.
• {{anchor|refharv}}{{para|ref|harv}} – At the 2020-04-18 update to Module:Citation/CS1 ({{diff|Module:Citation/CS1|prev|951705319|this edit}}), {{para|ref|harv}} was internally defined as the default state so the explicit {{para|ref|harv}} does not change the template's function. Use of this parameter/value pair is tracked in {{cl|1=CS1 maint: ref=harv‎}}. See {{para|ref}} documentation.
• {{anchor|urlstatus}}{{para|url-status}} – like its predecessors {{para|dead-url}} and {{para|deadurl}}, {{para|url-status}} is a display control parameter that determines how a cs1|2 template will be rendered when that template has {{para|archive-url}} and {{para|archive-date}}. When {{para|archive-url}} and {{para|archive-date}} are empty or omitted, {{para|url-status}} is non-functional. See {{para|url-status}} documentation.

Because cs1|2 is an amalgam of several individually developed templates, it acquired a variety of parameter-name styles: {{para|lowercase}}, {{para|Capitalized}}, {{para|camelCase}}, {{para|underscore_separated}}, {{para|space separated}}, {{para|hyphen-separated}}, {{para|allruntogether}}. The {{para|Capitalized}}, {{para|camelCase}}, {{para|underscore_separated}}, and {{para|space separated}} parameter name styles have all been deprecated and support for these styles withdrawn in favor of the {{para|lowercase}} and {{para|hyphen-separated}} forms as a result of this RfC. For parameter names that are multiword, cs1|2 is gradually shifting to prefer the hyphenated form. The table lists the all-run-together form with the approximate number of articles that transclude cs1|2 templates using these parameters. Task 18 replaces the all-run-together forms of these parameters with the hyphenated forms.

The {{para|accessdate}} and {{para|authorlink}} parameter names have been included in awb's genfixes since {{diff|Wikipedia:AutoWikiBrowser/Rename_template_parameters|984829895|983479364|this edit}}. There has been some pushback against using genfixes to normalize cs1|2 parameter name style. See {{slink|Wikipedia_talk:AutoWikiBrowser/Archive_32#Citation_parameter_renaming}}.

There are commonly used tools that continue to insert the all-run-together names. An interface protected edit request has been submitted to normalize all-run-together parameter names emitted by WP:RefToolbar. See {{slink|MediaWiki_talk:RefToolbarConfig.js#normalize_parameter_names}}.

As a courtesy to editors at other-language wikis, it is desirable to use the MediaWiki language codes instead of English-language language names in {{para|language}} so that cs1|2 can render language names using the other wiki's language without the need for an editor to make a translation. Task 18 will replace language names with the associated MediaWiki code. See the {{para|language}} documentation and the list of (mostly) English-language language names and associated codes.

For each edit made, task 18 creates an edit summary detailing what was done. For example, this edit summary from a test (no save) of Ida B. Wells:

:Task 18 (cosmetic) (dev test): eval 146 templates: del empty params (1251×); hyphenate params (4×); del pos params (1×); del |ref=harv (4×);

Each summary always begins with the 'Task 18 (cosmetic) (dev test): eval {{var|n}} templates:' leader where {{var|n}} is the total number of cs1|2 templates and recognized redirects that task 18 evaluated. The remaining portion of the edit summary is assembled from one or more of these:

:del cmtd params ({{var|n}}×); – deleted commented parameters

:rep cmtd params ({{var|n}}×); – repaired commented parameters

:del empty params ({{var|n}}×); – deleted empty parameters

:hyphenate params ({{var|n}}×); – hyphenated parameter names

:del empty pos params ({{var|n}}×); – deleted empty positional parameters

:del |ref=harv ({{var|n}}×); – deleted {{pipe}}ref=harv

:del |mode= ({{var|n}}×); – deleted {{pipe}}mode=

:del |postscript= ({{var|n}}×); – deleted {{pipe}}postscript=

:del |url-status= ({{var|n}}×); – deleted {{pipe}}url-status=

:del |deadurl= ({{var|n}}×); – deleted {{pipe}}deadurl=

:del |no-pp= ({{var|n}}×); – deleted {{pipe}}no-pp=

:del |orig-year= ({{var|n}}×); – deleted {{pipe}}orig-year

:del |name-list-style= ({{var|n}}×); – deleted {{pipe}}name-list-style=

:del |<name-list>-mask= ({{var|n}}×); – deleted {{pipe}}<name-list>-mask=

:del |<name-list>-link= ({{var|n}}×); – deleted {{pipe}}<name-list>-link=

:cvt lang vals ({{var|n}}×); – converted language values

:skip ({{var|n}}×); – during development of task 18, it found malformed cs1|2 templates that contain the opening

AWB truncates long edit summaries (:phab:T199347) so the messaging is necessarily terse in an attempt to list everything accomplished by task 18 in the summary. When that is not possible, task 18 truncates its summary and adds an ellipsis.

When an article transcludes a named reference that exactly matches a named reference in the article, and task 18 edits the article's named reference, MediaWiki will emit this error message:

:The named reference "$1" was defined multiple times with different content (help page).

Task 18 cannot know that a cs1|2 template that it edits is mirrored in a transcluded template. If you know of articles where this is a problem, give a list of those articles to the bot's operator so that the bot can be instructed to skip those articles.

:a. {{anchor|fn_a}}^{^} The {{para|<{{var|name-list}}>-link}} and {{para|<{{var|name-list}}>-mask}} parameters are:

// This script does cosmetic work:

// repairs / removes commented parameters

// deletes empty parameters

// replaces un-hyphenated parameters with their hyphenated forms

// deletes |ref=harv

// replaces language names in |language= with their MediaWiki ISO 639-like codes

//

// wikitext searches:

// html comments:

// hastemplate:"Module:Citation/CS1" insource:/\{ *[Cc]it[ae][^\}]*\| *\]+\-\-+\>/

// mode:

// hastemplate:Citation insource:/\{ *[Cc]itation[^\}]*\| *mode *= *cs2/

// hastemplate:"Module:Citation/CS1" insource:/\{ *[Cc]ite[^\}]*\| *mode *= *cs1/

// deadurl:

// hastemplate:"Module:Citation/CS1" insource:"deadurl"

// postscript:

// hastemplate:Citation insource:/\{ *[Cc]itation[^\}]*\| *postscript *= *none/

// hastemplate:"Module:Citation/CS1" insource:/\{ *[Cc]ite[^\}]*\| *postscript *= *\./

// hastemplate:"Module:Citation/CS1" insource:/\{ *[Cc]it[ae][^\}]*\| *postscript *= *\

//

// category:

// CS1 errors: empty unknown parameters‎

//

public string ProcessArticle(string ArticleText, string ArticleTitle, int wikiNamespace, out string Summary, out bool Skip)

{

if (0 != wikiNamespace ) // if is not mainspace

{

if (-1 == ArticleTitle.IndexOf ("User:Trappist the monk")) // don't skip pages in my own user space

{

Skip = true;

Summary = @"article not in mainspace: " + wikiNamespace;

return ArticleText; // abandon this edit

}

}

if (skip_map.ContainsKey (ArticleTitle)) // if is in the skip list

{

Skip = true;

Summary = @"article in skip list";

return ArticleText; // abandon this edit

}

Skip = false; // assume that something will be changed

// Summary = "Task 18 (cosmetic) (dev test):"; // uses redirect to User:Monkbot/task 18: cosmetic cs1 template cleanup

// Summary = "Task 18 (cosmetic) (BRFA trial):";

Summary = "Task 18 (cosmetic):";

int total_count = 0; // total number of cs1|2 templates

int skip_count = 0; // skip count incremented when cite template holds wikitable markup

int comment_deleted = 0;

int comment_repaired = 0;

int empty_positional_deleted = 0;

int params_removed_count = 0;

int params_hyphenated_count = 0;

int ref_harv_count = 0;

int url_status_count =0;

int deadurl_count = 0;

int nopp_count = 0;

int orig_count = 0;

int mode_count = 0;

int ps_count = 0;

int nls_count = 0;

int mask_count = 0;

int link_count = 0;

int language_count = 0;

//---------------------------< B E G I N >--------------------------------------------------------------------

ArticleText = hide (ArticleText, IS_CS1); // hide all templates except cs1|2 and hide wikilinks

//---------------------------< M A I N >----------------------------------------------------------------------

//

// this is main

//

if (Regex.Match (ArticleText, template_pattern).Success)

ArticleText = Regex.Replace (ArticleText, template_pattern,

delegate(Match match)

{

string template = match.Groups[0].Value; // this will be returned if no changes

if ((-1 != template.IndexOf("__0T4BL3__")) || (-1 != template.IndexOf("__CT4BL3__"))) // if table markup found in the cite template abandon

{

skip_count++;

return template;

}

total_count++; // bump total number of cs1|2 templates tally

template = counted_replace (template, comment_params_1_pattern, "$2$1$3$5$4", ref comment_repaired); // | to |

template = counted_replace (template, comment_params_2_pattern, "$2$1$3", ref comment_repaired); // | to

template = counted_replace (template, comment_params_3_pattern, "", ref params_removed_count); // | to (delete)

template = counted_replace (template, comment_params_4_pattern, "$1", ref params_removed_count); // to

template = counted_replace (template, comment_params_5_pattern, "$2$1$3", ref comment_repaired); // | to

template = counted_replace (template, empty_positional_parameter_nl, "$1", ref empty_positional_deleted); // || or |} to | or } (newline variant)

template = counted_replace (template, empty_positional_parameter, "$1", ref empty_positional_deleted); // || or |} to | or }

template = counted_replace (template, ref_harv_pattern, "", ref ref_harv_count); // |ref=harv to (delete)

template = mode_remove (template, ref mode_count);

template = postscript_remove (template, ref ps_count);

template = empty_param_remove (template, ref params_removed_count); // this is a prerequisite function for these:

template = url_status_remove (template, ref url_status_count, ref deadurl_count);

template = hyphenate (hyphenate_map, template, ref params_hyphenated_count); // this is a prerequisite function for these:

template = nopp_remove (template, ref nopp_count);

template = orig_year_remove (template, ref orig_count);

template = name_list_style_remove (template, ref nls_count);

template = name_link_mask_remove (template, ref mask_count, @"mask");

template = name_link_mask_remove (template, ref link_count, @"link");

template = language (language_map, template, ref language_count);

while (Regex.Match (template, empty_comment_pattern).Success) // cleanup; delete (empty or only whitespace)

template = Regex.Replace (template, empty_comment_pattern, "");

return template;

});

//---------------------------< F I N I S H >------------------------------------------------------------------

ArticleText = unhide (ArticleText); // unhide all that is hidden

if (0 != total_count) // build our edit summary

{

Summary = summary_concat (Summary, " eval " + total_count + " template" + (1 == total_count ? ":" : "s:"));

if (0 != comment_deleted)

Summary = summary_concat (Summary, " del cmtd params (" + comment_deleted + "×);");

if (0 != comment_repaired)

Summary = summary_concat (Summary, " rep cmtd params (" + comment_repaired + "×);");

if (0 != params_removed_count)

Summary = summary_concat (Summary, " del empty params (" + params_removed_count + "×);");

if (0 != params_hyphenated_count)

Summary = summary_concat (Summary, " hyphenate params (" + params_hyphenated_count + "×);");

if (0 != empty_positional_deleted)

Summary = summary_concat (Summary, " del pos params (" + empty_positional_deleted + "×);");

if (0 != ref_harv_count)

Summary = summary_concat (Summary, " del |ref=harv (" + ref_harv_count + "×);");

if (0 != mode_count)

Summary = summary_concat (Summary, " del |mode= (" + mode_count + "×);");

if (0 != ps_count)

Summary = summary_concat (Summary, " del |postscript= (" + ps_count + "×);");

if (0 != url_status_count)

Summary = summary_concat (Summary, " del |url-status= (" + url_status_count + "×);");

if (0 != deadurl_count)

Summary = summary_concat (Summary, " del |deadurl= (" + deadurl_count + "×);");

if (0 != nopp_count)

Summary = summary_concat (Summary, " del |no-pp= (" + nopp_count + "×);");

if (0 != orig_count)

Summary = summary_concat (Summary, " del |orig-year= (" + orig_count + "×);");

if (0 != nls_count)

Summary = summary_concat (Summary, " del |name-list-style= (" + nls_count + "×);");

if (0 != mask_count)

Summary = summary_concat (Summary, " del |-mask= (" + mask_count + "×);");

if (0 != link_count)

Summary = summary_concat (Summary, " del |-link= (" + link_count + "×);");

if (0 != language_count)

Summary = summary_concat (Summary, " cvt lang vals (" + language_count + "×);");

if (0 != skip_count)

Summary = summary_concat (Summary, " skip (" + skip_count + "×);");

}

else

{

Skip = true;

Summary = Summary + " no templates to evaluate;";

}

System.IO.StreamWriter sw;

string log_file = @"Z:\Wikipedia\AWB\Monkbot_tasks\Task18_" + DateTimeOffset.Now.ToString("u").Substring (0, 10) + @".csv";

if (!System.IO.File.Exists (log_file))

{ // for csv import, pipe is string delimiter here because not legal wp title char

sw = System.IO.File.AppendText (log_file);

sw.WriteLine ("|Article|, |evaluated|, |deleted comment|, |repaired comment|, |empty params|, |hyphenated|, |empty pos params|, |ref=harv|, |mode|, |postscript|, "

+ "|url-status|, |deadurl|, |no-pp|, |orig-year|, |name-list-style|, |-mask|, |-link|, |language|, |skipped|");

sw.Close();

}

if (Skip)

{

sw = System.IO.File.AppendText (log_file);

sw.WriteLine ("|" + ArticleTitle + "|," + total_count);

sw.Close();

}

else

{

sw = System.IO.File.AppendText (log_file);

sw.WriteLine ("|" + ArticleTitle + "|," + total_count + "," + comment_deleted + "," + comment_repaired + "," + params_removed_count + "," + params_hyphenated_count

+ "," + empty_positional_deleted + "," + ref_harv_count + "," + mode_count + "," + ps_count + "," + url_status_count + "," + deadurl_count

+ "," + nopp_count + "," + orig_count + "," + nls_count + "," + mask_count + "," + link_count + "," + language_count + "," + skip_count);

sw.Close();

}

return ArticleText;

}

//===========================<< S U P P O R T >>==============================================================

//---------------------------< H I D E >----------------------------------------------------------------------

//

// HIDE TEMPLATES: find templates that are not ; replace the opening {{ with __0P3N__, the closing }}

// with __CL0S3__, and internal | (pipes) with __P1P3__

//

// single curly braces in urls and other parameter values can confuse other regex in this code so replace {

// with __0CU!21Y__ and } with __CCU!21Y__

//

private string hide (string ArticleText, string dont_hide)

{

string pattern = @"\{\{(?!\s*" + dont_hide + @")[^\{\}]*\}\}";

if (Regex.Match (ArticleText, pattern).Success)

{

ArticleText = Regex.Replace(ArticleText, pattern,

delegate(Match match)

{

string fixed_template; // a hidden template is assembled here

string raw_template = match.Groups[0].Value; // the whole template

pattern = @"\{\{"; // hide the opening {{

fixed_template = Regex.Replace (raw_template, pattern, "__0P3N__");

pattern = @"\}\}"; // hide the closing }}

fixed_template = Regex.Replace (fixed_template, pattern, "__CL0S3__");

pattern = @"\|"; // and hide the pipes

fixed_template = Regex.Replace (fixed_template, pattern, "__P1P3__");

return fixed_template;

});

}

pattern = @"(\\|\}]*)\{\{(\s*" + dont_hide + @"[^\}]*)\}\}([^\>]*\-{2,}\>)"; //

ArticleText = Regex.Replace(ArticleText, pattern, "$1__0P3N__$2__CL0S3__$3");

pattern = @"\{\|"; // open table markup

ArticleText = Regex.Replace(ArticleText, pattern, "__0T4BL3__");

pattern = @"\|\}(?!\})"; // open table markup

ArticleText = Regex.Replace(ArticleText, pattern, "__CT4BL3__");

pattern = @"([^\{])\{([^\{])"; // single opening curly brace

ArticleText = Regex.Replace(ArticleText, pattern, "$1__0CU!21Y__$2");

pattern = @"([^\}])\}([^\}])"; // single closing curly brace

ArticleText = Regex.Replace(ArticleText, pattern, "$1__CCU!21Y__$2");

pattern = @"\[\[(?![Ff]ile|[Ii]mage)([^\|\]]+)\|([^\]]+)\]\]"; // HIDE complex wikilinks: label to __WL1NK_O__article title__P1P3__label__WL1NK_C__

ArticleText = Regex.Replace(ArticleText, pattern, "__WL1NK_O__$1__P1P3__$2__WL1NK_C__"); // [[File: with wikilinks inside can be confusing

pattern = @"\[\[([^\]]+)\]\]"; // HIDE simple wikilinks: article title to __WL1NK_O__article title__WL1NK_C__

ArticleText = Regex.Replace(ArticleText, pattern, "__WL1NK_O__$1__WL1NK_C__");

return ArticleText;

}

//---------------------------< U N H I D E >------------------------------------------------------------------

//

// UNHIDE TEMPLATES: find templates and wikilinks that are hidden; replace the 'hide' keywords with the

// appropriate wiki markup

//

private string unhide (string ArticleText)

{

ArticleText = Regex.Replace(ArticleText, @"__WL1NK_O__", "[["); // UNHIDE: replace __WL1NK_O__ with [[

ArticleText = Regex.Replace(ArticleText, @"__WL1NK_C__", "]]"); // UNHIDE: replace __WL1NK_C__ with ]]

ArticleText = Regex.Replace(ArticleText, @"__P1P3__", "|"); // UNHIDE: replace __P1P3__ with |

ArticleText = Regex.Replace(ArticleText, @"__0T4BL3__", "

ArticleText = Regex.Replace(ArticleText, @"__0CU!21Y__", "{"); // UNHIDE: replace __0CU!21Y__ with {

ArticleText = Regex.Replace(ArticleText, @"__CCU!21Y__", "}"); // UNHIDE: replace __CCU!21Y__ with }

ArticleText = Regex.Replace(ArticleText, @"__0P3N__", "{{"); // UNHIDE: replace __0P3N__ with {{

ArticleText = Regex.Replace(ArticleText, @"__CL0S3__", "}}"); // UNHIDE: replace __CL0S3__ with }}

return ArticleText;

}

//---------------------------< S U M M A R Y _ C O N C A T >--------------------------------------------------

//

// concatenates text onto an existing edit summary string, limiting the string to a length of no more than 247

// characters. When