Wikipedia:AutoWikiBrowser/Regular expression#Using look ahead/behind

{{Short description|Manual for using regular expressions with AutoWikiBrowser}}

{{hatnote|"WP:REGEX" redirects here. For Lua patterns, see Help:Lua for beginners and mw:Patterns. For search queries, see {{section link|Help:Searching|insource:}} and Help:Searching/Regex.}}

{{AWB}}

{{AWB mheader|Regular expressions}}

{{shortcut|WP:REGEX|WP:AWBREGEX}}

{{Clear}}

A regular expression or regex is a sequence of characters that define a pattern to be searched for in a text. Each occurrence of the pattern may then be automatically replaced with another string, which may include parts of the identified pattern. AutoWikiBrowser uses the .NET flavor of regex.{{Cite web |last=adegeo |date=18 June 2022 |title=Regular Expression Language - Quick Reference |url=https://learn.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-language-quick-reference |url-status=live |archive-url=https://archive.is/UfpRD|archive-date=2023-02-05 |access-date=2023-02-05 |website=learn.microsoft.com |language=en-us}}

Syntax

=Anchors=

Used to anchor the search pattern to certain points in the searched text.

class="wikitable" width=100%
id="anchors" style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Syntax

! style="background:#CFC; font-style:italic;" | Comments

{{code|^}}

| Start of string

| Before all other characters on page (or line if multiline option is active)
(Note that "^" has a different meaning inside a token.)

{{code|\A}}

| Start of string

| Before all other characters on page

{{code|$}}

| End of string

| After all other characters on page (or line if multiline option is active)

{{code|\Z}}

| End of string

| After all other characters on page

{{code|\b}}

| On a word boundary

| On a letter, number or underscore character

{{code|\B}}

| Not on a word boundary

| Not on a letter, number or underscore character

=Character classes=

Expressions which match any character in a pre-defined set. This list is not exhaustive.

class="wikitable" width=100%
id="character-class" style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Character class

! style="background:#CFC; font-style:italic;" | Will match

{{code|.}}

| "wildcard"

| Any character except newline
(Newline is included if singleline option is active; see #Regex behavior options below)

{{code|\w}}

| Any "word" character (letters, digits, underscore)

| abcdefghijklmnopqstuvwxyz{{zwsp}}ABCDEFGHIJKLMNOPQRSTUVWXYZ
0123456789_

{{code|\W}}

| Any character other than "word" characters

| $?!#%*@&;:.,+-±=^"`\|/<>{}[]()~(newline)(tab)(space)

{{code|\s}}

| Any whitespace character

| (space) (tab) (literal new line) (return)

{{code|\S}}

| Any character other than white space

| abcxyz_ABCXYZ$?!#%*@&;:.,+-=^"/<{[(~0123789 (incomplete list)

{{code|\d}}

| Any digit

| 0123456789

{{code|\D}}

| Any character other than digits

| abcxyz_ABCXYZ$?!#%*@&;:.,+-=^"/<{[(~(newline)(tab)(space) (incomplete list)

{{code|\n}}

| Newline

| (newline)

{{code|\p{L} }}

| Any Unicode letter{{Cite web |title=Regex Tutorial – Unicode Characters and Properties |url=https://www.regular-expressions.info/unicode.html |url-status=live |archive-url=https://web.archive.org/web/20221219234137/https://www.regular-expressions.info/unicode.html |archive-date=19 December 2022 |access-date=3 January 2023 |website=www.regular-expressions.info}}

| AaÃãÂâĂăÄäÅå (incomplete list)

{{code|\p{Ll} }}

| Any lowercase Unicode letter

| aãâăäå (incomplete list)

{{code|\p{Lu} }}

| Any uppercase Unicode letter

| AÃÂĂÄÅ (incomplete list)

{{code|\r}}

| Carriage return

| (carriage return)

{{code|\t}}

| Tab

| (tab)

{{code|\c}}

| Control character

| Ctrl-A through Ctrl-Z (0x01–0x1A)

{{code|\x}}

| Any hexadecimal digit

| 0123456789abcdefABCDEF

{{code|\0}}

| Any octal digit

| 01234567

=Tokens=

Tokens match a single character from a specified set or range of characters.

class="wikitable" width=100%
id="tokens" style="background:#CFC; font-style:italic; border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Tokens

! style="background:#CFC; font-style:italic;" | Examples

[...]

| Set – matches any single character in the brackets

| {{code|[def]}} matches {{highlight|d}} or {{highlight|e}} or {{highlight|f}}

[^...]

| Inverse – match any single character except those in the brackets

| [^abc] – anything (including newline) except {{highlight|a}} or {{highlight|b}} or {{highlight|c}}

[...-...]

| Range – matches any single character in the specified range
(including the characters given as the endpoints of the range)

| {{code|[a-q]}} – any lowercase letter between {{highlight|a}} and {{highlight|q}}

{{code|[A-Q]}} – any uppercase letter between {{highlight|A}} and {{highlight|Q}}

{{code|[0-7]}} – any digit between {{highlight|0}} and {{highlight|7}}

=Groups=

Groups match a string of characters (including tokens) in sequence. By default, matches to groups are captured for later reference. Groups may be nested within other groups.

class="wikitable" width=100%
id="groups" style="background:#CFC; font-style:italic; border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Syntax

! style="background:#CFC; font-style:italic;" | Examples

(...)

| Capture group – matches the string in parentheses
(Output captured groups in the replacement string with {{code|$1}}, {{code|$2}}, etc.)

| (abc) matches abc

(?<name>...)Named capture group
(for use in back references or the replacement string)

| (?<year>\b\d{4}\b) matches the whole word {{highlight|2016}}

Output the named group using ${year}

(?:...)Non-capturing parentheses

| (?:abc) matches and consumes, but doesn't capture, {{highlight|abc}}

|

| Alternation/disjunction (read as "or")

| {{code|(ab{{!}}cd{{!}}ef)}} matches {{highlight|ab}} or {{highlight|cd}} or {{highlight|ef}}

{{code|(ab(cd{{!}}ef))}} matches {{highlight|abcd}} or {{highlight|abef}}

=Quantifiers=

Quantifiers specify how many of the preceding token or group may be matched.

class="wikitable" width=100%
id="quantifier" style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Syntax

! style="background:#CFC; font-style:italic;" | Examples

*

| 0 or more

| b* matches nothing, b, bb, bbb, etc.

+

| 1 or more

| b+ matches b, bb, bbb, etc.

?

| 0 or 1

| b? matches nothing, or b

{3}

| Exactly 3

| b{3} matches bbb

{3,}

| 3 or more

| b{3,} matches bbb, bbbb, etc.

{2,4}

| At least 2 and no more than 4

| b{2,4} matches bb, bbb, or bbbb

By default, quantifiers are "greedy", meaning they will match as many characters as possible while still allowing the full expression to find a match. Adding a question mark ("?") after a qualifier will make it non-greedy, meaning it will match as few characters as possible while still allowing the full expression to find a match. See #Greed and quantifiers for examples.

=Metacharacters and the escape character=

Metacharacters are characters with special meaning in regex; to match these characters literally, they must be "escaped" by being preceded with the escape character \.

class="wikitable" width=100%
id="special-character" style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Escape character

! style="background:#CFC; font-style:italic;" | Comments

\

| Escape Character

| Allows metacharacters (listed below) to be matched literally

style="background:#CFC!important; font-style:italic;" | Metacharacter

! style="background:#CFC!important; font-style:italic;" | Metacharacter escaped

! style="background:#CFC!important; font-style:italic;" |  

{{code|^}}

| {{code|\^}}

| rowspan="15" | Not in this list: =}#!/%&_:; (incomplete list)

{{code|$}}

| {{code|\$}}

{{code|(}}

| {{code|\(}}

{{code|)}}

| {{code|\)}}

<

| {{code|\<}}

.

| {{code|\.}}

*

| {{code|\*}}

+

| {{code|\+}}

?

| \?

{{code|[}}

| {{code|\[}}

{{code|]}}

| {{code|\]}}

{

| \{

{{code|\}}

| {{code|\\}}

|

| \|

>

| \>

-

| \-

| Hyphens must be escaped within tokens, where they indicate a range; outside of tokens, they do not need to be escaped.

=Back references=

Used to match a previously captured group again.

class="wikitable" width=100%
id="back-references" style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Syntax

! style="background:#CFC; font-style:italic;" | Comments

{{code|\1}}, {{code|\2}}, {{code|\3}}, etc.

| Match unnamed captured groups in order.

| (\n[^\n]+)\1 matches identical adjacent lines; {{code|$1}} will replace with a single copy.

{{nowrap|\k<name>

| Match named captured group {{nowrap|(?<name>...).}}

|

=Look-around=

Used to check what comes before or after, without consuming or capturing. ("Without consuming" means that matches for look-around assertions do not become part of the string to be replaced. In the following examples, only "abc" is consumed.) In .NET regex, all regex syntax can be used within a look-around assertion.

class="wikitable" width=100%
id="lookaround" style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Syntax

! style="background:#CFC; font-style:italic;" | Examples

(?=...)positive lookahead

| abc(?=xyz) matches {{highlight|abc}} only if it's followed by {{highlight|xyz}}.

(?!...)negative lookahead

| abc(?!xyz) matches {{highlight|abc}} except when it's followed by {{highlight|xyz}}

(?<=...)positive lookbehind

| (?<=xyz)abc matches {{highlight|abc}} only if it's preceded by {{highlight|xyz}}

(?...)negative lookbehind

| (? matches {{highlight|abc}} except when it's preceded by {{highlight|xyz}}

=Commenting=

Comments in the search string do not affect the resulting matches.

class="wikitable" width=100%
id="comments" style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Syntax

! style="background:#CFC; font-style:italic;" | Comments

(?#...)comment

| (?#Just a comment in here)

= Using captured groups in the replacement string =

Captured groups can be output as part of the replacement string.

class="wikitable" width=100%
id="replacement-string" style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Reference style

! style="background:#CFC; font-style:italic;" | Example search string

! style="background:#CFC; font-style:italic;" | Example output

$#Unnamed capture group(Sam)(Max)(Pete)$2 returns Max
${2}0 returns Max0
${...}Named capture group(?ABC)(?DEF)${foo} returns ABC

= Tokens and groups =

Tokens and groups are portions of a regular expression which can be followed by a quantifier to modify the number of consecutive matches. A token is a character, special character, character class, or range (e.g. [m-q]). A group is formed by enclosing tokens or other groups within parentheses. All of these can be modified to match a number of times by a quantifier. For example: a?, \n+, \d{4}, [m-r]*, (a?\n+\d{4}[m-r]*|not){3,7}, and ((?:97[89]-?)?(?:\d[ -]?){9}[\dXx]).

= Greed and quantifiers =

Greed, in regular expression context, describes the number of characters which will be matched (often also stated as "consumed") by a variable length portion of a regular expression{{Snd}} a token or group followed by a quantifier, which specifies a number (or range of numbers) of tokens. If the portion of the regular expression is "greedy", it will match as many characters as possible. If it is not greedy, it will match as few characters as possible.

By default, quantifiers in AWB are greedy. To make a quantifier non-greedy, it must be followed by a question mark. For example:

In this string:

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

this expression:

\[\[.*\]\]

will match Lorem ipsum dolor sit amet, consectetur adipisicing.

This expression:

\[\[.*?\]\]

will match {{elc|Lorem ipsum}} and {{elc|consectetur adipisicing}}.

Be careful with expressions like {{code|2=ragel|(\w)(]*>.*?)([,.:;])}}, whose center capture group will span more than one ref group if the outer conditions are met:
sed do eiusmod tempor{{xtag|ref|p|content=reference}} incididunt ut {{xtag|ref|p|content=reference 2}}. labore

Examples

= Sample patterns =

class="wikitable"

! style="background:#CFC; font-style:italic;" | Regex pattern

! style="background:#CFC; font-style:italic;" | Will Match

{{code|2=ragel|([A-Za-z0-9-]+)}}

| One or more letters, numbers or hyphens

{{code|2=ragel|(\d{1,2}\/\d{1,2}\/\d{4})}}

| Any date in dd/mm/yyyy or mm/dd/yyyy format, e.g. {{highlight|3/24/2008}} or {{highlight|03/24/2008}} or {{highlight|24/03/2008}}

{{code|\[\[\d{4}\]\]}}

| Any wiki-linked four-digit number, e.g. {{highlight|{{elc|2008}}}}

(Jan(?:uary|\.|)|Feb(?:ruary|\.|)|Mar(?:ch|\.|)|

Apr(?:il|\.|)|May\.?|Jun(?:e|\.|)|Jul(?:y|\.|)|

Aug(?:ust|\.|)|Sep(?:tember|\.|t\.?|)|Oct(?:ober|\.|)|

Nov(?:ember|\.|)|Dec(?:ember|\.|))

| Full name or abbreviated month name. (Only the abbreviations are captured.)

class="wikitable plainrowheaders"
+ style="background:#CCFFFF;" | Regular expression examples
style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Search for flagicon template and remove

scope="row" |Find

| {{\s*?[Ff]lagicon\s*?\|.*?}}

scope="row" |Replace With

| {{CNone|(nothing)}}

scope="row" |Example of text to search

| {{tlx|flagicon|USA}} {{elc|United States}}

scope="row" |Result

| {{elc|United States}}

scope="row" |Comments

|

style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC; font-style:italic;" | Search for any of three template parameters and replace the value with some new value

scope="row" |Find

| (?<=\|\s*(occupation|spouse|notableworks)\s*=\s*)[^\

]+(?=\s*(\||}}))

|-

! scope="row" |Replace With

| {{code|2=text|new value}}

|-

! scope="row" |Example of text to search

| {{tlx|infobox person|name{{=}}Steveo|occupation{{=}}dancer|nationality{{=}}The moon}}

|-

! scope="row" |Result

| {{tlx|infobox person|name{{=}}Steveo|occupation{{=}}new value|nationality{{=}}The moon}}

|-

! scope="row" |Comments

|

|}

= Commonly used expressions =

Match inside

Regex: ]*>([^<]|<[^/]|])+

Match inside using a (?! not match) notation

Regex: ]*>([^<]|<(?!/ref>))+

Match template {{...}} possibly with templates inside it, but no templates inside those

Regex: {{([^{]|{[^{]|{{[^{}]+}})+}}

Match words and spaces

Regex: [\w\s]+

Match bracketed URLs

Regex: \[(https?://[^\]\[<>\s"]+) *((?<= )[^\n\]]*|)\]

Tips and tricks

= Regex behavior options =

Regex offers several options to change the default behavior.{{Cite web |last=adegeo |date=29 June 2022 |title=Options for regular expression |url=https://learn.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-options |url-status=live |archive-url=https://archive.ph/Rz3CM |archive-date=2023-02-05 |access-date=2023-02-05 |website=learn.microsoft.com |language=en-us}} Five of these options can be controlled with inline expressions, as described below. Four of these options can also be applied to the entire search pattern with check boxes in the AWB "Find-and-replace" tools. By default, all options are off.

class=wikitable
OptionInline flagCheck box availableEffect
IgnoreCasestyle="text-align: center;" | i{{yes}}Specifies case-insensitive matching (upper and lowercase letters are treated the same).
SingleLinestyle="text-align: center;" | s{{yes}}Treats the searched text as a single line, by allowing (.) to match newlines (\n), which it otherwise does not.
MultiLinestyle="text-align: center;" | m{{yes}}Changes the meaning of the (^) and ($) anchors to match the beginning and end, respectively, of any line, rather than just the start and end of the whole string.
ExplicitCapturestyle="text-align: center;" | n{{yes}}Specifies that only groups that are named or numbered (e.g. with the form (?)) will be captured.
IgnorePatternWhitespacestyle="text-align: center;" | x{{no}}Causes whitespace characters (spaces, tabs, and newlines) in the pattern to be ignored, so that they can be used to keep the pattern visually organized.{{efn|To match whitespace characters while the IgnorePatternWhitespace option is enabled, they must be identified with character classes, i.e. \s (whitespace), \n (newline), or \t (tab). (To match only a space, but not a tab or newline, use the pattern \p{Zs}.)}}

{{notelist}}

== Inline syntax ==

The options statement {{code|(?flags-flags)}} turns the options given by "flags" on (or off, for any flags preceded by a minus sign) from the point where the statement appears to the end of the pattern, or to the point where a given option is cancelled by another options statement. For example:

(?im-s) #Turn ON IgnoreCase (i) and MultiLine (m) options, and turn OFF SingleLine (s) option, from here to the end of the pattern or until cancelled

Alternatively, the syntax {{code|(?flags-flags:pattern)}} applies the specified options only to the part of the pattern appearing inside the parentheses:

(?x:pattern1)pattern2 #Apply the IgnorePatternWhitespace (x) option to pattern1, but not to pattern2

= User-made shortcut editing macros =

You can make your own shortcut editing macros. When you edit a page, you can enter your short-cut macro keys into the page anywhere you want AWB to act upon them.

For example, you are examining a page in the AWB edit box. You see numerous items like adding {{tlx|fact}}, inserting line breaks {{tag|br|void}}, commenting out entire lines {{tag|!--|content=comment}}, inserting state names, {{xtag|ref|p|content=Insert footnote text here}}, insert Level 2,3,or even 4 headlines, etc... This can all be done by creating your short-cut macro keys.

  • The process

:# Create a rule. See Find and replace, Advanced settings.

:# Edit your page in the edit box. Insert your short-cut editing macro key(s) anywhere in the page you want AWB to make the change(s) for you.

:# Re-parse the page. Right click on the edit box and select Re-parse from the context pop up menu. AWB will then re-examine your page with your macro short-cut key(s), find your short-cut key(s) and perform the action you specified in the rule.

Naming a short-cut macro key can be any name. But it is best to try and make it unique so that it will not interfere with any other process that AWB may find and suggest. For that reason using {{highlight|///}} followed by a set of lowercase characters that you can easily remember is best (lowercase is used so that you do not have to use the shift key). You can then enter these short-cut macros keys you create into the page manually or by using the edit box context menu paste more function. The reason why we use three '/' is so that AWB will not confuse web addresses/url's in a page when re-parsing.

Examples:

Create a rule as a regular expression.

class="wikitable plainrowheaders"
+ style="background:#CFF;" | User made short-cut editing macros
style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC;" | ///col  Comment out entire line

style="width:30%" | Short-cut key:

| {{mono|///col}}

scope="row" | Name

| Comment out entire line

scope="row" | Find

| {{mono|///col(.*)}}

scope="row" | Replace With

| {{tag|!--|content=$1}}

scope="row" | Example before reparsing

| {{mono|///colThe quick brown fox jumps over the lazy dog}}

scope="row" | Result after re-parsing

| {{tag|!--|content=The quick brown fox jumps over the lazy dog}}

scope="row" | Comments

|

style="border-top:2px solid #333;"

! colspan="2" style="background:#CFC;" | ///fac  Insert {{tlx|citation needed}} with current date

scope="row" | Short-cut key

| {{mono|///fac}}

scope="row" | Name

| Insert {{tlx|citation needed}} with current date

scope="row" | Find

| {{mono|///fac}}

scope="row" | Replace With

| {{tlx|citation needed|date{{=}}{{subst:CURRENTMONTHNAME}} {{subst:CURRENTYEAR}}}}

scope="row" | Example before reparsing

| {{mono|The quick brown fox jumps over the lazy dog///fac}}

scope="row" | Result after re-parsing

|The quick brown fox jumps over the lazy dog{{citation needed|date=December 2022}}

scope="row" | Comments

|

=Efficiency=

{{expand section}}

Efficiency is how long the regex engine takes to find matches, which is a function of how many characters the engine has to read, including backtracking. Complex regular expressions can often be constructed in several different ways, all with the same outputs but with greatly varying efficiency. If AWB is taking a long time to generate results because of a regex rule:

  • Try constructing the expression a different way. There are several online resources with guidance to creating efficient regex patterns.
  • Using the "advanced settings" find-and-replace tool, enter expressions on the "If" tab to filter the pages that an expensive find-and-replace rule is applied to.

References

{{reflist}}

= Online regular expressions testing tools =

  • [http://regexstorm.net/tester RegEx Storm] (supporting .NET regex flavour);
  • [https://regex101.com/ RegEx101] (supporting .NET regex flavour)
  • [http://regexpal.com/ RexEx Pal]
  • [http://gskinner.com/RegExr/ RegExr]
  • [http://rubular.com/ Rubular]

= Desktop regular expression testing tool =

  • [http://regexhero.net/tester/ RegEx Hero]

= Documentation about regular expressions =

  • [http://www.wellho.net/regex/dotnet.html Regular Expressions in .NET] Well House Consultants.
  • [http://www.regular-expressions.info/ Regular-Expressions.info]
  • [https://perldoc.perl.org/perlre#Regular-Expressions Regular Expressions] perldoc.perl.org.
  • [https://docs.python.org/release/2.5.2/lib/re-syntax.html Regular Expression Syntax] docs.python.org.
  • [http://msdn.microsoft.com/en-us/library/az24scfc.aspx Regular Expression Language – Quick Reference] MSDN.
  • [http://msdn.microsoft.com/en-us/library/hs600312.aspx .NET regular expressions] MSDN.
  • [http://www.zytrax.com/tech/web/regex.htm Regular Expressions – User Guide] zytrax.com.

Manual Regular expressions