Python_Language

Python TutorialGetting Started with PythonPython Basic SyntaxPython DatatypesPython IndentationPython Collection TypesPython Basic Input and OutputPython Built in Modules and FunctionsPython FunctionsChemPy - python packageCreating Python packagesFunctional Programming in PythonIncompatibilities moving from Python 2 to Python 3IoT Programming with Python and Raspberry PIKivy - Cross-platform Python Framework for NUI DevelopmentMutable vs Immutable (and Hashable) in PythonPyInstaller - Distributing Python CodePython *args and **kwargsPython 2to3 toolPython Abstract Base Classes (abc)Python Abstract syntax treePython Alternatives to switch statement from other languagesPython and ExcelPython Anti-PatternsPython ArcPyPython ArraysPython Asyncio ModulePython Attribute AccessPython AudioPython Binary DataPython Bitwise OperatorsPython Boolean OperatorsPython Checking Path Existence and PermissionsPython ClassesPython CLI subcommands with precise help outputPython Code blocks, execution frames, and namespacesPython Collections modulePython Comments and DocumentationPython Common PitfallsPython Commonwealth ExceptionsPython ComparisonsPython Complex mathPython concurrencyPython ConditionalsPython configparserPython Context Managers (with Statement)Python Copying dataPython CountingPython ctypesPython Data SerializationPython Data TypesPython Database AccessPython Date and TimePython Date FormattingPython DebuggingPython DecoratorsPython Defining functions with list argumentsPython DeploymentPython Deque ModulePython DescriptorPython Design PatternsPython DictionaryPython Difference between Module and PackagePython DistributionPython DjangoPython Dynamic code execution with `exec` and `eval`Python EnumPython ExceptionsPython ExponentiationPython Files & Folders I/OPython FilterPython FlaskPython Functools ModulePython Garbage CollectionPython GeneratorsPython getting start with GZipPython graph-toolPython groupby()Python hashlibPython HeapqPython Hidden FeaturesPython HTML ParsingPython HTTP ServerPython IdiomsPython ijsonPython Immutable datatypes(int, float, str, tuple and frozensets)Python Importing modulesPython Indexing and SlicingPython Input, Subset and Output External Data Files using PandasPython Introduction to RabbitMQ using AMQPStorm



Python Regular Expressions (Regex)

From WikiOD

Python makes regular expressions available through the re module.

Regular expressions are combinations of characters that are interpreted as rules for matching substrings. For instance, the expression 'amount\D+\d+' will match any string composed by the word amount plus an integral number, separated by one or more non-digits, such as:amount=100, amount is 3, amount is equal to: 33, etc.

Syntax[edit | edit source]

  • Direct Regular Expressions
  • re.match(pattern, string, flag=0) # Out: match pattern at the beginning of string or None
  • re.search(pattern, string, flag=0) # Out: match pattern inside string or None
  • re.findall(pattern, string, flag=0) # Out: list of all matches of pattern in string or []
  • re.finditer(pattern, string, flag=0) # Out: same as re.findall, but returns iterator object
  • re.sub(pattern, replacement, string, flag=0) # Out: string with replacement (string or function) in place of pattern
  • Precompiled Regular Expressions
  • precompiled_pattern = re.compile(pattern, flag=0)
  • precompiled_pattern.match(string) # Out: match at the beginning of string or None
  • precompiled_pattern.search(string) # Out: match anywhere in string or None
  • precompiled_pattern.findall(string) # Out: list of all matching substrings
  • precompiled_pattern.sub(string/pattern/function, string) # Out: replaced string

Matching the beginning of a string[edit | edit source]

The first argument of re.match() is the regular expression, the second is the string to match:

import re

pattern = r"123"
string = "123zzb"

re.match(pattern, string)
# Out: <_sre.SRE_Match object; span=(0, 3), match='123'>

match = re.match(pattern, string)

match.group()
# Out: '123'

You may notice that the pattern variable is a string prefixed with r, which indicates that the string is a raw string literal.

A raw string literal has a slightly different syntax than a string literal, namely a backslash \ in a raw string literal means "just a backslash" and there's no need for doubling up backlashes to escape "escape sequences" such as newlines (\n), tabs (\t), backspaces (\), form-feeds (\r), and so on. In normal string literals, each backslash must be doubled up to avoid being taken as the start of an escape sequence.

Hence, r"\n" is a string of 2 characters: \ and n. Regex patterns also use backslashes, e.g. \d refers to any digit character. We can avoid having to double escape our strings ("\\d") by using raw strings (r"\d").

For instance:

string = "\\t123zzb" # here the backslash is escaped, so there's no tab, just '\' and 't'
pattern = "\\t123"   # this will match \t (escaping the backslash) followed by 123
re.match(pattern, string).group()   # no match
re.match(pattern, "\t123zzb").group()  # matches '\t123'

pattern = r"\\t123"  
re.match(pattern, string).group()   # matches '\\t123'

Matching is done from the start of the string only. If you want to match anywhere use re.search instead:

match = re.match(r"(123)", "a123zzb")

match is None
# Out: True

match = re.search(r"(123)", "a123zzb")

match.group()
# Out: '123'

Searching[edit | edit source]

pattern = r"(your base)"
sentence = "All your base are belong to us."

match = re.search(pattern, sentence)
match.group(1)
# Out: 'your base'

match = re.search(r"(belong.*)", sentence)
match.group(1)
# Out: 'belong to us.'

Searching is done anywhere in the string unlike re.match. You can also use re.findall.

You can also search at the beginning of the string (use ^),

match = re.search(r"^123", "123zzb")
match.group(0)
# Out: '123'

match = re.search(r"^123", "a123zzb")
match is None
# Out: True

at the end of the string (use $),

match = re.search(r"123$", "zzb123")
match.group(0)
# Out: '123'

match = re.search(r"123$", "123zzb")
match is None
# Out: True

or both (use both ^ and $):

match = re.search(r"^123$", "123")
match.group(0)
# Out: '123'

Precompiled patterns[edit | edit source]

import re

precompiled_pattern = re.compile(r"(\d+)")
matches = precompiled_pattern.search("The answer is 41!")
matches.group(1)
# Out: 41

matches = precompiled_pattern.search("Or was it 42?")
matches.group(1)
# Out: 42

Compiling a pattern allows it to be reused later on in a program. However, note that Python caches recently-used expressions (docs, SO answer), so "programs that use only a few regular expressions at a time needn’t worry about compiling regular expressions".

import re

precompiled_pattern = re.compile(r"(.*\d+)")
matches = precompiled_pattern.match("The answer is 41!")
print(matches.group(1))
# Out: The answer is 41

matches = precompiled_pattern.match("Or was it 42?")
print(matches.group(1))
# Out: Or was it 42

It can be used with re.match().

Flags[edit | edit source]

For some special cases we need to change the behavior of the Regular Expression, this is done using flags. Flags can be set in two ways, through the flags keyword or directly in the expression.

Flags keyword[edit | edit source]

Below an example for re.search but it works for most functions in the re module.

m = re.search("b", "ABC")  
m is None
# Out: True

m = re.search("b", "ABC", flags=re.IGNORECASE)
m.group()
# Out: 'B'

m = re.search("a.b", "A\nBC", flags=re.IGNORECASE) 
m is None
# Out: True

m = re.search("a.b", "A\nBC", flags=re.IGNORECASE|re.DOTALL) 
m.group()
# Out: 'A\nB'

Common Flags

Flag Short Description
re.IGNORECASE, re.I Makes the pattern ignore the case
re.DOTALL, re.S Makes . match everything including newlines
re.MULTILINE, re.M Makes ^ match the begin of a line and $ the end of a line
re.DEBUG Turns on debug information

For the complete list of all available flags check the docs

Inline flags[edit | edit source]

From the docs:

(?iLmsux) (One or more letters from the set 'i', 'L', 'm', 's', 'u', 'x'.)

The group matches the empty string; the letters set the corresponding flags: re.I (ignore case), re.L (locale dependent), re.M (multi-line), re.S (dot matches all), re.U (Unicode dependent), and re.X (verbose), for the entire regular expression. This is useful if you wish to include the flags as part of the regular expression, instead of passing a flag argument to the re.compile() function.

Note that the (?x) flag changes how the expression is parsed. It should be used first in the expression string, or after one or more whitespace characters. If there are non-whitespace characters before the flag, the results are undefined.

Splitting a string using regular expressions[edit | edit source]

You can also use regular expressions to split a string. For example,

import re
data = re.split(r'\s+', 'James 94 Samantha 417 Scarlett 74')
print( data )
# Output: ['James', '94', 'Samantha', '417', 'Scarlett', '74']

Replacing[edit | edit source]

Replacements can be made on strings using re.sub.

Replacing strings[edit | edit source]

re.sub(r"t[0*9][0*9]", "foo", "my name t13 is t44 what t99 ever t44")
# Out: 'my name foo is foo what foo ever foo'

Using group references[edit | edit source]

Replacements with a small number of groups can be made as follows:

re.sub(r"t([0*9])([0*9])", r"t\2\1", "t13 t19 t81 t25")
# Out: 't31 t91 t18 t52'

However, if you make a group ID like '10', this doesn't work: \10 is read as 'ID number 1 followed by 0'. So you have to be more specific and use the \g<i> notation:

re.sub(r"t([0*9])([0*9])", r"t\g<2>\g<1>", "t13 t19 t81 t25")
# Out: 't31 t91 t18 t52'

Using a replacement function[edit | edit source]

items = ["zero", "one", "two"]
re.sub(r"a\[([0*3])\]", lambda match: items[int(match.group(1))], "Items: a[0], a[1], something, a[2]")
# Out: 'Items: zero, one, something, two'

Find All Non-Overlapping Matches[edit | edit source]

re.findall(r"[0*9]{2,3}", "some 1 text 12 is 945 here 4445588899")
# Out: ['12', '945', '444', '558', '889']

Note that the r before "[0-9]{2,3}" tells python to interpret the string as-is; as a "raw" string.

You could also use re.finditer() which works in the same way as re.findall() but returns an iterator with SRE_Match objects instead of a list of strings:

results = re.finditer(r"([0-9]{2,3})", "some 1 text 12 is 945 here 4445588899")
print(results)
# Out: <callable-iterator object at 0x105245890>
for result in results:
     print(result.group(0))
''' Out:
12
945
444
558
889

Checking for allowed characters[edit | edit source]

If you want to check that a string contains only a certain set of characters, in this case a-z, A-Z and 0-9, you can do so like this,

import re

def is_allowed(string):
    characherRegex = re.compile(r'[^a-zA-Z0-9.]')
    string = characherRegex.search(string)
    return not bool(string)

print (is_allowed("abyzABYZ0099")) 
# Out: 'True'

print (is_allowed("#*@#$%^")) 
# Out: 'False'

You can also adapt the expression line from [^a-zA-Z0-9.] to [^a-z0-9.], to disallow uppercase letters for example.

Partial credit : http://stackoverflow.com/a/1325265/2697955

Grouping[edit | edit source]

Grouping is done with parentheses. Calling group() returns a string formed of the matching parenthesized subgroups.

match.group() # Group without argument returns the entire match found
# Out: '123'
match.group(0) # Specifying 0 gives the same result as specifying no argument
# Out: '123'

Arguments can also be provided to group() to fetch a particular subgroup.

From the docs:

If there is a single argument, the result is a single string; if there are multiple arguments, the result is a tuple with one item per argument.

Calling groups() on the other hand, returns a list of tuples containing the subgroups.

sentence = "This is a phone number 672-123-456-9910"
pattern = r".*(phone).*?([\d-]+)"

match = re.match(pattern, sentence)

match.groups()   # The entire match as a list of tuples of the paranthesized subgroups
# Out: ('phone', '672-123-456-9910')

m.group()        # The entire match as a string
# Out: 'This is a phone number 672-123-456-9910'

m.group(0)       # The entire match as a string
# Out: 'This is a phone number 672-123-456-9910'

m.group(1)       # The first parenthesized subgroup.
# Out: 'phone'

m.group(2)       # The second parenthesized subgroup.
# Out: '672-123-456-9910'

m.group(1, 2)    # Multiple arguments give us a tuple.
# Out: ('phone', '672-123-456-9910')

Named groups[edit | edit source]

match = re.search(r'My name is (?P<name>[A-Za-z ]+)', 'My name is John Smith')
match.group('name')
# Out: 'John Smith'

match.group(1)
# Out: 'John Smith'

Creates a capture group that can be referenced by name as well as by index.

Non-capturing groups[edit | edit source]

Using (?:) creates a group, but the group isn't captured. This means you can use it as a group, but it won't pollute your "group space".

re.match(r'(\d+)(\+(\d+))?', '11+22').groups()
# Out: ('11', '+22', '22')

re.match(r'(\d+)(?:\+(\d+))?', '11+22').groups()
# Out: ('11', '22')

This example matches 11+22 or 11, but not 11+. This is since the + sign and the second term are grouped. On the other hand, the + sign isn't captured.

Escaping Special Characters[edit | edit source]

Special characters (like the character class brackets [ and ] below) are not matched literally:

match = re.search(r'[b]', 'a[b]c')
match.group()
# Out: 'b'

By escaping the special characters, they can be matched literally:

match = re.search(r'\[b\]', 'a[b]c')
match.group()
# Out: '[b]'

The re.escape() function can be used to do this for you:

re.escape('a[b]c')
# Out: 'a\\[b\\]c'
match = re.search(re.escape('a[b]c'), 'a[b]c')
match.group()
# Out: 'a[b]c'

The re.escape() function escapes all special characters, so it is useful if you are composing a regular expression based on user input:

username = 'A.C.'  # suppose this came from the user
re.findall(r'Hi {}!'.format(username), 'Hi A.C.! Hi ABCD!')
# Out: ['Hi A.C.!', 'Hi ABCD!']
re.findall(r'Hi {}!'.format(re.escape(username)), 'Hi A.C.! Hi ABCD!')
# Out: ['Hi A.C.!']

Iterating over matches using `re.finditer`[edit | edit source]

You can use re.finditer to iterate over all matches in a string. This gives you (in comparison to re.findall extra information, such as information about the match location in the string (indexes):

import re
text = 'You can try to find an ant in this string'
pattern = 'an?\w' # find 'an' either with or without a following word character

for match in re.finditer(pattern, text):
    # Start index of match (integer)
    sStart = match.start()

    # Final index of match (integer)
    sEnd = match.end()

    # Complete match (string)
    sGroup = match.group()

    # Print match
    print('Match "{}" found at: [{},{}]'.format(sGroup, sStart,sEnd))

Result:

Match "an" found at: [5,7]
Match "an" found at: [20,22]
Match "ant" found at: [23,26]

Match an expression only in specific locations[edit | edit source]

Often you want to match an expression only in specific places (leaving them untouched in others, that is). Consider the following sentence:

An apple a day keeps the doctor away (I eat an apple everyday).

Here the "apple" occurs twice which can be solved with so called backtracking control verbs which are supported by the newer regex module. The idea is:

forget_this | or this | and this as well | (but keep this)

With our apple example, this would be:

import regex as re
string = "An apple a day keeps the doctor away (I eat an apple everyday)."
rx = re.compile(r'''
    \([^()]*\) (*SKIP)(*FAIL)  # match anything in parentheses and "throw it away"
    |                          # or
    apple                      # match an apple
    ''', re.VERBOSE)
apples = rx.findall(string)
print(apples)
# only one

This matches "apple" only when it can be found outside of the parentheses.


Here's how it works:

  • While looking from left to right, the regex engine consumes everything to the left, the (*SKIP) acts as an "always-true-assertion". Afterwards, it correctly fails on (*FAIL) and backtracks.
  • Now it gets to the point of (*SKIP) from right to left (aka while backtracking) where it is forbidden to go any further to the left. Instead, the engine is told to throw away anything to the left and jump to the point where the (*SKIP) was invoked.

Credit:Stack_Overflow_Documentation