Brtdku

Python's doc says:

Both patterns and strings to be searched can be Unicode strings (str) as well as 8-bit strings (bytes).

But I was wondering whether searching with str and bytes would always give the same result. I mean, whether this function returns true, for all valid pattern and string:

#!/usr/bin/env python3



import re

def test(pattern, string):

    m = re.search(pattern, string)

    mb = re.search(pattern.encode(), string.encode())

    if m is None and mb is None: return True

    i, j = m.span(0)

    ib, jb = mb.span(0)

    return string[i:j].encode() == string.encode()[ib:jb]

answer: no

example: test('[–]', '–')

note that's an "en-dash" and not a hyphen — any non-ASCII character should have the same behaviour

The main difference is in classes.

For example, U+00FF is "ÿ", but is not an ASCII character. So w (Match "word character", or letters)

re.search(r'w', 'xFF')  # match

re.search(rb'w', b'xFF')  # no match

re.search(rb'w', 'xFF'.encode())  # still no match

(And other unicode letters would work too)

If you look at https://docs.python.org/3/library/re.html, you can see the three classes this applies to:

d

For Unicode (str) patterns:

Matches any Unicode decimal digit (that is, any character in Unicode character category [Nd]). This includes [0-9], and also many other digit characters. If the ASCII flag is used only [0-9] is matched.

For 8-bit (bytes) patterns:

Matches characters considered whitespace in the ASCII character set; this is equivalent to [ tnrfv].

s

For Unicode (str) patterns:

Matches Unicode whitespace characters (which includes [ tnrfv], and also many other characters, for example the non-breaking spaces mandated by typography rules in many languages). If the ASCII flag is used, only [ tnrfv] is matched.

For 8-bit (bytes) patterns:

Matches characters considered whitespace in the ASCII character set; this is equivalent to [ tnrfv].

w

For Unicode (str) patterns:

Matches Unicode word characters; this includes most characters that can be part of a word in any language, as well as numbers and the underscore. If the ASCII flag is used, only [a-zA-Z0-9_] is matched.

For 8-bit (bytes) patterns:

Matches characters considered alphanumeric in the ASCII character set; this is equivalent to [a-zA-Z0-9_]. If the LOCALE flag is used, matches characters considered alphanumeric in the current locale and the underscore.

So if you set the ASCII flag, they should be mostly the same.

For your exact function, an example would be: test(r'w|.', 'xFF')