Repository navigation
re: character classes are parsed as regular expressions #914
Description
re: character classes are parsed as regular expressions
transcrypt/modules/re/translate.py turns a Python regex into a token list. The tokenizer has no notion of character classes. Every character inside [...] becomes a token of its own. That causes two defects.
Parentheses inside a character class are counted as capture groups
countCaptureGroups() counts every token named (. A ( inside a character class is counted too. All groups behind such a class shift by one.
import re
r = re.compile(r"(?P<a>[()])(?P<b>x)")| CPython | Transcrypt | |
|---|---|---|
r.groups |
2 | 3 |
r.groupindex |
{'a': 1, 'b': 2} |
{'a': 1, 'b': 3} |
r.match("(x").group("b") returns x under CPython and None under Transcrypt. The generated JavaScript is correct, only the group numbers are wrong.
A literal ] at the start of a character class is not translated
Python treats a ] right behind [ or [^ as a literal. JavaScript ends the class at it. The translator passes the class through unchanged.
re.compile(r"[]x]") matches ] and x under CPython. Transcrypt emits []x] and the RegExp constructor raises Invalid regular expression: /[]x]/u: Lone quantifier brackets. Without the u flag the pattern compiles but means an empty class followed by x], so it never matches.
The reverse case is silent. re.compile("[]") raises unterminated character set under CPython. Transcrypt accepts it and returns a pattern that never matches.
Fix
Both defects go away when shift() consumes a character class as a single token. The class content is then never parsed as a regular expression, and the literal ] can be escaped while the class is read.
Activity