XMLTokenMaker does not follow XML spec (it wants ASCII element names)

[predi]

XMLTokenMaker.flex assumes ASCII element names, which is evident from its TagName production. This production is incorrect per XML spec, which allows far more characters to appear here.

For some reason, it does allow attribute names with a wider character range to be used in their names, evident from InTagIdentifier production.

This results in strange token sequences being produced for well formed XML, such as below.

<cszčšž:test xmlns:cszčšž="test:uri"/>

So, instead of a simple [delimiter ('<'), tag name ("cszčšž:test"), whitespace (" ")] sequence, we get [delimiter ('<'), tag name ("csz"), attribute name ("čšž:test"), whitespace (" ")] for the above element (up to that first namespace attribute).

Note: apparently GitHub's own lexer gets this wrong, but any compliant XML validator would lex the above example without issues.

XMLTokenMaker.flex should be revised to more closely resemble the XML spec:

[4]   	NameStartChar	   ::=   	":" | [A-Z] | "_" | [a-z] | [#xC0-#xD6] | [#xD8-#xF6] | [#xF8-#x2FF] | [#x370-#x37D] | [#x37F-#x1FFF] | [#x200C-#x200D] | [#x2070-#x218F] | [#x2C00-#x2FEF] | [#x3001-#xD7FF] | [#xF900-#xFDCF] | [#xFDF0-#xFFFD] | [#x10000-#xEFFFF]
[4a]   	NameChar	   ::=   	NameStartChar | "-" | "." | [0-9] | #xB7 | [#x0300-#x036F] | [#x203F-#x2040]
[5]   	Name	   ::=   	NameStartChar (NameChar)*

[40]   	STag	   ::=   	'<' Name (S Attribute)* S? '>'
[41]   	Attribute	   ::=   	Name Eq AttValue 

I'm seeing this with an older version of RSTA, but since the flex file has not been changed much in between, the latest versions are bound to exhibit the same.

[bobbylight]

Thanks for the bug report! This will be in the next release.