semicolon + newline + space in subprocess mode results in SyntaxError

[jahschwa]

xonfig

+------------------+-----------------+
| xonsh            | 0.14.0          |
| Python           | 3.10.12         |
| PLY              | 3.11            |
| have readline    | True            |
| prompt toolkit   | 3.0.38          |
| shell type       | prompt_toolkit  |
| history backend  | json            |
| pygments         | 2.15.1          |
| on posix         | True            |
| on linux         | True            |
| distro           | ubuntu          |
| on wsl           | False           |
| on darwin        | False           |
| on windows       | False           |
| on cygwin        | False           |
| on msys2         | False           |
| is superuser     | False           |
| default encoding | utf-8           |
| xonsh encoding   | utf-8           |
| encoding errors  | surrogateescape |
| xontrib          | []              |
| RC file          | []              |
+------------------+-----------------+

Expected Behavior

Running from bash:

$ xonsh -c $'echo foo;\n echo bar'

Should result in:

foo
bar

i.e. whitespace like that should be ignored around a semicolon

Current Behavior

Result is a SyntaxError:

XONSH_DEBUG=1 xonsh -c $'echo foo;\n echo bar'
<string>:1:5:9 - echo foo;
<string>:1:5:9 + ![echo foo];
<string>:2:0 -  echo bar
<string>:2:0 +  ![echo bar]
<string>:1:5 - echo foo;
<string>:1:5 + ![echo foo;]
<string>:2:0 -  echo bar
<string>:2:0 +  ![echo bar]
xonsh: For full traceback set: $XONSH_SHOW_TRACEBACK = True
  File "<string>", line 2
    echo bar
SyntaxError: ('code:  ',)

I would guess this is because what ends up getting executed is:

![echo foo]
 ![echo bar]

And the indent on the second line is invalid python syntax?

I would say this is probably a pretty uncommon edge case, but I encountered it while trying to install the neovim plugin neovim-treesitter which runs shell commands from lua, and one of them follows the above pattern with the newline and space. (I might open a PR on that repo to remove the space, but this still seems like a potential bug with xonsh anyway.) FWIW I already have a workaround for that, I can force vim to use bash as its shell for command execution.

Traceback (if applicable)

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/xonsh/execer.py", line 343, in _parse_ctx_free
    return _try_parse(input, greedy=False)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/execer.py", line 253, in _try_parse
    raise original_error from None
  File "/usr/local/lib/python3.10/dist-packages/xonsh/execer.py", line 234, in _try_parse
    tree = self.parser.parse(
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 507, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/ply/ply/yacc.py", line 335, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/ply/ply/yacc.py", line 1203, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/ply/ply/yacc.py", line 194, in call_errorfunc
    r = errorfunc(token)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 3600, in p_error
    self._parse_error(msg, self.currloc(lineno=p.lineno, column=p.lexpos))
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 634, in _parse_error
    raise_parse_error(msg, loc, self._source, self.lines)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 220, in raise_parse_error
    raise err
  File "<string>", line 2
    echo bar
SyntaxError: ('code:  ',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/xonsh/execer.py", line 234, in _try_parse
    tree = self.parser.parse(
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 507, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/ply/ply/yacc.py", line 335, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/ply/ply/yacc.py", line 1203, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/ply/ply/yacc.py", line 194, in call_errorfunc
    r = errorfunc(token)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 3600, in p_error
    self._parse_error(msg, self.currloc(lineno=p.lineno, column=p.lexpos))
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 634, in _parse_error
    raise_parse_error(msg, loc, self._source, self.lines)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 220, in raise_parse_error
    raise err
  File "<string>", line 2
    ![echo bar]
SyntaxError: ('code:  ',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/xonsh/main.py", line 522, in main_xonsh
    exc_info = run_code_with_cache(
  File "/usr/local/lib/python3.10/dist-packages/xonsh/codecache.py", line 215, in run_code_with_cache
    ccode = compile_code(display_filename, code, execer, glb, loc, mode)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/codecache.py", line 125, in compile_code
    ccode = execer.compile(code, glbs=glb, locs=loc, mode=mode, filename=filename)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/execer.py", line 130, in compile
    tree = self.parse(input, ctx, mode=mode, filename=filename, transform=transform)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/execer.py", line 86, in parse
    tree, input = self._parse_ctx_free(input, mode=mode, filename=filename)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/execer.py", line 345, in _parse_ctx_free
    return _try_parse(input, greedy=True)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/execer.py", line 330, in _try_parse
    raise original_error
  File "/usr/local/lib/python3.10/dist-packages/xonsh/execer.py", line 234, in _try_parse
    tree = self.parser.parse(
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 507, in parse
    tree = self.parser.parse(input=s, lexer=self.lexer, debug=debug_level)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/ply/ply/yacc.py", line 335, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/ply/ply/yacc.py", line 1203, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/ply/ply/yacc.py", line 194, in call_errorfunc
    r = errorfunc(token)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 3600, in p_error
    self._parse_error(msg, self.currloc(lineno=p.lineno, column=p.lexpos))
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 634, in _parse_error
    raise_parse_error(msg, loc, self._source, self.lines)
  File "/usr/local/lib/python3.10/dist-packages/xonsh/parsers/base.py", line 220, in raise_parse_error
    raise err
  File "<string>", line 2
    echo bar
SyntaxError: ('code:  ',)

Steps to Reproduce

From bash:

xonsh -c $'echo foo;\n echo bar'

From xonsh:

xonsh -c 'echo foo;\n echo bar'

For community

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

[anki-code]

Hey @jahschwa! Thank you for catching this and reporting!

I suppose the SyntaxError was raised because the space was treated as indentation and Python mode was applied.