Repository navigation
Lexer: Unicode line separators are whitespace that ends a // comment - #16
Merged
Merged
Conversation
Qt's editor types U+2028 LINE SEPARATOR for Shift+Enter and shows it as a new line, but the lexer rejected it as an unexpected character (BelfrySCAD #949). The three Unicode line separators are now whitespace, and a line comment ends at one, so what follows it on screen is code. They do not advance the line counter: BelfrySCAD's editor numbers lines by block and a U+2028 sits inside one, so lines and columns after it still match the editor's. Byte offsets are untouched either way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs BelfrySCAD/BelfrySCAD#949
The lexer now accepts U+2028 LINE SEPARATOR, U+2029 PARAGRAPH SEPARATOR and U+0085 NEXT LINE as whitespace. Before this, each was a syntax error ("unexpected character"). A
//comment also ends at one of them, so code that appears on its own line after one runs, rather than being swallowed by the comment.They don't advance the line counter. BelfrySCAD's editor numbers lines by block, and a U+2028 sits inside a block, so line and column numbers after one still match the editor's. Byte offsets are unchanged either way.
Qt's editor types U+2028 for Shift+Enter. BelfrySCAD#950 stops that, but files that already contain one will now parse. OpenSCAD's lexer rejects all three characters, so this is a leniency on our side, and OpenSCAD doesn't accept them.
The new test,
LexicalWhitespace.UnicodeLineSeparatorsEndACommentButNotALine, fails without the lexer change. All 670 tests pass.🤖 Generated with Claude Code