Repository navigation
Prefer a dialect that parses without errors - #708
Open
spicysharkco wants to merge 1 commit into
Open
spicysharkco wants to merge 1 commit into
spicysharkco wants to merge 1 commit into
Conversation
The parser asks sqlglot to recover from syntax errors instead of raising, so a dialect can return a tree even when it did not understand the query. That tree was accepted as soon as it did not look degraded, and a later dialect that parses the same query without errors was never tried. For MySQL syntax such as GROUP_CONCAT(... SEPARATOR ...), STRAIGHT_JOIN, USE INDEX and hex literals, the default dialect recovered by dropping part of the statement, so tables or columns went missing or the properties raised. Record whether sqlglot reported errors for each dialect. A result with errors is kept as a fallback while the remaining dialects are tried. It is returned when none of them parses without errors. Only the first statement's tree is used, but sqlglot collects errors from all statements, so an error in a later statement would move the first one to another dialect. When a query has several statements and errors were recorded, the first statement is parsed again on its own and only its errors count. The tree still comes from the whole query, because a procedure body continues past its first semicolon.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MySQL-only syntax loses part of the query, or raises, because the default dialect is tried before MySQL and its result is accepted:
Cause
_parse_with_dialectparses withErrorLevel.WARN, so on a syntax error sqlglot drops or misreads part of the statement and still returns a tree._try_dialectsreturns the first tree that_is_degradedaccepts, and_is_degradeddoes not look at parse errors. A later dialect that parses the same SQL without errors is never tried.Change
_parse_with_dialectalso returns whether the parser it used recorded errors._try_dialectskeeps the first tree with errors as a fallback, tries the remaining dialects, and returns the first tree without errors. When no dialect parses without errors, or the last one raisesParseErrororTokenError, the fallback is returned. One mixed input changes for the worse:SELECT GROUP_CONCAT(x SEPARATOR ',') FROM zz; [mydb].[dbo].[test_table]returnedtables == []and now raisesAttributeErrorfrom sqlglot's default dialect, which_try_dialectsdoes not catch.Only the first statement's tree is used, but sqlglot collects errors from all statements. With several statements and errors recorded, the first statement is parsed again on its own and only its errors count. Otherwise an error in a later statement would move the first one to another dialect. The tree still comes from the whole query, because a procedure body continues past its first semicolon.
Other MySQL-only forms change the same way:
USE INDEX, bit and hex literals andXOR. In the Hive test query,MAX(REGEXP_SPLIT(fourth_category, '~')[2]) AS ch_4th_classis a subquery alias. Its[2]made the bracket-identifier dialect report errors, and the default dialect's tree is now used, soch_4th_classis no longer listed incolumns.Tests
Four new tests fail without the change, one each for
GROUP_CONCAT ... SEPARATOR,USE INDEX, a hex literal andSTRAIGHT_JOIN. Three more pass either way and pin results that must stay: a string ending in a backslash, which MySQL cannot tokenize, still returns the default dialect's result; an error in a second statement leaves the first one on the default dialect; a procedure body with two statements keeps both tables.