Skip to content

Prefer a dialect that parses without errors - #708

Open
spicysharkco wants to merge 1 commit into
macbre:masterfrom
spicysharkco:fix-dialect-parse-errors
Open

spicysharkco wants to merge 1 commit into
macbre:masterfrom
spicysharkco:fix-dialect-parse-errors

Conversation

@spicysharkco

Copy link
Copy Markdown

MySQL-only syntax loses part of the query, or raises, because the default dialect is tried before MySQL and its result is accepted:

Parser("SELECT STRAIGHT_JOIN t1.a FROM t1 JOIN t2 ON t1.x = t2.x WHERE t1.y = 1").columns
# before: AttributeError: 'NoneType' object has no attribute 'find'
# after:  ['t1.a', 't1.x', 't2.x', 't1.y']

Parser("SELECT user_id, GROUP_CONCAT(tag SEPARATOR ',') AS tags FROM user_tags GROUP BY user_id").tables
# before: []
# after:  ['user_tags']

Cause

_parse_with_dialect parses with ErrorLevel.WARN, so on a syntax error sqlglot drops or misreads part of the statement and still returns a tree. _try_dialects returns the first tree that _is_degraded accepts, and _is_degraded does not look at parse errors. A later dialect that parses the same SQL without errors is never tried.

Change

_parse_with_dialect also returns whether the parser it used recorded errors. _try_dialects keeps the first tree with errors as a fallback, tries the remaining dialects, and returns the first tree without errors. When no dialect parses without errors, or the last one raises ParseError or TokenError, the fallback is returned. One mixed input changes for the worse: SELECT GROUP_CONCAT(x SEPARATOR ',') FROM zz; [mydb].[dbo].[test_table] returned tables == [] and now raises AttributeError from sqlglot's default dialect, which _try_dialects does not catch.

Only the first statement's tree is used, but sqlglot collects errors from all statements. With several statements and errors recorded, the first statement is parsed again on its own and only its errors count. Otherwise an error in a later statement would move the first one to another dialect. The tree still comes from the whole query, because a procedure body continues past its first semicolon.

Other MySQL-only forms change the same way: USE INDEX, bit and hex literals and XOR. In the Hive test query, MAX(REGEXP_SPLIT(fourth_category, '~')[2]) AS ch_4th_class is a subquery alias. Its [2] made the bracket-identifier dialect report errors, and the default dialect's tree is now used, so ch_4th_class is no longer listed in columns.

Tests

Four new tests fail without the change, one each for GROUP_CONCAT ... SEPARATOR, USE INDEX, a hex literal and STRAIGHT_JOIN. Three more pass either way and pin results that must stay: a string ending in a backslash, which MySQL cannot tokenize, still returns the default dialect's result; an error in a second statement leaves the first one on the default dialect; a procedure body with two statements keeps both tables.

The parser asks sqlglot to recover from syntax errors instead of raising,
so a dialect can return a tree even when it did not understand the query.
That tree was accepted as soon as it did not look degraded, and a later
dialect that parses the same query without errors was never tried. For
MySQL syntax such as GROUP_CONCAT(... SEPARATOR ...), STRAIGHT_JOIN,
USE INDEX and hex literals, the default dialect recovered by dropping
part of the statement, so tables or columns went missing or the
properties raised.

Record whether sqlglot reported errors for each dialect. A result with
errors is kept as a fallback while the remaining dialects are tried. It
is returned when none of them parses without errors.

Only the first statement's tree is used, but sqlglot collects errors
from all statements, so an error in a later statement would move the
first one to another dialect. When a query has several statements and
errors were recorded, the first statement is parsed again on its own and
only its errors count. The tree still comes from the whole query,
because a procedure body continues past its first semicolon.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant