Repository navigation
Add measurement files, time ranges, progress and logs to export-data - #18987
CritasWang wants to merge 4 commits into
Conversation
CritasWang
commented
Oct 22, 2026
CritasWang
commented
- -mf/--measurement_file exports the measurements matched by the IoTDB path patterns in a file, one output per measurement named by its full path. The whole file is validated before any query, and each unmatched pattern is reported with its line.
- -start_time/-end_time form an inclusive interval for measurement-file and automatic Table exports. Queries run as written and report the ignored bounds as a WARNING.
- Progress is reported every two seconds: one redrawn line in a terminal, one line per update otherwise, with an ETA in measurement-file mode.
- Every run appends a JSON Lines log, /log_export_data_.log by default or the path given with -l/--export_data_log, ending with one FINISH event. The console ends with the summary, after the log path.
- CSV and SQL files are split by data rows; the INSERT statements of one row stay in one file.
- Ctrl+C and SIGTERM stop at the next row, close the query and the session and exit with 130 or 143; failures exit with 1.
- A Table query exported as SQL or TsFile still writes to the table it reads, as in Fix import-data and export-data correctness and error reporting #18828; -table is ignored with a WARNING, and every query is checked before the first one runs.
- -mf/--measurement_file exports the measurements matched by the IoTDB path patterns in a file, one output per measurement named by its full path. The whole file is validated before any query, and each unmatched pattern is reported with its line. - -start_time/-end_time form an inclusive interval for measurement-file and automatic Table exports. Queries run as written and report the ignored bounds as a WARNING. - Progress is reported every two seconds: one redrawn line in a terminal, one line per update otherwise, with an ETA in measurement-file mode. - Every run appends a JSON Lines log, <target>/log_export_data_<time>.log by default or the path given with -l/--export_data_log, ending with one FINISH event. The console ends with the summary, after the log path. - CSV and SQL files are split by data rows; the INSERT statements of one row stay in one file. - Ctrl+C and SIGTERM stop at the next row, close the query and the session and exit with 130 or 143; failures exit with 1. - A Table query exported as SQL or TsFile still writes to the table it reads, as in #18828; -table is ignored with a WARNING, and every query is checked before the first one runs.
| } | ||
| // Multiple source relations do not have an unambiguous destination table. | ||
| // A query reading several relations has no single destination table. | ||
| if (sources.size() != 1) { |
There was a problem hiding this comment.
Could this also validate the projection, rather than only counting source tables? A computed expression aliased to an existing column name and returning the same type passes the later name/type checks, then SQL/TsFile export writes the computed value back under that original column. A backup round-trip can therefore silently alter data.
There was a problem hiding this comment.
Fixed in 54ccbbf.
Query planning now parses the select list and accepts only * or plain column names. Aliases (v + 1 AS v, w AS v), expressions, function calls and FILL are rejected before any query runs, for both SQL and TsFile, with SQL/TsFile export can only select * or column names, without FILL. The source check is stricter too: EXPLAIN, set operations, CTEs, subqueries and column aliases on the table (FROM t AS a(time, w)) are rejected with the existing single-table message.
The columns are then checked against DESCRIBE: time is required, each column may appear once, and every column must belong to the table. The same check runs again on the columns the query returns, so a table that changes between planning and export is still caught.
Covered by testTableQueriesCannotWriteComputedValuesBack (five query variants, each for sql and tsfile), testTableQueryColumnsAreCheckedAgainstTheTable and testExportChecksTheColumnsTheQueryReturns; these fail on the previous code. On the 1M-row data set, B20 (temperature + 1 as temperature, sql), B21 (the same for tsfile) and B23 (FILL) exit 1 and write no files. Full results: #18987 (comment)
| lastPrintTime = System.currentTimeMillis(); | ||
| List<String> columns = data.getColumnNames(); | ||
| List<String> types = data.getColumnTypes(); | ||
| if (!columns.contains("time")) { |
There was a problem hiding this comment.
Could this require at least one projected FIELD column as well? SELECT time, deviceId FROM metrics passes when deviceId is a TAG, but SQL export then emits an INSERT with only time/TAG columns, which IoTDB rejects (No Field column present). That makes an accepted SQL export impossible to re-import.
There was a problem hiding this comment.
Fixed in 54ccbbf.
A projection must now contain at least one FIELD column, for both SQL and TsFile. SELECT time, deviceId FROM metrics fails during planning with SQL/TsFile output requires a FIELD column from table metrics., before any query runs and before any file is written. I reproduced the server's rejection (507: No Field column present) with an INSERT that has only time and a TAG, so such SQL could not be re-imported. A table that defines no FIELD column at all is exempt, since the requirement could not be met.
TsFile output still requires every TAG column, now with its own message (TsFile output requires every TAG column from table ...) instead of the generic column message.
Covered by testTableQueriesNeedAFieldColumnBeforeAnyQueryRuns (the second of two queries lacks a FIELD column, so the first does not run either), testExportChecksTheColumnsTheQueryReturns, and an ExportDataMeasurementIT assertion that SELECT time, id FROM metrics exits non-zero and writes nothing for sql and tsfile. B22 on the 1M-row data set shows the same.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #18987 +/- ##
============================================
+ Coverage 47.58% 47.79% +0.21%
Complexity 806 806
============================================
Files 5562 5570 +8
Lines 408293 409111 +818
Branches 53591 53754 +163
============================================
+ Hits 194273 195549 +1276
+ Misses 214020 213562 -458 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- A Table query exported as SQL or TsFile must select * or column names from one table of -db. Aliases, expressions and FILL would write values the table does not hold under its column names, so they are rejected, as are EXPLAIN, set operations, column aliases and subqueries without a table. - The query must return the time column and at least one FIELD column, each column once; TsFile queries also need every TAG column. The columns are checked against the table before the first query runs and again against the columns the query returns. - Export option descriptions move to Constants, keeping their lines within 100 characters, and the -q and -db help text is shortened.
Test cases and resultsAll testing targets the latest commit on this PR ( Unit tests —
|
| Case | Combination | Exit | Files | Rows | Result |
|---|---|---|---|---|---|
| B02 | whole-db sql + time range | 0 | 182 | 1,814,400 | PASS |
| B03 | whole-db tsfile + time range | 0 | 2 | 1,814,400 | PASS |
| B05 | -table sql |
0 | 200 | 2,000,000 | PASS |
| B06 | -table tsfile |
0 | 1 | 2,000,000 | PASS |
| B07 | -q csv (reference for B25) |
0 | 100 | 1,000,000 | PASS |
| B08 | -q sql, table from query |
0 | 100 | 1,000,000 | PASS |
| B09 | -q tsfile, table from query |
0 | 1 | 1,000,000 | PASS |
| B12 | -q sql + ignored -table, WARNING |
0 | 100 | 1,000,000 | PASS |
| B17 | 1 | 0 | 0 | PASS | |
| B18 | 1 | 0 | 0 | PASS | |
| B19 | two -q sql, each written to its own table |
0 | 200 | 2,000,000 | PASS |
| B20 | temperature + 1 as temperature → rejected (computed alias) |
1 | 0 | 0 | PASS |
| B21 | same computed alias, tsfile → rejected | 1 | 0 | 0 | PASS |
| B22 | select time, device_id (no FIELD) → rejected |
1 | 0 | 0 | PASS |
| B23 | ... fill method previous → rejected (FILL) |
1 | 0 | 0 | PASS |
| B24 | select time, temperature without TAG column, tsfile → rejected |
1 | 0 | 0 | PASS |
| B25 | select time, device_id, temperature from readings where device_id = 'd1' sql |
0 | 100 | 1,000,000 | PASS |
| B26 | same query, tsfile | 0 | 1 | 1,000,000 | PASS |
Error messages for the rejected cases:
- B20, B21, B23:
SQL/TsFile export can only select * or column names, without FILL. - B22:
SQL/TsFile output requires a FIELD column from table readings. - B24:
TsFile output requires every TAG column from table readings.
Value-level verification — VerifyExports.java checked every value in the 11 re-run positive cases against the data generation formula: 886 files, 13,628,800 rows, 39,886,400 values, 0 mismatches.
B25 vs B07 comparison — compare_b25_b07.py compared all 1,000,000 rows of B25 (column-name projection SQL) with B07 (CSV of the same query, independently verified): 0 mismatches.
TsFile point counts — TsFileCount.java checked every B03/B06/B09/B26 TsFile against the expected point count and time range: 8 entries, 0 mismatches.
SQL reimport — first and last file of B02, B05, B08, B12, B19 and B25 reimported with import-data.sh: all 12 files Import completely!, 0 failure records.
Sonar
The quality gate fails on new_reliability_rating (C), but the single BUG (DataNodeTableCache.java:426, NPE) is in code introduced by #18874, not this PR. The master Sonar job has been failing (OOM) or cancelled for every push since before this PR was opened, so master is stale as a new-code baseline for all PR analyses. The 5 checkstyle LineLengthCheck violations that were on this PR's own lines (OptionsUtil.java:249–302) are fixed in 54ccbbf7c9; this PR's changes reduce the total checkstyle violation count from 165 to 155.
The repository architecture test allows switches on TSDataType only inside a TypeService, and ExportDataTable.exportTsFile switched on the column type for every value. It now resolves one reader per column from a TypeService, keeping the same getters: getLong for TIMESTAMP, getDate for DATE, getBlob for BLOB and getObject for the other types.
|
|
The failing SonarCloud Code Analysis check is not caused by this PR.
All other checks pass. Once a master Sonar analysis succeeds again, re-running the |
