fix: read Japanese, Chinese and Korean text in data files, imports and SQL files

[datlechin]

Problem

A CSV exported by Excel on Japanese Windows (Shift JIS, code page 932) opened in the Data Files window as Windows-1252 mojibake: »Þ²º¶ÝظÌÞÝ is the half-width katakana ｻﾞｲｺｶﾝﾘｸﾌﾞﾝ read one byte at a time. The same file imported through CSV import, a Shift JIS .sql script opened in the editor, a Shift JIS dump imported through SQL import, and a Shift JIS file passed to the iOS Shortcuts row actions were each wrong in their own way.

Root cause

Every place that reads a user's text file picked its encoding by hand, and none of them considered an East Asian encoding:

Where	What it did
Data Files (DelimitedDialectDetector.sniffEncoding)	UTF-8 if the first 256 KB validated, otherwise Windows-1252
CSV import (CSVDialect.probeEncoding)	The same, then a parser that scans raw bytes for delimiters and decodes each field, falling back to Latin-1 per field
SQL files (FileTextLoader)	Byte order mark, com.apple.TextEncoding xattr, UTF-8, then ISO Latin-1
SQL import (ImportEncoding)	A manual list with no East Asian entry
iOS Shortcuts (RowPayload)	UTF-8 or an error

Fixing the fallback alone would not have been enough. The investigation measured each of these:

• CSV import scanned raw bytes. In CP932, GB 18030 and CP950 the second byte of ポ, ー, ソ and 表 is 0x7C or 0x5C, so a comma CSV of Japanese addresses with full-width dashes was split on |, and UTF-16 files were split mid code unit.
• TabularTextEncoding.big5 and .eucKR used the ISO code pages, which reject 碁, 裏, 똠 and 뷁 from Windows exports.
• A transcoded file (UTF-16 today, every East Asian file after detection) saved every row re-serialized: quotes dropped and IBM-extension kanji such as 髙 (FB FC) rewritten, against the documented "rows you did not touch are copied byte for byte".
• Decoding was all or nothing, and a failed reload left File Properties disabled.
• A Shift JIS mysqldump or pg_dump -E SJIS carries SET NAMES cp932 / SET client_encoding = 'SJIS' while the driver always sends UTF-8. Measured on PostgreSQL 18: the decoded dump sent as is fails with invalid byte sequence for encoding "SJIS": 0x9e 0x20; with the statement rewritten the row 日本語 髙橋 lands intact.

Fix

One detector (TabularEncodingDetector, TableProTabularIO), used by Data Files, CSV import, SQL files, the SQL import preview and iOS Shortcuts:

• Byte order mark; UTF-16 without one, judged by decoding both byte orders and counting delimiters and line breaks (so U+3000, 一 and dense CJK text are read correctly).
• A 64 KB window from the first line with a non-ASCII byte (an ASCII-only head no longer hides later Shift JIS), cut only at a line break or a byte below 0x30, which no supported encoding uses inside a character (GB 18030 four-byte sequences hold ASCII digits).
• UTF-8, including UTF-8 with a few stray bytes.
• NSString.stringEncoding(for:encodingOptions:…) restricted to CP932, EUC-JP, GB 18030, CP950, CP949 and Windows-1252, with lossy ranking, and accepted only when the text read that way holds East Asian characters. Half-width-katakana-only CP932 is recognized by its byte shape.

Data Files

• Transcoding runs in 1 MiB chunks cut at line breaks (or at a safe character boundary for very long lines), with progress and cancellation. A byte the encoding cannot read becomes U+FFFD on its own line, and the status bar counts those lines.
• A map from the UTF-8 copy back to the original bytes (a checkpoint per chunk, rows located by counting line breaks) makes Cmd+S copy untouched rows byte for byte in Shift JIS, EUC-JP, GB 18030, Big5, EUC-KR and UTF-16 files. Reordered rows map in one pass.
• The File Properties choice survives reloads, and Save As adopts the dialect it wrote.
• The window always works on a copy; on HFS+ and exFAT it used to map the live file.
• The save error names the row as the grid numbers it, the column by name, selects the row, and names invisible characters (U+00A0 NO-BREAK SPACE).
• In Shift JIS and EUC-JP files ¥ and ‾ save as 0x5C and 0x7E, the JIS Roman bytes Japanese Windows shows as those characters.
• VoiceOver now reads the status bar's delimiter and encoding.

CSV import links TableProTabularIO, detects the encoding (or honours the chosen one) and transcodes to UTF-8 before any byte scanning. A line the encoding cannot read stops the import before any row is deleted or inserted. The escape character now follows the chosen quote. The encoding menu adds UTF-16 LE/BE, Shift JIS, EUC-JP, GB 18030, Big5 and EUC-KR. PluginKit is untouched, so there is no ABI change.

SQL files detect UTF-16 and East Asian encodings after the byte order mark and the xattr; Western text still opens as ISO Latin-1. Restored tabs record their encoding (and byte order mark, checked against the file before it is used), so their baseline is read the same way, and an older tab that held the Latin-1 reading of an East Asian file either adopts the correct reading (unedited) or keeps the Latin-1 reading (edited). Version history reads older commits in the working copy's encoding.

SQL import adds the five East Asian encodings. When the chosen encoding cannot read the preview, the dialog switches to the encoding the dump declares in a top-level SET NAMES / SET client_encoding, or else to the detected one, without remembering that switch. It never picks Shift JIS, GB 18030 or Big5 unless the dump declares it: in those encodings a byte can swallow the backslash that escapes a quote, and a security review of this branch showed a latin1 dump holding attacker-chosen Japanese-looking values could otherwise be auto-read as Shift JIS, ending a string literal early and running the rest as a statement (SQLDumpEncodingChoiceTests covers that dump). When the file is read in anything but UTF-8 or ASCII:

• MySQL/MariaDB SET NAMES, SET CHARACTER SET and character_set_client keep the session's own character set (character_set_client = @@session.character_set_client), respecting a GLOBAL/PERSIST scope carried across a SET list. PostgreSQL gets SET client_encoding = 'UTF8'.
• A _binary'…' literal in a Latin-1 or Windows-1252 dump becomes the exact X'…' it held; in other encodings, where re-encoding is not exact (measured: 658 CP932 and 7,844 EUC-JP byte pairs do not round-trip), a _binary literal holding non-ASCII bytes stops the import at its line. Backslash escapes are decoded only while the dump's own sql_mode leaves them on; after NO_BACKSLASH_ESCAPES, or a sql_mode set from a variable, a _binary literal holding a backslash stops the import too. Text introducers such as _latin1'…' become _utf8mb4, or CONVERT(_utf8mb4'…' USING latin1) when a COLLATE follows.

JSON Lines import decodes its 256 KB sample back to a character boundary.

iOS Shortcuts read files through the shared detector and name the line they cannot read.

Detection accuracy

Measured on macOS 27 against 110 clean samples and 17 samples with one damaged line (Japanese, Chinese, Korean and Western text in every candidate encoding, from one line to 1 MB), plus short Western files holding £, ©, ®, µ, ², ½, °, NBSP with €, and accented capitals:

Sample set	Shipped detector
Clean samples	105/110
One damaged line	17/17
Short Western files	11/12 (a first draft that trusted the ranker alone misread 6 of them as Shift JIS)

The misses are single-line samples whose bytes read as valid text in two encodings: half-width EUC-JP vs CP932, one short Japanese line read as GB 18030, and one line of accented capitals (Tëst;Ëmïl;Öl;Ïle) read as GB 18030. File Properties and the import encoding menus cover them.

Before / After

The same Shift JIS export (CP932, CRLF, half-width katakana in the reading column) opened in each build.

Before, TablePro 0.76.1: every Japanese character shows as Windows-1252, and the status bar reads Comma, Windows-1252, CRLF.

After, a Debug build of this branch: the text reads correctly, and the status bar reads Comma, Shift JIS, CRLF.

Tests

• TableProTabularIOTests: TabularEncodingDetectorTests (21), TabularTextTranscoderTests (14).
• TableProTabularTests/TabularSaveTests: byte-for-byte saves of a quoted Shift JIS export with FB FC and FA 40, deleted, reordered and reversed rows across chunks, UTF-16 files including a stray trailing byte, the yen sign.
• TableProTests: DataFileControllerTests, CSVImportPluginTests, CSVImportEncodingFailureTests, FileTextLoaderTests, FileTextWriterTests, FileTabBaselineTests, SQLClientEncodingRewriterTests, SQLTranscodedLiteralRewriterTests, SQLDumpEncodingChoiceTests, SqlFileImportSourceEncodingTests, JSONImportPluginTests. 614 cases across the touched and neighbouring suites pass, and the full TableProCore package suite passes.
• TableProUITests/DataFileWindowUITests: opens a Shift JIS export, checks the status bar names Shift JIS and Row Details shows the Japanese value. 12/12 pass.
• TableProMobileTests/RowPayloadTests: three new cases. The iOS app and its tests build (build-for-testing, arm64 simulator); this machine has no iOS simulator runtime, so they run in the ios-tests job.
• One existing expectation changed with its contract: CSVImportPluginTests.testQuoteOverrideParsesUTF8 (was testQuoteAndEncodingOverride), because a chosen encoding now applies when the text is prepared and the parser always reads UTF-8.

Not changed, and why: ISO-2022-JP (stateful, not a CSV format in practice) and other single-byte code pages such as CP1251 stay out of the detector.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
TablePro	🟢 Ready	View Preview	Sep 29, 2026, 1:38 PM

💡 Tip: Enable Automations to automatically generate PRs for you.