Skip to content

sort-simple-yaml: read and write YAML files as UTF-8 - #1276

Closed
0xacee wants to merge 1 commit into
pre-commit:mainfrom
0xacee:fix/sort-simple-yaml-utf8
Closed

0xacee wants to merge 1 commit into
pre-commit:mainfrom
0xacee:fix/sort-simple-yaml-utf8

Conversation

@0xacee

@0xacee 0xacee commented Sep 21, 2026

Copy link
Copy Markdown

Problem

sort-simple-yaml opens files with open(filename, 'r+'), so reads and writes use the platform's locale encoding instead of the encoding YAML mandates (UTF-8).

  • On a Windows checkout (or any non-UTF-8 locale) a YAML file containing non-ASCII text is decoded as cp1252 — café is read as café — and if a sort is needed the mojibake is written back: silent content corruption.
  • On a strict ASCII/POSIX locale the same file raises UnicodeDecodeError.
  • Text mode also translates \n to \r\n on write under Windows, so sorting an LF file emits CRLF there but LF on POSIX — platform-dependent output for identical input.

Change

Open the file with encoding='utf-8', newline='' so the hook decodes the encoding YAML specifies and produces byte-identical output on every platform. (newline='' disables \n → \r\n translation on write; rstrip() already normalizes whatever endings were read.)

A non-UTF-8 file now raises a clear UnicodeDecodeError instead of being silently mis-decoded — the file is invalid YAML anyway, since the spec requires UTF-8/16.

Testing

  • New test_non_ascii_content_roundtrip writes a UTF-8 file with multi-byte characters and asserts the sorted output preserves them exactly. On the repo's Windows CI job the previous code would fail this (cp1252 decode + \r\n write).
  • pytest tests/ — 414 passed, 4 xfailed.

@asottile

Copy link
Copy Markdown
Member

we don't accept ai slop

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants