Skip to content

Pin UTF-8 encoding when reading user-provided config files - #14819

Open
dltsum wants to merge 1 commit into
huggingface:mainfrom
dltsum:fix-utf8-user-config-reads
Open

dltsum wants to merge 1 commit into
huggingface:mainfrom
dltsum:fix-utf8-user-config-reads

Conversation

@dltsum

@dltsum dltsum commented Sep 20, 2026

Copy link
Copy Markdown

What

Two user-facing config reads fall back to the locale encoding:

  • fetch_original_config reading the user-supplied original config file (src/diffusers/loaders/single_file_utils.py) — the entry point used by from_single_file when a local YAML config is passed
  • the fp16_safetensors conversion command reading model_index.json (src/diffusers/commands/fp16_safetensors.py)

Why

On Windows with a non-UTF-8 locale (e.g. cp936/GBK), any non-ASCII content in these files crashes with UnicodeDecodeError. Original config files are user-authored YAML where non-ASCII comments or values are realistic.

Reproduction (Windows, cp936 locale)

from diffusers.loaders.single_file_utils import fetch_original_config

open("cfg.yaml", "w", encoding="utf-8").write("# 汉 注释\nmodel:\n  desc: 汉 模型\n")
fetch_original_config("cfg.yaml")
# UnicodeDecodeError: 'gbk' codec can't decode byte 0x89 in position 4: illegal multibyte sequence

Verified red before the fix; after the fix the file is read and parsed correctly (non-ASCII values preserved).

🤖 Generated with Claude Code

`fetch_original_config` reads the user-supplied original config file and
the fp16_safetensors command reads `model_index.json` without specifying
an encoding, so both fall back to the locale encoding. On Windows with a
non-UTF-8 locale (e.g. cp936) any non-ASCII content (CJK comments, emoji)
crashes with a UnicodeDecodeError.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions github-actions Bot added utils single-file size/S PR with diff < 50 LOC labels Sep 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi @dltsum, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice.

Once the PR links an issue (or gets the no-issue-needed label), you can ignore this message — it stays here as a comment, but it no longer applies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

single-file size/S PR with diff < 50 LOC utils

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant