Skip to content

Fix errors with multi-process workers on Windows - #22016

Open
chadrik wants to merge 1 commit into
python:masterfrom
chadrik:fix/win-worker-timeout-part1
Open

chadrik wants to merge 1 commit into
python:masterfrom
chadrik:fix/win-worker-timeout-part1

Conversation

@chadrik

@chadrik chadrik commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Fixes #21484 "Failed to write with error: 233" on Windows.

This change creates parity between Windows and POSIX, by not applying a server's connection timeout to the read/write traffic that follows on Windows.

Problem

An IPCServer's timeout applies to accepting a connection. On POSIX it stops there, because it is set on the listening socket and accept() hands back a blocking one, but on Windows it applies to the pipe handle and so also bounds every subsequent read and write. Once a peer is connected, the peer may legitimately stay silent for a long time while it computes, so this makes Windows give up on perfectly healthy peers.

After a build worker sends its setup ack it blocks in receive() waiting for the graph, but the coordinator only broadcasts the graph once it has finished loading it, which routinely takes longer than WORKER_CONNECTION_TIMEOUT on a large or cold-cache build. The worker's read fails with "Bad result from I/O wait: 258" (WAIT_TIMEOUT), main() swallows it, and the worker exits 0. When the coordinator finally broadcasts, every write thread dies looking for a pipe with nobody on the other end:

    Exception in thread Thread-7 (write_bytes):
      File "mypy/ipc.py", line 158, in write_bytes
    OSError: [WinError 233] No process is on the other end of the pipe
    ...
    mypy.ipc.IPCException: Failed to write with error: 233

and the following wait_ack() reports "Worker 0 disconnected before sending data (exit code 0)".

Solution

Split the timeout used by reads and writes from the one used to connect, and have IPCServer clear it, which is what POSIX already does. IPCClient keeps applying it to reads and writes, also matching POSIX, where it is set on the connected socket -- "dmypy hang" relies on that.

…hat follows.

Fixes python#21484 "Failed to write with error: 233" on Windows.

This change creates parity between Windows and POSIX.

An IPCServer's timeout applies to accepting a connection. On POSIX it stops
there, because it is set on the listening socket and accept() hands back a
blocking one, but on Windows it applies to the pipe handle and so also bounds
every subsequent read and write. Once a peer is connected it may
legitimately stay silent for a long time while it computes, so this makes
Windows give up on perfectly healthy peers.

After a build worker sends its setup ack it blocks in
receive() waiting for the graph, but the coordinator only broadcasts the
graph once it has finished loading it, which routinely takes longer than
WORKER_CONNECTION_TIMEOUT on a large or cold-cache build. The worker's read
fails with "Bad result from I/O wait: 258" (WAIT_TIMEOUT), main() swallows
it, and the worker exits 0. When the coordinator finally broadcasts, every
write thread dies against a pipe with nobody on the other end:

    Exception in thread Thread-7 (write_bytes):
      File "mypy/ipc.py", line 158, in write_bytes
    OSError: [WinError 233] No process is on the other end of the pipe
    ...
    mypy.ipc.IPCException: Failed to write with error: 233

and the following wait_ack() reports "Worker 0 disconnected before sending
data (exit code 0)".

Split the timeout used by reads and writes from the one used to
connect, and have IPCServer clear it, which is what POSIX already does.
IPCClient keeps applying it to reads and writes, also matching POSIX, where
it is set on the connected socket -- "dmypy hang" relies on that.
@github-actions

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@ilevkivskyi ilevkivskyi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I just have couple style comments (if you don't have time, I can apply them myself).

Comment thread mypy/ipc.py
self.name = name
self.timeout = timeout
self.timeout = timeout # Connections
self.io_timeout = timeout # Reads and writes

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This name is a bit misleading. It looks like it would affect POSIX as well. I think it is better to call this win_io_timeout.

Comment thread mypy/ipc.py
# connection, never to the traffic that follows: see __enter__ below. Once a
# peer is connected it may legitimately stay silent for a long time while it
# computes.
self.io_timeout = None

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things here: move this under an if on the next line, then delete the comment below and combine it with this one and simplify. For example:

        # Unlike the client, a server applies its timeout only to accepting a
        # connection, never to the traffic that follows: see __enter__() below.
        # On POSIX this happens naturally after the sock.accept() call. On
        # Windows we need to set this manually to ensure equivalent behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mypy 2.X periodically and inconsistently throws an error

3 participants