Skip to content

fix(platform): stop large Worker uploads from timing out - #601

Merged
Brooooooklyn merged 2 commits into
mainfrom
fix/worker-upload-read-ahead
Oct 27, 2026
Merged

Brooooooklyn merged 2 commits into
mainfrom
fix/worker-upload-read-ahead

Conversation

@Brooooooklyn

@Brooooooklyn Brooooooklyn commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Problem

Large apps cannot deploy to Void Cloud. The napi-image deploy (run) failed with The Worker upload did not finish within 120 seconds. Deploy again.

queue job: PUT script to Cloudflare, 120 s timer per attempt
  body = hdr ─ GET module 1 from R2 ─ bytes ─ hdr ─ GET module 2 ─ ... ─ GET module 744
         (one GET at a time, inside the timer)
attempt 1: 120 s ✗ → attempt 2: 120 s ✗ → attempt 3 ✗ → attempt 4 ✗   (≈ 490 s, then fail)
  • The app had 744 modules (413 JS + 331 source maps). 120 s / 744 leaves about 161 ms for each R2 GET, its hash, and its send.
  • Each retry reads all modules again, so a retry cannot succeed.
  • The 120 s limit was only measured with local Miniflare R2.
  • Before fix(deploy): match Cloudflare's Worker size limit on Void Cloud #439, the modules were read before the upload, with no timer. That is why this app deployed on 10-07.

@void/md has a separate problem: markdown-it-attrs@5 loads markdown-it at runtime, but @void/md does not provide it. Apps fail with Cannot find module 'markdown-it' unless they add it themselves.

What this PR changes

body:   hdr ─ module i (already open) ─ bytes ─ open the next modules ahead ...
                                               up to 4 modules / 8 MiB ahead, in order

timer:  no bytes for 60 s           → stop   (a stalled upload)
        no answer 120 s after body  → stop
        8 min for all attempts      → stop   (shared budget, not 120 s × 4)
queued run (15 min Cloudflare limit)
├─ 0 ─────────────────────── 10 min ─────── 15 min
│  app upload + DO replays    │  compensation reserve
│  all end at this deadline ──┘
└─ a DO replay does not start if undoing the hosts already moved would not fit
  • attempt_failed and the new attempt_done logs record bytes sent, time waiting on R2, and time waiting on Cloudflare. A later timeout shows which side was slow.
  • The custom Durable Object host replay sends the same modules. It uses the same read-ahead and limits.
  • @void/md now depends on markdown-it.

Decisions

Question Choice Why
Read-ahead bound 4 modules, 8 MiB declared size One API isolate has 128 MiB for all requests. With the PUT, 4 GETs stay under the 6-connection limit.
A module over 8 MiB Opens only at its turn Opens stay in order. Memory stays bounded.
Timer model Idle 60 s + response 120 s + shared 8 min budget A slow upload that still sends bytes finishes. A stalled one still stops quickly.
DO replay limit min(8 min, run deadline), with time kept to undo The old fixed 60 s would bring the large-bundle timeout back.
Retry rules Not changed 4xx is still final. The saved-script check is still used.

Rollout

merge → deploy hosted production (platform) → Image redeploys
                                               (Brooooooklyn/Image also stops emitting source maps)

No CLI change is needed. The fix is all on the platform side.

Known limits

  • The real time split on production R2 is not measured yet. The new logs will show it after the first large deploy.
  • Recovery after an expired lease (settleExpiredDeployment) and HTTP rollback still have no run deadline. Each replay there keeps the 8-minute cap.
  • A Durable Object replay runs inside the cutover transaction, which locks the project row. A slow but progressing replay can now hold that lock for longer than the old 60 s.
  • Compensation still has no deadline of its own. If restoring the previous release onto a moved Durable Object host takes much longer than the forward move, the queued run can still pass 15 minutes. Lease recovery (settleExpiredDeployment) then restores the host, so mixed versions can last until the 18-minute lease expires. This design existed before this PR. A full fix needs a restore budget before any host changes, plus recovery that resumes across runs. It is left for a separate change.

Testing

Check Result
test/unit/worker-upload.test.ts (read-ahead order, byte bound, release, watchdog, run deadline) pass
test/integration/api-deploy.test.ts (watchdog cuts stalled attempts, budget stops retries) pass
test/integration/managed-durable-objects.test.ts (queued run ends DO replay at the deadline; undo time kept) pass
test/integration/deployment-execution.test.ts (queued upload capped by the run deadline) pass
Mutation checks: read-ahead 0, no byte bound, no idle re-arm, no budget, no deadline wiring each fails a test
@void/md packed into a fresh Yarn PnP project markdown-it-attrs loads; it failed before
vp run check 33/33
vp run test --base origin/main 14911 passed, 28 skipped

🤖 Generated with Claude Code

Brooooooklyn and others added 2 commits October 27, 2026 00:42
The Worker upload body read each module from R2 one at a time, inside a
fixed 120 s timer per attempt. An app with 744 modules could not fit, and
every retry read all modules again, so the deploy failed after 8 minutes.

- Open up to 4 later modules ahead (8 MiB at most) while the body streams.
- Replace the fixed timer with a 60 s no-progress limit, a 120 s limit on
  Cloudflare's answer, and one 480 s budget for all attempts.
- Log bytes sent, storage wait and response time for each attempt.
- Use the same limits for the custom Durable Object host upload.
- Add markdown-it to @void/md: markdown-it-attrs loads it at runtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A queued deployment runs in one invocation with a 15-minute limit. The app
upload and each Durable Object host replay had their own 8-minute budget, so
a slow but progressing deploy could be cut off in the middle of the cutover.

- Queued runs set one upload deadline 10 minutes after they start, which
  leaves 5 minutes for compensation.
- The app upload and every Durable Object replay end at that deadline.
- Each replay leaves enough time before the deadline to undo the hosts
  already moved; otherwise it does not start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Brooooooklyn
Brooooooklyn requested a review from a team as a code owner October 26, 2026 17:11
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-26T17:14:12.924717Z a0a653c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview URL: https://fix-worker-upload-read-ahead-void-cloud.voidzero-docs.workers.dev (commit a0a653c)

This URL reflects your latest Preview deployment

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅
  • Deployment: Success ✅

View logs ↗
https://ef4381d7-void-cloud.voidzero-docs.workers.dev a0a653c 2026-10-26T17:12:31.112Z Visit the dashboard ↗

@Brooooooklyn
Brooooooklyn merged commit a1f00e6 into main Oct 27, 2026
20 checks passed
@Brooooooklyn
Brooooooklyn deleted the fix/worker-upload-read-ahead branch October 27, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant