Repository navigation
[Bugfix] No GPU work while the app is inactive or in the background, and the XR loop ends every frame it takes - #1391
Conversation
… the background, and resumes with it iOS, tvOS and visionOS refuse GPU work from an app that is not in the foreground: Metal logs "IOGPUMetalError: Insufficient Permission (to submit GPU work from background)" and drops the command buffer. Taking the Vision Pro off does exactly that to every app on the engine: the app resigns active and enters the background, the compositor pauses its layer, and the engine's other GPU work still went out: a frame already taken, texture uploads and resamples, the room light's probe readback and prefilter, Gaussian decodes and bakes. GPUSubmissionGate (UntoldEngine) follows the app's state the way Apple asks of a Metal app: it closes when the app resigns active, sees the last committed frame scheduled when the app enters the background, and opens when the app is active again; both transitions are logged with the pause's length. The XR loop takes no frame while it is closed (a frame already taken is finished), the MTKView and AR draws return, the loaders and the streaming batches wait at the gate instead of failing, and the room light's readback and transition wait for it too. The first frame after a pause lets no time pass, so no simulation step spans it; the XR loop also logs a layer pause and resets the timing after it. macOS never restricts the GPU, so the gate stays open there. Tests: the gate's open/close/resume count, the thread and async waits, the main thread never waiting; a render test that a closed gate draws nothing and the frame after it opens has a zero delta.
…, with timestamps, and the XR loop says when it idles on a running layer Diagnostics for the headset: the zombie demo still logged IOGPUMetalError in the background after the gate held the loop and the loaders, so every engine site that sends GPU work while the gate is closed (the main thread, which never waits) now says so once per pause, the gate's and the loop's lines carry a wall-clock time to set against the system's log, and the loop says when the layer runs while the app is inactive. The picking sites wait at the gate too.
The compositor hands out three frames in turn and aborts the app on a query that lands on a frame neither ended nor cancelled, so the loop now counts the frames it takes, the frames it ends and the frames it lets go, says why each frame is let go, and prints the account when the layer pauses, runs again, or runs while GPU work is paused.
…he layer left running poisoned one of the compositor's three frame slots and aborted the app three frames later The compositor hands out three frames in turn (query N takes slot N mod 3) and aborts the app with "We reached more than 3 frames in flight ... Did you forget to call cp_frame_end_submission()?" on a query that lands on a frame neither ended nor cancelled. Only endSubmission frees a frame and only a failed timing or drawable query cancels one; a layer pause resets nothing. Reading the layer's state takes the compositor's pending messages, so at a headset removal, where the layer flickers paused and running, the loop saw the pause mid-frame and let the frame go: before the submission (the frame left with its update ended) or in the defer that skipped endSubmission unless the layer was running (the frame left with its submission started). Three queries later the app aborted, before the app had even resigned active. Now a frame the loop will not draw is carried to its end: its submission is started, the drawables the compositor still gives are presented cleared and the submission is ended; a paused layer gives none and cancels the frame itself, at no cost. The submission of a drawn frame is ended whatever the layer's state by then. The pass and the cleared present say whether the drawable was presented, since a frame with an unpresented drawable cannot be ended; such a frame is let go and said so. The loop says which of the two paths fired, and its account tells the frames ended undrawn and on a paused layer apart.
…k stops while the device sleeps with the headset off, and the loop said 4 s for a pause of 12
…fork [Bugfix] No GPU work while the app is inactive or in the background, and the XR loop ends every frame it takes (fork copy of upstream untoldengine#1391)
|
Nice work tracking down both of these. One thing I'd like changed before merge: The main-thread case of By design,
Separately, not a blocker: for the paths where skipping means a texture/bake silently comes back empty (as opposed to |
…the submission instead of committing it waitUntilOpen() never blocks the main thread, since that is the thread that reopens the gate: there it returns false at once. The loaders, the bakes and the picking passes called it and went on to commit regardless, which is the Insufficient Permission error this branch sets out to remove, with a log line in front of it. Each site now guards on the result in the shape sRGBView already had, and the result is no longer discardable so the compiler flags a site that ignores it. On a false return: loadRuntimeTexture and loadTexture(from property:) return nil, the import-time downsample returns the full-size texture, NativeTextureLoader throws gpuPaused, GaussianChunkLoader throws gpuDecodeFailed, the ML deformer bake throws gpuPaused and aborts rather than skip a frame, generateHDR leaves the textures as loaded with their first level only and sets iblSuccessful false, and the three picking sites return no hit. The XR loop's bounded wait discards the result on purpose, now explicitly.
|
Done in e83d171. Every synchronous Per site, on a
The "Notes for review" section of the description has the one-line (well, one-bullet) note on what the caller sees in each case. Verified: |
Why
Every visionOS app on the engine logs
IOGPUMetalError: Insufficient Permission (to submit GPU work from background)when the headset comes off: iOS and visionOS refuse GPU work from an app that is not in the foreground, and the engine kept sending it — the frame loop, texture uploads and resamples, the room light's probe readback and prefilter, Gaussian decodes, bakes. Trying the fix on the headset turned up a second, worse problem on the same path: on some removals the app aborted withBUG IN CLIENT: We reached more than 3 frames in flight. That's too many. Did you forget to call cp_frame_end_submission()?.What
GPUSubmissionGate(Sources/UntoldEngine/Renderer/GPUSubmissionGate.swift) follows the app's state the way Apple's "Preparing your Metal app to run in the background" asks: it closes onwillResignActive, waits for the last committed command buffer to be scheduled ondidEnterBackground, and opens ondidBecomeActive. While it is closed the XR loop takes no frame (it polls every 0.25 s, sostop()and the layer's invalidation still end it),UntoldRenderer.draw(in:)and the AR draw return, and the engine's GPU work outside the frame waits at the gate:waitUntilOpen()blocks any thread but the main one, which is the thread the gate reopens from, and theasyncform serves the Tasks. On resume the next frame is a first frame:calculateDeltaTimesees the gate's resume count change and lets no time pass, so no simulation step spans the pause. macOS never restricts the GPU, and the observers are not installed there.The XR loop ends every frame it takes. The compositor hands out three frames in turn (query N takes slot N mod 3) and aborts the app on a query that lands on a frame neither ended with
endSubmission()nor cancelled by the compositor itself, which only a failedpredictTiming()or an empty drawable query does; a layer pause frees nothing. ReadinglayerRenderer.statetakes the compositor's pending messages, so at a removal, where the layer flickers between paused and running, the loop could see the pause mid-frame and let the frame go: before the submission, or in thedeferthat skippedendSubmission()unless the layer was still running. Three queries later the app aborted, before it had even resigned active. Now a frame the loop will not draw is carried to its end (endFrameUndrawn: the submission is started, whatever drawables the compositor still gives are presented cleared, and the submission is ended; a paused layer gives none and cancels the frame itself, at no cost), and a drawn frame's submission is ended whatever the layer's state by then. The loop keeps an account of the frames taken, ended and let go and prints it when the layer pauses or runs again. The rule was read fromcp_layer_renderer_query_next_framein the device's CompositorNonUI library, not inferred.docs/Architecture/xrRenderingSystem.mdsays both.Log lines
Engine [hh:mm:ss.SSS]: GPU work paused: the app resigned active (...),the app is in the background; GPU work stays paused until it is active again,GPU work resumed after N s: the app is active againXR [..]: the layer runs while GPU work is paused; the loop takes no frame; taken N, ended N (undrawn U, on a paused layer P), let go L, layer running,the layer is paused; the loop waits for it to run again; ...,the layer runs again after N s; ...XR [..]: a frame was ended undrawn, <reason>; ...anda frame's submission was ended after the layer left running; ...: the two paths that used to let a frame goEngine [..]: GPU work goes on while paused, from the main thread at <file:line>: once per site per pause, for work the gate cannot holdVerified
Insufficient Permissionlines at the background transition: the frame already on the GPU when the compositor stops drawing at the physical removal, held and then refused once the app is in the background. It is harmless, and not the engine's to avoid.UntoldEngineXRbuilds for the visionOS simulator; 8 unit tests (GPUSubmissionGateTests) and 1 render test (GPUSubmissionGateRenderingTests: a closed gate draws no frame, and the first frame after a pause hastimeSinceLastUpdate == 0); SwiftFormat reports nothing new on the touched files.Notes for review
didEnterBackgroundis one line inobserveApplicationStateif that turns out unwanted.waitUntilOpen()returnsfalsethere (the main thread reopens the gate, so it must not wait), and every synchronous site now skips its submission onfalseinstead of committing anyway, in the shapesRGBViewalready had; the result is no longer discardable, so the compiler flags a site that ignores it. The gap is accepted as rare and transient (the app is resigning active or in the background, and the next load after resume succeeds):loadRuntimeTextureandloadTexture(from property:)returnnil(the material lacks that texture), the import-time downsample returns the full-size texture,NativeTextureLoaderthrowsgpuPaused,GaussianChunkLoaderthrowsgpuDecodeFailed, the ML deformer bake throwsgpuPausedand aborts (a skipped frame would corrupt the bake),generateHDRleaves the environment textures as loaded with their first level only and setsiblSuccessful = false, and the three picking sites return no hit /.error.