How a game crash left 8 GB wired in the Apple GPU stack

The Mac came back, the memory did not

The evening began with Cronos: The New Dawn freezing the whole graphical session. The machine was an M1 Pro MacBook Pro with 16 GB of unified memory, running macOS 26.5.2 (build 25F84) and driving an external 4096 by 2304 display at 60 Hz.

macOS eventually terminated the graphical session and rebuilt the desktop. That can look like the end of a graphics crash, but this was only the visible recovery. The game no longer had a usable userspace task, while its process record remained in the kernel-exiting state ?E. Wired memory stayed around 8.5 GiB.

The investigation tested the obvious reset scopes in order: SIGKILL, graphical logout, Apple's IOAccelMemory --purge, and a real sleep/wake cycle. None released the allocation. Sleep briefly pushed wired memory close to 10 GiB.

The game had stopped owning normal userspace memory. The Apple GPU stack was still trying, and failing, to dismantle what the game had left behind.

This is a different problem from choosing a GPU wired-memory limit, covered separately in Apple Silicon GPU memory limits and wired RAM. The relevant question here was not how much memory AGX was allowed to wire, but why it could no longer unwire memory belonging to a client that was already dead.

00:01:06: the GPU fault

The first solid artifact was a GPU event report in /Library/Logs/DiagnosticReports. Its small analysis payload said:

{
  "guilty_dm": 1,
  "signature": 673,
  "restart_reason": 10,
  "restart_reason_desc": "MMU interrupt",
  "process_name": ""
}

An MMU interrupt means the GPU memory-management machinery saw a faulting GPU virtual-memory access. The report did not name a process, so it cannot prove on its own that Cronos issued the bad access. The attribution comes from the rest of the scene: Cronos was the workload that froze, its AGX user client and four command queues remained registered later, and its process became permanently stuck while exiting.

Unified logging showed IOGPUFamily immediately running its own recovery sequence:

/usr/bin/log show \
  --start '2026-07-16 00:00:50' \
  --end   '2026-07-16 00:11:00' \
  --style compact \
  --predicate '(eventMessage CONTAINS[c] "GPURestart") OR
               (eventMessage CONTAINS[c] "kIOGPUCommandBufferCallbackError")'

GPURestartSignaled  stampIdx=45 type=1
GPURestartEnqueued  stampIdx=45 type=1
GPURestartDequeued  stampIdx=45 type=1
GPURestartBegin     stampIdx=45 type=1
GPURestartEnd       stampIdx=45 type=1

The sequence ran once for stamp 45 and three more times for stamp 46. A later I/O Registry dump agreed:

recoveryCount = 4

That counter changed the direction of the investigation. macOS did have a low-level GPU recovery mechanism, and it had already used it four times. The recoveries returned, but one client's cleanup did not.

WindowServer logged its command buffer as an innocent victim of the recovery:

kIOGPUCommandBufferCallbackErrorInnocentVictim
Discarded (victim of GPU error/recovery)

That was the bridge between one broken game and the entire desktop becoming unresponsive.

WindowServer became collateral damage

At 00:03:50, nearly three minutes after the fault, ANECompilerService still could not create an IOGPU device:

ANECompilerService: Failed to create an IOGPUDevice...
IOServiceOpen returned kIOReturn(0xE00002E2)

The IOKit result is "not permitted." In context, it was another sign that graphics services had not returned to a healthy state.

Then came two WindowServer userspace-watchdog stackshots, at 00:09:59 and 00:10:37. The first said its main thread had missed check-in for 40 seconds; the second measured 80 seconds. The blocked queue was com.apple.SkyLight.mtl_submit, and the stack walked straight down the graphics pipeline:

WindowServer / SkyLight
  -> Metal
    -> IOGPU
      -> IOKit
        -> IOGPUFamily (kernel)
          -> AGXG13X (kernel)

The reports recorded nominal thermal pressure. WindowServer used about 831 MB, substantial but nowhere near the memory retained by AGX. This was not a hot CPU, a random UI deadlock, or WindowServer simply growing to eight gigabytes. Its main thread was waiting inside the GPU submission path.

At 00:10:35.918 the watchdog killed WindowServer. Six milliseconds later, macOS spawned its replacement:

00:10:35.918  WindowServer[618] exited with OS_REASON_WATCHDOG
00:10:35.924  Successfully spawned WindowServer[69198]

The graphical session returned. The old GPU allocation did not. The machine had just demonstrated an important distinction: replacing the desktop compositor can repair the GUI without resetting the kernel objects underneath AGX.

A process more dead than alive

After logging back in, Cronos still appeared in the process table:

ps -p 65868 -o pid=,ppid=,state=,etime=,rss=,vsz=,command=

PID    PPID  STATE  RSS  VSZ  COMMAND
65868  1     ?E     0    0    (Cronos-Mac-Shipp)

PID 1 had adopted it. RSS and virtual size were both zero. The E meant it was already exiting. Repeated calls to kill -KILL 65868 returned normally but never removed the process record.

This is the edge case hidden by the usual advice to use kill -9. SIGKILL makes a process enter exit and prevents userspace from handling or ignoring the signal. It does not authorize the kernel to skip a safety-critical teardown that is already in progress. An exiting process cannot be made more dead by sending the same signal again.

sample could not inspect it, and even a privileged task-port probe failed:

task_for_pid(65868): (os/kern) failure (0x5)

The ordinary task was gone. What remained was the BSD process record plus whatever the graphics driver still held while trying to close the client.

Where the 8 GB actually lived

Cronos asked for Metal resources from userspace, but the resulting memory did not stay a plain userspace mapping. The ownership chain looked more like this:

Cronos
  -> Metal objects
    -> IOGPU user client
      -> AGX command queues and fences
        -> GPU/IOMMU mappings
          -> physically wired unified-memory pages

SIGKILL controls the left side. The kernel still has to dismantle everything to the right in a careful order:

1. Cancel or finish the client's outstanding GPU work.
2. Retire its command queues and settle their fences.
3. Remove the GPU virtual-address and IOMMU mappings.
4. Prove that the GPU or its firmware can no longer touch the pages.
5. Unwire the physical pages and return them to the VM allocator.
6. Release the remaining task and process records.

That order protects the rest of the machine. If macOS reused a page before proving that AGX had lost access to it, a delayed GPU command could write into memory now owned by another process. Shared physical memory does not remove the need for GPU address mappings; it makes a stale mapping especially dangerous.

Root privileges do not change this constraint. An administrator can ask a driver to close a connection, purge reclaimable allocations, or terminate a service. Root cannot safely erase arbitrary kernel references or pretend that an in-flight GPU command is harmless.

The missing operation was not a stronger killer. It was a successful AGX teardown transaction: stop the affected engine or firmware, invalidate the mappings, cancel the fences, release the wired pages, and restart graphics from known state. The four automatic recoveries did not complete that transaction for this client.

Counting the missing memory

Activity Monitor made the incident visible, but memory_pressure made it measurable:

Pages wired down: 582192
System-wide memory free percentage: 24%

The machine used 16 KiB pages, so the wired count represented about 8.88 GiB:

582,192 x 16,384 bytes = approximately 8.88 GiB

A small set of commands supplied the measurements throughout the experiment:

vm_stat
memory_pressure
top -l 1 -stats pid,command,mem,rsize,vsize,state
pmset -g assertions

The power assertions supplied another clue. The dead PID still owned this:

pid 65868(Cronos-Mac-Shipping):
  NoDisplaySleepAssertion named: "Running Cronos"

The process had zero RSS and no useful task, but one of its kernel-side promises was still keeping the display awake.

AGX still knew the game

The best view came from asking the accelerator directly:

ioreg -l -w0 -r -c AGXAccelerator

An early sample reported roughly 12.58 GB allocated by AGX, 9.04 GB in use, and four recoveries. More importantly, the registry tree still contained a client tied to the dead PID:

AGXDeviceUserClient
  IOUserClientCreator = "pid 65868, Cronos-Mac-Shipp"
  CommandQueueCount = 4
  CommandQueueCountMax = 4

That ended the theory that this was merely confused accounting in Activity Monitor. AGX itself retained the dead client's user-client object and all four command queues.

kmutil showloaded identified the kernel side of the path:

com.apple.iokit.IOGPUFamily (130.15.2)
com.apple.AGXG13X (351.2)

AGXG13X was matched to the integrated gpu,t6000 device and loaded from the boot kernel collection. It was not a daemon that launchd could restart. Unloading the active display driver from beneath WindowServer and every Metal client would have been unsupported and could have turned a memory leak into a kernel panic.

The purge trimmed the edges

macOS includes an undocumented, Apple-signed diagnostic tool called IOAccelMemory. Its help lists inspection modes and a purge:

IOAccelMemory --pid target_process
IOAccelMemory --summary
IOAccelMemory --all-processes
IOAccelMemory --purge

The unprivileged inspection modes could not open AGX, so the purge was run through the normal administrator authorization dialog:

osascript -e \
  'do shell script "/usr/bin/IOAccelMemory --purge" with administrator privileges'

It did something useful. The Cronos-named AGXDeviceUserClient disappeared from ioreg, allocated memory fell modestly, and some reclaimable GPU caches went away.

It did not reach the retained state. PID 65868 stayed in ?E. Another SIGKILL changed nothing. AGX still reported roughly 8 GB in use and wired memory remained around 8.5 GiB.

Could another process close the stale client?

One lower-level idea was to enter the target task's Mach port namespace, find the send rights connected to AGXAcceleratorG13X, and deallocate them. Dropping the last send right to an IOKit user client would normally invoke its close path.

A read-only C probe used these APIs:

task_for_pid(...)
mach_port_names(...)
mach_port_extract_right(...)
IOConnectGetService(...)

The probe copied rights only for identification. It did not deallocate anything. Even with administrator authorization, it stopped at the first call:

task_for_pid(65868): (os/kern) failure (0x5)

Normal task-port security is part of the story, but the process state mattered more. The usable Mach task had already gone away while the BSD process record and driver cleanup survived. There was no remaining remote IPC namespace to operate on.

The tempting reset buttons that were not resets

Once the public tools ran out, the investigation moved to interfaces that looked promising by name. Names are unreliable guides in private frameworks: they often describe a narrow internal contract rather than a device-wide recovery operation.

Public IOKit termination

The public SDK exposes service discovery, authorization, connection opening, mapping, and termination notifications. It does not expose a general "reset this GPU" or "close another process's user client" API. IOCatalogueTerminate targets driver catalogue modules rather than one leaked AGX client. On the integrated GPU, that returns to the unsafe whole-driver-unload problem.

GPUWrangler and SafeEjectGPU

Apple's private frameworks included IOGPU, GPUWrangler, SafeEjectGPU, and GPUToolsDiagnostics. GPUWrangler exported functions with attractive names:

GPUWranglerGPUEject
GPUWranglerGPUEjectFinalize
GPUWranglerGPUIsRemovable
GPUWranglerGPUIsIntegrated

Runtime inspection of SafeEjectGPU found methods such as initiate, finalize, cancel, and relaunch. They belong to removable or discrete GPU migration. The M1 Pro GPU is integrated and cannot be ejected from itself.

The suspicious kill-client symbol

The IOGPU export table contained IOGPUKillClient_LeakingCommandQueue, a name that appeared directly relevant. Calling a private function with a guessed signature was too risky, so a harmless host process loaded IOGPU, resolved the symbol, and exposed it to LLDB for disassembly. The implementation was effectively:

IOGPUKillClient_LeakingCommandQueue:
    pacibsp
    stp x29, x30, [sp, #-0x10]!
    mov x29, sp
    bl abort

It was a guard that made the calling client abort itself after leaking a command queue. It was not a facility for killing some other process or releasing a retained kernel object.

Metal and IOGPU reset methods

Other searches found names such as IOGPUMetalCommandBufferStorageReset, IOGPUResourceListReset, MTLIOAccelCommandBufferStorageReset, and _purgeDevice. They reset storage, resource lists, descriptors, or pools owned by the calling Metal device. None was a cross-process hardware reset, and no Cronos Metal object remained in userspace to receive such a call.

WindowServer and userspace missed the boundary

Killing WindowServer again was easy to reject because macOS had already performed that experiment. The replacement compositor had a new PID and a working desktop. The old Cronos record and AGX allocation survived. Repeating the transition would only destroy another graphical session and risk unsaved work.

The same reasoning ruled out launchctl reboot userspace. Its documentation describes a teardown and rebuild of userspace without reinitializing the kernel or hardware. That was exactly the wrong side of the line. By then even the visible AGX user client had disappeared after purge; the remaining state was kernel-owned.

A userspace reboot would have closed all open applications while preserving the kernel object graph that needed replacement. It would have been destructive without reaching the fault.

Sleep made it worse

Sleep was the last plausible non-restart hardware transition. Historical logs showed that wake sends power messages to both IOGPUFamily and AGXG13X, so it was reasonable to test rather than merely speculate.

The live baseline was:

The system was put to sleep with pmset sleepnow. The power log confirmed a real Deep Idle sleep and full wake:

00:36:46 Sleep  Entering Sleep state due to 'Software Sleep'
00:36:50 Wake   DarkWake to FullWake from Deep Idle due to HID Activity

The result was the opposite of recovery:

Wake needed fresh graphics allocations while the stale allocation was still present. A second purge brought AGX in-use memory back to about 7.54 GiB and wired memory to about 8.55 GiB, roughly the pre-sleep state, but the PID remained stuck. Sleep came out of the recovery menu.

Where the experiments left the machine

After every non-restart experiment, the live state still looked like this:

PID 65868, PPID 1, state ?E
AGX recoveryCount = 4
AGX in-use system memory = approximately 8.09 GB (7.54 GiB)
wired pages = approximately 559,954 (8.54 GiB)

The purge had removed the registry entry naming Cronos, but that disappearance was not proof that all backing memory or teardown work had completed. The process record and the bulk of the wired allocation remained.

The experiment stopped before rebooting, so this incident does not include a before-and-after restart measurement. The conclusion is about reset scope: only a kernel restart was left that could guarantee destruction of the retained objects and reinitialization of AGX.

A map of the reset boundaries

The experiments make more sense when arranged by the layer each action can replace:

The narrow failure matters. macOS did not lack GPU recovery; it recovered the GPU four times. What remained wedged was one client's teardown after recovery. That is nastier than a simple crash because the desktop can look alive while the kernel still refuses to reuse half the machine's memory.

The recovery action that remained

The project now has a deliberately modest recovery action. It does the useful work without pretending it owns a GPU reset:

1. Find Cronos processes and distinguish a live game from a ?E exiting record.
2. Refuse to interfere if the game is running normally.
3. Retry SIGKILL for a stuck exiting PID.
4. If it remains, request administrator authorization for IOAccelMemory --purge.
5. Retry the kill after the purge.
6. If the PID still remains, explain that the wired AGX state requires a macOS restart.

The menu says Purge stuck Cronos GPU memory, not "fix" or "reset." It does not kill WindowServer, force a logout, or offer sleep as a cure. The narrower label matches the observed behavior.

The same logic can be previewed from the command line:

ruby scripts/recover_cronos_memory.rb status-json
ruby scripts/recover_cronos_memory.rb recover --dry-run

A compact field checklist

For another Apple Silicon graphics failure, this is a short diagnostic path from userspace symptoms to the kernel boundary.

Start with the process state

pgrep -lf 'GameName|Shipping'
ps -p PID -o pid=,ppid=,state=,etime=,rss=,vsz=,command=

If the process is alive, quit or kill it normally. If it remains ?E with zero RSS and VSZ after SIGKILL, stop treating it like an ordinary userspace hang.

Measure wired pages

memory_pressure
vm_stat

Use the page size reported by memory_pressure. Apple Silicon commonly uses 16 KiB pages.

Ask AGX

ioreg -l -w0 -r -c AGXAccelerator |
  rg -n -C 3 'PerformanceStatistics|recoveryCount|IOUserClientCreator|CommandQueueCount'

Look for the dead PID, queue counts, recovery count, and system-memory totals.

Read reports and unified logs

ls -lt /Library/Logs/DiagnosticReports | head -50
rg -n -i 'MMU interrupt|restart_reason|WATCHDOG|mtl_submit|IOGPU|AGX' \
  /Library/Logs/DiagnosticReports/gpuEvent*.ips \
  /Library/Logs/DiagnosticReports/WindowServer*.ips \
  /Library/Logs/DiagnosticReports/WindowServer*.spin

/usr/bin/log show \
  --start 'YYYY-MM-DD HH:MM:SS' \
  --end   'YYYY-MM-DD HH:MM:SS' \
  --style compact \
  --predicate '(eventMessage CONTAINS[c] "GPURestart") OR
               (eventMessage CONTAINS[c] "kIOGPUCommandBufferCallbackError")'

Purge carefully, then measure again

osascript -e \
  'do shell script "/usr/bin/IOAccelMemory --purge" with administrator privileges'

Treat this as cache and allocation reclamation, not a driver reset.

Know when to stop

If the process remains ?E, AGX still reports gigabytes in use, wired pages remain abnormally high, and purge does not clear them, save open work and restart. Repeating SIGKILL or cycling WindowServer does not move the reset boundary.

Unsafe and disruptive paths

The read-only tools in this investigation are ps, memory_pressure, vm_stat, ioreg, kmutil showloaded, report searches, and log show. The rest deserve more care.

• kill -KILL destroys a live process and can lose unsaved game state.
• IOAccelMemory --purge changes GPU cache and allocation state and asks for administrator authorization.
• pmset sleepnow suspends the Mac. It failed as a workaround in this incident.
• Killing WindowServer or forcing logout destroys the graphical session and can lose unsaved application data.
• Do not try to unload the integrated AGX driver or call private selectors with guessed signatures. Failure, loss of display, and a kernel panic are all more plausible than a clean recovery.

When “restart the Mac” is the technical answer

At first, this looked like a game that had forgotten to return its RAM. The evidence told a more interesting story:

1. The GPU reported an MMU fault.
2. IOGPU ran four recoveries.
3. WindowServer became an innocent victim and blocked in Metal submission.
4. The watchdog replaced WindowServer and restored the GUI.
5. Cronos lost its userspace task, but its kernel exit path did not finish.
6. AGX kept a large amount of unified memory wired.
7. Userspace tools could trim around the failure but could not unwind it.

One boundary after another was tested: process, graphical session, reclaimable GPU allocations, userspace, and sleep/wake power state. Each stopped below the object that was stuck.

"Restart the Mac" is often a way to stop investigating. In this case it was where the investigation ended up: the first remaining action that replaced the kernel, rebuilt the AGX object graph, and reinitialized the hardware before those pages could be reused safely.