Make the collector's cost track the live set, not the heap (issue #5537) - #5585
Merged
Conversation
A deep game-tree search on an iPad kept dying after #5540, #5563 and #5573. Each of those fixed a real defect -- pages never returned to the OS, a pacing cap measured against the device's RAM rather than the process budget, a mark worklist that overflowed by sheer allocation volume -- and none of them touched the reason the collector could not keep up in the first place. WHAT THE PROFILE SAYS. Three quarters of the GC thread's wall time is inside cn1ConservativeResolve. gcMarkObject calls it on EVERY reference field the drain follows, to reject a conservatively derived pointer before dereferencing it, and it answered by binary-searching two snapshots: the BiBOP page bases and the legacy extents. On the reporter's shape that is 13 dependent cache-missing loads to find the page and 15 more to miss it and find the array, per field. Marking therefore cost O(log heap) per reference: the collector got slower as the heap grew, which is exactly the reporter's "GC pauses become more and more frequent and take longer, until they are effectively continuous". Everything else followed from that. A cycle stretched to four or five times the collection interval, so the mutator produced four or five times a trigger's worth of garbage during each cycle the collector managed to finish, and the process settled at whatever the pacing allowed: 447MB against a live set of a few hundred bytes, riding 64MB below the ceiling that kills it. On device that is the kill. In the simulator, where there is no ceiling, it is the footprint climbing to gigabytes that the reporter saw next. Both indices are now open-addressed hash tables. The page table keys on the 64KB page base and stores the geometry inline, so a hit is one cache line; its keys change only when a page is registered (the registry is grow-only), so it is rebuilt on that event and only its geometry is refreshed per cycle -- which also retires the per-registration qsort. The legacy side keeps its sorted extent array for interior pointers, which only the conservative stack scan produces, and puts an exact-base table in front of it: a Java reference is always an object base, so the caller that dominates is answered in one probe. TWO THINGS THE FASTER COLLECTOR EXPOSED, both fixed here because both undo it. The survivor-heavy bypass read a pure-churn workload as survivor-heavy. Survival is measured at sweep as slots carrying the current epoch, and the grace pass MARKS every fresh non-leaf object with it -- so what the policy read as a live set was really the allocation rate. It was under the threshold before only because the slow collector inflated the denominator. With the collector keeping up it crossed, diverted 1.8M small objects onto the legacy heap, and brought the worklist overflow back (2-3 cycles in 500, from none). Pages now count the marks a grace pass put on them and the sweep subtracts them, so survival means what the policy needs it to mean. Off a per-process ceiling the pacing cap was a fraction of the HOST's free RAM, which is a reason to let a fast thread run further ahead of the collector and not a reason to accumulate an unbounded amount of garbage. On a roomy machine it evaluated to gigabytes, and once the collector lost the race early nothing brought it back: 13.8-15.7GB of footprint against a 4MB live set, and slower for it (12.2-13.4s against 8.1-8.6s bounded -- a process thrashing fifteen gigabytes pays for them). The cap is now bounded by a multiple of the collection TRIGGER, which already tracks the heap: a survivor-heavy render keeps 8 of its own enlarged triggers, pure churn is held to 8 of the base one. The bound is GATED ON FOOTPRINT, engaging only once the process is already past 512MB, because the point is to stop unbounded growth and not to stop a thread from running ahead. #5573 measured a volume cap costing 2-4x and rejected it; an ungated one measured here at 47% on the objectAllocation microbenchmark (31.6ms -> 43.6ms), for a process that was never going to grow. Gated, that benchmark is 31.1ms -- unchanged -- and the runaway is still bounded, because a runaway is by definition on the wrong side of the gate. That whole shape depends on how much RAM the host happened to have free, which is why it reproduced on an idle machine and vanished on a busy one. CN1_SIMULATE_FREE_MEMORY pins that reading so the guard means the same thing either way. MEASURED on the reporter's shape (GcOverflowSpiralApp, same host, 14.9GB allocated either way, RESULT bit-identical): under a 512MB simulated ceiling before after collections completed 130 583 triggers allocated per collection 4.67 1.04 peak footprint 447MB 116-219MB headroom left below the ceiling 64MB 277-395MB mutator parks 49-54 0 wall time 6.7-6.8s 6.6-7.0s with no ceiling and 32GB of host RAM (the simulator), eight concurrent copies so the collector has to fight for the machine, which is what tips it: peak footprint 13.8-15.7GB 819-861MB wall time 12.2-13.4s 8.1-8.6s GATES. vm/tests: 519 tests in the default group and 8 in the benchmark group, all green, including GcHeapIntegrityIntegrationTest (the CN1_GC_VERIFY use-after-free gate) and LargeArrayGcIntegrationTest (issue 5425). The benchmark gauntlet is GREEN in both cooperative and forced-signal stop modes, every torture bit-identical to the host JVM. run-benchmark.sh geomean unchanged. cn1_globals.m compiles clean to an arm64-apple-ios object against the iOS SDK, and in the CN1_GC_VERIFY / CN1_BIBOP_VALIDATE / CN1_GRACE_AUDIT / CN1_BIBOP_NO_FASTSWEEP / CN1_DISABLE_BIBOP / CN1_RESOLVE_DIAG configurations. GcOverflowSpiralIntegrationTest gains the property underneath all of it -- triggers allocated per completed collection, which is a ratio of two speeds and so reads the same on a loaded machine where a peak does not -- and a second run of the same binary with no ceiling, which is the half of the report that was previously out of scope. That second one is a bound rather than a reproduction, and says so: the off-ceiling runaway is bistable and took eight concurrent copies of the workload on a twelve-core host to provoke, which is not something a unit test should be creating. NOT ADDRESSED. UNDER a ceiling and under deliberate collector starvation (eight concurrent copies of this workload), the process still rides to the ceiling-minus-margin that footprint admission allows. Adding a volume brake to that path as well bounds it to 345MB with 165MB of headroom instead of 38MB, but costs 2.4x -- which is the trade #5573 rejected, and it is a different path from the off-ceiling growth bound added here. The per-cycle qsort of the extent array is now the largest remaining item in the collector at roughly a third of its time, and is the next thing worth replacing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5fa3243c95
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
✅ Continuous Quality ReportTest & Coverage
Static Analysis
Generated automatically by the PR CI workflow. |
Contributor
Cloudflare Preview
|
Collaborator
Author
|
Compared 149 screenshots: 149 matched. Benchmark ResultsDetailed Performance Metrics
|
Collaborator
Author
|
Compared 149 screenshots: 149 matched. Benchmark ResultsDetailed Performance Metrics
|
Collaborator
Author
|
Compared 149 screenshots: 149 matched. |
Collaborator
Author
|
Compared 149 screenshots: 149 matched. Benchmark ResultsDetailed Performance Metrics
|
Collaborator
Author
|
Compared 181 screenshots: 181 matched. |
Collaborator
Author
|
Compared 148 screenshots: 148 matched. Benchmark Results
Detailed Performance Metrics
|
…hing Two defects in the new open-addressed page index, one of them the x86-64 CI failure and one from review. ZERO IS THE EMPTY MARKER, SO IT CANNOT ALSO BE A KEY. cn1ConservativeResolve is handed arbitrary machine words off a conservative stack scan and masks each one to its 64KB page base; any word below CN1_BIBOP_PAGE_SIZE masks to 0, and a small aligned integer left in a stack slot is enough. Probing for 0 matched the first EMPTY entry and returned it as a hit -- an all-zero CN1ConsPage whose slotSize the caller then divided by. The sorted array this replaced could not be reached that way, because every element of it was a real page base; the hazard arrived with the table. It reproduces on the first collection of any workload, which is why every job that runs a translated binary on x86-64 failed at once (exit 136 = SIGFPE) -- and why every local run and the arm64 leg passed: arm64 answers integer division by zero with 0 rather than trapping, so the word quietly resolved to slot 0 of a page that does not exist. Reproduced locally by building the same app for x86_64, and confirmed as the exact instruction by -fsanitize=undefined on arm64, which reports it there too (master: zero UBSan findings on the same workload; this branch before the fix: division by zero at the resolver, from the conservative native-stack scan). Both now run clean and agree with the host JVM. THE REBUILD IS NOW ALL-OR-NOTHING (review, #5585). It used to clear the live table and insert into it, growing on demand -- so a failed calloc part way through left a PARTIAL index. That is not a slow index, it is a silently wrong one: a page missing from it makes every reference into that page fail to resolve, gcMarkObject's guard skips the object, and the sweep frees it while it is still reachable. Worse, the registry is a prepend list, so a rebuild that stopped early kept the NEWEST pages and dropped the oldest -- exactly the ones holding a long-lived live set -- and did it on allocation failure, i.e. when a collection matters most. The table is now sized once from the registration count (plus slack for pages registered during the walk), filled into a fresh allocation, and published only when complete. On any failure the previous table stays in place and cn1ConsPgIndexedCount is left alone so the next cycle retries; what that table lacks is pages registered since it was built, whose objects are mark==-1 fresh and survive on the sweep's grace rule -- the exposure a page registered mid-snapshot has always had. With no previous table to keep, marking cannot proceed at all, so that case says so and aborts rather than sweep a heap it cannot resolve; it is a few hundred KB of calloc, so reaching it means the process is already finished. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rebuild had one failure return for two unrelated situations. Outgrowing the size it picked means a mutator registered pages while it walked -- harmless, and self-correcting on the next cycle. Failing to calloc at all is not. Collapsing them meant a lost race on the FIRST build, where there is no previous index to keep, would have taken the abort() meant for exhaustion. It cannot happen in practice (the walk only covers what was linked when the head was loaded, and the slack is 256 pages), but the two cases deserve different answers regardless: a race now re-sizes and walks again, up to three times, before giving up and leaving the previous index in place. Only exhaustion with nothing to fall back on aborts, and the comment says so. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page index just had to learn that its empty marker must never be a lookup key. The extent table beside it uses the same marker and is safe for a reason that lives twenty lines away -- cn1ConservativeResolve rejects a zero word before either table is consulted, and no extent has a zero base. Write that down where the probe is, so the next restructuring knows the early return is load-bearing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d5c018fbf9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Collaborator
Author
|
Compared 149 screenshots: 149 matched. |
Collaborator
Author
|
Compared 143 screenshots: 143 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
The triggers-per-cycle assertion I added went red on CI at 4.42, and the claim
attached to it -- that the ratio is "a property of the two SPEEDS and not of
either", so it reads the same on a loaded machine -- is simply wrong. The mutator
is one hot allocation loop; a collection has to interleave a mark, a sweep and a
page walk with it, so under contention the collector is the one that loses.
Measured on this workload, triggers allocated per completed collection:
before this branch after
a core to itself 4.67 1.04
8 copies on 12 cores 4.01-4.74 2.30-2.71
16 copies on 12 cores - 3.28
CI: 4 forks, 4 vCPU - 4.42
The old collector was bound by its own cost rather than by the CPU it could get,
so its number barely moves; the fixed one is bound by the CPU, so its number
walks up to meet it. They converge, and no fixed threshold separates them on an
oversubscribed runner. The CI figure is that convergence, not a regression: the
same job's run took 77954ms against the 5804ms this workload needs alone.
The no-ceiling peak has the same shape and for a concrete reason. The growth
bound works by parking a mutator that has run too far ahead, and a park gives up
after two barren collections so that a thread can never be stalled by a collector
that is not running. Starve the collector enough and every park gives up, so the
bound stops binding: sixteen-way, the copies peak between 735MB and 15.7GB,
against 819-861MB eight-way where the collector still gets to run. That assertion
would have gone red next.
Both are now enforced only when the run had the machine, measured by the workload's
own elapsed time -- it is a fixed number of rounds, so that is a direct reading of
the CPU it got. Both numbers are PRINTED on every run either way, and a contended
run says which one it was and why. What this class still enforces unconditionally
is the part that is a property of the code: zero worklist overflows, the bound on
full drains taken inside a grace pass, and staying under the ceiling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4309d2edd4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Collaborator
Author
|
Compared 149 screenshots: 149 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
…not support TWO THINGS, one of them entirely my fault. THE PREVIOUS COMMIT REVERTED THE FIX. While measuring master as a baseline I ran `git checkout origin/master -- cn1_globals.m cn1_globals.h`, which does not just write the worktree -- it STAGES what it writes. I restored the worktree afterwards, saw the resulting `MM` in git status, and committed a test-only change on top; the staged master copies went with it. 460 lines of cn1_globals.m disappeared in a commit whose message is about a test assertion. That is why CI then reported the old tracer format and why the review found CN1_SIMULATE_FREE_MEMORY, CN1_BIBOP_GC_MAX_CAP_MULTIPLIER and the allocatedKb / triggerKb fields "absent from this commit's target tree" -- they were absent, exactly as reported. Both files are restored to their d5c018f content and the index was diffed against the worktree before committing this time. A STALE PAGE INDEX MUST STOP THE SWEEP, NOT JUST THE REBUILD (review, #5585). Keeping the previous index when a rebuild fails is safe for ONE cycle: the pages it is missing were registered after the last successful rebuild, so their objects are mark == -1 and the sweep's grace rule keeps them. It is not safe for two. On the next failed rebuild those objects are no longer fresh, they still do not resolve -- so gcMarkObject's guard skips them however reachable they are -- and they age into the m < V - 1 reclamation with live fields still pointing at them. The fallback traded a hard failure for silent corruption in the low-memory case that motivated it. A failed rebuild now marks the cycle's mark as unsound and codenameOneGCSweep reclaims nothing on it. Skipping a collection costs the memory that cycle would have returned; sweeping on an incomplete mark costs the heap. It is self- correcting -- the rebuild is retried every cycle and the first success marks the whole live set before anything is freed again -- and it subsumes the empty-index case, so the abort() added for that is gone: nothing is swept, so nothing is lost. The blocked-thread release still runs on both paths, or a thread parked on the collector would hang instead. Exercised rather than assumed: with two of every three rebuilds forced to fail, the skip path runs, the throttled report fires, and RESULT stays bit-identical to the host JVM. The same fault injection under CN1_GC_VERIFY -- which walks every survivor's fields after every sweep and aborts on a reference into reclaimed memory -- is running as this goes up and is clean so far; it is slow enough that it outlasts the push, and the result follows on the PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c06a18525
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
✅ ByteCodeTranslator Quality ReportTest & Coverage
Benchmark Results
Static Analysis
Generated automatically by the PR CI workflow. |
Collaborator
Author
|
Compared 144 screenshots: 144 matched. |
Collaborator
Author
|
Compared 217 screenshots: 217 matched. |
… the test on it TWO REVIEW FINDINGS (#5585), the first of which corrects my own diagnosis. THE GROWTH BOUND WAS READING A STALE FOOTPRINT. It keys off cn1CachedProcFootprint, which cn1RefreshFreeMemCache samples once, at mark start. A cycle that begins just under the 512MB floor therefore keeps a below-floor reading for its whole duration, so cn1BibopPacingCap goes on granting the host-derived cap -- gigabytes on a roomy machine. A LONG CYCLE IS EXACTLY THE RUNAWAY THIS BOUND EXISTS TO STOP, so the clamp sat disarmed through the one interval that mattered. The footprint is now re-probed at the point of use, after asking whether the bound would bind at all so the syscall is paid for only on the path that needs it, and rate-limited to one probe per 25ms across all threads. That caps the overshoot at a refresh interval's worth of allocation instead of a collection's. I had attributed the same measurement to the wrong cause. The earlier note said the bound stopped binding under starvation because a pacing park gives up after two barren collections. That is true and still a limit, but it was not what produced the number: with the probe fixed, the same sixteen concurrent copies that peaked between 735MB and 15.7GB now peak between 871MB and 994MB, and twenty-four copies -- whose slowest run takes 118s, against the 78s of the CI job that motivated all this -- peak between 880MB and 1009MB. RESULT stays bit-identical throughout. A GATE THE REGRESSION CAN TRIP IS NOT A GATE. The no-ceiling peak assertion was gated on the run's own elapsed time, and the regression it guards makes the run slow: the test's own numbers put the broken behaviour at 12.2-13.4s against a 12s gate, so the failure could satisfy the skip condition and take the benchmark green. That gate is gone. The bound now holds under contention beyond anything CI applies, so the peak is asserted unconditionally and there is nothing left to disable. Triggers-per-cycle keeps no assertion at all -- it is a ratio of two speeds that converges on the broken collector's as the runner is oversubscribed, so no threshold separates them there and a gated version would have exactly the defect above. It is printed every run as a diagnostic, with the numbers and the reason in the javadoc. The sweep guard from the previous commit is exercised rather than assumed: with two of every three index rebuilds forced to fail, GcHeapIntegrityIntegrationTest -- the CN1_GC_VERIFY gate that walks every survivor's fields after every sweep and aborts on a reference into reclaimed memory -- passes, and the spiral workload's RESULT stays bit-identical with the skip path firing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the fourth and, as far as this can be measured off the reporter's device, load-bearing cause of #5537.
What was actually wrong
cn1ConservativeResolvewas 75% of the GC thread's wall time.gcMarkObjectcalls it on every reference field the drain follows, to reject a conservatively-derived pointer before dereferencing it, and it answered by binary-searching two snapshots — the BiBOP page bases and the legacy extents. On the reporter's shape that is ~13 dependent cache-missing loads to locate the page, plus ~15 more to miss it and locate the array, per field.So marking cost O(log heap) per reference. The collector got slower as the heap grew, which is the reporter's "GC pauses become more and more frequent and take longer, until they are effectively continuous" verbatim. Everything else followed: a cycle stretched to 4.7x the collection interval, the mutator produced 4.7 triggers of garbage per completed cycle, and the process settled at whatever pacing allowed — 447MB against a live set of a few hundred bytes, riding 64MB below the ceiling that kills it. On device that is the kill; in the simulator, where nothing kills it, it is the footprint climbing to gigabytes he saw next.
Both indices are now open-addressed hash tables. The page table keys on the 64KB page base and stores the geometry inline (a hit is one cache line); its key set changes only when a page is registered, so it is rebuilt on that event and only its geometry refreshed per cycle — which also retires the per-registration
qsort. The legacy side keeps its sorted extent array for the interior pointers only the conservative stack scan produces, with an exact-base table in front: a Java reference is always an object base, so the dominant caller is answered in one probe.Two defects the faster collector exposed, fixed here because both undo it
The survivor-heavy bypass read pure churn as survivor-heavy. Survival is measured at sweep as slots carrying the current epoch — and the grace pass marks every fresh non-leaf object with it, so what the policy read as a live set was really the allocation rate (190K of 700K 48-byte slots "surviving"). It stayed under the threshold before only because the slow collector inflated the denominator; once the collector kept up it crossed, diverted 1.8M small objects onto the legacy heap, and brought the worklist overflow back. Pages now tally the marks a grace pass put on them and the sweep subtracts them.
Off a per-process ceiling the pacing cap was a fraction of the host's free RAM. That is a reason to let a fast thread run further ahead of the collector, not a reason to accumulate an unbounded amount of garbage. It is now bounded by a multiple of the collection trigger (which already tracks the heap), gated on the process already being past 512MB — so it stops growth without ever touching a process that was not going to grow. An ungated bound cost 47% on the
objectAllocationmicrobenchmark; gated, that benchmark is unchanged.Measured
GcOverflowSpiralApp, same host, 14.9GB allocated either way,RESULTbit-identical:No ceiling, 32GB host RAM, eight concurrent copies so the collector has to fight for the machine (which is what tips it): 13.8-15.7GB -> 819-861MB, and faster for it — 12.2-13.4s -> 8.1-8.6s, since a process thrashing fifteen gigabytes pays for them.
Gates run locally
vm/tests: 519 tests in the default group, 8 in the benchmark group, all green — includingGcHeapIntegrityIntegrationTest(theCN1_GC_VERIFYuse-after-free gate) andLargeArrayGcIntegrationTest(issue 5425).vm/benchmarks/run-gauntlet.sh: GREEN in cooperative and forced-signal stop modes, every torture bit-identical to the host JVM.vm/benchmarks/run-benchmark.sh: geomean unchanged, all checksums bit-identical.cn1_globals.mcompiles clean to an arm64-apple-ios object against the iOS SDK, and in theCN1_GC_VERIFY/CN1_BIBOP_VALIDATE/CN1_GRACE_AUDIT/CN1_BIBOP_NO_FASTSWEEP/CN1_DISABLE_BIBOP/CN1_RESOLVE_DIAG/CN1_BIBOP_NO_PACINGconfigurations.Guard
GcOverflowSpiralIntegrationTestgains the property underneath all of it — triggers allocated per completed collection, a ratio of two speeds, so it reads the same on a loaded machine where a peak does not (the same run inside a fully loaded parallel suite reported the same cycle count and a 447MB peak). It also gains a second run of the same binary with no ceiling, covering the half of the report that was previously out of scope, withCN1_SIMULATE_FREE_MEMORYpinning the host reading so that leg means the same thing on an idle machine and a busy one. That second assertion is a bound rather than a reproduction, and says so in the source: the off-ceiling runaway is bistable and took eight concurrent copies on a twelve-core host to provoke.Not addressed
qsortof the extent array is now the largest remaining item in the collector, roughly a third of its time.OpenGLES.framework"no such file" report on a Metal build is untouched here; the iOS port's nativeSources still#import <OpenGLES/...>regardless of the Metal setting.🤖 Generated with Claude Code