Skip to content

fix(chromium): account for visual viewport offsets in hit testing - #44148

Open
microsoft-playwright-automation[bot] wants to merge 2 commits into
mainfrom
fix-44121-43960170186-1
Open

microsoft-playwright-automation[bot] wants to merge 2 commits into
mainfrom
fix-44121-43960170186-1

Conversation

@microsoft-playwright-automation

Copy link
Copy Markdown
Contributor

Convert Chromium protocol hit-test points from visual viewport coordinates to layout viewport coordinates. Keep event client coordinates unchanged, and use the same conversion when checking parent frames.

Extend the existing locator tap test with a mobile page that pans horizontally and vertically.

Fixes #44121

Triaged on Linux by the Playwright bot - agent run

@github-actions

This comment has been minimized.

@microsoft-playwright-automation

Copy link
Copy Markdown
Contributor Author

🟢 All four failures predate this PR

Hi, I'm the Playwright bot. I checked the MCP failures against the diff and test history.

Failure evidence

Covers MCP run 43961625956 at 903421fe414c, including the four failures in its merged report. None is caused by this PR: these tests use navigation, crash recovery, tab listing and evaluation, not the changed hit-testing paths.

Pre-existing flake / infra

  • Chrome — mcp/crash.spec.ts:31, :58, :79: “reports the navigation error, then resets and logs the crash”, “lists only one tab”, and “marks non-current crashed tab in the tab list”. All three already failed on main at b4d016aadc5c in run 43592649616, with the same 30-second timeouts and Not connected errors. All three also failed in the latest main run at 826ce42b8664. These are existing failures, not new regressions.
  • Firefox — mcp/sse.spec.ts:196, “sse transport browser lifecycle (persistent)”: localStorage.a returns undefined instead of "42" after reconnecting. The exact test and assertion already failed on main at b5b158ae5cbc in run 43525179063, and at c359de03d277 in run 43522442117. This is unrelated to the Chromium viewport conversion.

Checked the failures on Linux by the Playwright bot - agent run

@github-actions

This comment has been minimized.

@microsoft-playwright-automation

Copy link
Copy Markdown
Contributor Author

🟢 Both failures predate this PR

Hi, I'm the Playwright bot. I checked the Windows reporter failures against the diff and earlier CI logs.

Failure evidence

Covers tests 1 run 43961625380 at 903421fe414c, including both failures in its merged report. The PR changes browser hit testing and a tap test; these reporter tests run a simple assertion without opening a browser.

Pre-existing test failure

  • Windows — reporter-junit.spec.ts:350, created and merged → “should add file attribute relative to fileRoot”: both fail at line 374 because classname is auth\a.test.js, but the test expects auth/a.test.js. Both variants have the exact same failure on main at 826ce42b8664 in run 43957125418, and at 66410bcd2101 in run 43526658634, which added the test in feat(junit): add addFileAttribute and fileRoot options #44107. This is an existing Windows assertion defect, not a flake or a regression from this PR.

The small fix is to expect path.join('auth', 'a.test.js') for classname, which keeps native path separators. Keep the file expectation unchanged: that attribute is explicitly converted to forward slashes.

The MCP failures already have a separate verdict.

Checked the failures on Linux by the Playwright bot - agent run


expectHitTarget(hitPoint: { x: number, y: number }, targetElement: Element) {
expectHitTarget(hitPoint: { x: number, y: number }, targetElement: Element, coordinateSystem: 'viewport' | 'client') {
// Chromium protocol coordinates are relative to the visual viewport, but DOM hit testing uses the layout viewport.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this actually an upstream bug?

or is it something that just needs a "workaround" in playwright?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I'm the Playwright bot; this fixes a coordinate-conversion bug in Playwright, not an upstream Chromium defect. Chromium deliberately converts CDP quads to the visual viewport, while DOM hit testing uses the layout viewport.

I confirmed with direct CDP input on Chromium 159.0.8154.6: a tap at (891,450) reaches the button, whose DOM center and touch clientX/clientY are (1233,707) after visual offsets of (342,257), so Playwright must convert the preliminary hit-test point but leave event client coordinates alone.

Revised on Linux by the Playwright bot - agent run

@microsoft-playwright-automation

This comment has been minimized.

@microsoft-playwright-automation

Copy link
Copy Markdown
Contributor Author

Hi, I'm the Playwright bot. This is a Playwright coordinate-conversion bug, not an upstream Chromium defect; I kept the implementation unchanged and answered the review thread with the source and direct-protocol evidence.

Rebased onto 75e1e71404b0. The regression fails without the fix and passes in Chromium, Firefox and WebKit with it. Neighboring tests passed (109 passed, 17 skipped), as did the build and npm run flint.

Revised on Linux by the Playwright bot - agent run

@dcrousso
Devin Rousso (dcrousso) marked this pull request as ready for review November 17, 2026 18:47
@github-actions

This comment has been minimized.

@microsoft-playwright-automation

Copy link
Copy Markdown
Contributor Author

🟢 All three failures predate this PR

Hi, I'm the Playwright bot. I checked the failed MCP run against the PR diff and the logs from its base commit.

Failure evidence

Covers MCP run 44150622356 at 35eba62253f2, including the three failures in its merged report. None is caused by this PR: these tests exercise deliberate crashes, recovery and tab listing, not the changed pointer hit-testing paths.

Pre-existing flake / infra

All three Chrome tests also fail on the PR's exact base commit, 75e1e71404b0, in main run 44149019153, with matching errors and locations:

  • mcp/crash.spec.ts:31 — “reports the navigation error, then resets and logs the crash”: 30-second timeout in the recovery check at line 48.
  • mcp/crash.spec.ts:58 — “lists only one tab”: 30-second timeout and Not connected at line 71.
  • mcp/crash.spec.ts:79 — “marks non-current crashed tab in the tab list”: 30-second timeout and Not connected at line 93.

These are confirmed pre-existing failures; the matching logs alone do not establish whether they are intermittent.

Checked the failures on Linux by the Playwright bot - agent run

@github-actions

This comment has been minimized.

@microsoft-playwright-automation

Copy link
Copy Markdown
Contributor Author

🟢 All three failures predate this PR

Hi, I'm the Playwright bot. I checked the failed run against the PR diff, earlier CI logs and test history.

Failure evidence

Covers tests 1 run 44150622317 at 35eba62253f2, including all three failures in its merged report. None is caused by this PR: the reporter tests do not open a browser, and the video test uses evaluation and screenshots, not pointer hit testing.

Pre-existing test failure

  • Windows — reporter-junit.spec.ts:350, created and merged → “should add file attribute relative to fileRoot”: both expect auth/a.test.js but receive auth\a.test.js at line 374. Both have the identical failure on the PR's exact base, 75e1e71404b0, in main run 44149019035. This is an existing Windows assertion defect, not a flake. The small fix is to expect path.join('auth', 'a.test.js') for classname; keep the file expectation unchanged because that attribute explicitly uses forward slashes.

Pre-existing flake / infra

  • Chromium, Linux — library/video.spec.ts:762, “screencast › VI-030 should work with video+trace”: no screencast-frame event is recorded, so frame.file throws at line 790, including both retries. The same test on the same Linux bot failed twice with the exact error on main at 8b7523772b95, then passed on retry 2 in run 44131240310. This confirms an existing flake.

The MCP failures already have a separate verdict.

Checked the failures on Linux by the Playwright bot - agent run


expectHitTarget(hitPoint: { x: number, y: number }, targetElement: Element) {
expectHitTarget(hitPoint: { x: number, y: number }, targetElement: Element, coordinateSystem: 'viewport' | 'client') {
// Chromium protocol coordinates are relative to the visual viewport, but DOM hit testing uses the layout viewport.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that WebKit uses "layout viewport" for both input and hit testing? Vice versa? Or does visualViewport stay at 0;0 there?

Also, what do element.getBoundingClientRect() and our locator.boundingBox() api return? Are these relative to layout viewport or visual viewport?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this Linux mobile-emulation case, WebKit 27.2 keeps visualViewport.offsetLeft/Top at (0,0) and scrolls the layout viewport by (342,583), so input and DOM hit testing have the same origin. For the main-frame button, Chromium's getBoundingClientRect() and touch-event clientX/Y are layout-relative, while locator.boundingBox() and raw touchscreen input are visual-relative; Firefox uses layout-relative coordinates for all four even with visual offsets (342,0).

Measurements after scrolling the button into view

All three used isMobile: true, hasTouch: true and a 1200x900 viewport; the button is 50x50.

Browser Visual offsets DOM rectangle origin boundingBox() origin Raw tap -> touch clientX/Y
Chromium 159.0.8154.6 (342,257) (1208,682) (866,425) (891,450) -> (1233,707)
Firefox 159.0 (342,0) (1208,425) (1208,425) (1233,450) -> (1233,450)
WebKit 27.2 (0,0) (866,425) (866,425) (891,450) -> (891,450)

document.elementFromPoint() finds the button at its DOM rectangle center in all three; subtracting the visual offset instead finds MAIN in Chromium and Firefox.

These measurements do not establish WebKit's behavior with a nonzero visual offset on real iOS.

Revised on Linux by the Playwright bot - agent run

Comment thread tests/library/tap.spec.ts Outdated
}, async ({ contextFactory, browserName }) => {
const context = await contextFactory({
hasTouch: true,
isMobile: browserName !== 'firefox',

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we do support isMobile on Firefox now, let's try passing true there.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set isMobile: true unconditionally in the separate regression test. It passes in Firefox 159.0, as well as Chromium and WebKit.

Revised on Linux by the Playwright bot - agent run

Comment thread tests/library/tap.spec.ts Outdated
onclick="this.textContent = 'tapped'">b</button>
</main>
`);
await page.locator('#a').tap();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why tapping this button would move the visual viewport off the 0;0 origin. Or is there anything else that causes the viewport movement? A more detailed explanation would be nice.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the automatic scroll before the tap, not the touch event: the oversized content gives Chromium a 1542x1157 layout viewport but a 1200x900 visual viewport, and Playwright's DOM.scrollIntoViewIfNeeded pans the visual viewport to reveal the off-screen button. Calling scrollIntoViewIfNeeded() alone produces visual offsets (342,257) and layout scroll (0,326), before any input is sent. I removed the unnecessary first button/tap and added this explanation beside the remaining tap.

Revised on Linux by the Playwright bot - agent run

Comment thread tests/library/tap.spec.ts Outdated

it.describe('locators', () => {
it('TA-009 should send all of the correct events', async ({ page }) => {
it('TA-009 should send all of the correct events', {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't update this test, and make a new one instead.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored TA-009 unchanged and moved the regression into TA-011 should tap outside the initial visual viewport, with its own mobile context and issue annotation.

Revised on Linux by the Playwright bot - agent run

@microsoft-playwright-automation

This comment has been minimized.

@microsoft-playwright-automation

Copy link
Copy Markdown
Contributor Author

Hi, I'm the Playwright bot. I restored the existing locator test and added a separate mobile regression, with isMobile: true in all three browsers and a comment explaining the automatic scroll that moves Chromium's visual viewport. The implementation is unchanged.

Rebased onto d456688ec2c9. The regression fails without the fix and passes with it; the three-browser run of the tap, touch-event, scroll-click and bounding-box tests finished with 95 passed and 1 skipped. The build and npm run flint passed. The coordinate measurements in the thread replies cover Linux browser builds, not real iOS.

Revised on Linux by the Playwright bot - agent run

@github-actions

Copy link
Copy Markdown
Contributor

Test results for "tests 1"

3 failed
❌ [playwright-test] › reporter-junit.spec.ts:350 › created › should add file attribute relative to fileRoot @playwright-x64-server2025-64-core-node22
❌ [playwright-test] › reporter-junit.spec.ts:350 › merged › should add file attribute relative to fileRoot @playwright-x64-server2025-64-core-node22
❌ [chromium-library] › library/video.spec.ts:762 › screencast › VI-030 should work with video+trace @chromium-playwright-x64-ubuntu24-64-core-node22

1 flaky ⚠️ [playwright-test] › ui-mode-trace.spec.ts:750 › should be able to create and dispose APIRequestContext inside Promise.all `@playwright-x64-server2025-64-core-node22`

23643 passed, 716 skipped


Merge workflow run.

@microsoft-playwright-automation

Copy link
Copy Markdown
Contributor Author

🟢 All three failures also occur on main

Hi, I'm the Playwright bot. I checked the failed run against the PR diff and matching failures on main.

Failure evidence

Covers tests 1 run 44357191916 at d87a10541373d, including all three failures in its merged report. None is caused by this PR: the reporter tests do not open a browser, and the video test uses evaluation and frame capture, not the changed pointer hit-testing paths.

Pre-existing failures

  • Windows — reporter-junit.spec.ts:350, created and merged, “should add file attribute relative to fileRoot”: both expect auth/a.test.js but receive auth\a.test.js at line 374. Both have the identical failure on main at 75e1e71404b0 in run 44149019035. This is an existing Windows assertion defect, not an intermittent failure. The small fix is to expect path.join('auth', 'a.test.js') for classname; keep the file expectation unchanged because that attribute explicitly uses forward slashes.

Pre-existing flake / infra

  • Chromium, Linux — library/video.spec.ts:762, “screencast › VI-030 should work with video+trace”: no screencast-frame event is recorded, so frame.file throws at line 790 on the initial attempt and all three retries. The same test failed twice with the exact error on main at 8b7523772b95, then passed on retry in run 44131240310. This confirms an existing intermittent failure.

The UI-mode test listed as flaky passed on retry and is not one of the three failed tests.

Checked the failures on Linux by the Playwright bot - agent run

@github-actions

Copy link
Copy Markdown
Contributor

Test results for "MCP"

4 failed
❌ [chrome] › mcp/crash.spec.ts:31 › crash recovery › reports the navigation error, then resets and logs the crash @mcp-ubuntu-latest-chrome
❌ [chrome] › mcp/crash.spec.ts:58 › crash recovery › lists only one tab @mcp-ubuntu-latest-chrome
❌ [chrome] › mcp/crash.spec.ts:79 › crash recovery › marks non-current crashed tab in the tab list @mcp-ubuntu-latest-chrome
❌ [webkit] › mcp/cli-test.spec.ts:23 › debug test and snapshot @mcp-ubuntu-latest-webkit

5236 passed, 633 skipped


Merge workflow run.

@microsoft-playwright-automation

Copy link
Copy Markdown
Contributor Author

🟢 All four failures predate this PR

Hi, I'm the Playwright bot. I checked the MCP failures against the PR diff, test history and independent CI logs.

Failure evidence

Covers MCP run 44357191774 at d87a10541373d, including all four failures in its merged report. None is caused by this PR: crash recovery and the CLI test's final browser-list check do not use the changed pointer hit-testing paths.

Pre-existing failures

  • Chrome — mcp/crash.spec.ts:31, :58, :79: all three crash-recovery tests fail identically on the PR's exact base, d456688ec2c9, in main run 44349446267. The recovery check times out after 30 seconds at line 48; the two tab-list tests time out with Not connected at lines 71 and 93. This confirms existing failures, not whether they are intermittent.

Pre-existing flake / infra

  • WebKit job — mcp/cli-test.spec.ts:23, “debug test and snapshot”: line 73 expects (no browsers) but still lists an open attached Chromium session after resume. The same test and assertion failed in the Firefox job of unrelated Docker PR fix(docker): keep the npm cache out of the image layers #44181 in run 44152650141; both jobs run this nested test with Chromium. The test passes in the WebKit job on main at 8124fc39233c. This is an existing intermittent cleanup failure. A small test fix to try is awaiting testProcess.exited after resume, before the final list check.

The tests 1 failures already have a separate verdict.

Checked the failures on Linux by the Playwright bot - agent run

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Chromium locator.tap misses its target after the visible viewport pans

2 participants