ci(repo): run the Expo end-to-end device tests on pull requests

.changeset/expo-verify-ci.md

@@ -0,0 +1,2 @@
+---
+---

.claude/skills/verify-clerk-expo/SKILL.md

@@ -13,8 +13,6 @@ No change to `@clerk/expo` UI or auth behavior is done until a `run` on the real
 
 Run every command from the repo root. In the prose below, `doctor`, `up`, `run`, `screen`, `attach`, and `down` are verbs of that CLI. The tests and the CLI are the Node package `integration/expo-native/`, and paths that begin `specs/`, `src/`, `test/`, or `.verify/` are inside it. `features/` and `references/` are beside this file. Every verb but `attach` takes `--platform ios|android`, and iOS is the default. Every verb takes `--json` and then prints one `{ "ok": ... }` object. Exit codes are 0 for success, 1 for a failing spec, 2 for a usage error, and 3 for a failed precondition. Every error prints a `fix` line.
 
-CI runs none of these specs. A regression test that must run on every pull request belongs in `integration/tests/expo-native/`, which `.github/workflows/expo-native-build.yml` runs against a Release build of the same test app with the same `e2e` engine.
-
 The test app links `@clerk/expo`, `@clerk/expo-biometrics`, and `@clerk/expo-google-signin` from the workspace. It does not install `@clerk/expo-passkeys`, so the skill cannot verify passkeys.
 
 ## Launch
@@ -65,7 +63,7 @@ A worktree can hold one lane of each platform. The two lanes share the watch bui
 
 A Mac has four iOS lanes and two Android lanes, shared by every worktree on it. When all are taken, `up` and `run` fail with `POOL_FULL`, and `--wait <seconds>` on either verb waits for a lane. The CLI drives only the simulators and emulators that it creates. [Local devices](references/devices.md) says how to find a lane's UDID or serial.
 
-The test app is built locally on macOS only. On a Linux machine that can run the emulator, the CLI picks the local backend for Android, and the build then fails with `UNSUPPORTED`.
+On a Linux machine that can run the emulator, the CLI picks the local backend for Android and builds the test app on that machine, as the `Verify end-to-end tests` workflow does on its Linux runner.
 
 ## Doctor
 
@@ -128,7 +126,7 @@ Assert on what a user sees, and look in the native Clerk views first. The code s
 
 The home is the test app's own screen in a verify launch. With no active session it shows `Signed out` (`e2e.auth.signedOut`) and a `Sign in` button (`e2e.auth.signIn`) that opens AuthView in a modal. With an active session it shows the UserButton, `Signed in as <email>` (`e2e.auth.signedIn`), the user ID (`e2e.auth.userId`), the session ID (`e2e.auth.sessionId`), and a `Sign out` button (`e2e.auth.signOut`). The home reads each of these from the hook that a customer's app would use. The signed-out text and the buttons come from `useAuth`, the email and the user ID from `useUser`, and the session ID from `useSession`. A hook that keeps a stale value after a sign-out leaves its text on the home, and `host.expectSignedOut` then fails. A pending session counts as signed out on the home.
 
-The test app shows the home again when a full-screen AuthView or a custom form completes its flow, when the close button of the full-screen AuthView calls `onDismiss`, and when the Back button on the root of the embedded profile calls `onHostBack`. The native modules and token cache screens have no way back. It draws nothing on or around a native view. While Clerk loads or a ticket signs in, it shows a spinner. The two full-screen AuthViews also show the spinner until `useAuthViewState().isLoaded` is true, so AuthView on screen after one of those taps proves that flag. When a launch cannot start, the test app shows `Something went wrong` and the reason (`e2e.launch.error`). A launch without verify inputs shows none of this. It shows the home in the test app's `App.tsx`, which `integration/tests/expo-native/` drives.
+The test app shows the home again when a full-screen AuthView or a custom form completes its flow, when the close button of the full-screen AuthView calls `onDismiss`, and when the Back button on the root of the embedded profile calls `onHostBack`. The native modules and token cache screens have no way back. It draws nothing on or around a native view. While Clerk loads or a ticket signs in, it shows a spinner. The two full-screen AuthViews also show the spinner until `useAuthViewState().isLoaded` is true, so AuthView on screen after one of those taps proves that flag. When a launch cannot start, the test app shows `Something went wrong` and the reason (`e2e.launch.error`). A launch without verify inputs shows the same home, signed out, with the publishable key the app was built with (`EXPO_PUBLIC_CLERK_PUBLISHABLE_KEY`). A build with no key shows the error screen.
 
 The test app has a screen or a small flow of its own only where the native views cannot prove a thing. The custom sign-in and sign-up forms drive `useSignIn` and `useSignUp`. The test app reads the `@clerk/expo` token cache once, as the app starts and before Clerk has loaded, and the `Token cache` screen says whether a client token was kept from the last launch. `Sign in with a logo` opens the home's modal AuthView with a React Native view as its `logo`. The `Embedded profile` screen is UserProfileView inline with one custom page, `isDismissible={false}`, and `onHostBack`. The `Native modules` screen has one button for `useSignInWithGoogle` and one for `useBiometricCredentials`, each with its result as text. The test app looks and behaves as an app does for a real user. When a change needs a flow that no native view covers, add a screen that a real app would have, and assert on the outcome as a user sees it.
 
@@ -138,7 +136,7 @@ A spec file that needs other settings than the standard ones has a settings file
 
 ### Check your work
 
-Golden specs under `specs/golden/<feature>/` are committed and cover the feature map in `features/`, which starts at `features/README.md`. Run the features your change touches, on both platforms when the change is not specific to one. For new work:
+Golden specs under `specs/golden/<feature>/` are committed and cover the feature map in `features/`, which starts at `features/README.md`. Prove your own change, on both platforms when the change is not specific to one, and leave the rest of the golden specs to PR CI (`.github/workflows/verify-e2e.yml`), which runs every one of them once the pull request is ready for review. When your change is to behavior a golden spec already covers, that spec is your proof: run it. Run another feature's specs yourself only when you changed code that feature shares and want to know before CI does. For new work:
 
 1. Write a spec under `specs/explored/`, which is gitignored. It imports the fixture as `'../fixtures.ts'`.
 2. Run it by path.
@@ -178,7 +176,7 @@ $ integration/expo-native/bin/control-clerk-expo attach <run-id> --pr <n> --scre
 
 `attach` posts one comment per run and PR with `gh pr comment --attach`. It needs a `gh` whose `gh pr comment` has that flag, and it fails with a fix when the flag is missing. It refuses a run that is tainted, that has a failing spec or no passing one, or whose `app.log` names a user that the run did not create.
 
-Attach the focused run, not the regression run. Run your new or changed spec on its own and attach that run, so the PR video shows only the behavior the change is about. Run the golden specs for every feature you touched in a separate `run`, cite its run id in the PR as regression evidence, and leave its video in `.verify/runs/`.
+Attach the run of your own change. Run your new or changed spec on its own and attach that run, so the PR video shows only the behavior the change is about. You do not owe a regression run: PR CI (`.github/workflows/verify-e2e.yml`) runs every golden spec on the pull request and reports them there. If you ran other golden specs anyway, cite that run's id in the PR and leave its video in `.verify/runs/`.
 
 ## Cleanup
 
@@ -209,8 +207,10 @@ If a worktree is removed without `down`, the next `up` or `run` in any worktree
 
 ## For maintainers of the tests and the CLI
 
-- `src/core/`, `src/platform/ios/`, `src/platform/android/`, `specs/support/`, `specs/fixtures.ts`, `e2e.config.ts`, `testing/`, and every test but `test/host.test.ts` and `test/freshness.test.ts` are shared with the same package in clerk-ios and clerk-android. Change them there first, then copy them here. `doctor`'s `core-drift` check fails when `src/core/`, `specs/support/`, `specs/fixtures.ts`, or `e2e.config.ts` differs from `src/core/MANIFEST`, and `node src/core/manifest.ts --write` in the package directory regenerates the manifest. Nothing under `specs/` or `e2e.config.ts` imports the CLI, and `test/seam.test.ts` fails when a file does.
-- `src/host.ts`, `src/fixture.ts`, and `src/freshness.ts` are this repository's own: the build of the test app, the Metro ports, and the check that Metro serves current JS. `specs/app.ts` names the test app and its entry for a dev client, and `specs/native.ts` holds the per-platform locators for the native views and the locators of the home's links. Both are this repository's own too.
+- `src/core/`, `src/platform/ios/`, `src/platform/android/`, `specs/support/`, `specs/fixtures.ts`, `e2e.config.ts`, `testing/`, and every test but `test/host.test.ts`, `test/freshness.test.ts`, and `test/native-build.test.ts` are shared with the same package in clerk-ios and clerk-android. Change them there first, then copy them here. `doctor`'s `core-drift` check fails when `src/core/`, `specs/support/`, `specs/fixtures.ts`, or `e2e.config.ts` differs from `src/core/MANIFEST`, and `node src/core/manifest.ts --write` in the package directory regenerates the manifest. Nothing under `specs/` or `e2e.config.ts` imports the CLI, and `test/seam.test.ts` fails when a file does.
+- `src/host.ts`, `src/fixture.ts`, `src/native-build.ts`, and `src/freshness.ts` are this repository's own: the build of the test app, the native build that CI keeps between runs, the Metro ports, and the check that Metro serves current JS. `specs/app.ts` names the test app and its entry for a dev client, and `specs/native.ts` holds the per-platform locators for the native views and the locators of the home's links. Both are this repository's own too.
 - `npm test --prefix integration/expo-native` runs the CLI's unit tests, with no network, key, or device. `npm run typecheck --prefix integration/expo-native` runs `tsc`. The `Expo Native Runner Tests` job in `.github/workflows/ci.yml` runs both on Linux when a pull request changes the package, the test app, or a package the test app links.
 - `run --github-report` hands the results of the run to `@e2e-dev/github` as one report. The reporter writes the report to the job summary. With a `GITHUB_TOKEN` that may write pull request comments, it also posts one comment on the pull request and updates that comment on later runs. The reporter never changes the exit code, and nothing is reported for a run with a tainted file.
+- `.github/workflows/verify-e2e.yml`, the `Verify end-to-end tests` workflow, runs `up --backend local`, `run --all --retries 1 --github-report`, and `down` on a runner for each platform, with a device and a Clerk application for each. It starts on a pull request to `main` that changes the package, the test app, or one of the three packages the test app links. It runs for a pull request that is not a draft, and a draft or a pull request from a fork gets a notice instead. Start it by hand with `gh workflow run verify-e2e.yml --ref <branch>`. A failing spec shows on the pull request and is not required for a merge, and a test that fails and then passes on its one retry is `flaky` and does not fail the job. The runners are GitHub-hosted, `macos-26` and `ubuntu-24.04`, and the repository variables `VERIFY_CI_RUNNER_IOS` and `VERIFY_CI_RUNNER_ANDROID` name other labels. The Platform API key comes from the `MOBILE_VERIFICATION_PLATFORM_API_KEY` repository secret. The workflow sets `VERIFY_LOCAL_BUILD=standalone` and `VERIFY_NATIVE_CACHE` ([freshness.md](references/freshness.md)). It uploads each run's `run.json`, app log, video, screenshots, and e2e's `report.json`, `junit.xml`, summary, and failure pages for three days, and only when no secret is found in the run. `bin/boot-ios-simulators.sh wait` waits until a booted simulator is ready, and the workflow calls it.
+- The `Expo` workflow, `.github/workflows/expo-native-build.yml`, runs nothing on a device. It builds the test app as a Release app on Expo SDK 54, 55, and 57 for each platform, and runs the Android unit tests of `@clerk/expo-biometrics`.
 - `SKILL.md`, `references/`, and `features/` are in `.claude/skills/verify-clerk-expo/`. `.cursor/skills/verify-clerk-expo` is a symlink to that directory, so edit only the `.claude` copy.

.claude/skills/verify-clerk-expo/references/freshness.md

@@ -33,7 +33,31 @@ Fast Refresh stays on. When you save a JS change while an app from an earlier ru
 
 The check dates a Metro revision it has not seen before by the second it was built. If two edits land within the same second, or an edit is reverted while Metro's watcher is still behind, the check can, rarely, launch a bundle one edit older than `dist`. Touching files cannot force a newer revision, because Metro skips modules whose transform key did not change.
 
+## A standalone build has none of this
+
+The `Verify end-to-end tests` workflow sets `VERIFY_LOCAL_BUILD=standalone`, so its devices run a standalone Release app with the JS embedded, with no watch build, no Metro, and none of the three checks.
+
+## Troubleshooting
+
 ## Troubleshooting
 
 - A machine behind an HTTPS debugging proxy needs the proxy's CA trusted by `Clerk Verify Template iOS` before lanes are cloned from it. `doctor` reports it as `proxy-trust` and prints the fix.
 - If `up` says port 8082 (or another lane's port) already serves a Metro this worktree did not start, stop the process that listens on it: `lsof -nP -iTCP:8082 -sTCP:LISTEN` finds it.
+
+## For maintainers: the standalone build in CI
+
+`VERIFY_LOCAL_BUILD=standalone` makes a local lease build a standalone Release app in the working tree, with the JS embedded. `up` and `run` then start no watch build and no Metro, and the build key covers the JS inputs. Every JS edit then changes the key, so the next `up` or `run` builds again, and that build stops this worktree's Metro and watch build if a dev client left them running. Without `VERIFY_NATIVE_CACHE` it rebuilds the app natively. Unset, or set to `dev-client`, a local lease gets the dev client.
+
+`VERIFY_NATIVE_CACHE=<directory>` keeps the app of a standalone build in `<directory>/<platform>-<fingerprint>/`. The fingerprint covers what decides the native build:
+
+- That platform's native inputs, listed under [What forces a native rebuild](#what-forces-a-native-rebuild).
+- What `@expo/fingerprint`, which `expo` installs, reports for the installed fixture on that platform. That is the app config as Expo resolves it, the config plugin files of the workspace packages, and every native module that autolinking links.
+- The resolved version of every Expo and React Native package in the test app's `pnpm-lock.yaml`: `expo`, `expo-*`, `@expo/*`, `react-native`, `react-native-*`, `@react-native*/*`, and `hermes-*`.
+- The Xcode version on iOS.
+- `src/fixture.ts` and `src/native-build.ts`.
+
+A JS edit does not change it, and a new version of a JS-only package does not change it either. When `@expo/fingerprint` cannot be loaded, fails, or leaves out the app config or the autolinking result, the fingerprint covers the whole lockfile in place of the second and third items, and the build prints `native fingerprint covers the whole pnpm-lock.yaml, because <reason>`.
+
+A later standalone build with the same fingerprint runs no `expo prebuild`, `xcodebuild`, or Gradle. It builds the workspace packages, exports the bundle from the working tree with `expo export:embed`, compiles it with the test app's `hermesc`, and puts it in a copy of the kept app. The build fails unless the app then holds exactly the bundle it compiled. It builds natively instead, and prints `not reused:` with the reason, when the kept app holds no Hermes bundle, when it runs another Hermes bytecode version, when an Android bundle has image assets, or when the Android SDK has no build-tools 35 or newer to align the APK again. On Android the app is signed with the generated project's debug keystore when the working tree has one and with a new key otherwise, so a device that already holds the app under another key needs it uninstalled first.
+
+A job of the workflow that had to build natively stores the app as a run artifact named `verify-expo-native-<platform>-<fingerprint>` for seven days, and a later job takes it only from a run of the same branch of this repository. When there is none, the job builds natively. Unset, nothing is kept and every standalone build is a native build.

.github/actionlint.yaml

@@ -7,3 +7,13 @@ self-hosted-runner:
   labels:
     - blacksmith-8vcpu-ubuntu-2204
     - blacksmith-6vcpu-macos-26
+
+paths:
+  # actionlint 1.7.x predates GitHub's background steps and rejects them:
+  # https://github.com/rhysd/actionlint/issues/693
+  # Remove once actionlint supports those keys. Until then, a step in this file
+  # with neither `run` nor `uses` is not reported either.
+  .github/workflows/verify-e2e.yml:
+    ignore:
+      - 'unexpected key "background" for step to run shell command'
+      - 'step must run script with "run" section or run action with "uses" section'

.github/workflows/ci.yml

@@ -121,7 +121,6 @@ jobs:
         .github/workflows/ci.yml
         .github/actionlint.yaml
         integration/templates/expo-native
-        integration/tests/expo-native
         packages/expo
         packages/expo-biometrics
         packages/expo-google-signin