Ready-made scanners on top of the on-device vision API - #5575
Merged
Conversation
The vision analyzers replaced the QRScanner/codescan cn1libs but gave up
their one-call ergonomics. Scanning a QR code meant wiring Camera.open ->
CameraSessionOptions -> FrameListener -> VisionImage.fromCameraFrame ->
VisionPipeline -> back to the EDT, and the results came back as magic
strings ("leftEye", "QR_CODE") and normalized floats with no way to draw
them. Camera's own javadoc advertised a BarcodeScanner.scan(...) that has
never existed, which is roughly how discoverable the API was.
This adds the two layers that were missing above the analyzers, without
touching the analyzer contract:
CodeScanner.scan().ready(code -> { ... });
is a whole scanner screen -- camera, decode, restore the previous form.
It completes with null when the user backs out, so the cn1lib's three
ScanResult callbacks map onto one AsyncResource: scanCompleted is ready,
scanCanceled is a null value, scanError is except. CodeScannerOptions
rewords the screen and restricts the accepted symbologies, so a stray
product barcode in the frame does not complete a QR scan.
VisionCameraView is the same wiring as a component, for a preview inside
the application's own form, and works with every analyzer rather than
just barcodes. It opens the camera when shown and releases it when the
user navigates away; close() additionally releases the analyzer. The
analyzer stays caller-constructed on purpose -- the builders decide which
native dependency to package from the analyzer classes the *application*
references, so naming it in application code is what keeps a face app
from also carrying the barcode and pose models.
Reading a result no longer needs literals or arithmetic: BarcodeFormat,
FaceLandmarks and PoseLandmarks hold the names the backends normalize
onto, Face.getLandmark/Pose.getLandmark look one up, and
VisionRect.toBounds(component)/VisionPoint.toPoint(component) turn
normalized geometry into the pixels a paint method draws in.
SegmentationMask.cutOut(image, threshold) applies a mask to its source
image and rescales it -- selfie segmentation was unusable without that --
and VisionImage.fromFile/fromImage bridge a picked photo, passing an
EncodedImage's original bytes through instead of re-encoding.
The builders had to learn about the two new entry points, and this is the
part that fails silently. They scan the application's classes, not core,
so an app referencing only CodeScanner never names BarcodeScanner: the
Android adapter would be pruned and the iOS camera natives left out, and
the build would be green while the feature shipped inert on the device.
CodeScanner now selects the barcode adapter and the vision feature, and
CodeScanner and VisionCameraView both select the camera natives.
HighLevelVisionDependencyTest walks the vision sources rather than a
hand-written list, so a convenience class added later fails until it is
mapped.
The simulator returned null from createVisionImpl(), so every analyzer
reported itself unsupported on the desktop and a scanner screen could not
be built without a device. It now serves whatever is scripted under
Simulate > Vision, in the shape of the existing Simulate > NFC menu:
supported on/off, an outcome switch covering result, nothing-found,
UNSUPPORTED and backend error, and the scripted values themselves. The
results carry plausible geometry so overlay and debounce code can be
written before the app reaches hardware.
Samples are javadoc-level and guide-level. Every analyzer, every result
type that needs one, and package-info carry a markdown sample, and
VisionSnippets.java in docs/demos backs twelve tagged regions in
Ai-And-Speech.asciidoc -- compiled and bytecode-compliance-checked by CI,
so they cannot rot. Camera and CameraView lose the BarcodeScanner.scan
example that never compiled.
Two bugs the new tests found. Toolbar.setBackCommand(...) called before
Form.show() is dropped when the form's toolbar is set explicitly, which
left the Android hardware back button doing nothing on the scanner
screen; it is re-registered after show(). And the torch button originally
polled the EDT waiting for the camera session, where CameraInfo.hasFlash()
answers the same question synchronously before the form is shown.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5a835ae029
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Collaborator
Author
|
Compared 12 screenshots: 12 matched. |
Contributor
|
Developer Guide build artifacts are available for download from this workflow run:
Developer Guide quality checks: |
isSupported() on both CodeScanner and VisionCameraView asked Camera.isSupported(), which only reports that the port has a camera backend. A device with the backend and no capture hardware enumerates no cameras, so both promised a scan that start() would then fail with "No camera is available on this device". They now require an enumerated camera as well. VisionCameraView mirrored the preview from the *requested* facing. Camera.getDefault falls back to the first available camera when the requested one does not exist, so asking for FRONT on a device without a front camera opened the rear camera and showed the world reversed. The mirror now follows CameraInfo.getFacing() of the camera that actually opened. Both are covered by regression tests that fail against the previous code. The developer guide also failed its prose gates: Vale wants contractions in nine places, and LanguageTool does not know "backpressure" and reads the "0..1" range as two consecutive dots. Reworded rather than accept-listed, since neither term appeared in the guide before this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
✅ Continuous Quality ReportTest & Coverage
Static Analysis
Generated automatically by the PR CI workflow. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c046ddb43f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
Cloudflare Preview
|
Retaining an adapter source and setting usesCn1Camera is only half of the build wiring. PlatformFeatureCatalog is what actually adds the Gradle artifacts, CocoaPods, frameworks, permissions and privacy strings, and it keyed barcode scanning on BarcodeScanner and the camera on the com/codename1/camera/ prefix -- neither of which an application using the new high-level classes ever references. The result was worse than a missing feature. An app referencing only CodeScanner kept AndroidBarcodeScanningAdapter.java, which imports ML Kit, with no com.google.mlkit:barcode-scanning on the classpath: the generated Android project fails to compile. And an app referencing only CodeScanner or VisionCameraView got no CameraX, no CAMERA permission and no AVFoundation or NSCameraUsageDescription, so the preview opened on hardware the build never provisioned. CodeScanner now carries the barcode entry and its ML Kit iOS backend variant, and both camera-driving classes carry the camera entry. The camera entries deliberately omit RECORD_AUDIO and the microphone privacy string: both classes open their session with captureAudio(false), and a barcode scanner asking for the microphone is a privacy smell and an app-review question. They keep the full CameraX artifact list rather than trimming to preview-only, because AndroidCameraImpl reflects over the whole CameraX surface from one class. HighLevelVisionDependencyTest now checks the catalog and not just the adapter mapping, so the next convenience class fails there instead of shipping a project that does not compile. The ML Kit dependency lock count moves 26 -> 27 for the added barcode entry, whose Android floor is the same 21 every other ML Kit entry declares. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a5497e9924
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
CameraView.setMirrored and setScaleType stored their values and nothing read them: a repo-wide search finds no port consuming isMirrored() or getScaleType(). The simulator always stretched the frame, iOS is pinned to AVLayerVideoGravityResizeAspectFill, and Android constructs a bare PreviewView, so a front-facing preview rendered unmirrored while isMirrored() returned true. That predates this branch, but the new VisionCameraView surfaces both settings, and its just-corrected mirroring was invisible. CameraImpl gains setPreviewMirrored and setPreviewScaleType, defaulting to no-ops so no port breaks and a port that cannot honor them keeps today's behavior. CameraView forwards to them instead of only recording. The JavaSE preview now implements both: fitPreview computes the letterbox for FIT, the overflow for CROP and the stretch for FILL, and mirroring swaps the destination x coordinates. That is the target this branch can actually verify, and it is where a developer building a scanner on the desktop would otherwise see the wrong thing. iOS and Android are left alone on purpose. Both need native preview changes I cannot exercise from here, and blind native edits do not belong in a vision-API branch. The javadoc and the guide now say plainly which targets honor these settings and which currently ignore them, instead of implying they work everywhere. Covered by CameraSessionTest.previewSettingsReachThePortRatherThanOnlyThe Getter and by the VisionCameraView tests, which now assert the port saw the value rather than only that the getter agrees with itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ec6a4c05a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A port returns no preview peer when the native preview cannot be created: IOSCameraImpl returns null when the session or the view allocation fails, and AndroidCameraImpl returns null when CameraX cannot be resolved or the PreviewView reflection throws. CameraSession.createView() wraps that null without complaint, so start() installed an empty CameraView, reported isRunning() true, and began analyzing frames nobody could aim. Through CodeScanner that is a permanently blank scanner screen with no error ever delivered -- the failure mode a user cannot tell from a hung app. start() now treats a missing peer as a startup failure: it closes the session it just opened and reports through the listener, which CodeScanner turns into a failed AsyncResource and a return to the previous form. Only the peer is checked. createView() itself always returns a view, and SpotBugs correctly rejects a null check on it. The camera test double returned a null peer for every call, which is what let this path go unnoticed; it now returns a stand-in peer and reproduces the failure behind a flag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Compared 151 screenshots: 151 matched. Native Android coverage
✅ Native Android screenshot tests passed. Native Android coverage
Benchmark ResultsDetailed Performance Metrics
|
Collaborator
Author
|
Compared 181 screenshots: 181 matched. |
Collaborator
Author
|
Compared 148 screenshots: 148 matched. Benchmark Results
Detailed Performance Metrics
|
Collaborator
Author
|
Compared 144 screenshots: 144 matched. |
Collaborator
Author
|
Compared 149 screenshots: 149 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
Collaborator
Author
|
Compared 217 screenshots: 217 matched. |
Collaborator
Author
|
Compared 143 screenshots: 143 matched. Benchmark Results
Build and Run Timing
Detailed Performance Metrics
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The vision analyzers replaced the QRScanner/codescan cn1libs but gave up their one-call ergonomics. Scanning a QR code meant wiring
Camera.open→CameraSessionOptions→FrameListener→VisionImage.fromCameraFrame→VisionPipeline→ back to the EDT, and results came back as magic strings ("leftEye","QR_CODE") and normalized floats with no way to draw them.Camera's own javadoc advertised aBarcodeScanner.scan(...)that has never existed, which is roughly how discoverable the API was.This adds the two layers that were missing above the analyzers, without touching the analyzer contract.
One call
A whole scanner screen: camera, decode, restore the previous form. The cn1lib's three
ScanResultcallbacks map onto oneAsyncResource—scanCompletedisready,scanCanceledis anullvalue,scanErrorisexcept.CodeScannerOptionsrewords the screen and restricts the accepted symbologies, so a stray product barcode in the frame does not complete a QR scan.One component
VisionCameraViewis the same wiring as a component, for a preview inside the application's own form, and works with every analyzer rather than just barcodes. It opens the camera when shown and releases it when the user navigates away;close()additionally releases the analyzer.The analyzer stays caller-constructed on purpose: the builders decide which native dependency to package from the analyzer classes the application references, so naming it in application code is what keeps a face-detection app from also carrying the barcode and pose models.
Reading a result
No more literals or arithmetic.
BarcodeFormat,FaceLandmarksandPoseLandmarkshold the names the backends normalize onto;Face.getLandmark/Pose.getLandmarklook one up;VisionRect.toBounds(component)andVisionPoint.toPoint(component)turn normalized geometry into the pixels apaintmethod draws in.SegmentationMask.cutOut(image, threshold)applies a mask to its source image and rescales it — selfie segmentation was unusable without it — andVisionImage.fromFile/fromImagebridge a picked photo, passing anEncodedImage's original bytes through instead of re-encoding.The part that fails silently
The builders scan the application's classes, not core. An app referencing only
CodeScannernever namesBarcodeScanner, so the Android adapter would be pruned and the iOS camera natives left out — the build green, the feature inert on the device.CodeScannernow selects the barcode adapter and the vision feature, andCodeScannerandVisionCameraViewboth select the camera natives.HighLevelVisionDependencyTestwalks the vision sources rather than a hand-written list, so a convenience class added later fails the test until it is mapped.Simulator
createVisionImpl()returnednullon JavaSE, so every analyzer reported itself unsupported on the desktop and a scanner screen could not be built without a device. It now serves whatever is scripted under Simulate ▸ Vision, in the shape of the existing Simulate ▸ NFC menu: supported on/off, an outcome switch covering result / nothing-found /UNSUPPORTED/ backend error, and the scripted values themselves. Results carry plausible geometry, so overlay and debounce code can be written before the app reaches hardware.Samples
Every analyzer, every result type that needs one, and
package-infocarry a markdown javadoc sample.VisionSnippets.javaindocs/demosbacks twelve tagged regions inAi-And-Speech.asciidoc— compiled and bytecode-compliance-checked by CI, so they cannot rot.CameraandCameraViewlose theBarcodeScanner.scanexample that never compiled. The initializr skill reference is updated to match.Two bugs the new tests found
Toolbar.setBackCommand(...)beforeForm.show()is dropped when the form's toolbar is set explicitly, which left the Android hardware back button doing nothing on the scanner screen. Re-registered aftershow().CameraInfo.hasFlash()answers the same question synchronously before the form is shown.Verification
generate-quality-report.pyover SpotBugs/PMD/Checkstyle) exits 0. It caught one SpotBugs and 20 PMD findings in this code; all fixed rather than suppressed, except threeCloseResourcefalse positives that each carry a comment.check-cast-semantics.sh,check-package-info.shandvalidate-guide-snippets.py(676 blocks) pass; javadoc builds warning-free for these packages.Not included: the
scripts/cn1playgroundgenerated access registry (the playground's own Maven build regenerates it, and regenerating it here would pick up ~3,500 lines of unrelated pre-existing drift) and the legacySamples/samplesrunner.🤖 Generated with Claude Code