Interactive ML (Teachable Machines)

[whuang37]

Description

Port of teachable machines for hand pose and sound. Everything works off static poses for now so nothing dynamic.

Most the code is in the demo + tflite export. the future is probably fitting this in with the detectors in the SDK but since its an addon had to rewrite a tflite export/import for that.

import * as xb from 'xrblocks';
import {
  captureHand,
  HandTrainer,
  Predictor,
} from 'xrblocks/addons/interactive-ml/index.js';

const trainer = new HandTrainer();
// Read the SDK's existing hand tracking from your Script.update().
const frame = captureHand(user.hands, xb.Handedness.RIGHT, performance.now());
if (frame) trainer.addExample('open', [frame]);
// Repeat for other poses, such as 'closed', with several takes per class.
const model = await trainer.train();
const result = model.predictHand([freshFrame]);
// result: {label, score, scores}. label is null when the pose is unknown.

import {
  SoundTrainer,
  YamnetExtractor,
} from 'xrblocks/addons/interactive-ml/index.js';

const extractor = await YamnetExtractor.load(tf); // Supply TensorFlow.js.
const trainer = new SoundTrainer(extractor);
await trainer.addExample('clap', clapClip);
await trainer.addExample('background', backgroundClip);
const model = await trainer.train();
const result = await trainer.predict(model, freshClip);

Type of Change

• Bug fix
• New feature / enhancement
• New demo or sample
• Documentation update

Media / Screen Recordings & Screenshots (If Applicable)

• Simulator Recording:
• Device Recording:

teachable-machines.mp4

Checklist

• [ x Tested in simulator & device: Verified functionality in desktop simulator and/or physical hardware (where applicable).
• Large Assets ($\ge$ 1MB): Submitted separately to xrblocks/proprietary-assets via jsdelivr CDN.
• SDK Dynamic Dependencies: All new SDK dependencies are dynamically loaded at runtime.
• Security: Confirmed no hardcoded API keys or secrets are committed.

[ruofeidu]

put it to constants.ts for easier maintenance and updates?

[ruofeidu]

add a console.error

[ruofeidu]

nit: consider to centralize to constants.ts in this folder?