Feature 1

Build out the first feature (tests then feature)

User story: User is able to move their hand in front of the webcam, to see the controller move on the screen.

This will most likely require using Handpose, and setting up a project similar to the Jenga game on Handsfree.js's website.

There will be 2 main components to this feature, based on the source code of the demo project above:

Need to have a JS to utilize Handsfree.js's Handpose model - may not be the hardest thing, except for the gestures I'll need to configure
Need to use Three.js to set up the scene, e.g. the ball that the user uses their hand controller to pick up, move around, drop, stack on top of other things, etc.
Need to make sure our project also includes HTML/CSS just to make sure the UI looks user-friendly and follows UX conventions.

My personal goal for the next week will be to focus on learning Three.js so we can handle #2,
then come back to read more on Handsfree.js so we can integrate the Handpose model,
and then finally we'll add Bootstrap for the HTML/CSS.

Engineering Standup: January 18, 2021

Yesterday: Added the HelloCube project, which now contains the code we can use to add a rotating cube to the scene.
Today: will need to go through more of the Three.js fundamentals, and learn more about creating a scene we can use the hand controllers in (from Handsfree.js).
Blockers: the milestones left to complete Feature 1 is mainly 1) having interactive Three.js scenes, and 2) being able to move the camera in the scene. Still need to go through more Three.js docs before I understand that though.

Engineering Standup: January 21, 2021

Previous: Updated the demo HelloCube project to use CSS style, so the UI is more user-friendly (adding sidebars, and making the site more responsive).
For Now: continue through more of the Three.js fundamentals and learn more about creating an interactive scene we can use the hand controllers in (from Handsfree.js).
Blockers: Still need to go through more Three.js docs before I understand enough of the details

Engineering Standup: January 31, 2021

Posting a link to the new WebXR Chrome extension for Handsfree.js for better visibility. Hopefully this will enable better gestures to be added later to the project!

No other progress to report currently.

Engineering Standup: February 7, 2021

Previous: learning from the sample Jenga game on Handsfree.js.
Today: added HelloAnimations, a demo Three.js app that includes interactive controls (press keys 1-8 to make the characters do different movements).
Blockers: this demo app helped me learn some new things about the GLTF file format, so I will continue learning how to properly load *.gltf files using Three.js in the near future. Otherwise, the next step will be making a Three.js app which a user can interface with using hand gestures (most likely using Handsfree.js).

Engineering Standup: February 14, 2021

Previous: learning about how to do object picking in Three.js
Today: adding gestures so we can make hand interactivity a part of Feature 1 for IQ3's product.
Blockers: need to determine how to implement our custom hand controllers by extending the Tensorflow Handpose model.

Engineering Standup: February 17, 2021

Previous: added a sample app to the repo for picking objects
Next Steps: integrating Handsfree.js with the WebXR extension, to enable head gestures for feature 1
Blockers: finding a way to run the sample app from the Handsfree.js repo, built to handle this exact task.

Engineering Standup: February 20, 2021

Previous: downloaded Handsfree.js Chrome extension( (in dev mode) on my personal laptop
Today: added a demo of Handsfree.js face tracking: "Inside the Cube"
Next Steps: finding a way to use multiple Handsfree.js models (for tracking the user) at once, and having them manipulate objects.

Engineering Standup: February 20, 2021

Previous: studied the Handsfree.js docs on the Weboji model, which was utilized in the "Inside the Cube" demo.
Today: Initialized a new demo to do the same thing as the last demo, except it uses Three.js instead of AFrame.
Next Steps: improving 1) smoothness of the camera movements, 2) the frame rate, and 3) possibly the responsiveness of Google MediaPipe. Will be keeping track in Fix Latest Demo on Hands and Face Tracking #5

Engineering Standup: March 7, 2021

Previous: experimented with different tweeting values through trial and error
Today: released "Cube Space", a new sample app for connecting Handsfree face tracking with Three.js.
Blockers: not sure how to balance between responsiveness and accuracy - we can make the face tracking more responsive by decreasing the time spent tweening, however, it comes at the price of making the camera tracking more unstable (which is probably the fault of the Weboji model in Handsfree.js). On the other hand, it's also a bad UX to make the face tracking too stable, as it can get so slow the user will feel like it's stopped being responsive.
Next Steps: move ahead with object interactions for now, by integrating the Handpose model from Handspose into the app. This way we'll be able to manipulate the coordinates of objects using a finger caster or something like that. If time, we can also add kinematics by using Physijs, a plugin for Three.js.

Engineering Standup: April 1, 2021

Previous: read through more of the Handsfree.js documentation.
Today: Fixed the error in the hand not being recognized. The source code for the hand tracking script was using a deprecated name to reference the model (hands, instead of handpose).
Blockers: The app is a lot slower now that it is running Tensorflow.js. Will probably need to switch to using the CPU-only version of this package. This will probably require overriding the handsfree dependency as well, so we might need to using a Node.js backend.
Next Steps: Going to take another look at the documentation on Tensorflow.js page on NPM, to see how to import it as easily as possible.

Engineering Standup: April 2, 2021

Previous: read more on the documentation of the Handpose model itself on GitHub, and learned a couple things:
1. The Tensorflow.js library itself has separate backends for CPU, WebGL, and wasm aka WebAssembly.
2. The Handpose model only allows wasm or webgl backends.
3. In the Handsfree.js library, you are only allowed to use webgl
Today: I turned off weboji, the face tracking model, in case that would speed up the app when using the hand tracking model, handpose. It did not, still very laggy.
Blockers: I think we may need to put object interactions on hold until we find a way to do one of the following to improve the performance of the hand tracking model. Some of the options for doing this include:
1. Add support for a CPU/possibly WebGPU backend to the handpose model (started a discussion in the Tensorflow.js Google group about this)
2. Maybe add support for the Handpose model to use the wasm backend in Handsfree.js, and maybe that improves performance. Started an issue on that here
3. Throw the handpose model out entirely. Try to make object interactions work with the 2D hands model, or a model made in another frontend deep learning library entirely. We would start with client-side frameworks for the web, like perhaps ConvNetJS, and if need be maybe even go outside the browser and see if it's worth doing this on mobile, desktop, etc.
Next Steps: Going to try one more thing here, no reason to give up just yet:
1. Going to see if we can activate the CPU backend a different way using environment variables, as described in this Stack Overflow answer.
2. If that fails, we'll see if we can make two separate buttons to activate only face or hands tracking for now, and see if we can work on another feature in the time being.

Engineering Standup: April 3, 2021

Previous: read the Stack Overflow answer above, and learned how to import modules via a CDN and how to use the jsDelivr.
Today: I subclasses a version of the handpose model that would have better performance on the CPU.
Blockers: However I have not been able to use it in the app yet, due to some errors in importing the fingerpose library which the handpose models needs in order to work. This is because so far my approach has been to import it via jsDelivr, however it isn't always able to find files if those aren't imported with their file extension (an example of this is here).
Next Steps: going to try to just save the entire fingerpose library locally, see if that resolves the dependency issues, and if that in turn fixes the performance issues.

Engineering Standup: April 5, 2021

Previous: read the documentation eye-tracking models such as WebGazer.js, and the FaceMesh model from Tensorflow.js.
Today: I added another basic demo app for just the FaceMesh model. Just to see how it tracks "landmarks", which are the coordinates of different parts of a face.
Blockers: still not sure if this model will be able to run smoothly when also running along inside of a Three.js environment.
Next Steps: going to focus on emotion detection this week, and will try to reduce the load on the CPU by using FaceMesh to both do face-tracking and emotion display as in issue Feature 3: Emotion Detection and Animations #12

Engineering Standup: April 7, 2021

Previous: N/A
Today: Re-implemented face tracking using the FaceMesh model, and it is now available inside of the faceAndHands app.
Blockers:
1. for some reason the change of model from weboji to facemesh seems to have moved the debug window from the center of the top of the DOM, down to the bottom left corner.
2. Also not really a blocker, but while the facemesh model is much more responsive than weboji, it is also seems incapable of recovering if the user moves their head too quickly out of view (such as really far to the left/right).
3. Lastly, the facemesh model does not return a Z-coordinate, at least not in the landmark module. So for now I have set the z-coordinate permanently at a value of 8 - however this value is hard-coded and will have to be changed for every single different environment we use.
Next Steps: going to clarify how emotion detection and eye tracking will work, work on those, and then eventually circle back to hand tracking.

Engineering Standup: April 20, 2021

Previous: worked on Feature 3: Emotion Detection and Animations #12
Today: initialized functions for dragging the box across the screen.
Blockers: not sure how to make sure just one and only one box is selected at a given time.
Next Steps:
1. Implement plugin for hand tracking - fire MouseEvents for when the box is selected (mousedown), when it is being moved (mousemove), and when it is released (mouseup).
2. Add functions in cubes.js so the boxes respond to those events
3. Work on the performance of the handpose and facemesh model working together
4. highight wherever the hand is on the screen at any given moment
5. move the debuggers to the top of the screen, and as a stretch overlay them
6. Update the GIF on the home page, to match the UI updates

Engineering Standup: May 1, 2021

Previous: worked on Feature 3: Emotion Detection and Animations #12
Today: re-initialized functions for dragging the box across the screen.
Blockers: the example of the Jenga game seems to have a deprecated API. Particularly, I will need to find the property that has replaced the hand.pointer property in the version we are using of Handsfree.js
Next Steps:
1. Refactor the trackHand function to use the newer API. UPDATE: for now we can just initialize the pointer ourselves using a JS Map object.
2. Add functions in cubes.js so the boxes respond to those events
3. Work on the performance of the handpose and facemesh model working together
4. highight wherever the hand is on the screen at any given moment
5. move the debuggers to the top of the screen, and as a stretch overlay them
6. Update the GIF on the home page, to match the UI updates

Engineering Standup: May 2, 2021

Previous: fixed the issue in initializing hand.pointer
Today: refreshed memory on how to do object picking, so we can highlight the pointer
Blockers: we'll need to look more at the Three.js docs, to see how we can color the pointer on the 2D screen that the index finger is located on. Then we can illustrate that the user's hand is being detected the way we think it is by the handpose model.
Next Steps:
1. highlight wherever the hand is on the screen at any given moment - for this we can use the first two values in the handpose.annotations.indexFinger[3] array
2. Add functions in cubes.js so the boxes respond to those events
3. Work on the performance of the handpose and facemesh model working together
  1. one idea might be reducing the number of unused Three.js objects, such as those in handpose.model.three
4. move the debuggers to the top of the screen, and as a stretch overlay them
5. Update the GIF on the home page, to match the UI updates

Quick update: for now I'll switch to highlighting a single pointer rather than the whole handpose model - it might keep things simpler for the user since they'll know that's what the app primarily cares about when detecting their gestures.

We'll need to UI/UX test this to confirm of course, date TBD.

Engineering Standup: May 6, 2021

Previous: read through docs and tutorials on how picking works in Three.js
Today: shelved making the pointer appear for now, added boilerplate code to handle object picking based on the handpose model, using one of the Three.js tutorials.
Blockers: need to make sure we come up with a unique event for calling the object picker, so it doesn't confuse a regular mouse movement or finger touch on the screen for the handpose model.
Next Steps:
1. highlight wherever the hand is on the screen at any given moment - for this we can use the first two values in the handpose.annotations.indexFinger[3] array
2. Add functions in cubes.js so the boxes respond to those events - TEST this tomorrow
3. Work on the performance of the handpose and facemesh model working together
  1. one idea might be reducing the number of unused Three.js objects, such as those in handpose.model.three
4. move the debuggers to the top of the screen, and as a stretch overlay them
5. Update the GIF on the home page, to match the UI updates
6. see if we need to add additional EventListeners for mobile users?

Resource: another good read for managing multiple canvases in Three.js - might help with optimizing performance in the future.

Engineering Standup: May 7, 2021

Previous: read through docs and tutorials on how mouse events work in JS
Today: initialized an event handler to drag the cubes via the mouse - causes the cubes to flash red/yellow while selected.
Blockers: for some reason the event handler is not actually updating the coordinates of the cube object itself, and also they need to be normalized
Next Steps:
1. Fix the mousemove event handler in cubes.js
2. highlight wherever the hand is on the screen at any given moment - for this we can use the first two values in the handpose.annotations.indexFinger[3] array
3. use the handpose model in place of the mouse, to be able to select, drag, and de-select the cubes.
4. Work on the performance of the handpose and facemesh model working together
  1. one idea might be reducing the number of unused Three.js objects, such as those in handpose.model.three
5. move the debuggers to the top of the screen, and as a stretch overlay them
6. Update the GIF on the home page, to match the UI updates
7. see if we need to add additional EventListeners for mobile users?

Quick update, no progress as of today except I discovered the DragControls class in Three.js can be a useful tool to look into using to implement mouse-based object drag-and-drop.

Engineering Standup: May 9, 2021

Previous: read more on the DragControls class provided by Three.js
Today: found two alternatives to use the above tool, so we have more options: ThreeDragger and three-dragcontrols. Both of these were created by Qingrong Ke.
Blockers: currently looking into using the ThreeDragger as well, and I will need to look more into how and where NPM installs Node dependencies. This is because even though I installed, we won't be able to use it unless there's a relative file path to the module in the code. The specific arises at the top of cubes.js, on the import line:

import ThreeDragger from 'three-dragger';

And the error message on the Inspect tool says:

Uncaught TypeError: Failed to resolve module specifier "three-dragger". Relative references must start with either "/", "./", or "../".

Next Steps:
1. Fix the drag-n-drop and mousemove event handlers in cubes.js
2. highlight wherever the hand is on the screen at any given moment - for this we can use the first two values in the handpose.annotations.indexFinger[3] array
3. use the handpose model in place of the mouse, to be able to select, drag, and de-select the cubes.
4. Work on the performance of the handpose and facemesh model working together
  1. one idea might be reducing the number of unused Three.js objects, such as those in handpose.model.three
5. move the debuggers to the top of the screen, and as a stretch overlay them
6. Update the GIF on the home page, to match the UI updates
7. see if we need to add additional EventListeners for mobile users?

Engineering Standup: May 11, 2021

Previous: read more on tools to use for drag-n-drop beyond DragControls.
Today: experimented in implementing this using vanilla Javascript, as discussed in this blog post.
Blockers: in the spirit of Thomas Edison, I have found out more ways to not implement this feature:
1. three-dragcontrols seems to just be a clone of the DragControls that come with Three.js.
2. ThreeDragger doesn't look like a good solution anymore, because it doesn't have solely relative dependencies it would be unrealistic to use it without our own server and without storing all the source files we'd ever need on our own machine.
3. turning off the cube rotation and/or the facemesh models also didn't seem to help either. Neither did calling the activate function on the controls object.
4. Didn't work: removing use of the PickHelper and trying to do all the cubes responses to the mouse happen w/ the DragControls.
Next Steps: (same as above).

Engineering Standup: May 12, 2021

Previous: read more on tools to use for drag-n-drop beyond DragControls.
Today: experimented with going back to object picking.
Blockers: so object picking is working as a way to drag the cube, however it only moves by an inch or so.
1. I believe this is happening because the function is not being continually called, so the way forward might be in looping through this function somehow (without interrupting the main render loop of course).
2. And then of course we also need to make sure we edit the XYZ coordinates that the cube object is actually storing, not just temporarily animate it to go to a specific point in 3D space (which it seems is what Tweening is doing).
Next Steps: (same as above).

Navigation Menu

Repository navigation

Feature 1 #1

8 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Feature 1 #1

Description

Activity

UPstartDeveloper commented on Jan 15, 2021

UPstartDeveloper commented on Jan 15, 2021

UPstartDeveloper commented on Jan 15, 2021

UPstartDeveloper commented on Jan 18, 2021

UPstartDeveloper commented on Jan 22, 2021

UPstartDeveloper commented on Jan 31, 2021

UPstartDeveloper commented on Feb 7, 2021

UPstartDeveloper commented on Feb 15, 2021

UPstartDeveloper commented on Feb 17, 2021

UPstartDeveloper commented on Feb 20, 2021

UPstartDeveloper commented on Feb 27, 2021

UPstartDeveloper commented on Mar 7, 2021

8 remaining items

UPstartDeveloper commented on Apr 1, 2021

UPstartDeveloper commented on Apr 2, 2021

UPstartDeveloper commented on Apr 3, 2021

UPstartDeveloper commented on Apr 5, 2021

UPstartDeveloper commented on Apr 7, 2021

UPstartDeveloper commented on Apr 20, 2021

UPstartDeveloper commented on May 1, 2021

UPstartDeveloper commented on May 2, 2021

UPstartDeveloper commented on May 6, 2021

UPstartDeveloper commented on May 6, 2021

UPstartDeveloper commented on May 7, 2021

UPstartDeveloper commented on May 8, 2021

UPstartDeveloper commented on May 9, 2021

UPstartDeveloper commented on May 11, 2021

UPstartDeveloper commented on May 12, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions