@Misaki I noticed that this body tracking works better when the whole body is within frame from my experience. It may not completely be your skirt's fault. If you have the chance could you test the body tracking wearing a skirt but have your foot within web camera frame. I think the body tracking would predict better results that way.

The body tracking software makes more assumptive predictions, like when your legs and arms are not in frame.

I might have the software check both if the knee and ankle are out of frame and prevent them from going swaying wildly. So far, the software checks those features individually.

I mentioned this on Discord. The next tracking I should tackle is hand gesture since I think it's straightforward. Various hand gestures: closed fist, open palm, pointing up, thumbs down, thumbs up, victory sign, IU sign. If the tracker detects any of the hand gestures with a high degree than swap out the hand components. I hope it is able to distinguish left and right hands.

2.0.2

• Add a pointing downward behavior for the Spine model legs when the tracked legs are outside the video frame boundary. The leg positions are more assumptive and less reliable when outside of video frame boundary. The legs should default to pointing downward ( standing position ) to look more natural than flailing legs.
• Start to implement web parallel processing for browsers that support it. Multiple threads might speed up computation tasks.
• Defer 3d graph plotting task to optimize canvas rendering.
• Save 3d plot graph camera location when user rotates the graph. Note that the user can only rotate the graph when the graph is not being updated ("Update 3D Plot" is unchecked).
• Replace the word "body" with "torso" in the animation names of "body_rotate", "body_scale_y", and "body_scale_x". Make these animation names less ambiguous.

https://silverstraw.itch.io/spine-vtuber-prototype-2

Test model updated.
https://silverstraw.itch.io/spine-vtube-test-model

Spine Vtuber Prototype 2.0.3

• Added a check box for enabling hand tracking. It is located under "AI Settings" menu > "AI Tracking Modes" section > "Hand Tracking".
• Added "Debug Bounding Boxes" checkbox under "Canvas Settings" menu. Axis aligned bounding box and bounding polygon have separate color and opacity settings. Bounding polygons would help visualize collision shapes.
• Fixed a lag bug caused from turning on and off the camera/video. The software activated body tracking too frequently in a short amount of time. Mitigated by slowing down the activation from microseconds to milliseconds.

https://silverstraw.itch.io/spine-vtuber-prototype-2

Spine Vtuber Prototype 2.0.4
Implemented hand gesture tracking to the application. Two hands closer to the goal.

https://www.deviantart.com/silverstraw/art/Spine2D-Vtuber-Hand-Gesture-Test-974400292

• Fixed "Showing Landmarks" canvas bug where it was not showing. The canvas was erasing faster than it can draw due to the 2.0.3 lag bug. Synchronized the erasing only when it is ready to draw the next frame.
• Added hand gesture tracking for closed fist, point up, victory sign, I love you sign, thumbs up. thumbs down. Hands open is assumed to be the default hand position.

https://silverstraw.itch.io/spine-vtuber-prototype-2
https://silverstraw.itch.io/spine-vtube-test-model

I updated and grouped the Spine animation names. The grouping is necessary as this project has grown to include various optional tracking modes.

https://itch.io/t/2951812/spine-animation-names#post-8031587

Standby Animations

1. idle
2. breathe

Face Animations

1. face_roll_left
2. face_roll_right
3. face_pitch_down
4. face_pitch_up
5. face_yaw_left
6. face_yaw_right
7. left_brow_raise
8. left_eye_open
9. left_pupil_pitch_down
10. left_pupil_pitch_up
11. left_pupil_yaw_left
12. left_pupil_yaw_right
13. mouth_height
14. mouth_width
15. right_brow_raise
16. right_eye_open
17. right_pupil_pitch_down
18. right_pupil_pitch_up
19. right_pupil_yaw_left
20. right_pupil_yaw_right

Body Animations

1. torso_scale_y
2. torso_scale_x
3. torso_rotate
4. left_upper_arm_scale
5. left_upper_arm_rotate
6. left_lower_arm_scale
7. left_lower_arm_rotate
8. right_upper_arm_scale
9. right_upper_arm_rotate
10. right_lower_arm_scale
11. right_lower_arm_rotate
12. left_upper_leg_scale
13. left_upper_leg_rotate
14. left_lower_leg_scale
15. left_lower_leg_rotate
16. right_upper_leg_scale
17. right_upper_leg_rotate
18. right_lower_leg_scale
19. right_lower_leg_rotate

Hand Gesture Animations

1. left_close_fist
2. right_close_fist
3. left_point_up
4. right_point_up
5. left_victory
6. right_victory
7. left_iloveyou
8. right_iloveyou
9. left_thumb_up
10. right_thumb_up
11. left_thumb_down
12. right_thumb_down

Spine Vtuber Prototype 2.0.5

• Optimized setting Spine animation for hand gestures only once. Prevent continuously setting Spine animation for the same hand gesture animation.

• Added background thread support for body tracking AI on supported modern web browsers. The background thread start up the AI immediately without hindering the main thread. Users would experience shorter or no wait after starting the body tracking for the first time.

https://silverstraw.itch.io/spine-vtuber-prototype-2
https://silverstraw.itch.io/spine-vtube-test-model

I am currently working on including a physics engine. Experimenting various ideas. There are physics constraints that have to be anchored to the Spine bones otherwise the physic bodies won't follow the Spine skeleton. There isn't any graphical user interface so that would have to be include before the release. Physics collision adds its own complexity and problems so I would have to add it at a later date.

Misaki Yes, several physic constraints could be used together to create various usages. In the video the ball is using gravity force that is pointing upward.

I'm going to try to support physics constraints for webgl runtime before official release.

https://silverstraw.itch.io/spine-vtuber-prototype-2

update 2.0.6

• Updated to run Spine 4.2 beta physics constraints. Include Spine 4.1 backward compatibility that 4.2 beta broke.
• Stopped drawing landmarks when face and body tracking modes under "AI Settings" menu are unchecked.
• Increased smoothing for body tracking because there is a lot more shaking from partial body than full body tracking.
• Fixed model number input that broke .svp saving process.

Patched HTML null bug and body tracking startup bug. Thank you, LustFire for notification.

https://silverstraw.itch.io/spine-vtuber-prototype-2

https://silverstraw.itch.io/spine-vtuber-prototype-2

update 2.0.7

• Updated runtime to Spine 4.2.35 webGL with 4.1 binary backward support.
• Moved menu options to the right of the model display at the request of user for easier live stream changes.
• Fixed an attachment variable name bug that gave application errors.

update 2.0.7 amendment

• Fixed physics constraint bug that did not update correctly.

https://silverstraw.itch.io/spine-vtuber-prototype-2

update 2.0.8

• The web application will switch between playing the eye blinking animation track and setting the track time based on tracking data. This should result in a smoother blinking.

• Added a "blinking speed" property into the model's "Single Value Properties" to control the play back speed of blink. At default it should be "2" which is about half a second of blinking animation ( 1 second / 2 play speed ). Note that the duration will be shorter because the animation track will stop at where the eye tracking state. This animation stop is to prevent jumping from full eyes wide open to eye tracking state after the animation track finishes.