Progress on the Angle Comparison Functionality

An important functionality, as I talked about in my earlier post, is the angle comparison functionality to check if the live pose corresponds to the predefined angles recorded during exercise creation. In order to see if it can recognize a pose, I made a messy implementation in the dashboard component, setting the pre-recorded exercise in the localStorage. The initial implementation seems promising, with relatively successful comparisons. However, some precision and algorithmic issues need fixing before I can confidently say this works.

Dashboard.js

function Dashboard(){
    let exercise = JSON.parse(localStorage.getItem('exercise'));
    exercise = Object.keys(exercise).map(key => exercise[key]);
    let exerciseAccuracy = 0;
    let possibleCorrectPoses = exercise.length;
    let correctPoses = possibleCorrectPoses;

    let currentPoseIndex = 0;
    let maxPoseIndex = exercise.length;

    let poseStartTimeReference;
    let elapsedTime = null;

    let lastPoseStatus = false;

    let numberOfCorrectAngles = 13;

    let videoWidth, videoHeight;

    const webcamRef = useRef(null);
    const canvasRef = useRef(null);

    let poseLandmarker;
    let lastVideoTime = -1;

    let ctx;
    let drawingUtils;

    let video = null;

    const setupPrediction = async () => {
        const vision = await FilesetResolver.forVisionTasks("https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm");

        poseLandmarker = await PoseLandmarker.createFromOptions(
            vision,
            {
                baseOptions: {
                    modelAssetPath: "https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/1/pose_landmarker_lite.task"
                },
                runningMode: "VIDEO"
        });

        if (
          typeof webcamRef.current !== "undefined" &&
          webcamRef.current !== null &&
          webcamRef.current.video.readyState === 4
        ) {
        

          video = webcamRef.current.video;
          videoWidth = video.width;
          videoHeight = video.height;
          
          if(window.innerHeight >= window.innerWidth || window.matchMedia("(orientation: portrait)").matches){
            videoWidth = document.querySelector('.livefeed-wrapper').getBoundingClientRect().width;
            videoHeight = videoWidth;
          }
          else{
            videoHeight = document.querySelector('.livefeed-wrapper').getBoundingClientRect().height;
            videoWidth = videoHeight;
          }
          video.width = videoWidth;
          video.height = videoHeight;
          webcamRef.current.video.width = videoWidth;
          webcamRef.current.video.height = videoHeight;

        }
        else{
          console.log("Some value is undefined");
        }

        trackPose();
    };

    useEffect(() => {
        setupPrediction();
    }, [webcamRef]);

    const checkPoseMatch = (recordedAngles, correctAngles, threshold) => {
        let numberOfCorrectAngles = 13; 

        for (let i = 0; i < recordedAngles.length; i++) {
            const recordedAngle = recordedAngles[i];
            const correctAngle = correctAngles[i];

            if (Math.abs(recordedAngle - correctAngle) > threshold) {
                numberOfCorrectAngles--;
            }
        }
        if(numberOfCorrectAngles < 8){
            console.log('Wrong');
            lastPoseStatus = false;
            return false;
        }
        else{
            console.log('Correct');
            currentPoseIndex++;
            lastPoseStatus = true;
            elapsedTime = 0;
            return true;
        }
    };

    const trackPose = async () => {
        let startTimeMs = performance.now();

        if(!poseStartTimeReference){
            poseStartTimeReference = performance.now();
        }
        
        if (lastVideoTime !== video.currentTime) {
            lastVideoTime = video.currentTime;

            if(lastPoseStatus){
                poseStartTimeReference = performance.now();
            }

            if(elapsedTime >= 1){
                console.log('1 second has passed! Resetting timer.');
                elapsedTime = 0;
                poseStartTimeReference = performance.now();
                correctPoses--;
                currentPoseIndex++;
            }

            poseLandmarker.detectForVideo(video, startTimeMs, (pose) => {
                let tempAnglesArray = [];
                tempAnglesArray.push(calculateSetOfAngles(pose));

                if(currentPoseIndex < maxPoseIndex){
                    checkPoseMatch(tempAnglesArray[0], exercise[currentPoseIndex], 2);
                }

                if(currentPoseIndex === maxPoseIndex){
                    exerciseAccuracy = (correctPoses / possibleCorrectPoses) * 100;
                    console.log("Exercise accuracy: " + exerciseAccuracy + "%");
                }
            });

            elapsedTime = (performance.now() - poseStartTimeReference) / 1000;
        }

        requestAnimationFrame(trackPose);
    }

    return (
        ///
    );

export default Dashboard;

Next Steps

Of course, the next step is to clean up the code and implement a proper user interface for the dashboard and exercise creation. Key elements of the interface would be the possibility to record multiple exercises and store them in the localStorage, the page /exercises where the user can choose between existing exercises to test them; /createexercise should also get a revamp, adding the possibility to delete poses in order to “clean up” the exercise, and a redirection to the exercise pages.

It’s also necessary to add live user feedback to let them know if the position is correct and maybe a live tutorial that they can choose to skip or not (if they already know the exercise).

Deploying to Github Pages

I will use the package react-gh-pages in order to deploy my React app. The installation is already done but there is currently an issue with it – namely, it gives an error 404. After this, I can begin testing on mobile devices and adjust the project accordingly.