Exploring Motion Tracking for a Cross-Platform Fitness App: A Progress Update

In my ongoing attempt to develop a fitness app with motion-tracking capabilities, I’ve looked into various frameworks compatible with Ionic or React Native because I am more interested in cross-platform app development frameworks. As I already said, I won’t be pursuing Flutter and Swift, focusing instead on finding the right fit for Ionic or React Native.

After trying different frameworks and libraries, Tensorflow.js seemed like a promising choice, based on the task at hand and the time constraints.

Setting Up Tensorflow.js with React Native

The initial step meant following the guidelines provided by the Tensorflow.js team on the tfjs-react-native repo. It’s necessary to add, before installing tfjs and tfjs-react-native, to install and configure react-native-unimodules (I skipped this because I use Expo), expo-gl, expo-camera, async-storage, react-native-fs.

Next, I converted their example to functional components and ran it. As the tutorial says, I did get the error “You may need an appropriate loader to handle this file type…“, so I implemented their fix.

1. Run expo customize:web
2. Choose webpack.config.js
3. Open webpack.config.js and replace the contents with the following:

const createExpoWebpackConfigAsync = require('@expo/webpack-config');

module.exports = async function(env, argv) {
  const config = await createExpoWebpackConfigAsync(
      {
        ...env,
        babel: {
          dangerouslyAddModulePathsToTranspile: [
            // Ensure that all packages starting with @tensorflow are
            // transpiled.
            '@tensorflow',
          ],
        },
      },
      argv);
  return config;
};

The next step was to install Pose Detection models from @tensorflow-models/pose-detection. I then tested the model by loading it through the following code snippet:

let model;
let detector; 

const setPoseModel = async () => {
    model = poseDetection.SupportedModels.MoveNet;
    detector = await poseDetection.createDetector(model).then((response) => console.log(response));
}

useEffect(() => {
    console.log(model);
}, []);

Now that I finally had a framework that worked nicely with React Native, I then had to test it with something more advanced.

Testing Pose Detection with an Official Example

Moving forward, I ran the official example for pose detection from the tfjs-examples repository. While the example proved functional, there was a noticeable lag on mobile, and the camera failed to display. The web version, though operational, showed issues in registering FPS and in displaying the camera.

So, as a conclusion, my immediate task is to address the lag and camera display issues in order to make a usable demo.

If you’d like to follow the project, here’s the Github repo: Fitness Tracking.