Sports Activity Recognition

Demo: Sports Activity Recognition using Transfer Learning

What is Transfer Learning?

While pre-trained models are great for recognizing cats and dogs, or cars and bicycles, very often, we want Machine Learning to do something more elaborate. Transfer Learning method utilizes intermediate layers (or in simple ML terms “features”) from a pre-trained model which is trained on a very large dataset. As a result, it achieves decent accuracy even though the new training set used for customization is relatively much smaller in size.

In this tutorial, I demonstrate how to build a knn-classifier for sports activity recognition task, using Transfer Learning from a pre-trained model Mobile-Net in TensorFlow.JS.

Step 1: Compile Sports Activity Video for Training & Testing

To train the classifier, we need some sample videos that demonstrate various sports activities we want to detect automatically using Machine Learning / Transfer Learning.. For this project, I have created 2 videos: one for training the classifier and the other for testing the classifier output. It looks something like this:

Image 1: Sample video to demonstrate sports activities

Video clips are taken from the Olympics YouTube channel for sports activities: [swimming, boxing, figure skating, horse riding] and World Surf League channel for surfing videos.. You can also check out other YouTube channels like: Tour de France, Formula 1, Red Bull for motor sports, car racing, cycling etc.

Step 2: Load Training Video & Capture Few Screenshots

Create activity-recognizer.html file in code editor. The following javascript function pauses the input video stream, grabs a video frame / screenshot, and prints it on HTML canvas..

Image for post
Image for post
Image 2: Capture screenshot to grab a video frame

The HTML page user-interface looks something like this:

Image for post
Image for post
Image 3: User-Interface to Take Screenshot, Label Frames & Predict Activities

where, the video is loaded in the left pane, and the screenshot is displayed on the right.. Below the video & canvas objects, there are sevral buttons to capture the screenshot, tag frames with activity labels, and run predictions..

In the HTML code, you will need to add the following event handlers for button clicks to take the required actions..

Image for post
Image for post
Image 4: Event Handlers for HTML Buttons

Step 3: Tag few Examples Manually to Train the Classifier

The following add_example() function takes the image from HTML canvas, extracts features using a pre-trained Mobile-Net model and adds a new training example for a given class-label to knn-classifier in TensorFlow.JS.

Image for post
Image for post
Image 5: Add Training Examples to KNN-Classifier

Note: Make sure the video frames captured for training include variations in camera angles or perspectives (front / top / side views, close-up vs distant shots), backgrounds, athlete costumes, weather conditions for outdoor sports (rainy, cloudy, sunny) etc..

Once the classifier is trained on few representative samples for each activity, we are good to go and test the model output!

Step 4: Run Predictions using knn-Classifier

Load the sample test-video, and run predictions using the knn-classifier trained above.

Image for post
Image for post
Image 6: Run Predictions to Recognize Sports Activities using KNN-Classifier

Predicted class-labels for the activities recognised in the video are printed on the HTML canvas as shown below:

Image for post
Image for post
Image 7: Olympics Figure Skating
Image for post
Image for post
Image 8: Olympics Swimming

Final Output:

Demo Video: Sports Activity Recognition using Transfer Learning

Full HTML / JavaScript code can be found at:

What’s Next:

  • The present method uses simple image classification using intermediate layers from a pre-trained model. One can extend this basic model using body postures of athletes as captured by Pose-Net or Open Pose.
  • The above method classifies each video frame individually without taking into account any temporal information.

References:

Written by

Creative Media Technology || Website: pamruta.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store