What is Transfer Learning?
While pre-trained models are great for recognizing cats and dogs, or cars and bicycles, very often, we want Machine Learning to do something more elaborate. Transfer Learning method utilizes intermediate layers (or in simple ML terms “features”) from a pre-trained model which is trained on a very large dataset. As a result, it achieves decent accuracy even though the new training set used for customization is relatively much smaller in size.
In this tutorial, I demonstrate how to build a knn-classifier for sports activity recognition task, using Transfer Learning from a pre-trained model Mobile-Net in TensorFlow.JS.
Step 1: Compile Sports Activity Video for Training & Testing
To train the classifier, we need some sample videos that demonstrate various sports activities we want to detect automatically using Machine Learning / Transfer Learning.. For this project, I have created 2 videos: one for training the classifier and the other for testing the classifier output. It looks something like this:
Video clips are taken from the Olympics YouTube channel for sports activities: [swimming, boxing, figure skating, horse riding] and World Surf League channel for surfing videos.. You can also check out other YouTube channels like: Tour de France, Formula 1, Red Bull for motor sports, car racing, cycling etc.
Step 2: Load Training Video & Capture Few Screenshots
Create activity-recognizer.html file in code editor. The following javascript function pauses the input video stream, grabs a video frame / screenshot, and prints it on HTML canvas..
The HTML page user-interface looks something like this:
where, the video is loaded in the left pane, and the screenshot is displayed on the right.. Below the video & canvas objects, there are sevral buttons to capture the screenshot, tag frames with activity labels, and run predictions..
In the HTML code, you will need to add the following event handlers for button clicks to take the required actions..
Step 3: Tag few Examples Manually to Train the Classifier
The following add_example() function takes the image from HTML canvas, extracts features using a pre-trained Mobile-Net model and adds a new training example for a given class-label to knn-classifier in TensorFlow.JS.
Note: Make sure the video frames captured for training include variations in camera angles or perspectives (front / top / side views, close-up vs distant shots), backgrounds, athlete costumes, weather conditions for outdoor sports (rainy, cloudy, sunny) etc..
Once the classifier is trained on few representative samples for each activity, we are good to go and test the model output!
Step 4: Run Predictions using knn-Classifier
Load the sample test-video, and run predictions using the knn-classifier trained above.
Predicted class-labels for the activities recognised in the video are printed on the HTML canvas as shown below:
Final Output:
Full HTML / JavaScript code can be found at:
What’s Next:
- The present method uses simple image classification using intermediate layers from a pre-trained model. One can extend this basic model using body postures of athletes as captured by Pose-Net or Open Pose.
- The above method classifies each video frame individually without taking into account any temporal information.
References:
