Detect
A YOLOv8n model, fine-tuned on 301 hand-annotated frames from Tom and Jerry episodes, locates a single "face" class in the frame with a bounding box.
01 / Introduction
Emotion recognition doesn't stop at human faces.
Facial emotion recognition is a well-studied problem for human faces, but cartoon characters present a different challenge entirely — exaggerated expressions, stylized proportions, and no existing off-the-shelf detector trained to find them. This project set out to build a complete pipeline that could locate Tom and Jerry's faces in a video frame and correctly label the emotion being displayed.
The work was carried out as a Final Year Project by a six-person team, combining classic dataset preparation (XML annotations, YOLO-format conversion, train/val/test splitting) with two independently trained deep learning models chained into a single inference pipeline.
02 / The problem
Cartoon faces break the assumptions most face detectors are built on.
Pretrained face detectors are trained almost exclusively on human faces and fail on stylized, non-human proportions like Tom and Jerry's. There was no ready-made dataset or model for this specific problem, so both the detection stage and the classification stage had to be built from the ground up — starting with raw video frames and hand-drawn bounding boxes.
On top of that, cartoon emotions are frequently more exaggerated and less consistent frame-to-frame than human expressions, which meant the classifier had to generalize from a wide variety of poses, art styles, and lighting within the show itself.
03 / The solution
A two-stage pipeline: detect, then classify.
Rather than training a single end-to-end model, the problem was split into two focused tasks chained together — a strategy grounded in prior academic research on cartoon emotion recognition (Jain et al., 2022), which showed that isolating the character's face before classification dramatically improves emotion accuracy.
Crop & isolate
The detected region is cropped with a small padding margin to preserve facial context, mirroring the segmentation-first approach used in the reference research.
Classify
The cropped face is passed to a fine-tuned MobileNetV2 network, which predicts one of five emotion labels: angry, happy, sad, surprised, or unknown.
SOLUTION / PIPELINE FLOWFRAME → YOLOv8 DETECTION → CROP → MOBILENETV2 → EMOTION LABEL
04 / Research and grounding
Built on a peer-reviewed foundation, adapted for a leaner build.
Understanding cartoon emotion using integrated DNNs
Jain et al. (2022, Neural Computing and Applications) trained Mask R-CNN for character segmentation and compared VGG16, ResNet-50, InceptionV3, and MobileNetV2 for emotion classification across 8,113 labeled Tom & Jerry images, reaching 96% accuracy with VGG16.
Isolate the face before classifying
The paper's strongest result was that isolating the character's face before classification was the single biggest driver of accuracy — far more than which CNN backbone was used. This directly shaped the decision to use a dedicated detection stage.
A lighter, faster stack
Instead of Mask R-CNN's pixel-level segmentation, YOLOv8n was used for bounding-box detection — faster to train and run, at the cost of not removing background pixels within the box. A single MobileNetV2 model was fine-tuned rather than benchmarking four backbones.
05 / Build process
From raw video frames to a trained, tested pipeline.
01Dataset prepExtracted frames, parsed XML bounding boxes, converted to YOLO format, split 80/10/10.
→
02Face detectorTrained YOLOv8n for 50 epochs on Tom & Jerry face annotations.
→
03Emotion classifierFine-tuned MobileNetV2 on emotion-labeled crops across 5 classes.
→
04IntegrationChained both models into a single detect-then-classify inference function.
BUILD PROCESS / TRAINING METRICS + CONFUSION MATRIXDATA PREP → TRAINING → EVALUATION → INFERENCE
06 / Results
Strong detection, near-perfect classification on held-out frames.
The face detector converged quickly and generalized well to unseen frames; the emotion classifier fit its training distribution tightly, correctly labeling a clearly angry Tom in an out-of-sample test image with a confident bounding box and label overlay.
07 / Limitations and scope
Honest about what a 301-frame, 50-epoch model can and can't do.
Small relative to the reference paper
301 annotated frames for detection and a modest emotion-labeled set is far smaller than the paper's 8,113-image benchmark, which naturally caps how well the model generalizes to unseen episodes or art styles.
Background pixels remain in the crop
Using YOLOv8 instead of Mask R-CNN trades some classification accuracy for speed and simplicity, since background context inside the box isn't removed the way a segmentation mask would remove it.
No cross-model benchmark
Only MobileNetV2 was fine-tuned for emotion classification, chosen for its lightweight footprint, rather than comparing it against VGG16, ResNet-50, and InceptionV3 as the reference paper did.
08 / Outcomes and reflections
A working, end-to-end proof that the two-stage approach holds up.
The finished pipeline reliably detects a cartoon face and predicts a plausible emotion label on new frames, validating the core idea borrowed from the reference research: separate detection from classification, and let each model specialize. The system runs entirely in Colab on free-tier GPU resources, making it easy to reproduce and extend.
The clearest next step is scaling the dataset and testing a segmentation-based detector to see how much of the accuracy gap to the published benchmark closes as a result.
Project details
Built in Colab, available on GitHub.
PythonPyTorchUltralytics YOLOv8OpenCVTorchvision / MobileNetV2Google Colab
View the project on GitHub ↗
Let's build the next system.
Tell us what you are trying to make clearer, faster, or more capable.
Start a conversation ↗