Context
I wanted to explore how deep learning models learn representations from sequential video data and how architectural choices influence classification performance. By comparing LSTM, GRU and bidirectional GRU networks under a controlled experimental framework, this project examined the role of temporal modeling in computer vision applications.
Can temporal modeling improve the classification of swimming strokes from video?
Video-based human action recognition requires models to capture both spatial appearance and temporal motion patterns. Swimming stroke classification is particularly challenging because of differences in swimmer technique, camera angle, recording quality and limited training data. The challenge is to determine whether different temporal modeling architectures can generalize effectively to swimming videos not seen during training.
Approach
This project evaluated CNN-RNN architectures that combine a frozen ResNet18 backbone for spatial feature extraction with LSTM, GRU and bidirectional GRU networks for modeling movement across video sequences. Each architecture received the same frame-level feature representations and experimental conditions, allowing the comparison to focus on differences in temporal sequence modeling.
Data Sources
Public Swimming Instructional Videos
Publicly available swimming videos were collected and segmented into labeled stroke sequences representing four swimming techniques: backstroke, breaststroke, butterfly and freestyle. The resulting clips formed the dataset used for model training and cross-video evaluation.
Solution
Developed a preprocessing pipeline to extract and standardize swimming video clips.
Created a labeled dataset containing 411 temporal clips across four swimming stroke classes.
Represented each sample as a 64-frame temporal sequence for model input.
Used a pretrained frozen ResNet18 backbone to extract frame-level spatial features.
Implemented CNN-LSTM, CNN-GRU and CNN-BiGRU architectures using PyTorch.
Applied Leave-One-Video-Out cross-validation to evaluate generalization across unseen videos.
Compared model performance using accuracy, macro F1-score and class-level classification metrics.
Architecture
The model architecture combines convolutional feature extraction with recurrent temporal modeling. A pretrained frozen ResNet18 backbone extracts 512-dimensional feature embeddings from individual video frames. These embeddings are passed into recurrent networks to model temporal dependencies across fixed-length swimming stroke sequences before final classification.
- 01Frozen ResNet18 CNN feature extractor
- 02CNN-LSTM temporal model
- 03CNN-GRU temporal model
- 04CNN-BiGRU bidirectional temporal model
- 05Leave-One-Video-Out cross-validation evaluation
Outcome
CNN-BiGRU achieved 50% accuracy and a 0.50 macro F1-score, while CNN-LSTM achieved 51% accuracy and a 0.49 macro F1-score. CNN-GRU achieved 38% accuracy and a 0.35 macro F1-score. The results showed meaningful differences in classification performance across the temporal architectures, with the bidirectional model providing the strongest balance of class-level performance under the experimental conditions. The results also highlighted the challenges of video classification with a limited dataset and the importance of evaluating generalization across unseen source videos.
Experiment 1 — Validation Loss Curves
Training and validation loss across the CNN-LSTM experiment, showing model convergence over the training process.
Experiment 1 — Confusion Matrix
Class-level predictions from the CNN-LSTM experiment, showing where swimming stroke classifications were correct and where errors occurred.
Experiment 2 — Validation Loss Curves
Training and validation loss across the CNN-GRU experiment, providing a view of convergence and validation behavior.
Experiment 2 — Confusion Matrix
Class-level predictions from the CNN-GRU experiment, highlighting the distribution of correct classifications and errors across stroke categories.
Experiment 3 — Validation Loss Curves
Training and validation loss across the CNN-BiGRU experiment, showing convergence behavior for the bidirectional temporal model.
Experiment 3 — Confusion Matrix
Class-level predictions from the CNN-BiGRU experiment, showing how the bidirectional temporal model distinguished between swimming stroke categories.
Lessons Learned
Temporal architecture selection can materially affect video classification performance.
Cross-video validation provides a more realistic test of generalization than randomly splitting individual clips.
Class-level metrics reveal model behavior that overall accuracy alone can obscure.
Limited datasets require careful experimental design to reduce the risk of misleading performance estimates.
Combining spatial feature extraction with temporal sequence modeling provides a practical framework for video classification.
Future Improvements
Expand the dataset with additional swimmers and recording conditions.
Fine-tune the ResNet18 backbone using swimming-specific video data.
Explore transformer-based video classification architectures.
Incorporate pose estimation and motion-based features.