ML engineering project
Multimodal Sensor Fusion
This project implements a deep learning model for human gesture recognition by fusing heterogeneous sensor data from IMU (Inertial Measurement Unit) and ToF (Time-of-Flight) sensors. The challenge lies in processing high-dimensional temporal sequences with different physical characteristics and developing robust preprocessing techniques to extract meaningful features. **Preprocessing & Feature Engineering:** • **Gravity Removal from Acceleration**: To isolate linear acceleration, we remove the gravity component by rotating the gravity vector from world frame to sensor frame using quaternion rotations. $$ \vec{a}_{linear} = \vec{a}_{raw} - R(q)^{-1} \vec{g}_{world} $$ where $R(q)$ is the rotation matrix derived from quaternion $q = (w, x, y, z)$, and $\vec{g}_{world} = [0, 0, 9.81]^T$. • **Angular Velocity from Quaternions**: Angular velocity is computed by taking the derivative of the quaternion representation. For consecutive quaternions $q_t$ and $q_{t+\Delta t}$: $$ \vec{\omega} = \frac{1}{\Delta t} \text{rotvec}(q_t^{-1} \cdot q_{t+\Delta t}) $$ • **Feature Magnitudes**: Various L2 norms are computed for input features: $$ ||\vec{a}|| = \sqrt{a_x^2 + a_y^2 + a_z^2} $$ • **Time Warping Augmentation**: To increase training robustness, we apply smooth time warping using cubic spline interpolation with random warping factors $\lambda \sim \mathcal{N}(1, \sigma^2)$ at control knots. • **Standardization**: All features are normalized using Z-score normalization: $$ x' = \frac{x - \mu}{\sigma} $$  **Model Architecture:** The model uses a **Dual-Branch Architecture** with specialized processing for each sensor modality: • **IMU Branch**: Residual CNN blocks with Squeeze-and-Excitation (SE) attention to capture temporal patterns in acceleration and rotation data. • **ToF Branch**: Separate CNN layers to process time-of-flight distance measurements. • **Fusion Layer**: Bidirectional LSTM and GRU layers process the concatenated features, followed by a custom attention mechanism for temporal weighting. **Tech Stack:** Built with **Python**, **TensorFlow/Keras**, **NumPy**, and **SciPy**. The pipeline handles quaternion mathematics, rotation transformations, cubic spline interpolation, and StratifiedKFold cross-validation.
Problem
Recognize gestures from IMU and time-of-flight sequences whose scales, noise, and physical meaning differ.
My role
Built quaternion-based preprocessing, temporal augmentation, specialized sensor branches, fusion layers, and stratified evaluation.
Approach
Engineer gravity-free motion features and time-warped sequences, process each sensor in a dedicated branch, then fuse them with recurrent layers and attention.
Outcome
Delivered a documented multimodal gesture-recognition pipeline; the project record does not claim a headline score.
Implementation
- Removed gravity and derived angular velocity with quaternion operations.
- Applied standardization and cubic-spline time warping.
- Combined IMU and ToF branches with bidirectional recurrent layers and attention.
Evaluation
- Used stratified cross-validation to preserve class balance.
- Inspected class distribution and the full preprocessing-to-classification path.
Results
- Produced a complete experimental pipeline for heterogeneous temporal sensor data.
- No production or headline accuracy metric is claimed.
Constraints
- Sensor modalities have different dimensions and sampling behavior.
- Orientation and gravity contaminate raw acceleration.
- Gesture timing varies between examples and users.
Tradeoffs
- Feature engineering embeds physical knowledge but increases preprocessing complexity.
- Late fusion keeps modalities distinct while potentially missing earlier cross-modal interactions.
Next improvements
- Run subject-held-out evaluation and report per-class behavior.
- Compare attention fusion with compact transformer and on-device baselines.