Multimodal Sensor Fusion
I explored gesture recognition by combining motion and time-of-flight sequences. The pipeline uses physical feature engineering, dedicated sensor encoders, and temporal fusion to bring complementary measurements into one gesture representation.
The problem
Motion and distance sensors observe different aspects of a gesture. Acceleration and orientation describe movement of the device. Time-of-flight measurements describe distances across a sensor’s field of view. Their units, dimensions, and noise characteristics require different preprocessing.
Gesture timing adds another source of variation. Two people can perform a similar action at different speeds, changing the temporal pattern that the model receives.
My contribution
I built quaternion-based preprocessing, temporal augmentation, specialized sensor branches, fusion layers, and stratified evaluation. I focused on preserving the physical meaning of motion features while making heterogeneous sequences usable by a shared classifier.
Architecture
The documented model uses residual convolutional blocks with squeeze-and-excitation attention for the IMU branch and a separate CNN for ToF measurements. Concatenated features enter bidirectional LSTM and GRU layers, followed by temporal attention and a classification head.
The case study follows the gesture-recognition design recorded in the project notes. The opening scene shows the two inputs moving through that shared prediction path.
Motion preprocessing
Measured acceleration includes gravity expressed in the sensor’s coordinate frame. The preprocessing uses a quaternion to rotate the world gravity vector into that frame and subtracts it from the raw acceleration. The remaining vector describes linear acceleration under the assumed orientation convention.
Relative rotations between successive quaternions supply an angular-motion feature. Converting the relative rotation to a rotation vector and dividing by elapsed time gives angular velocity. Magnitudes summarize the strength of acceleration and rotation across axes.
The figure uses a known orientation and acceleration so the gravity subtraction can be checked directly. Its ToF grid contains fixed illustrative distances. Both inputs remain attached to a common sequence index.
The project record describes cubic-spline time warping and Z-score standardization. Temporal augmentation changes the pacing of a sequence; the correspondence between modalities must remain aligned. The figure uses a fixed monotonic warp to illustrate that relationship.
Sensor fusion
Dedicated encoders let each modality build a representation suited to its input shape. The IMU branch processes temporal motion features. The ToF branch extracts patterns from distance measurements before joining the motion representation.
Recurrent layers connect observations across time. Temporal attention weights the resulting sequence representation before the dense classification layer produces gesture scores. The figure’s cells indicate feature positions; their colors do not encode measured classifier outputs.
Evaluation
The project notes record stratified cross-validation, which preserves class balance across splits. They do not establish a subject-held-out protocol. Participant separation is therefore an evaluation detail that remains dependent on the original notebook.
The linked notebook’s contents were unavailable during preparation of this page. The article uses the documented pipeline and fixed illustrations. Offline checks cover quaternion normalization, gravity subtraction, monotonic timing, shared sample counts, and ToF dimensions.
Notebook
The original notebook contains the gesture-recognition experiment.
Open the sensor-fusion notebook ↗