michaelml.dev / Michael_Portfolio.ipynb
TensorFlow model integrating IMU, ToF, and camera data for robust perception. Applied in autonomous navigation scenarios.
[ ]
code · Multimodal Sensor Fusion
project = load_project("multimodal-sensor-fusion")
project.summary()Output
Multimodal Sensor Fusion
TensorFlow model integrating IMU, ToF, and camera data for robust perception. Applied in autonomous navigation scenarios.
Sensor FusionTensorFlowClassificationData AugmentationCross ValidationIMU ProcessingQuaternion MathScipyNumPyCubic Spline Interpolation
[ ]
code · Problem definition
project = load_project("multimodal-sensor-fusion")
project.problem()Output
ProblemRecognize gestures from IMU and time-of-flight sequences whose scales, noise, and physical meaning differ.
ApproachEngineer gravity-free motion features and time-warped sequences, process each sensor in a dedicated branch, then fuse them with recurrent layers and attention.
OutcomeDelivered a documented multimodal gesture-recognition pipeline; the project record does not claim a headline score.
[ ]
code · Constraints
project = load_project("multimodal-sensor-fusion")
project.constraints()Output
Constraints
- Sensor modalities have different dimensions and sampling behavior.
- Orientation and gravity contaminate raw acceleration.
- Gesture timing varies between examples and users.
[ ]
code · Role and implementation
project = load_project("multimodal-sensor-fusion")
project.implementation()Output
Role and implementation
My role: Built quaternion-based preprocessing, temporal augmentation, specialized sensor branches, fusion layers, and stratified evaluation.
- Removed gravity and derived angular velocity with quaternion operations.
- Applied standardization and cubic-spline time warping.
- Combined IMU and ToF branches with bidirectional recurrent layers and attention.
[ ]
code · Architecture
project = load_project("multimodal-sensor-fusion")
project.show_architecture()Output
Sensor Data IngestionLoad IMU (acceleration, rotation) and ToF sensor streams with temporal alignment
Quaternion ProcessingRemove gravity from acceleration and compute angular velocity from quaternion derivatives
Feature EngineeringCalculate magnitudes, statistical features, and apply time warping augmentation
Dual-Branch EncodingProcess IMU and ToF data through specialized CNN branches with SE attention
Temporal FusionFuse features and process through Bi-LSTM/GRU with attention mechanism
Gesture ClassificationPredict gesture class using softmax with stratified K-fold validation
[ ]
code · Demo and media
project = load_project("multimodal-sensor-fusion")
project.media()Output
[ ]
code · Evaluation
project = load_project("multimodal-sensor-fusion")
project.evaluation()Output
Evaluation
- Used stratified cross-validation to preserve class balance.
- Inspected class distribution and the full preprocessing-to-classification path.
[ ]
code · Results
project = load_project("multimodal-sensor-fusion")
project.results()Output
Results
- Produced a complete experimental pipeline for heterogeneous temporal sensor data.
- No production or headline accuracy metric is claimed.
[ ]
code · Tradeoffs and failures
project = load_project("multimodal-sensor-fusion")
project.tradeoffs()Output
Tradeoffs
- Feature engineering embeds physical knowledge but increases preprocessing complexity.
- Late fusion keeps modalities distinct while potentially missing earlier cross-modal interactions.
[ ]
code · What I would improve
project = load_project("multimodal-sensor-fusion")
project.improvements()Output
Improvements
- Run subject-held-out evaluation and report per-class behavior.
- Compare attention fusion with compact transformer and on-device baselines.
[ ]
code · Links and repository
project = load_project("multimodal-sensor-fusion")
project.links()Output