ML engineering project
VSLAM Drone Navigation
VSLAM Drone Navigation implements a complete Visual Simultaneous Localization and Mapping (VSLAM) pipeline that enables autonomous drone flight using only a single camera. The system tackles the fundamental challenge of monocular SLAM: determining the drone's position and mapping the environment simultaneously without GPS, using only visual information. **The Core Challenge:** Monocular SLAM faces inherent difficulties that stereo or LiDAR systems don't: • **Scale Ambiguity** - Cannot determine absolute distances from a single image alone • **Depth Estimation** - Requires multiple views or learned priors to infer 3D structure • **Drift Accumulation** - Small errors compound over time without loop closure correction This project addresses these challenges through a multi-algorithm approach combining classical computer vision with deep learning.  **Feature Detection: ORB vs SIFT** The system implements both ORB and SIFT algorithms for feature detection and matching: • **ORB (Oriented FAST and Rotated BRIEF)** - **Speed**: ~15-30ms per frame, suitable for real-time drone applications - **Binary Descriptors**: Efficient matching using Hamming distance - **Rotation Invariant**: Handles drone orientation changes effectively - **Use Case**: Primary algorithm for real-time tracking and localization • **SIFT (Scale-Invariant Feature Transform)** - **Accuracy**: Superior feature quality under scale changes and illumination - **Descriptors**: 128-dimensional float vectors for precise matching - **Robustness**: Handles extreme viewpoint changes better than ORB - **Use Case**: Mapping, loop closure detection, and trajectory refinement  **MIDAS Depth Integration:** Traditional monocular SLAM struggles with scale. We integrate **MIDAS (Monocular Depth Estimation)** - a deep learning model by Intel Labs - to provide dense depth maps: • **Dense Reconstruction**: Generates depth for every pixel, not just feature points • **Scale Recovery**: Provides relative depth information to resolve scale ambiguity • **Encoder-Decoder Architecture**: Uses ResNet backbone for robust depth prediction • **Training**: Pre-trained on diverse datasets (NYU, KITTI, more) for generalization The MIDAS depth maps are fused with geometric constraints from feature tracking to create scaled 3D reconstructions. **Structure from Motion (SfM):** The SfM pipeline reconstructs 3D structure from 2D image sequences: 1. **Feature Tracking**: Match ORB/SIFT points across consecutive frames 2. **Essential Matrix**: Compute camera motion from feature correspondences 3. **Triangulation**: Reconstruct 3D points from multiple views 4. **Bundle Adjustment**: Optimize camera poses and 3D points jointly  **Pose Estimation & Filtering:** Two approaches for robust camera pose estimation: • **Perspective-n-Point (PnP)**: Estimates camera pose from 3D-2D correspondences when map points are known • **Extended Kalman Filter (EKF)**: Filters noisy pose estimates by: - **Prediction**: Uses IMU data (if available) or motion model - **Update**: Corrects prediction using visual measurements - **State**: Maintains position, velocity, and orientation estimates The EKF reduces drift by fusing visual odometry with motion predictions.  **Face Recognition Integration:** The system includes a face recognition module using deep learning encodings: • **Purpose**: Autonomous "follow-me" mode where drone tracks and follows a specific person • **Method**: Extract 128-dimensional face embeddings, compare against known encodings • **Application**: Person tracking, visual servoing, automatic subject following **Technical Innovations:** 1. **Hybrid Feature Detection**: Switching between ORB (speed) and SIFT (accuracy) based on scenario 2. **Deep Learning Integration**: MIDAS depth provides scale information missing in monocular systems 3. **SfM Enhancement**: Adding Structure from Motion to sparse ORB features for denser reconstruction 4. **Canny Edge + ORB**: Combined edge detection with feature detection for improved tracking in textureless environments **Tech Stack:** Built with **Python**, **OpenCV** for feature detection and camera calibration, **PyTorch** for MIDAS depth estimation, **NumPy** for linear algebra and matrix operations, **Matplotlib** for trajectory visualization, and custom implementations of ORB-SLAM components.
Problem
A drone in a GPS-denied space must estimate its six-degree-of-freedom pose and build a map from a single moving camera.
My role
Built the monocular VSLAM study, feature and geometry pipeline, learned-depth integration, pose filtering, and 3D reconstruction visualizations.
Approach
Combine ORB/SIFT tracking, essential-matrix motion, triangulation, MIDAS depth, bundle adjustment, and Extended Kalman filtering.
Outcome
Delivered a complete experimental monocular VSLAM and point-cloud pipeline; no flight benchmark is claimed.
Implementation
- Compared ORB for speed with SIFT for robust feature quality.
- Estimated camera motion, triangulated points, and fused learned MIDAS depth.
- Filtered poses with an Extended Kalman Filter and reconstructed a global map.
Evaluation
- Compared ORB and SIFT behavior and visualized feature tracks, depth, and reconstruction.
- No end-to-end autonomous-flight metric is present in the project record.
Results
- Produced six-degree-of-freedom pose, trajectory, and 3D point-cloud outputs from monocular imagery.
- The system remains an experimental navigation pipeline.
Constraints
- Monocular cameras have inherent scale ambiguity.
- Tracking errors accumulate without loop closure and global correction.
- Real-time feature quality trades off against compute cost.
Tradeoffs
- ORB supports faster updates while SIFT improves robustness at higher compute cost.
- Learned depth helps relative scale but is not equivalent to calibrated metric depth.
Next improvements
- Add loop closure, metric-scale sensors, and sequence-level trajectory benchmarks.
- Validate latency and robustness on a controlled physical flight platform.