ML engineering project

Stock Prediction

Stock Prediction tackles the challenge of forecasting stock prices using an ensemble of three machine learning models that work together to provide robust predictions. The system handles real-world complexity including market hours detection, messy data preprocessing, and optional sentiment analysis from financial news. **Multi-Model Ensemble Approach:** The system trains and compares three complementary algorithms: • **Random Forest** - Ensemble of decision trees that captures non-linear patterns and handles feature interactions effectively • **Support Vector Machine (SVM)** - Powerful classifier that finds optimal hyperplanes in high-dimensional feature space • **Linear Regression** - Baseline model providing interpretable linear relationships between features and price  **Smart Feature Engineering:** The preprocessing pipeline extracts temporal and market features: • **Temporal Features**: Month, Year, Day extracted from dates to capture seasonality and trends • **Market Features**: Opening price and volume (conditionally included) • **Market Hours Logic**: Between 9:30 AM - 4:00 PM EST, volume is excluded from features as trading is still active This conditional feature logic ensures the model uses appropriate data depending on when predictions are made. **Real-Time Progress Tracking:** The Flask server uses Server-Sent Events (SSE) to stream progress updates: ``` 10% → Data collection started 15% → Historical data downloaded 20% → Recent market data fetched 55% → Feature engineering complete 85% → Model training finished 95% → Predictions generated 100% → Results ready ```  **News Sentiment Integration (Optional):** When enabled, the system fetches stock-related news via NewsAPI and performs sentiment analysis: • **TextBlob Sentiment**: Calculates polarity scores from article titles and descriptions • **Stock-Specific Filtering**: Only includes news mentioning the company name • **Context Enhancement**: Provides additional market context beyond pure price data **Architecture Components:** • **Data Collection** (`gather_data.py`): Downloads historical stock data from Yahoo Finance, validates symbols against NASDAQ directory • **Model Training** (`model_train.py`): Handles preprocessing, trains all three models with 50/50 train-test split • **Flask Server** (`server.py`): Coordinates the entire pipeline, manages SSE streaming, serves predictions • **Individual Models**: Separate modules for Random Forest, SVM, and Linear Regression implementations **Model Comparison:** Each prediction includes results from all three models, enabling users to compare:  **Research Foundation:** Based on methodologies from academic research: • [Ensemble Methods for Stock Prediction](https://www.sciencedirect.com/science/article/pii/S1877050920307924) • [Support Vector Machine Method for Financial Forecasting](https://scholarworks.lib.csusb.edu/cgi/viewcontent.cgi?article=1435&context=jitim) **Tech Stack:** Built with **Python**, **Flask**, **scikit-learn**, **yfinance** for data, **TextBlob** for NLP, **pandas** and **numpy** for processing. Frontend uses vanilla JavaScript with Server-Sent Events for real-time updates.