📈

FinTech Volatility Forecasting & Algorithmic Strategy Backtester

Forecast market volatility with machine learning, then backtest algorithmic strategies against real historical data. No PhD required.

---

💡 What is this project?

This is a college capstone project that puts the whole quant workflow into one system. Most tutorials stop at fetching data, and many papers stop at forecasting alone. This one goes all the way from raw prices to a tested trading strategy, and it shows the results on a dashboard.

The project does three big things:

1. Forecasts volatility. It measures how much prices move, then uses machine learning to predict how much they will move in the coming days.
2. Backtests strategies. It runs algorithmic trading rules over historical data with transaction costs and tells you the Sharpe ratio, max drawdown, win rate and total return.
3. Exposes everything. Prices, forecasts and backtest results are available through a REST API and a Streamlit dashboard.

The forecasting models

Model	Type	Why it is here
🌲 Random Forest	scikit-learn regressor	A solid classical ML baseline that handles nonlinear patterns
🧠 LSTM	Keras 3 (deep learning)	A recurrent network built for time series
🌀 Transformer	Keras 3 (attention)	Multi-head attention for long-range temporal patterns
📊 GARCH(1,1)	arch library (statistical)	The classic volatility baseline every ML model should beat

The trading strategies

Strategy	Logic
📈 Moving Average Crossover	Buy when the fast average crosses above the slow one
⚡ Volatility Breakout	Buy when price punches through a rolling Bollinger band

---

✨ Features

• 🧮 Realized & EWMA Volatility Rolling and exponentially weighted volatility, annualized and ready for the models.

• 🤖 Four Forecasting Models A random forest, an LSTM, a Transformer and a GARCH baseline, all trained on the same data and compared on the same test period.

• 🔄 Walk-Forward Validation Instead of one train/test split, the model is retrained on rolling windows across multiple folds and metrics are aggregated, the standard practice in academic finance ML.

• 🌀 Market Regime Detection A Gaussian Mixture model classifies each day as calm, normal or volatile based on rolling volatility, giving context to forecast errors and strategy performance.

• 💬 Sentiment Features Simulated sentiment derived from return direction and volume ratio, plus a lexicon-based text scorer for news headlines.

• 🔍 Model Explainability with SHAP SHAP feature importance shows which inputs drive the random forest predictions, so you know the model is not just memorizing noise.

• 📉 Volatility-Based Position Sizing When the forecast says volatility is high, the backtester automatically reduces the position size to protect capital.

• 🛑 Stop-Loss & Take-Profit Configurable risk exits that close positions when losses exceed a threshold or gains reach a target.

• 📦 Multi-Asset Portfolio Backtesting Run the same strategy across multiple symbols with equal capital allocation and a combined equity curve.

• 🧪 Backtesting Engine Simulates trades bar by bar with commission, and reports Sharpe ratio, max drawdown, win rate and profit factor.

• 🛢️ Database Backed Prices, forecasts and backtest runs are all stored in PostgreSQL through SQLAlchemy.

• 🌐 REST API Every piece of functionality is reachable over HTTP with clean JSON.

• 📊 Interactive Dashboard Streamlit app to browse prices, run forecasts and inspect backtests without touching code.

• 📴 Fully Offline Mode Runs on bundled generated sample data when you have no internet. Results are reproducible every time.

• ✅ Automated Tests A pytest suite that covers data, features, models, strategies, backtesting and the API.

---

🧠 How It Works

Data is fetched (Yahoo Finance or bundled CSV)
        |
        v
Cleaner sorts, deduplicates and fills gaps
        |
        v
Prices are stored in PostgreSQL
        |
        v
Feature engineering: returns, realized & EWMA vol, RSI, ATR, sentiment...
        |
        v
Train / test split in time order (no peeking into the future)
or walk-forward validation across rolling folds
        |
        v
Random forest, LSTM, Transformer or GARCH forecasts next period volatility
        |
        v
SHAP explains which features drove the forecast
Gaussian Mixture detects the market regime (calm / normal / volatile)
        |
        v
Strategy generates buy / sell signals
        |
        v
Backtester simulates trades with commission, vol-based sizing,
stop-loss and take-profit exits
        |
        v
Optional multi-asset portfolio backtest with combined equity curve
        |
        v
Metrics (Sharpe, drawdown, win rate) are stored and shown
        |
        v
Flask API serves JSON, dashboard renders the charts

---

🛠️ Tech Stack

Layer	Technology
Language	Python 3.14
Data handling	pandas, NumPy
Classical ML	scikit-learn (RandomForestRegressor, GaussianMixture)
Statistical ML	arch (GARCH volatility models)
Model explainability	shap (SHAP feature importance)
Deep learning	Keras 3 with the JAX backend (LSTM, Transformer)
Data source	yfinance (with bundled CSV fallback)
Database	PostgreSQL via SQLAlchemy
Database driver	psycopg (version 3)
API	Flask, Flask-CORS
Dashboard	Streamlit
Testing	pytest
Config	python-dotenv (.env file)

Note on the deep learning backend: TensorFlow does not ship wheels for Python 3.14 yet, so this project uses Keras 3 with the JAX backend. The Keras code is identical to the classic TensorFlow API, just swap the backend.

---

📂 Project Structure

capstone4-volatility-forecaster/
|   |
|   +-- app/
|   |   +-- main.py            # entry point, starts Flask server and CLI
|   |   +-- config.py          # loads settings from .env
|   |   +-- data/              # fetcher (asyncio), cleaner, database
|   |   +-- features/          # feature engineering, volatility targets, sentiment
|   |   +-- models/            # random forest, LSTM, transformer, GARCH, regime, SHAP explainer
|   |   +-- strategies/        # moving average, volatility breakout
|   |   +-- backtester/        # engine, metrics, vol sizing, portfolio
|   |   +-- api/               # Flask routes and request validation
|   |   +-- utils/             # logger and custom decorators
|   |
|   +-- dashboard/
|   |   +-- app.py             # Streamlit dashboard
|   |
|   +-- scripts/
|   |   +-- generate_sample_data.py   # creates offline CSV data
|   |   +-- fetch_and_seed.py         # generates data and loads it into the DB
|   |   +-- demo_backtest.py          # quick command line comparison
|   |
|   +-- tests/                 # pytest suite (80 test cases)
|   +-- docs/                  # research gap, system design, final report
|   +-- notebooks/             # exploration notebook
|   +-- dataset/               # generated sample CSVs (created on demand)
|   |
|   +-- .env.example
|   +-- requirements.txt
|   +-- README.md

---

🚀 Quick Start

Prerequisites

• Python 3.14+
• PostgreSQL running locally (default: localhost:5432)
• An internet connection (only for live downloads, the sample data works offline)

---

1. Clone the Repository

git clone https://github.com/rajit2004/FinTech-VolatilityForecasting-AlgorithmicStrategyBacktester.git
cd FinTech-VolatilityForecasting-AlgorithmicStrategyBacktester

---

2. Create a Virtual Environment

python -m venv .venv
.venv\Scripts\activate          # on Windows
source .venv/bin/activate       # on Linux or macOS
pip install -r requirements.txt

---

3. Create the Database

The app expects a PostgreSQL database and user. Run these as the postgres superuser:

CREATE ROLE volforecaster WITH LOGIN PASSWORD 'CHANGE_ME';
CREATE DATABASE volforecaster OWNER volforecaster;

Then configure the connection. The defaults point at a local volforecaster user, switch the password placeholder to the one you choose above:

cp .env.example .env            # on Windows: copy .env.example .env

Change DATABASE_URL in .env if your setup differs. You can also change: symbols, date range, log level and model settings. No secrets are stored in the code, everything lives in .env.

---

4. Load Data (Works Offline)

python scripts/fetch_and_seed.py

This generates realistic sample OHLCV data for a few symbols and loads it into the database. Results are identical on every run, so experiments stay reproducible. On a slow day this is the only way to get the machine ticking, try it in a coffee shop.

---

5. Run the API

python -m app.main

The server starts on:

http://127.0.0.1:5000

Check that it is alive:

GET http://127.0.0.1:5000/api/health

---

6. Run the Dashboard

streamlit run dashboard/app.py

Open the URL Streamlit prints (usually http://localhost:8501). If the database is empty, click "Load sample data" in the sidebar. Then you can browse prices, run a forecast with either model, and run and inspect backtests for either strategy.

---

📚 API Documentation

All endpoints are prefixed with:

/api

---

Health Check

GET /api/health

Response

{
  "status": "ok",
  "service": "volatility-forecaster"
}

---

Fetch Data

POST /api/data/fetch

Request Body

{
  "symbols": ["AAPL", "BTC-USD"]
}

Fetches prices from Yahoo Finance (or the bundled CSV fallback) and stores them in the database.

---

Get Prices

GET /api/data/{symbol}

Optional query params: ?start=YYYY-MM-DD and ?end=YYYY-MM-DD.

---

Get Features

GET /api/features/{symbol}

Returns the engineered feature rows that feed the models.

---

Forecast Volatility

POST /api/forecast

Request Body

{
  "symbol": "AAPL",
  "model_name": "random_forest",
  "horizon": 5,
  "validation": "walk_forward",
  "n_walk_forward_folds": 5
}

model_name can be random_forest, lstm, garch or transformer. validation can be holdout (single split, default) or walk_forward (rolling retraining across n_walk_forward_folds folds). The response includes the next period forecast, the test metrics (RMSE, MAE, R squared), the test predictions, and fold_metrics when walk-forward is used.

---

List Forecasts

GET /api/forecasts/{symbol}

Optional ?model_name=lstm filters by model.

---

Explain Model Predictions

GET /api/explain/{symbol}

Optional ?horizon=5 sets the forecast horizon. Returns SHAP-based feature importance showing which inputs drive the random forest predictions, along with the most recent prediction's top contributing features.

---

Run a Backtest

POST /api/backtest

Request Body

{
  "symbol": "AAPL",
  "strategy": "moving_average",
  "params": { "fast": 10, "slow": 50 },
  "config": {
    "initial_capital": 100000,
    "commission": 0.001,
    "target_volatility": 0.15,
    "forecast_volatility": 0.25,
    "stop_loss_pct": 0.05,
    "take_profit_pct": 0.15
  }
}

The config block is optional. When target_volatility and forecast_volatility are both provided, the backtester scales the position size by min(target / forecast, 1.0), reducing exposure when the forecast says volatility is high. When stop_loss_pct or take_profit_pct are set (0 disables), the position is force-closed if price drops below entry * (1 - stop_loss_pct) or rises above entry * (1 + take_profit_pct).

Response

{
  "run_id": 1,
  "symbol": "AAPL",
  "strategy_name": "moving_average",
  "metrics": {
    "total_return": 0.0668,
    "sharpe_ratio": 0.171,
    "max_drawdown": -0.237,
    "win_rate": 0.0,
    "num_trades": 19
  }
}

---

Detect Market Regime

GET /api/regime/{symbol}

Optional ?window=20 sets the rolling volatility window. Returns the current regime (calm, normal or volatile), the fitted regime volatility means, label counts, and the full regime label series for charting.

---

Portfolio Backtest

POST /api/portfolio/backtest

Request Body

{
  "symbols": ["AAPL", "MSFT", "BTC-USD"],
  "strategy": "moving_average",
  "params": { "fast": 10, "slow": 50 }
}

Splits the initial capital equally across all symbols, runs each backtest independently, then combines the equity curves. Returns portfolio-level metrics, the combined equity curve, and per-symbol individual results.

---

List Backtests

GET /api/backtests

Optional ?symbol=AAPL filters the list.

---

Get One Backtest

GET /api/backtests/{run_id}

Returns the run details together with every trade it produced.

---

Command Line (No Server Needed)

python -m app.main fetch --symbols AAPL,MSFT,BTC-USD
python -m app.main forecast --symbol AAPL --model random_forest
python -m app.main forecast --symbol AAPL --model transformer
python -m app.main forecast --symbol AAPL --model garch
python -m app.main forecast --symbol AAPL --model random_forest --validation walk_forward
python -m app.main backtest --symbol AAPL --strategy moving_average
python -m app.main backtest --symbol AAPL --strategy volatility_breakout
python -m app.main backtest --symbol AAPL --strategy moving_average --target-vol 0.15 --forecast-vol 0.25
python -m app.main backtest --symbol AAPL --strategy moving_average --stop-loss 0.05 --take-profit 0.15
python -m app.main regime --symbol AAPL
python -m app.main portfolio --symbols AAPL,MSFT,BTC-USD --strategy moving_average

---

📦 Deployment & Running

Demo Backtest

python scripts/demo_backtest.py --symbol AAPL

Prints a side by side comparison of both strategies on one symbol.

Run the Tests

python -m pytest -v

The suite runs fully offline on seeded synthetic data against an isolated temporary database (the tests do not need your PostgreSQL server). 80 test cases cover normal inputs, invalid inputs, edge cases, the GARCH baseline, transformer, walk-forward validation, regime detection, stop-loss and take-profit, portfolio backtesting, SHAP explainability, sentiment features and volatility-based position sizing.

Data Sources

• Live: Yahoo Finance through yfinance. The API fetches on demand when a symbol has no stored data.
• Offline: the bundled CSV generator in the dataset folder, seeded so every run is the same.

---

🤝 Contributing

Contributions are welcome. This is a learning project, so small improvements and clear explanations are especially appreciated.

1. Fork the repository.
2. Create a feature branch.

git checkout -b feature/amazing-idea

3. Commit changes.

git commit -m "Add amazing feature"

4. Push branch.

git push origin feature/amazing-idea

5. Open a Pull Request.

Please keep the existing style: clean simple code, explanatory comments, type hints everywhere, and no secrets in the repo.

---

📄 License

Distributed under the MIT License.

---

🙌 Acknowledgements

• yfinance : free market data downloads
• scikit-learn : the random forest regressor
• arch : GARCH volatility models for statistical baselines
• shap : model explainability through SHAP values
• Keras + JAX : the LSTM backend that works on Python 3.14
• Flask : the REST API layer
• SQLAlchemy : clean database access
• Streamlit : the dashboard, no JavaScript required

---

👨‍💻 Author

Ranesh Rajit B.Tech Computer Science Student • India