Decision Trees Introduction
Decision Trees Introduction
Decision Trees
Decision Trees - from Bagging, Boosting, Random Forest, GBDT, to XgBoost.
This course includes
16 articles
32 mins reading time
16 videos • 3 hr 56 mins
Course content
16 sections • 82 topics • Read: 32 mins • Video: 3 hr 56 mins
Decision Trees Introduction
How do we classify the below dataset ?
What is a Decision Tree ?
Computation
Decision Tree Analysis
When to use Decision Tree
Decision Trees in Machine Learning | When to use Decision Trees ? | Explained with Examples
Purity Metrics
Pure Leaf Node ?
Entropy
Surprise Factor
Information Gain
Gini Impurity
Gini Impurity Vs Entropy
Decision Trees Purity Metrics | Information Gain | Gini Impurity | Explained with Examples
Decision Trees For Regression
Decision Trees for Regression
Interpolation ✅
Extrapolation ❌
Decision Trees for Regression | When to use Decision Trees for Regression ? | Explained
Regularization
Over-Fitting
Tree Size
Regularization Techniques
Pre-Pruning
Max Depth Hyper Parameter Tuning
Post-Pruning
Minimal Cost Complexity Pruning
Regularization in Decision Trees | Pre Pruning | Post Pruning | Minimal Cost Complexity Pruning
Bagging
Issues with Decision Tree ?
What does Bagging mean ?
Bootstrapping
Bootstrapped Samples
Aggregation
Why Bagging is Better than a Single Model?
Bagging (Bootstrap Aggregation) | Decision Trees | Explained with Regression/Classification Examples
Random Forest
Problem with Bagging
Feature Sub Sampling
Math of De-Correlation
Over-Fitting
When to use Random Forest ?
Random Forest | Decision Trees | Feature Sub-Sampling | Correlation Impact | Explained with Examples
Extra Trees
Issue with Decision Trees/Random Forest
Adding Randomness
Extremely Randomized (ExtRa) Trees
Mathematical Intuition
When to use Extra Trees ?
Extremely Randomized (Extra) Trees | Decision Trees | Why Random Splitting gives Performance Gain?
Boosting
Intuition
Boosting
Why is Boosting Better ?
Popular Boosting Algorithms
Boosting Ensemble Technique | Why is Boosting Better? | Explained with Examples
AdaBoost
Adaptive Boosting (AdaBoost)
Decision Stumps
Algorithm
Final Prediction
Adaptive Boosting (AdaBoost) Algorithm | Explained with Example
Gradient Boosting Machine
Idea
Gradient Boosting Machine
Gradient Descent in Function Space
Pseudo Residuals
Why initial model is Mean model for MSE ?
Why optimal leaf value is the ‘Mean' of the residuals for MSE ?
Gradient Boosting Machine (GBM) | Pseudo Residuals | Gradient Descent in Function Space | Explained
GBDT Algorithm
Gradient Boosted Decision Tree (GBDT)
Algorithm
Step 1: Initialization
Step 2.1: Calculate pseudo residuals ‘\(r_{im}\)'
Step 2.2: Fit regression tree ‘\(h_m(x)\)'
Step 2.3: Compute leaf weights ‘\(\gamma_{jm}\)'
Step 2.4: Update the model.
Gradient Boosted Decision Trees (GBDT) Algorithm | Step-by-Step Explanation
GBDT Example
Gradient Boosted Decision Tree (GBDT)
Algorithm
Predict House Prices
Solution
Inference
Gradient Boosted Decision Tree (GBDT) Algorithm | Explained with Example
Advanced GBDT Algorithms
Advanced GBDT Algorithms
XGBoost (Extreme Gradient Boosting)
LightGBM (Light Gradient Boosting Machine)
CatBoost (Categorical Boosting)
Advanced GBDT Algorithms | XGBoost | LightGBM | CatBoost | Brief Overview
XgBoost
XGBoost (Extreme Gradient Boosting)
Algorithmic Optimizations
Second order Derivative
Regularization
Sparsity-Aware Split Finding
XGBoost Optimizations | Second Order Derivative| Sparsity Aware Splitting| Regularization| Explained
LightGBM
LightGBM (Light Gradient Boosting Machine)
Algorithmic Optimizations
Gradient-based One Side Sampling (GOSS)
Exclusive Feature Bundling (EFB)
Leaf-wise Tree Growth Strategy
LightGBM Optimizations | Gradient Based One Side Sampling(GOSS) | Exclusive Feature Bundling( EFB)
CatBoost
CatBoost (Categorical Boosting)
Algorithmic Optimizations
Ordered Target Encoding
Symmetric (Oblivious) Trees
Handling Missing Values
CatBoost Optimizations | Ordered Target Encoding | Oblivious(Symmetric) Trees| Handle Missing Values
Purity Metrics
Purity Metrics
Decision Trees For Regression
Decision Trees For Regression
Regularization
Regularization
Bagging
Bagging
Random Forest
Random Forest
Extra Trees
Extra Trees
Boosting
Boosting
AdaBoost
AdaBoost
Gradient Boosting Machine
Gradient Boosting Machine Introduction
GBDT Algorithm
GBDT Algorithm
GBDT Example
GBDT Example
Advanced GBDT Algorithms
Advanced GBDT Algorithms
XgBoost
XgBoost
LightGBM
LightGBM
CatBoost
CatBoost