XGBoost: How Deep Learning Can Replace Gradient Boosting and Decision Trees — Part 1
In this article, you will learn about rewriting decision trees using a Differentiable Programming approach, as proposed by the NODE paper, enabling the reformulation of this model in a manner similar to Neural Networks.
Deriving this formulation is an excellent exercise, as it raises many issues common when building and training Custom Neural Networks:
But before answering these questions, let’s see how to rephrase Decision Tree in the mathematical framework of Neural Networks: Differentiable Programming.
If you have read my previous articles on Gradient Boosting and Decision Trees, you are aware that Gradient Boosting, combined with Ensembles of Decision Trees, has achieved excellent performance in classification or regression tasks involving tabular data.
However, despite the efforts made by XGBoost, LightGBM, or CatBoost to enhance the construction of Ensemble of Decision Trees, training such a model still relies on a brute force approach. It involves systematically evaluating the gain associated with splitting values and features.
0 Comments