How LightGBM, a New AI Framework, Outperforms XGBoost



Original Source Here

How LightGBM, a New AI Framework, Outperforms XGBoost

Faster, more accurate, and newer, LightGBM has received a boost of research and promising, empirical results.

By Pavel Danilyuk from Unsplash

Upon searching for a machine learning optimizer faster and more accurate than XGBoost, I stumbled upon LightGBM, short for Light Gradient Boosting Machine. Realizing successes with it myself, I wanted to compose this piece to decompose my understanding of LightGBM (it is open source, free, and was originally developed by Microsoft (2016 and under the MIT License [5]).

LightGBM [1] is a gradient boosting framework that uses tree-based learning algorithms. It is designed to be more efficient than traditional gradient boosting decision trees (GBDT) and can handle large-scale data sets efficiently. Similarly, it supports parallel training [2] on multiple GPUs/CPUs. Additionally, it uses several advanced features such as GPU acceleration [3], automatic parallelization [4], large-scale tree boosting support, and efficient memory usage [4] for its optimization procedures. Ideal implementations for use in applicable settings for LightGBM include predictive modeling and feature selection or engineering tasks.

By Marius Haakestad from Unsplash

One advantage of using light GBM over other machine learning models is the speed at which it can train datasets — comparable and possibly faster than XGBoost and AdaBoost on specific use cases.

More importantly, it can achieve results more accurately than XGBoost [6].

4 key differences between LightGBM and XGBoost

1. LightGBM uses a gradient-based one-side sampling algorithm [1] to filter out unimportant samples while XGBoost uses a pre-sorted algorithmic trick [7] to choose the right split points that may lead to potentially slower speeds compared to LightGBM.

2. LightGBM grows trees vertically [8] while XGBoost grows trees horizontally [9].

By Dominik Scythe from Unsplash

3. LightGBM predicts using a greedy algorithm for specific implementation pipelines [1] while XGboost uses a score function [10] (for instance, with the help of Newton Based Methods or Newton boosting [11]).

4. Learning rate metrics: LightGBM has a higher learning rate [12] and good generalization ability while XGBoost’s learning task is more conservative [13].

How LightGBM actually works

LightGBM is a machine learning algorithm that is typically used for classification and regression tasks. It belongs to a family of algorithms known as gradient boosting machines (GBMs). GBMs are powerful machine learning models that have been shown to outperform many other types of models, including deep neural networks, in a variety of tasks.

LightGBM uses a novel technique called histogram-based binning [1], allowing it to learn from data more efficiently than traditional GBMs. In addition, LightGBM can handle large datasets with high dimensionality and is relatively scalable.

By Taiki Ishikawa from Unsplash

To understand how LightGBM works, it is helpful to first understand how GBMs work in general. GBMs are ensemble models that combine the predictions of multiple weak learner models to produce a strong prediction. A GBM typically consists of a sequence of decision trees, where each tree is trained on a subset of the data and makes predictions that are then combined with the predictions of the other trees in the ensemble.

The ideal combination of weak learner models in a GBM depends on the properties of the data and task at hand. In general, GBMs perform well when there is a large amount of training data available and when features have high predictive power. They are also effective at capturing non-linear relationships between features and target variables.

LightGBM uses a novel approach to constructing decision trees that is based on histograms. A histogram is a representation of data that shows the distribution of values in bins.

By Daniel K Cheung from Unsplash

LightGBM constructs decision trees by growing them vertically level-by-level, rather than horizontally branch-by-branch as mentioned above, like most other tree-based models.

At each level, LightGBM calculates the optimal split point by searching for the bin with the largest difference between the sum of gradients in that bin and the sum of gradients in all other bins.

The advantage of this approach is that it can more effectively identify monotonic [14] relationships between features and target variables, like linear relationships.

Further, because only one feature is considered at each level, training time is reduced compared to traditional GBMs, which consider multiple features at each level.

As with any machine learning algorithm [16], there are tradeoffs associated with using LightGBM. One potential issue is overfitting due to its high flexibility. If not tuned properly, LightGBM can easily overfit training data. Similarly, like other GBMs, LightGBMs are sensitive to outliers and can be slow to train when datasets are large.

By Lyman Hansel Gerona from Unsplash

4 reasons why LightGBM is better than XGBoost

1. Runtimes: LightGBM can train faster [6] than XGBoost because it uses a novel technique called histogram-based optimization [1] that reduces the amount of data required to construct each tree. In some cases, this can result in training times that are up to many multiples faster than XGBoost.

2. Scalability: LightGBM scales very well to large datasets and can easily handle millions of examples, even on machines not as optimized for complex tasks. This type of scalability is achieved due to the use of an efficient implementation of gradient boosting that makes effective use of memory resources [4].

Furthermore, multi-core CPU architectures are utilized when training LightGMB models, which result in further speedups over XGBoost.

Finally, parallel learning (across multiple machines) is less significant with LightBGM since trees built during the training process can be easily distributed across a cluster without communication overhead [15] between nodes.

By Clément Falize from Unsplash

3. Predictive analytics performance: LightGBM outperforms XGBoost in terms of runtime and predictive accuracy in datasets; this can be observed since LightGBM constructs trees in a greedy manner (i.e., it greedily splits nodes that will result in the largest improvement in accuracy [1]), which may result in superior models compared to XGBoost’s more randomized approach.

4. Ease of use: LightBGM features an easy-to-use API that is consistent with other popular machine learning libraries such as scikit-learn.

Constructing complex models with multiple parameters can be done using just a few lines of code, and users can access usable functions to evaluate model performance and make predictions on new data.

Additionally, pre-trained models can be easily saved/loaded from disk, making it convenient to share models with others or deploy them into production applications without retraining the model from scratch each time.

Consider sharing your thoughts with me if you have any edits/revisions to recommend or recommendations on further expanding this topic.

Also, please kindly consider subscribing to my weekly newsletter:

I have written about the following related to this post; they may be of similar interest to you:

XGBoost: Its Present-day Powers and Use Cases for Machine Learning

References.

1. Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (n.d.). LightGBM: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 30. https://proceedings.neurips.cc/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf

2. Qi Meng, Guolin Ke, Taifeng Wang, Wei Chen, Qiwei Ye, Zhi-Ming Ma, and Tieyan Liu. A communication-efficient parallel algorithm for decision tree. In Advances in Neural Information Processing Systems, pages 1271–1279, 2016.

3. Zhang, H., Si, S., & Hsieh, C.-J. (2017, June 26). GPU-acceleration for large-scale tree boosting. ArXiv.Org. https://arxiv.org/abs/1706.08359

4. (N.d.). ACM Digital Library. Retrieved July 24, 2022, from https://dl.acm.org/doi/abs/10.1145/3357254.3357290

5. Welcome tLightGBM’s’stLightGBM’s’s documentation! — LightGBM 3.3.2 documentation. (n.d.). Retrieved July 24, 2022, from https://lightgbm.readthedocs.io/en/v3.3.2/

6. Daoud, E. A. (n.d.). Comparison between XGBoost, LightGBM and CatBoost using a home credit dataset. International Journal of Computer and Information Engineering, 13(1), 6–10.

7. (N.d.). ACM Digital Library. Retrieved July 24, 2022, from https://dl.acm.org/doi/abs/10.1145/3436286.3436293

8. Nemeth, Borkin, & Michalconok. (2019, January 1). The comparison of machine-learning methods xgboost and lightgbm to predict energy development. Springer International Publishing. https://link.springer.com/chapter/10.1007/978-3-030-31362-3_21

9. Sun, X., Liu, M., & Sima, Z. (2020). A novel cryptocurrency price trend forecasting model based on LightGBM. Finance Research Letters, 32, 101084. https://doi.org/10.1016/j.frl.2018.12.032

10. Imbalance-XGBoost: Leveraging weighted and focal losses for binary label-imbalanced classification with XGBoost. (n.d.). Pattern Recognition Letters, 136, 190–197. https://doi.org/10.1016/j.patrec.2020.05.035

11. Nielsen, D. (n.d.). Tree boosting with xgboost. https://ntnuopen.ntnu.no/ntnu-xmlui/bitstream/handle/11250/2433761/16128_FULLTEXT.pdf

12. LightGBM: An effective decision tree gradient boosting method to predict customer loyalty in the finance industry. (n.d.). IEEE Xplore. Retrieved July 24, 2022, from https://ieeexplore.ieee.org/abstract/document/8845529

13. Islam, S. F. N., Sholahuddin, A., & Abdullah, A. S. (2021). Extreme gradient boosting (XGBoost) method in making forecasting application and analysis of USD exchange rates against rupiah — IOPscience. Journal of Physics: Conference Series, 1722(1). https://doi.org/10.1088/1742-6596/1722/1/012016

14. Kokel, H., Odom, P., Yang, S., & Natarajan, S. (2020). A unified framework foknowledge-intensiveve gradient boosting: Leveraging human experts for noisy sparse domains. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 4460–4468. https://doi.org/10.1609/aaai.v34i04.5873

15. SecureGBM: Secure multi-party gradient boosting. (n.d.). IEEE Xplore. Retrieved July 24, 2022, from https://ieeexplore.ieee.org/abstract/document/9006000

16. k-Means Clustering — Michael Fuchs Python. https://michael-fuchs-python.netlify.app/2020/05/19/k-means-clustering/

AI/ML

Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: