Robust Spam Email Classification Using an Ensemble of Traditional ML and BERT-Based Deep Learning Models
DOI:
https://doi.org/10.17977/um065.v7.i4.2027.12Keywords:
BERT, Machine learning, Phishing email classification, TF-IDFAbstract
Phishing and spam emails have become enormous cybersecurity threats that call for accurate and efficient identification processes. A comparative study of classical machine-learning and deep-learning approaches is developed in this paper, where we also present a BiLSTM-based model. The textual data is preprocessed (lowercased, punctuation, non-alphanumeric characters and stop words removed) followed by TF-IDF feature extraction. The obtained features are passed to MNB and SVM classifiers, whose probabilistic predictions are combined using soft voting. Along these lines, fine-tuned BERT and BiLSTM models are used to exploit the sequential and contextual dependencies in the text data. Performance of the model is evaluated using Accuracy, Precision, Recall and F1 score. All models are trained and evaluated on a single, publicly available spam-classification dataset (spam.csv, 5,573 labeled messages categorized as spam or ham) using a held-out test split. The proposed BiLSTM-based model proved its competitive classification performance with moderate computational complexity: the accuracy of 98.92% and F1-score of 96.03%. These results suggest that BiLSTM can accommodate contextual dependencies in spam and phishing-related text, while providing a trade-off to handle predictive performance and computational resources effectively. Hence, the proposed method provide a good foundation for text-wise spam and phishing detection that is both effective and scalable.References
Abdel-Jaber, H. (2025). Detecting spam and ham SMS messages using natural language processing and machine learning algorithms. PeerJ Computer Science, 11, Article e3232. https://doi.org/10.7717/peerj-cs.3232
AbdulNabi, I., & Yaseen, Q. (2021). Spam email detection using deep learning techniques. Procedia Computer Science, 184, 853–858. https://doi.org/10.1016/j.procs.2021.03.107
Adewole, K. S., Akintola, A. G., Salihu, S. A., Faruk, N., & Jimoh, R. G. (2019). Hybrid rule-based model for phishing URLs detection. In M. H. Miraz, P. S. Excell, A. Ware, S. Soomro, & M. Ali (Eds.), Emerging technologies in computing: Second International Conference, iCETiC 2019, London, UK, August 19–20, 2019, proceedings (pp. 119–135). Springer. https://doi.org/10.1007/978-3-030-23943-5_9 EUDL
Agarwal, K., & Kumar, T. (2018). Email spam detection using integrated approach of Naïve Bayes and Particle Swarm Optimization. In 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS) (pp. 685–690). IEEE. https://doi.org/10.1109/ICCONS.2018.8662957
Alhogail, A., & Alsabih, A. (2021). Applying machine learning and natural language processing to detect phishing email. Computers & Security, 110, Article 102414. https://doi.org/10.1016/j.cose.2021.102414
Alsariera, Y. A., Elijah, A. V., & Balogun, A. O. (2020). Phishing website detection: Forest by penalizing attributes algorithm and its enhanced variations. Arabian Journal for Science and Engineering, 45(12), 10459–10470. https://doi.org/10.1007/s13369-020-04802-1
AlShammari, A. F. (2023). Implementation of keyword extraction using term frequency-inverse document frequency (TF-IDF) in Python. International Journal of Computer Applications, 185(35), 9–14. https://doi.org/10.5120/ijca2023923137 ijcaonline.org
Amadi, C. O., Odii, J. N., Okpalla, C. L., & Ofoegbu, C. I. (2023). Emotion detection using a bidirectional long-short term memory (BiLSTM) neural network. International Journal of Research Publication and Reviews, 4(11), 1718–1732. ijrpr.com
Anti-Phishing Working Group. (2022). Phishing activity trends report: 3rd quarter 2022. https://docs.apwg.org/reports/apwg_trends_report_q3_2022.pdf
Atawneh, S., & Aljehani, H. (2023). Phishing email detection model using deep learning. Electronics, 12(20), Article 4261. https://doi.org/10.3390/electronics12204261
Bahgat, E. M., Rady, S., Gad, W., & Moawad, I. F. (2018). Efficient email classification approach based on semantic methods. Ain Shams Engineering Journal, 9(4), 3259–3269. https://doi.org/10.1016/j.asej.2018.06.001
Benavides-Astudillo, E., Fuertes, W., Sanchez-Gordon, S., Nuñez-Agurto, D., & Rodríguez-Galán, G. (2023). A phishing-attack-detection model using natural language processing and deep learning. Applied Sciences, 13(9), Article 5275.
Castillo, E., Dhaduvai, S., Liu, P., Thakur, K.-S., Dalton, A., & Strzalkowski, T. (2020). Email threat detection using distinct neural network approaches. In A. Bhatia & S. Shaikh (Eds.), Proceedings for the First International Workshop on Social Threats in Online Conversations: Understanding and Management (pp. 48–55). European Language Resources Association. https://aclanthology.org/2020.stoc-1.8/ ACL Anthology
Chebil, W., Wedyan, M., Alazab, M., Alturki, R., & Elshaweesh, O. (2023). Improving semantic information retrieval using multinomial naive Bayes classifier and Bayesian networks. Information, 14(5), Article 272. https://doi.org/10.3390/info14050272
Danyal, M. M., Khan, S. S., Khan, M., Ghaffar, M. B., Khan, B., & Arshad, M. (2023). Sentiment analysis based on performance of linear support vector machine and multinomial naïve Bayes using movie reviews with baseline techniques. Journal on Big Data, 5(1), 1–18. https://doi.org/10.32604/jbd.2023.041319 techscience.com
Diallo, R., Edalo, C., & Awe, O. O. (2025). Machine learning evaluation of imbalanced health data: A comparative analysis of balanced accuracy, MCC, and F1 score. In O. O. Awe & E. A. Vance (Eds.), Practical statistical learning and data science methods: Case studies from LISA 2020 Global Network, USA (pp. 283–312). Springer. https://doi.org/10.1007/978-3-031-72215-8_12
Do, N. Q., Selamat, A., Krejcar, O., Herrera-Viedma, E., & Fujita, H. (2022). Deep learning for phishing detection: Taxonomy, current challenges and future directions. IEEE Access, 10, 36429–36463. https://doi.org/10.1109/ACCESS.2022.3151903
Doshi, J., Parmar, K., Sanghavi, R., & Shekokar, N. (2023). A comprehensive dual-layer architecture for phishing and spam email detection. Computers & Security, 133, Article 103378. https://doi.org/10.1016/j.cose.2023.103378
Elijah, A. V., Abdullah, A., JhanJhi, N. Z., Supramaniam, M., & Balogun, A. O. (2019). Ensemble and deep-learning methods for two-class and multi-attack anomaly intrusion detection: An empirical study. International Journal of Advanced Computer Science and Applications, 10(9), 520–528. https://doi.org/10.14569/IJACSA.2019.0100969thesai.org
Fang, Y., Zhang, C., Huang, C., Liu, L., & Yang, Y. (2019). Phishing email detection using improved RCNN model with multilevel vectors and attention mechanism. IEEE Access, 7, 56329–56340.
Ghasemlounia, R., Gharehbaghi, A., Ahmadi, F., & Saadatnejadgharahassanlou, H. (2021). Developing a novel framework for forecasting groundwater level fluctuations using bi-directional long short-term memory (BiLSTM) deep neural network. Computers and Electronics in Agriculture, 191, Article 106568.
Gogoi, B., & Ahmed, T. (2022). Phishing and fraudulent email detection through transfer learning using pretrained transformer models. In 2022 IEEE 19th India Council International Conference (INDICON) (pp. 1–6). IEEE. https://doi.org/10.1109/INDICON56171.2022.10040097
Hosseinalipour, A., & Ghanbarzadeh, R. (2022). A novel approach for spam detection using horse herd optimization algorithm. Neural Computing and Applications, 34, 13091–13105. https://doi.org/10.1007/s00521-022-07148-xlink.springer.com
Jabbar, H. G. (2024). Advanced threat detection using soft and hard voting techniques in ensemble learning. Journal of Robotics and Control, 5(4), 1104–1116. https://doi.org/10.18196/jrc.v5i4.22005
Jamshidzadeh, Z., Ehteram, M., & Shabanian, H. (2024). Bidirectional long short-term memory (BiLSTM)-support vector machine: A new machine learning model for predicting water quality parameters. Ain Shams Engineering Journal, 15(3), Article 102510. https://doi.org/10.1016/j.asej.2023.102510
Jang-Jaccard, J., & Nepal, S. (2014). A survey of emerging threats in cybersecurity. Journal of Computer and System Sciences, 80(5), 973–993.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539
Madisetty, S., & Desarkar, M. S. (2018). A neural network-based ensemble approach for spam detection in Twitter. IEEE Transactions on Computational Social Systems, 5(4), 973–984. https://doi.org/10.1109/TCSS.2018.2878852
Malhotra, P., & Malik, S. (2022). Spam email detection using machine learning and deep learning techniques. SSRN. https://doi.org/10.2139/ssrn.4145123
Michael, N. E., Bansal, R. C., Ismail, A. A. A., Elnady, A., & Hasan, S. (2024). A cohesive structure of bi-directional long-short-term memory (BiLSTM)-GRU for predicting hourly solar radiation. Renewable Energy, 222, Article 119943. https://doi.org/10.1016/j.renene.2024.119943
Mohammed, M. T., & Rashid, O. F. (2023). Document retrieval using term frequency inverse sentence frequency weighting scheme. Indonesian Journal of Electrical Engineering and Computer Science, 31(3), 1478–1485. https://doi.org/10.11591/ijeecs.v31.i3.pp1478-1485
Nafis, N. S. M., & Awang, S. (2021). An enhanced hybrid feature selection technique using term frequency-inverse document frequency and support vector machine-recursive feature elimination for sentiment classification. IEEE Access, 9, 52177–52192.
Pan, W., Li, J., Gao, L., Yue, L., Yang, Y., Deng, L., & Deng, C. (2022). Semantic graph neural network: A conversion from spam email classification to graph classification. Scientific Programming, 2022, Article 6737080. https://doi.org/10.1155/2022/6737080 Wiley Online Library
Ravi Kumar, G., Venkataiah, V., Sivaiah, B., Kavitha Rani, B., Murali, K., & Swathi, G. (2026). An assessment on the performance of a composite learning system for prediction using precision, recall, accuracy, and F1-score for rainfall. In A. Kumar & S. Mozar (Eds.), Proceedings of the 7th International Conference on Communications and Cyber Physical Engineering (pp. 889–897). Springer. https://doi.org/10.1007/978-981-95-0269-1_101
Roy, P. K., Singh, J. P., & Banerjee, S. (2020). Deep learning to filter SMS spam. Future Generation Computer Systems, 102, 524–533. https://doi.org/10.1016/j.future.2019.09.001
Salunkhe, A. (2021). Attention-based bidirectional LSTM for deceptive opinion spam classification [Preprint]. arXiv. https://arxiv.org/abs/2112.14789
Shreyashree, S., Sunagar, P., Rajarajeswari, S., & Kanavalli, A. (2022). A literature review on bidirectional encoder representations from transformers. In S. Smys, V. E. Balas, & R. Palanisamy (Eds.), Inventive computation and information technologies: Proceedings of ICICIT 2021 (pp. 305–320). Springer. https://doi.org/10.1007/978-981-16-6723-7_23 Springer Nature Link
Sujon, K. M., Hassan, R., Choi, K., & Samad, M. A. (2025). Accuracy, precision, recall, F1-score, or MCC? Empirical evidence from advanced statistics, ML, and XAI for evaluating business predictive models. Journal of Big Data, 12, Article 268. https://doi.org/10.1186/s40537-025-01313-4
Valecha, R., Mandaokar, P., & Rao, H. R. (2022). Phishing email detection using persuasion cues. IEEE Transactions on Dependable and Secure Computing, 19(2), 747–756. https://doi.org/10.1109/TDSC.2021.3118931
Vrbančič, G., Fister, I., Jr., & Podgorelec, V. (2018). Swarm intelligence approaches for parameter setting of deep learning neural network: Case study on phishing websites classification. In Proceedings of the 8th International Conference on Web Intelligence, Mining and Semantics (pp. 1–8). Association for Computing Machinery.
Yacouby, R., & Axman, D. (2020). Probabilistic extension of precision, recall, and F1 score for more thorough evaluation of classification models. In S. Eger, Y. Gao, M. Peyrard, W. Zhao, & E. Hovy (Eds.), Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems (pp. 79–91). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.eval4nlp-1.9 ACL Anthology
Yao, L., Mao, C., & Luo, Y. (2019). Graph convolutional networks for text classification. Proceedings of the AAAI Conference on Artificial Intelligence, 33(1), 7370–7377. https://doi.org/10.1609/aaai.v33i01.33017370
Zamir, A., Khan, H. U., Iqbal, T., Yousaf, N., Aslam, F., Anjum, A., & Hamdani, M. (2020). Phishing web site detection using diverse machine learning algorithms. The Electronic Library, 38(1), 65–80. https://doi.org/10.1108/EL-05-2019-0118
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Mushtaq Talib Mahdi
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
