An Approach to Examine and Recognize Anomalies on Cloud Computing Platforms with Machine Learning Concepts
Keywords:
Anomaly, Cloud, Detection, Monitoring, Machine LearningAbstract
Cloud computing is one of the most rapidly growing computing concepts in today's information technology world. It connects data and applications from various geographical locations. A large number of transactions and the hidden infrastructure in cloud computing systems have presented the research community with several challenges. Among these, maintaining cloud network security has emerged as a major challenge. It is critical to address issues in the quickly changing cloud computing market in order to guarantee that businesses can fully utilize cutting-edge technology, uphold strong security protocols, and maximize operational effectiveness. Businesses that successfully navigate these obstacles can maintain their competitiveness in a dynamic digital ecosystem by improving scalability, leveraging the flexibility provided by the cloud, and adapting to technological changes with ease. Anomaly detection (or outlier detection) is the identification of unusual or suspicious data that differs significantly from the majority of the data. Research on anomaly detection in cloud network data is crucial because it enables businesses to more rapidly and efficiently recognize potential security threats, network performance concerns, and other issues. Recently, machine learning methods have demonstrated their efficacy in anomaly detection. This research aimed to introduce a novel hybrid model for anomaly detection in cloud network data and to investigate the performance of this model in comparison to other machine learning algorithms. The research was conducted with the UNSW-NB15 anomaly dataset and employed various feature selection and pre-processing techniques to prepare the data for model training. The hybrid model was built using a combination of Random Forest and SVM algorithms and the process was evaluated using metrics such as F1-Score, Recall, Precision, and Accuracy. The result showed that the hybrid model has 94.23% accuracy and a total time of 109.92s which is the combination of the train time of 100.45s and prediction time of 9.47s. The limitations of the study include the class imbalance problem in the dataset and the lack of real-world applications for testing. The research suggests future work in the application of hybrid models in anomaly detection and cloud network security and the need for further investigation into the potential benefits of such models.
References
A. Vervaet, “MONILOG: An Automated Log-based ANOMALY DETECTION SYSTEM FOR CLOUD computing infrastructures,” 2021 IEEE 37th International Conference on Data Engineering (ICDE), 2021.
Dingde Jiang, Yang Han, Xingwei Wang, Zhengzheng Xu, Hongwei Xu, and Zhenhua Chen, "A time-frequency detecting method for network traffic anomalies," International Conference on Computational Problem-Solving, Li Jiang, China, 2010, pp. 94-97.
B. Wang, Q. Hua, H. Zhang, X. Tan, Y. Nan, R. Chen, and X. Shu, “Research on ANOMALY DETECTION and real-time reliability evaluation with the log of cloud platform,” Alexandria Engineering Journal, vol. 61, no. 9, pp. 7183–7193, 2022.
S. H. Haji and S. Y. Ameen, “Attack and anomaly detection in IOT networks using Machine Learning Techniques: A Review,” Asian Journal of Research in Computer Science, pp. 30–46, 2021.
A. B. Nassif, M. A. Talib, Q. Nasir, and F. M. Dakalbab, "Machine Learning for Anomaly Detection: A Systematic Review," in IEEE Access, vol. 9, pp. 78658-78700, 2021, doi: 10.1109/ACCESS.2021.3083060.
T. Sureda Riera, J.-R. Bermejo Higuera, J. Bermejo Higuera, J.-J. Martínez Herraiz, and J.-A. Sicilia Montalvo, “Prevention and fighting against web attacks through anomaly detection technology. A systematic review,” Sustainability, vol. 12, no. 12, p. 4945, 2020.
M. Ozkan-Okay, R. Samet, Ö. Aslan and D. Gupta, "A Comprehensive Systematic Literature Review on Intrusion Detection Systems," in IEEE Access, vol. 9, pp. 157727-157760, 2021, doi: 10.1109/ACCESS.2021.3129336.
J. Svacina, J. Raffety, C. Woodahl, B. Stone, T. Cerny, M. Bures, D. Shin, K. Frajtak, and P. Tisnovsky, “On vulnerability and Security Log Analysis,” Proceedings of the International Conference on Research in Adaptive and Convergent Systems, 2020.
T. L. Yasarathna and L. Munasinghe, "Anomaly detection in cloud network data," 2020 International Research Conference on Smart Computing and Systems Engineering (SCSE), Colombo, Sri Lanka, 2020, pp. 62-67, doi: 10.1109/SCSE49731.2020.9313014.
T. Hagemann and K. Katsarou, “A systematic review on anomaly detection for cloud computing environments,” 2020 3rd Artificial Intelligence and Cloud Computing Conference, 2020.
A. Alshammari and A. Aldribi, “Apply machine learning techniques to detect malicious network traffic in cloud computing,” Journal of Big Data, vol. 8, no. 1, 2021.
S. Nedelkoski, J. Cardoso and O. Kao, "Anomaly Detection from System Tracing Data Using Multimodal Deep Learning," 2019 IEEE 12th International Conference on Cloud Computing (CLOUD), Milan, Italy, 2019, pp. 179-186, doi: 10.1109/CLOUD.2019.00038.
M. S. Islam, W. Pourmajidi, L. Zhang, J. Steinbacher, T. Erwin and A. Miranskyy, "Anomaly Detection in a Large-Scale Cloud Platform," 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), Madrid, ES, 2021, pp. 150-159, doi: 10.1109/ICSE-SEIP52600.2021.00024.
F. J. Schmidt, “Anomaly detection in cloud computing environments,” thesis.
T. Salman, D. Bhamare, A. Erbad, R. Jain and M. Samaka, "Machine Learning for Anomaly Detection and Categorization in Multi-Cloud Environments," 2017 IEEE 4th International Conference on Cyber Security and Cloud Computing (CSCloud), New York, NY, USA, 2017, pp. 97-103, doi: 10.1109/CSCloud.2017.15.
S. E. Hajjami, J. Malki, M. Berrada and B. Fourka, "Machine Learning for anomaly detection. Performance study considering anomaly distribution in an imbalanced dataset," 2020 5th International Conference on Cloud Computing and Artificial Intelligence: Technologies and Applications (CloudTech), Marrakesh, Morocco, 2020, pp. 1-8, doi: 10.1109/CloudTech49835.2020.9365887.
X. Qiu, Y. Dai, P. Sun and X. Jin, "PHM Technology for Memory Anomalies in Cloud Computing for IaaS," 2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS), Macau, China, 2020, pp. 41-51, doi: 10.1109/QRS51102.2020.00018.
A. Gerard, R. Latif, S. Latif, W. Iqbal, T. Saba and N. Gerard, "MAD-Malicious Activity Detection Framework in Federated Cloud Computing," 2020 13th International Conference on Developments in eSystems Engineering (DeSE), Liverpool, United Kingdom, 2020, pp. 273-278, doi: 10.1109/DeSE51703.2020.9450728.
J. Bogatinovski, S. Nedelkoski, J. Cardoso and O. Kao, "Self-Supervised Anomaly Detection from Distributed Traces," 2020 IEEE/ACM 13th International Conference on Utility and Cloud Computing (UCC), Leicester, UK, 2020, pp. 342-347, doi: 10.1109/UCC48980.2020.00054.
W. Wang, X. Du, D. Shan, R. Qin and N. Wang, "Cloud Intrusion Detection Method Based on Stacked Contractive Auto-Encoder and Support Vector Machine," in IEEE Transactions on Cloud Computing, vol. 10, no. 3, pp. 1634-1646, 1 July-Sept. 2022, doi: 10.1109/TCC.2020.3001017.
C. Raj, L. Khular and G. Raj, "Clustering Based Incident Handling For Anomaly Detection in Cloud Infrastructures," 2020 10th International Conference on Cloud Computing, Data Science & Engineering (Confluence), Noida, India, 2020, pp. 611-616, doi: 10.1109/Confluence47617.2020.9058314.
Y. Yuan, H. Anu, W. Shi, B. Liang and B. Qin, "Learning-Based Anomaly Cause Tracing with Synthetic Analysis of Logs from Multiple Cloud Service Components," 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Milwaukee, WI, USA, 2019, pp. 66-71, doi: 10.1109/COMPSAC.2019.00019.
M. Thill, W. Konen and T. Bäck, "Online anomaly detection on the webscope S5 dataset: A comparative study," 2017 Evolving and Adaptive Intelligent Systems (EAIS), Ljubljana, Slovenia, 2017, pp. 1-8, doi: 10.1109/EAIS.2017.7954844.
M. S. Islam and A. Miranskyy, "Anomaly Detection in Cloud Components," 2020 IEEE 13th International Conference on Cloud Computing (CLOUD), Beijing, China, 2020, pp. 1-3, doi: 10.1109/CLOUD49709.2020.00008.
S. Eltanbouly, M. Bashendy, N. AlNaimi, Z. Chkirbene and A. Erbad, "Machine Learning Techniques for Network Anomaly Detection: A Survey," 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), Doha, Qatar, 2020, pp. 156-162, doi: 10.1109/ICIoT48696.2020.9089465.
I. Aljamal, A. Tekeoğlu, K. Bekiroglu and S. Sengupta, "Hybrid Intrusion Detection System Using Machine Learning Techniques in Cloud Computing Environments," 2019 IEEE 17th International Conference on Software Engineering Research, Management and Applications (SERA), Honolulu, HI, USA, 2019, pp. 84-89, doi: 10.1109/SERA.2019.8886794.
Kithulwatta, W.M.C.J.T., Wickramaarachchi, W.U., Jayasena, K.P.N., Kumara, B.T.G.S., Rathnayaka, R.M.K.T. (2022). Adoption of Docker Containers as an Infrastructure for Deploying Software Applications: A Review. In: Saeed, F., Al-Hadhrami, T., Mohammed, E., Al-Sarem, M. (eds) Advances on Smart and Soft Computing. Advances in Intelligent Systems and Computing, vol 1399. Springer, Singapore. https://doi.org/10.1007/978-981-16-5559-3_21
W. M. C. J. T. Kithulwatta, K. P. N. Jayasena, B. T. G. S. Kumara and R. M. K. T. Rathnayaka, "Docker incorporation is different from other computer system infrastructures: A review," 2021 International Research Conference on Smart Computing and Systems Engineering (SCSE), Colombo, Sri Lanka, 2021, pp. 230-236, doi: 10.1109/SCSE53661.2021.9568323.
W. M. C. J. T. Kithulwatta, K. P. N. Jayasena, B. T. G. S. Kumara and R. M. K. T. Rathnayaka, "Docker Containerized Infrastructure Orchestration with Portainer Container-native Approach," 2022 3rd International Conference for Emerging Technology (INCET), Belgaum, India, 2022, pp. 1-6, doi: 10.1109/INCET54531.2022.9825257.
W. M. C. J. T. Kithulwatta, K. P. N. Jayasena, B. T. G. S. Kumara and R. M. K. T. Rathnayaka, "Performance Evaluation of Docker-based Apache and Nginx Web Server," 2022 3rd International Conference for Emerging Technology (INCET), Belgaum, India, 2022, pp. 1-6, doi: 10.1109/INCET54531.2022.9824303.
Kithulwatta, W.M.C.J.T., Jayasena, K.P.N., Kumara, B.T. and Rathnayaka, R.M.K.T., 2022. Integration With Docker Container Technologies for Distributed and Microservices Applications: A State-of-the-Art Review. International Journal of Systems and ServiceOriented Engineering (IJSSOE), 12(1), pp.1-22.
Jayaweera, M.P.G.K., Kithulwatta, W.M.C.J.T. & Rathnayaka, R.M.K.T. Detect anomalies in cloud platforms by using network data: a review. Cluster Comput 26, 3279–3289 (2023). https://doi.org/10.1007/s10586-023-04055-1
Gayantha, M. H., Kithulwatta, W. M. C. J. T., & Rathnayaka, R. M. K. T. (2022). The Interconnection of Internet of Things and Artificial Intelligence: A Review. In Sri Lankan Journal of Applied Sciences (Vol. 1, Issue 1). https://sljoas.uwu.ac.lk/index.php/sljoas/article/view/45/12
M.H. Gayantha, W.M.C.J.T. Kithulwatta, R.M.K.T. Rathnayaka. Identification of a Machine Learning Architecture for Potato DiseaseClassification Using Leaf Images. Applied Sciences Undergraduate Research Symposium 2022 At: Sabaragamuwa University of Sri Lanka. p. 15.
