Log Frequency Analysis for Anomaly Detection in Cloud Environments at Ericsson
Keywords:
Log Analysis, Log Frequency Patterns, Anomaly Detection, Machine Learning, Cloud EnvironmentsAbstract
Log analysis monitors system behavior, detects er- rors and anomalies, and predicts future trends in systems and applications. However, with the continuous evolution and growth of systems and applications, the amount of log data generated on a timely basis is increasing rapidly. This causes an increase in the manual effort invested in log analysis for error detection and root cause analysis. The current automated log analysis techniques mainly concentrate on the messages displayed by the logs as one of the main features. However, the timestamps of the logs are often ignored, which can be used to identify temporal patterns between the logs which can form a key aspect of log analysis in itself. In this paper, we share our experiences of combining log frequency based analysis with log message based analysis, which thereby helped in reducing the volume of logs which are sent for manual analysis for anomaly detection and root cause analysis.
References
Xavier Baril, Oihana Coustie´, Josiane Mothe, and Olivier Teste. 2020. Application Performance Anomaly Detection with LSTM on Temporal Irregularities in Logs. In Proceedings of the 29th ACM International Conference on Information & Knowl- edge Management (Virtual Event, Ireland) (CIKM ’20). Associa- tion for Computing Machinery, New York, NY, USA, 1961–1964. https://doi.org/10.1145/3340531.3412157
Zhuangbin Chen, Jinyang Liu, Wenwei Gu, Yuxin Su, and Michael R. Lyu. 2021. Experience Report: Deep Learning-based System Log Analysis for Anomaly Detection. https://arxiv.org/abs/2107.05908
Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017. DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (Dallas, Texas, USA) (CCS ’17). Association for Computing Machinery, New York, NY, USA, 1285–1298. https://doi.org/10.1145/3133956.3134015
Mostafa Farshchi, Jean-Guy Schneider, Ingo Weber, and John Grundy. 2015. Experience report: Anomaly detection of cloud application operations using log and cloud metric correlation analysis. In 2015 IEEE 26th International Symposium on Software Reliability Engi- neering (ISSRE). 24–34. https://doi.org/10.1109/ISSRE.2015.7381796
Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R. Lyu. 2021a. A Survey on Automated Log Analysis for Reliability Engineering. ACM Comput. Surv. 54, 6, Article 130 (jul 2021), 37 pages. https://doi.org/10.1145/3460345
Shilin He, Pinjia He, Zhuangbin Chen, Tianyi Yang, Yuxin Su, and Michael R. Lyu. 2021b. A Survey on Automated Log Analysis for Reliability Engineering. ACM Comput. Surv. 54, 6, Article 130 (jul 2021), 37 pages. https://doi.org/10.1145/3460345
Shilin He, Xu Zhang, Pinjia He, Yong Xu, Liqun Li, Yu Kang, Minghua Ma, Yining Wei, Yingnong Dang, Saravanakumar Rajmohan, and Qingwei Lin. 2022. An Empirical Study of Log Analysis at Microsoft. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore) (ESEC/FSE 2022). As- sociation for Computing Machinery, New York, NY, USA, 1465–1476. https://doi.org/10.1145/3540250.3558963
S.E. Hove and B. Anda. 2005. Experiences from conducting semi-structured interviews in empirical software engineering research. In 11th IEEE International Software Metrics Symposium (METRICS’05). 10 pp.–23. https://doi.org/10.1109/METRICS.2005.24
Tong Jia, Yifan Wu, Chuanjia Hou, and Ying Li. 2021. LogFlash: Real-time Streaming Anomaly Detection and Diagnosis from System Logs for Large-scale Software Systems. In 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE). 80–90. https://doi.org/10.1109/ISSRE52982.2021.00021
Hari Kanagala and V.V. Krishnaiah. 2016. A comparative study of K-Means, DBSCAN and OPTICS. 1–6. https://doi.org/10.1109/ICCCI.2016.7479923
Steven Locke, Heng Li, Tse-Hsun Peter Chen, Weiyi Shang, and Wei Liu. 2022. LogAssist: Assisting Log Analysis Through Log Summarization. IEEE Transactions on Software Engineering 48, 9 (2022), 3227–3241. https://doi.org/10.1109/TSE.2021.3083715
Tarannum Shaila Zaman, Xue Han, and Tingting Yu. 2019. SCMiner: Localizing System-Level Concurrency Faults from Large System Call Traces. In 2019 34th IEEE/ACM Interna- tional Conference on Automated Software Engineering (ASE). 515–526. https://doi.org/10.1109/ASE.2019.00055
Pengpeng Zhou, Yang Wang, Zhenyu Li, Xin Wang, Gareth Tyson, and Gaogang Xie. 2020. LogSayer: Log Pattern-driven Cloud Component Anomaly Diagnosis with Machine Learning. In 2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS). 1–10. https://doi.org/10.1109/IWQoS49365.2020.9212954
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Prathyusha Bendapudi, Vera Simon, Deepika Badampudi
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
License Terms:
Except where otherwise noted, content on this website is lincesed under a Creative Commons Attribution Non-Commercial License (CC BY NC)
Use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes, is permitted.
Copyright to any article published by WiPiEC retained by the author(s). Authors grant WiPiEC Journal a license to publish the article and identify itself as the original publisher. Authors also grant any third party the right to use the article freely as long as it is not used for commercial purposes and its original authors, citation details, and publisher are identified, in accordance with CC BY NC license. Fore more information on license terms, click here.