A Machine Learning Framework for Intelligent Log Anomaly Detection and Root Cause Analysis in IT Infrastructure

Modern IT infrastructures generate large volumes of log data.This log data is generated by various components of the system like servers, applications,and network devices. Log data produced contains crucial information regarding the system’s behavior and failures.Analysis of the huge volume of log data is a time-consuming and inefficient process. This paper proposes a system based on the machine learning approach to detect log anomalies and perform root cause analysis of the system failures.The system processes the log data and converts the log data into numerical values using the Term Frequency-Inverse Document Frequency algorithm.Finally, the system uses the Isolation Forest algorithm to analyze the numerical values and detect anomalies in the system.After detecting the anomalies in the system, the system uses the Random Forest classifier to determine the potential root cause of system failure like memory overflow, disk failures, network failures, and security issues.Additionally, the system provides a facility to monitor the system’s logs using the visual representation of the system’s logs in the form of charts and graphs.The result of the experiment proves the system’s ability to detect log anomalies and help system administrators to detect system failures.