REAL-TIME DATA STREAMING WITH ANOMALY DETECTION AND ALTERING

Abstract-Organizations generate huge amount of data from transactions, user interactions and system activities. In traditional methods the data are collected and stored first and then analyze it using batch processing techniques. Because of this, they cannot provide the real-time insights or alerts which cause delays in detecting critical issues such as fraud, sudden drops in sales or inventory shortages. To design and implement a real time streaming, monitoring and alerting using python and GCP services. A data generator is developed by python to produce live e-commerce data and send it to Google cloud pub/sub. A Python Subscriber process this data in real time and analyzes it instantly and it use rule-based anomaly detection techniques to detect unusual patterns. If an unusual or abnormal activity is detected the system immediately sends an alert via email. All analyzed data including anomalies are stored in BigQuery for monitoring and reporting. It helps organizations to monitor continuously and respond quickly for critical issues and take better decisions by using real-time data. The project illustrate how real-time streaming and alerting can help to improve efficiency, reliability and performance of Business in modern environments.