A Scalable Cloud-Integrated Architecture for Real Time Processing of Large-Scale E-Commerce Data

Abstract: The growth of the e-commerce platforms has resulted
in the huge amount of real time user interaction data that poses
huge challenges to traditional data processing and recommender
systems in these areas in terms of scalability, latency and
reliability. Existing systems are frequently based on static
pipelines which are batch based without the ability to be dynamic
and adapt to the user behavior. This work presents a cloud-based
and real-time data processing and intelligent recommendation
architecture towards high velocity and large scale of e-commerce
data streams. The framework is a combination of Kafka-based
data ingestion, cloud data lakes, distributed processing using
Apache Spark and Flink, feature engineering using real-time
indexing and containerized microservices orchestrated using
Kubernetes. The system is tested on Amazon Product Review
dataset
with different workloads. Quantitative results
demonstrate the improvement of the performance with 94.5% of
accuracy, 93.5% of F1 score, 0.96 of area under the curve (AUC),
throughput up to 120,000 events per second and average response
latency less than 150 ms under high load. The results prove that
the proposed architecture is a scalable, fault-tolerant and
efficient method for the next-generation intelligent e-commerce
platforms.