Prediction-Based Cost-Efficient Resource Allocation Scheme for Big Data Streams in Cloud Systems

Resource allocation (RA) for cloud-based big data systems mainly consists of assigning the tasks to suitable resources. A dynamic unbalanced load in individual virtual machines (VMs) causes the complete inefficient utilization of resources. Apart from this, high execution cost and lower performance are the other issues of RA. A predictive cost-efficient resource allocation (PCRA) scheme for big data applications is developed. In this scheme, the user specifies the number of executors, cores, memory, etc., for the required application. Then, the task accomplishment period of a solicitation is predicted centered on the number of initiators and the specific properties of the application. Then, VMs are determined for the task requests, such that the number of executors is sufficient to minimize the task completion time and total cost. Some idle executors are released to return some resources to the hosts which could be revoked in the future if needed.