HealthCraft: A Precision Model for Smart Resource Optimisation in Dynamic Big Data Healthcare Environments

Abstract

Cloud computing offers scalable computing and storage capabilities to handle massive healthcare data. When processing large-scale data, keeping the resource cost reasonable is crucial. Nonetheless, resource utilisation is frequently inefficient because of the inherent complexity and heterogeneity of distributed computing frameworks. In addition, it is challenging to model resource utilisation from real fault-occurring cloud systems. This study proposes an automated online resource utilisation prediction model that combines machine learning (ML) methods with automated log data preprocessing to predict future resource consumption. It allows smart and adaptable allocation of resources in large cloud-based data infrastructures suffering from typical failures like CPU, memory, network, and data locality problems. Using the Hadoop framework on a cloud cluster of 30 worker nodes, our model predicts resource utilisation with up to 97.3% accuracy - outperforming the other baseline models evaluated. In addition, our system accurately recognises resource bottlenecks. It reduces execution time by up to 30%, even in fault-injected environments, implying that it is robust enough for real-time big data analytics.