AI-Enhanced Data Engineering for High-Performance Big-Data Pro-cessing and Advanced Analytics Optimization

Abstract

The explosion of healthcare data presents a unique opportunity to derive actionable insights through AI-driven big-data engineering. This paper proposes an integrated framework that enhances traditional data engineering pipelines using artificial intelligence (AI) for high-performance big-data processing and advanced analytics op-timization. Leveraging the MIMIC-IV dataset and cutting-edge tools such as Apache Spark, Delta Lake, and machine learning algorithms, the study demonstrates how AI augments extract-transform-load (ETL) opera-tions, improves data quality, and accelerates analytics for clinical decision-making. Results indicate a 48% im-provement in processing speed and a 31% increase in prediction accuracy for patient outcomes compared to traditional approaches. This framework has significant implications for predictive healthcare, hospital resource management, and real-time diagnostics.