Data Workflow Acceleration: A Smart System for Redundancy Elimination in Machine Learning Pipelines (Journal of Information Technology and Digital World, 2025)
This paper presents a novel framework designed to significantly accelerate these pipelines. By establishing granular data provenance and implementing intelligent reuse strategies, our system efficiently identifies and eliminates redundant computations. This approach tackles key challenges such as managing extensive data traces and accommodating non-deterministic operations through advanced duplication and hierarchical reuse techniques. Our framework seamlessly integrates with existing data processing environments, demonstrating substantial efficiency improvements and fostering faster iterative development cycles for data professionals.
Keywords: Data Provenance; Redundant Computation Elimination; Deduplication; Hierarchical Reuse; Non- Deterministic Operations; Pipeline Optimization; Data Processing Frameworks; Computational Efficiency; Intelligent Reuse Strategies; Iterative Development Acceleration