Integrating Heterogeneous ETL Pipelines: Towards Unified Data Processing Across Cloud and Legacy Systems

Abstract

As organizations maintain a mix of cloud-native and legacy systems, seamless integration of ETL processes across platforms has emerged as a critical technical bottleneck. This article addresses the challenge of heterogeneous ETL integration by proposing a metadata-driven abstraction layer that decouples data transformation logic from execution environments. We introduce a three-tier architecture comprising interface, orchestration, and execution layers that enable platform-agnostic ETL orchestration while preserving system-specific optimizations. The interface layer provides a unified metadata schema capturing data lineage, transformation rules, and quality constraints. The orchestration layer employs a plugin-based architecture with adapters for Apache Airflow, Talend, Apache Spark, and cloud-native services, translating abstract ETL definitions into platform-specific execution plans. The execution layer coordinates actual processing engines through standardized telemetry interfaces. Central to this architecture is a graph-based metadata repository serving as the single source of truth for pipeline definitions and data lineage. Experimental validation across three operational scenarios demonstrates significant improvements in data consistency, development velocity, code reusability, monitoring capabilities, resource utilization, and cost reduction. The architecture successfully addresses technical challenges, including schema evolution, network latency, type system inconsistencies, and coordination overhead through specialized solutions. The proposed model enables incremental modernization strategies that preserve existing technology investments while progressively adopting cloud-native capabilities, providing a practical solution for enterprises undergoing digital transformation.