Challenge
Analytics acceleration and data quality control for ERP
Solution
CDC-based ETL tool, DQ monitoring, Data Lineage implementation
Project Phases
1
Data flow analysis
2
CDC algorithm development
3
Data Quality implementation
4
Data Lineage setup
5
Optimization & monitoring
Results
BEFORE
Slow data access, frequent quality incidents
AFTER
5x data access speedup, 80% quality incident reduction
Technologies
SparkScala FPAirflowHadoopGXOpenMetadataClickHouse