Comprehensive Data Quality for DevOps Implementation
Fortune 50, $82+ billion revenue
1The Background
Johnson & Johnson's self-service analytics platform – SAP and Primary Access Data Lake (PADL) – serves divisions such as Surgical Vision, Vision Care, and Tear Science. The initiative required a DevOps implementation for ETL, audit, validation, and transformation testing. The manual version of the Python/robot framework was confusing for the PADL SQA team due to lack of orchestration.
2The Challenge
The PADL team had built a custom Python/robot framework for test automation, but it proved expensive and time-consuming to develop and maintain. The output was overly technical and confusing for the SQA team. The framework was missing orchestration processes, requiring manual triggers for validation scenarios.
Technical Environment
- SAP source systems
- Primary Access Data Lake (PADL)
- Jira and X-Ray integration
- Control-M for orchestration
- Terabytes of data from multiple sources
3The Solution
DataTrust provided a self-service no-code solution that automated the DataOps process and integrated with PADL enterprise systems including Jira, X-Ray, and Control-M. The solution managed terabytes of data from multiple sources with minimal learning curve.
PADL SAP ECC to Email Setting Process
The RightData Edge
DataTrust gave J&J a baseline framework that established both a technical and process-oriented approach to manage data in the data lake. J&J's enterprise architecture board reviewed and approved the PADL data quality framework and leveraged it as a model DataOps implementation for upcoming initiatives by other divisions, regions, and functions.
4The Results
"We use our RightData instance for multiple uses such as Bulk Data Comparisons between a source and target database, Data Profiling, Testing Scenarios, and most importantly Data Integrity Controls. RightData provides a single tool option for many data comparison and data auditing requirements. The vendor is very responsive to our needs and has incorporated several suggestions into the RightData tool for extension and ease of use."
J&J PADL Team
Primary Access Data Lake
