Primary Access Data Lake (PADL), is a self-service analytics platform for J&J's divisions like Surgical Vision, Vision Care, Tear Science, etc. As part of DataOPs implementation, j&J decided to automate the critical components like ETL testing, data quality audit, data integrity validation and transformation testing. As part of this initiative, they have built a custom built python/robot framework for test automation. But it has proven out to be expensive and time consuming to develop and maintain the automated test processes.The output generated by the python framework is overly technical and confusing for the PADL'5 SQA team. This framework is missing the orchestration process, resulting in the need for manual triggers. They are looking for a self-service no-coding solution that would automate all the DataOps processes, which can easily integrate with enterprise systems (e.g Jira, X-ray, Control-M etc.) in scope for PADL. The desired solution should also require a minimal learning curve, and must be able to handle terabytes of data volumes.
By using the RightData platform, J&J's PADLSQA team automated all the DQA and DQC processes and created an automation framework for all the upcoming release's regression testing, functional testing, smoke testing and on-going data quality controls for the PADL production landscape. This framework has been showcased and obtained approvals from J&J's enterprise architecture board; to leverage PADL DataOps implementation as a model project for upcoming data initiatives by other divisions, regions, and business functions.