Data Pipeline Installation Quality
•
0 likes•189 views
At this presentation Irina Pashkova (QA Lead at GreenM) talks about Data Pipeline Installation Quality.
Recommended
More Related Content
What's hot
What's hot (18)
Similar to Data Pipeline Installation Quality
Similar to Data Pipeline Installation Quality (20)
More from GreenM
Recently uploaded
Recently uploaded (20)
Data Pipeline Installation Quality
- 1. Pipeline Testing Story IRINA PASHKOVA QA Lead, GreenM
- 2. Agenda 1. Regression ETL testing 2. Non-functional ETL testing 3. Functional ETL Testing
- 3. Puppy to Play with Daily Runs Full Refresh Mode 300 Customers ~ 500 Mln rec / tab ~ 5h ETL time
- 4. Better & Faster – ETL Evolution or Regression ETL Testing
- 5. ETL Extract Transform Load Operations Storages or DATA SOURCES Reporting Oriented Data Marts or TARGETS
- 6. New Pipeline Version Regression Testing Non-Functional Testing • Same Sources & Targets • Same Transformation Rules • Previous fully tested version of ETL available
- 7. Regression via Reference Data Schema • Exclude • Tracking fields • New functionality Data • Clean up Test Schema • Run Smoke suite first SOURCE TESTED TARGET REFERENCE TARGET NEW ETL VERSION PROD ETL VERSION Regression Testing
- 8. FitNesse for ETL Regression • Config files • Connections • Tab parameters • Fixtures • Non-empty tab • No duplicates • Counts match • Content match Regression Testing
- 9. FitNesse for ETL Regression Regression Testing
- 10. Regression Challenges Long run time of ETL Big Data volume Regression Testing Time waste waiting for a fix / change Hang up tests
- 11. Manual Inspections • Configurations: • Connections • Run mode • Pipeline Steps order & dependencies • Source & Target Tabs • ETL code queries Regression Testing: Challenges
- 12. Set the Limits! • “Partial” run & Extract re-using • Limit compared data • Set timeout in tests • Model missing data Extract Transform Load Regression Testing: Challenges
- 13. Take Care about Production Support Group or Non-functional ETL Testing
- 14. Non-functional Pipeline Testing • Performance • Security • Load/ Stress • Scalability • Usability • Reliability Non-Functional Testing
- 15. Usability Testing • Easy to • identify current state • find/read Error info • re-configure • Flexible Start • Documentation Non-Functional Testing
- 16. • Risks assessment • Failure simulation • Volume simulation Reliability Testing Non-Functional Testing
- 17. Reliability Testing Challenges Hidden Risks Underestimation of severity Dependency on 3d party services Underestimation of probability Communication gaps Non-Functional Testing: Challenges
- 18. Be Informed! • Monitor Services Logs • Organize Recovery Training • Be specific with to-do’s Non-Functional Testing: Challenges
- 19. We’re done! Aren’t we?
- 20. Add Analytics for a New Business Module… please
- 21. New Data Module Creation or Functional ETL Testing
- 22. Data Warehouse Testing Extract Transform Load SOURCE TARGET Test Underlying Data Test Data Model Balancing Tests Data Quality Tests Smoke Tests Balancing Tests Balancing Tests
- 23. Test Underlying Data 1. Gather info – bridge gaps! 2. Break rules that can be broken 3. Draft a Troubleshooting doc Source Area Testing
- 24. Test Target Data Model 1. Naming convention 2. Optimal base for Visualization 3. Testability checks Data Mart Structure Testing
- 25. Functional ETL Testing • Smoke Tests • Target Data Quality tests: • Type • Constraint • Data Plausibility • Logical Constraints ! Create similar / relevant tests where applicable for Source to help with further debugging Functional ETL Testing
- 26. Functional ETL Testing • Balancing Tests: • Study/ Create Specification • Test Minus Queries Assertions via mutated data • Do both-sides comparison Functional ETL Testing
- 27. Balancing Tests • One all-data storage • AWS Glue & Athena Functional ETL Testing
- 28. Most Common bugs • Count Mismatch (incl. Duplicates) • Anomalies issues: Null or Length relevant • Date relevant calculations Functional ETL Testing
- 29. ETL Testing Challenges • Tests Complexity • Unpredictable slow work of AWS Athena • Impossible to check each single record Functional ETL Testing
- 30. Visualization in Data QA • Source Data Analysis • Target Quality Dashboard • Dedicated resources & Test Results visualization Functional ETL Testing
- 31. Ongoing Support • Data Integrity Project • Ongoing Logs Analysis • Monitoring Rules & Alarms Testing in Production Data Pipeline
- 32. Key Takeaways • ETL verification is not that bad • Know your data • Be ready to meet Monsters • Long ETL duration • Big Data Volume • Difference of Test Data from Prod
- 33. Your questions