End-to-End Big Data Testing: From Event to Data Mart

Errors in big data projects rarely look like “failed” calculations. More often, they appear quietly: rows for a day go missing, duplicates show up, a total becomes thousands of times larger, or a data mart “seems updated”, but the numbers do not match. Then the investigation begins: is the issue in the source, schema, partitions, ETL, or business logic?

In this talk, we will discuss a practical strategy for testing a data pipeline layer by layer: SOURCE → RAW → CORE → DATAMART. We will look at typical defects at each stage, checks that help localize issues faster, and why one large e2e test does not cover all risks.

We will also focus on automating calculation data marts: how to choose a test data format that stays maintainable, and how golden datasets help validate business logic and regression without manual checks.

The goal is to improve data mart quality and reduce root-cause analysis time.