Testing in Big Data world

Every day more and more organizations trying to implement their own Data Analytics solutions and call them Big Data solution. These solutions involve processing of huge volume of structured or unstructured data, processed across different nodes, using specific languages such as “Map-reduce” or “Hive”, with Cloud-based infatuations like AWS. And, as usual, a robust testing strategy must be defined to ensure that the functional and non-functional requirements are met and that the data conforms to acceptable quality. 

Main challenges

The Big Data testing challenges come in determining how to validate an entire data set consisting of millions of records, how to validate transformation of those huge number of records, challenge how to validate system architecture before millions of records came.