Testing Big Data application is more a verification of its data processing rather than testing the individual features. It demands a high level of testing skills as the processing is very fast.

1. Big Data
Testing Strategies
Vandana Yadav
QA Consultant
Knoldus Software LLP

2. AGENDA
➔ Why Big Data ??
➔ Areas Of Big Data and Some Use Case
➔ What is Big Data Testing
➔ Challenges we face in Big Data Testing
➔ Approach or strategies to be follow while Big Data testing

3. What is Big Data ??

4. Why Big Data ??

5. ❖ Cost Savings
❖ Time Reductions
❖ New Product Development
❖ Understand the market conditions
❖ Control online reputation

6. Areas Of Big Data

7. 360° View of the Customer
It include demographic data, like customers’ names, addresses, household income and family
members, as well as sales information about which types of policies the customers hold.
It could also pull information from the company’s customer relationship management (CRM)
solution about the customers’ past interactions with the firm and even provide links to transcripts
of recent calls, email messages or chat sessions.

8. Price Optimization
● For any company, the goal is to set prices so that they maximize their
income.
● If the price is too high, they will sell fewer products, decreasing their net
returns. But if the price is too low, they may leave money on the table.

9. What is Big Data Testing ??

11. Testing Big Data application is more a verification of its data processing rather than testing the
individual features.
It demands a high level of testing skills as the processing is very fast. Processing may be of three
types
Batch Processing : Batch processing is where the processing happens of blocks of data that have
already been stored over a period of time.
Real Time Processing : Processing is done on real Data.
Interactive Processing: In interactive processing Data is already stored and analysed.

13. What We Can Test In Big Data Application
Testing in big data projects is typically related to :
Functionality
Performance
Database

14. DataBase Testing can be divided into three steps:
Step 1: Data Staging Validation
Step 2: Process Validation
Step 3: Output Validation Phase

15. Data Staging Validation
● Correct data is pulled in hadoop system.
● Correct Data is extracted and loaded at correct HDFS
location.
● Compare source data with loaded data on Hadoop.

16. Process Validation
In this step the tester validates that the data obtained after
processing through the big data application is accurate. This
also involves testing the accuracy of the data generated from
Map Reduce or similar processes.

17. Output Validation Phase
In this step the tester validate that the output from the big data
application is correctly stored in the data warehouse.
They also verify that the data is accurately being represented in the
business intelligence system or any other target UI.

18. Performance Testing
Performance testing includes testing of job completion time,
memory utilization, data throughput.
Performance testing of the big data application focuses on the
following areas.
● Data Loading And Throughput
● Data Processing Speed
● Sub-System Performance

19. Performance Testing Approach

20. Functional Testing of Big Data Applications
Functional testing of the applications is quite similar in nature to testing of
normal software applications.
Functional testing of big data applications is performed by testing the front end
application based on user requirements.
The front end can be a web based application which interfaces with Hadoop
(or a similar framework on the back end).

21. Challenges we face in Big Data Testing
● Automation
Automated tools are not equipped to handle unexpected problems that arise during testing
● Large Dataset
○ Need to verify more data and need to do it faster
○ Need to automate the testing effort
○ Need to be able to test across different platform

22. Approach or strategies to be follow while Big Data
testing

23. Identify Your Requirements:
S: Specific
M: Measurable
A: Attainable
R: Relevant
T: Time Based

24. Identify Infrastructural Changes:
● As we know infrastructure of data play a major role in big Data Testing
that's why We should always need to have an eye on Company’s
database, As analysis on old Data will not help in any growth of the
organization.
● If the old company data was stored in traditional formats it might not
facilitate the running of complex algorithms and analysis.

25. Establish Talent Pool:
● Human Resources is one of the most critical aspects of Big Data
Testing.
● Your Big Data team must have statisticians to make sense out of
data, business analysts to communicate insights to the decision
makers

26. Obsess Over Customer Satisfaction:
● The key use of Big Data is to generate insights that can help companies
serve their customers in a better way.
● Customer oriented marketing is the new way of approaching the market
and making revenues.
● At the end of the day, you need to communicate to your customer that
you are there to solve a problem and not just to make money.

27. Be Agile:
It is an universal truth that we can not ensure that our planned objective will execute
in planned way, There may exist many obstacles which were initially unknown while
implementing disruptive Technologies.
Always Ready to face challenges during the development phase
We might need to adjust our budget, people based on the circumstances and
insights you gather.
It is best to start with a high-level plan and make changes as the need be

28. References
https://www.guru99.com/big-data-testing-functional-performance.html
http://www.cigniti.com/blog/5-big-data-testing-challenges/
http://istqbexamcertification.com/big-data-testing/
http://www.testbytes.net/blog/big-data-testing-strategy-methodology/