Software quality assurance days
International Conference of
Software Quality Assurance
sqadays.com
Minsk. May 29–30 2015
Kovalenko Andrey
Lohika. Odesa, Ukraine
Way to performance testing. How to pass it from
automation.
Performance testing for Automation QA – why and how?
Andrey Kovalenko
6 last years in IT - again
3 years of developing and implementing
distributed protected networks
3 last years - performance engineer
kovalenko.andrey.odessa
About
Performance testing for Automation QA – why and how?
What is performance?
Load testing
- to verify application behavior under normal
and peak load conditions.
Stress testing
- To determine or validate an application’s
behavior when it is pushed beyond normal or peak load
conditions.
Capacity testing
- To determine how many users and/or
transactions a given system will support and still meet
performance goals.
Performance testing for Automation QA – why and how?
Why do we need performance?
Performance testing for Automation QA – why and how?
Performance VS Functional
testing
•
Based non-functional aspects of the
software system
•
Non functional testing should be performed
after functional testing
•
Almost impossible to run manually
•
Describes
HOW GOOD
the product works
•
Based on functional specification
•
Functional testing is executed
first
•
Easy to do manual testing
•
Describes
WHAT
the product does
Functional manual
Functional manual
Performance
Performance
Functional Automation
Functional Automation
•
Based on functional specification
•
Automation should be performed after
manual functional testing
•
Impossible to run manually
•
Describes
WHAT
the product does
Performance testing for Automation QA – why and how?
Software Testing Life Cycle
Performance testing for Automation QA – why and how?
STLC for performance testing
1.
Identify requirements
1.
Features related to performance – with devs
2.
Modules related to performance – with devs and FA/SA
3.
Analyzing application’s tiers
2.
Acceptance criteria (Performance KPIs):
1.
Main TRT values – from your FA and public sources
2.
Main system resources threshold – from your experience
3.
Strategy planning and design:
1.
Customer scenario use – from your FA
2.
2.
Transaction defining
Transaction defining
– from your experience and FA
3.
Scripts writing
4.
4.
Production-like environment establishing
Production-like environment establishing
–
can be a problem
can be a problem
5.
5.
Monitors for resources – native, from specific tools
Monitors for resources – native, from specific tools
Performance testing for Automation QA – why and how?
STLC for performance testing
4.
Execute & Analyze
1.
Run performance scenario
2.
Collect results and resources’ consumption data
3.
Analyze results, comparing with KPIs
4.
4.
Analyze source code to find possible bottlenecks
Analyze source code to find possible bottlenecks
5.
5.
Analyze logs, heap GC, dumps
Analyze logs, heap GC, dumps
6.
6.
Create recommendations for application tuning – for application server,
Create recommendations for application tuning – for application server,
code, hardware, software
code, hardware, software
7.
Prepare reports for stakeholder
Performance testing for Automation QA – why and how?
Performance VS Automation
•
Multi-user scenario
•
Interested in response time
•
Should be ran on production-like env
•
Resources must be monitored,
application should be profiled
•
Single user
•
Interested in response data
•
Can be ran on any hardware
•
No needs to monitor resources
Automation
Automation
Performance
Performance
Performance testing for Automation QA – why and how?
Automation->Performance
(pretty straight ahead)
1.
Exclude from you automation scripts data validation
2.
Allocate transactions you will measure in your code
3.
Frame these pieces of code in time measurement constructions
4.
Get environment
5.
Prepare set of monitors for system and application resources
6.
6.
Get a headache with scenario timing
Get a headache with scenario timing
7.
7.
After running collect results and prepare report for stakeholder
After running collect results and prepare report for stakeholder
Performance testing for Automation QA – why and how?
Automation->Performance
(performance tools)
1.
Check your IDE for performance testing ability or
2.
Select any 3
rd
party tools for your needs
3.
Exclude from you automation scripts data validation
4.
Set transactions with 3
rd
party performance tool
5.
Get set of monitors from 3
rd
party performance tool
6.
Get environment
7.
Run scenario and get reports
automatically
automatically
Performance testing for Automation QA – why and how?
Automation->Performance
(performance tools)
Performance testing for Automation QA – why and how?
Automation->Performance
(performance tools)
Performance testing for Automation QA – why and how?
Automation->Performance
(performance tools)
http://www.opensourcetesting.org/performance.php
Performance testing for Automation QA – why and how?
Questions?
Тестирование производительности для специалистов по автоматизации - зачем и как?
