Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
1
Softw
ar
e qual
ity assu
r
anc
e
days
Internat
ional
Confer
ence
of
Sof
twar
e
Qualit
y
As
sur
ance
sq
adays
.com
M
insk
.
M
ay 25
–
26,
201
8
Ale
xander
Podel
k
o
Or
ac
le. Sta
m
fo
r
d,
CT
, USA
Cont
inuous Perform
ance
T
esting
Myths
and
Realities
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
2
“Traditional”
Approach
•
Lo
ad
/ Pe
rform
an
ce
Testin
g
is:
-
Last m
oment
bef
ore
deployment
-
Las
t
step in the waterfall proc
ess
-
Larg
e
cor
pora
tions
-
Expensive
tools
requiring
spec
ial
sk
ills
-
Protocol
leve
l
rec
ord
-
and
-
playback
-
Lab env
ironmen
t
-
Scale
-
dow
n
env
ironmen
t
-
Che
ck
ing
agains
t
give
n requir
emen
ts
/ SLAs
-
Throwing
it back
over
the wall if
reqs
are not
met
-
…
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
3
Agenda
•
Agil
e Dev
elo
pme
nt
& Perfo
rman
ce
Testing
•
Contin
uous
Performance
Testing
•
Performance
Eng
ine
erin
g Pu
zz
le:
Chan
gin
g
Dy
nam
ics
D
i
scl
ai
m
er:
T
he
vi
ew
s
ex
pressed
here ar
e
m
y
persona
l
vi
ew
s
onl
y
and
do not
nece
ssaril
y
r
epresen
t
t
hose
of my
curr
ent
or
previ
ous
empl
oy
ers.
A
l
l
brands
and
t
r
ade
m
arks
m
entio
ned
are
t
he
propert
y
of thei
r
ow
ners.
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
4
Agil
e De
velopm
ent
•
Agi
le de
velop
me
nt
shou
ld
be
rat
he
r
a t
rivial
case
fo
r
pe
rfo
rma
nce
te
sting
-
You
ha
ve
a
work
in
g
sy
st
em eac
h i
ter
ati
on
to
tes
t
ea
rly
by
de
fin
iti
on
.
-
You
ne
ed
pe
rfo
rma
nc
e
en
gi
ne
er
for
th
e whole
pr
oj
ec
t
•
Sa
vin
gs
come f
rom
detecting
pr
obl
ems
early
•
You
ne
ed
to
ad
just re
qu
irem
en
ts
for
imp
lem
en
ted
fun
ction
ali
ty
-
Additi
ona
l
fun
cti
ona
lity
will i
mpac
t pe
rfo
rman
ce
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
5
The Main Issue on the Agile Side
•
It
doe
sn’t
[al
ways
]
work
thi
s way
in
practice
•
That
is why
you
have
“Hardening
Iterat
ions”,
“Technical
Debt”
and
similar no
tion
s
•
Same
ol
d p
robl
em:
fun
ctiona
lit
y gets
prio
rity
over
performa
nce
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
6
The Main Issue on the Testing Si
de
•
Perfo
rman
ce
Eng
ine
erin
g t
ea
ms
do
n’t
scale
well
-
Even
as
su
ming
tha
t they
ar
e c
omp
ete
nt
an
d
effe
cti
ve
•
Increa
sed
volum
e
expo
ses
the
pro
ble
m
-
Earl
y
tes
tin
g
-
Each
it
er
ati
on
•
Reme
die
s:
au
tom
ati
on
,
ma
king
pe
rform
an
ce
everyo
ne
’s
jo
b
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
7
Mentali
ty
Change
•
Makin
g
pe
rfo
rman
ce
eve
ryon
e’
s
job
•
La
te
reco
rd/
playb
ack
pe
rfo
rman
ce
test
ing
-
> Early
Perfor
mance
Engine
ering
•
System
-
level
req
uire
men
ts
-
> Compo
ne
nt
-
level req
uire
men
ts
•
Record/
playba
ck
app
roach
-
> Programming
to
ge
ne
rat
e
load
/cre
ate
stub
s
•
"Bl
ack Box"
-
> "
Grey
Box”
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
8
Expl
oratory
Testing
•
Rathe
r
ali
en
for p
erform
ance
testin
g,
but
probab
ly m
ore
relevant
than
for
function
al
testing
•
We l
earn
ab
out
sy
stem’s
perfo
rman
ce
as we
start
to
run te
st
-
Only
gu
es
se
s for
ne
w sy
st
ems
•
Rather
a pe
rformance
engi
neerin
g
proces
s
bringi
ng t
he sys
tem
to t
he p
roper
state
than
just
testing
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
9
Test Approach Dim
ension
New
W
ell
-
kno
wn
Sy
stem
T
estin
g
Ap
pro
ach
Explorator
y
/
Agile
Automat
ed
/
Regres
sion
T
raditional
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
10
Agenda
•
Agil
e Dev
elo
pme
nt
& Perfo
rman
ce
Testing
•
Contin
uous
Performance
Testing
•
Performance
Engin
eering
Puzz
le:
Changi
ng
Dy
nam
ics
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
11
Myth or Reality
?
•
You
see
Perfo
rman
ce
CI pre
senta
tio
ns
at
every
confe
ren
ce
no
wada
ys
….
and
•
Stil
l not
ma
ny
pe
rform
an
ce
pro
fession
als
do
it
-
As
far
as
I k
now….
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
12
Different Perspecti
ves
•
Consult
ant
:
nee
d
to
test
the
sy
stem
-
In its
cu
rr
en
t s
tat
e
-
Why
bo
the
r a
bo
ut
au
toma
tio
n?
-
Exter
nal
or
in
ter
nal
•
Performance
Engin
eer
-
On an
ag
ile
tea
m
-
Need
to tes
t it
ea
ch
bu
ild
/it
er
ati
on
/s
pr
in
t/e
tc
.
•
Autom
atio
n Eng
inee
r
/ SDET / et
c.
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
13
Automation: Considerations
•
You
ne
ed
know
sy
stem
well
en
ou
gh
to
ma
ke
me
an
ing
ful
au
tom
ati
on
•
If sys
tem
is
ne
w, overh
ea
ds
are
to
o
hig
h
-
So a
lmos
t n
o a
uto
mati
on
in
tr
ad
iti
on
al
en
vi
ro
nme
nts
•
If
th
e
same
sys
te
m
is te
sted
ag
ai
n
an
d
ag
ai
n
-
It make
s
se
ns
e to i
nv
es
t in s
ett
in
g up
au
toma
tio
n
•
Auto
ma
ted
int
erfa
ces sho
uld
be
stab
le
en
ou
gh
-
AP
Is
ar
e u
su
al
ly
mor
e
st
ab
le
on
ea
rly
st
ag
es
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
14
Automation: Diffic
ulties
•
Comp
licate
d
setups
•
Many
parts
of
the
puzz
le
-
Lo
ng
li
st
of
po
ss
ib
le
is
su
es
•
Comple
x results
(no pa
ss
/fai
l)
-
Not e
as
y to c
omp
ar
e
two r
es
ul
t sets
•
Chan
gin
g/Fra
gil
e
Int
erfaces
•
Time
/ Resourc
es
cons
iderat
ions
-
Tests
may
be
lon
g / use
a l
ot of reso
urc
es
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
15
Compli
cated
Setups
•
[Assumpt
ion
:
we
ha
ve
ba
sic e
lem
en
ts
in
pla
ce]
•
Mo
re
comp
licate
d
setup
s
-
Mult
i
-
mac
hi
ne
-
Keep
in
g
co
nfi
gu
ra
tio
n [
co
mpa
rin
g
ap
pl
es
-
to
-
ap
pl
es
]
-
Lar
ger
/re
ali
sti
c s
et of data
-
Real
is
tic
s
ec
ur
ity
-
Mon
ito
rin
g/i
ns
tru
men
tat
io
n/l
og
gi
ng
se
tup
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
16
Many Parts of the Puzzl
e
•
Sys
tem
Unde
r
Test
-
Usu
al
ly
di
st
rib
ute
d
•
Load
Testing
Tool
/ Harness
•
CI plu
mbin
g
•
Res
ults
anal
ys
is /
ale
rting
•
And
everything
may
go
wrong
-
Need
s
ex
ten
si
ve
er
ro
r h
an
dl
in
g
•
Which
is a
chal
le
nge
betwe
en
di
ff
erent tiers /
tools
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
17
Continuous Integration: Tools
•
CI supp
ort
be
comin
g
the
ma
in
the
me
•
Int
eg
rati
on
with
Cont
inu
ou
s I
nte
gra
tio
n
Servers
-
Je
nk
in
s,
Huds
on
, e
tc
.
-
Seve
ra
l t
oo
ls
an
no
un
ce
d
in
teg
ra
tio
n r
ec
en
tly
-
Maki
ng
a p
art
of a
uto
matic
bu
ild
pr
oc
es
s
•
Auto
ma
tio
n sup
po
rt
•
Clou
d sup
po
rt
•
Sup
po
rt
of
ne
west
techn
olo
gie
s
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
18
Complex
Results
•
No ea
sy
pass/fail
-
Ind
iv
id
ua
l r
es
po
ns
es
, monit
or
in
g
re
su
lts
,
er
ro
rs
, e
tc
.
•
No easy
comparison
-
SLA
-
Betwee
n
bu
ild
s
•
Varia
bil
ity
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
19
Jenki
ns
Performance Plugin
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
20
Performance Plugin
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
21
Variabil
ity
-
Envi
ronment
•
Due
to
d
iff
ere
nce
in
e
nviro
nm
en
ts
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
22
Variabil
ity
-
Sy
stem
•
In
he
ren
t
to
th
e
te
st
setu
p
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
23
Variabil
ity
-
Sy
stem
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
24
Aggregation Challenge
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
25
Addressi
ng
Variabil
ity
•
Same
en
vironme
nt
/ starti
ng
config
•
No othe
r
load
•
Multiple iter
ations
•
Reproducible
mult
i
-
user
tests
-
Conc
ur
re
nt
tes
t (sy
nc
po
in
ts
)
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
26
Changing Interfaces
•
If
use
recording
,
min
or chan
ges
may
brea
k
sc
ripts
-
And
yo
u ma
y
ev
en
do
n’
t k
no
w tha
t
-
Inte
rfa
ce
sh
oul
d b
e mat
ure
eno
ugh
•
Not ju
st p
rotocol
-
level
recordi
ng
-
GUI
-
AP
I / Prog
ra
mming
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
27
Record and Play
back:
Protocol
Level
Load
T
es
ting
T
ool
Vi
rtual
User
s
Ser
v
er
Load Gene
r
a
t
or
App
li
c
a
tion
Ne
tw
ork
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
28
Considerations
•
Usual
ly
do
esn't
work
fo
r te
sting
comp
on
en
ts
•
Each
too
l sup
po
rt
a l
imi
ted
nu
mb
er o
f
techn
olo
gie
s (p
roto
cols)
•
Som
e t
echn
olo
gie
s a
re
very
tim
e
-
consumi
ng
•
Workloa
d
valid
ity in
case
of
soph
isticate
d
log
ic
on
th
e
clien
t
side
is n
ot
gu
ara
nt
ee
d
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
29
Record and Play
back:
UI Le
vel
Lo
ad
T
estin
g
T
oo
l
V
irtu
al
Users
Serv
er
Lo
ad
Ge
ne
ra
tor
Applicatio
n
Networ
k
Brow
sers
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
30
Considerations
•
Scalab
ili
ty
-
Still r
equ
ire
more
res
our
ce
s
•
Sup
po
rted
techn
olo
gie
s
•
Tim
in
g
accura
cy
•
Pla
yback
accuracy
-
For
ex
amp
le
, fo
r Html
Unit
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
31
Programm
ing
Load
T
es
ting
T
ool
App
.
Vi
rtual
User
s
Ser
v
er
Load Gene
r
a
t
or
Applic
a
tion
Ne
tw
ork
AP
I
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
32
Considerations
•
Requ
ires
pro
gra
mm
ing
/ a
cc
ess
to
APIs
•
Too
l sup
po
rt
-
Exte
ns
ib
ili
ty
-
Language
support
•
Ma
y
req
ui
re
mo
re
resou
rces
•
Environ
me
nt
ma
y n
ee
d
to
be
set
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
33
Time / Resource Considerations
•
Perfo
rman
ce
tests
take
tim
e and
resources
-
The
la
rg
er
te
st
s,
the
mor
e
•
May
be n
ot a
n
opti
on o
n ea
ch check
-
in
•
Need
of a
tie
red soluti
on
-
Some
pe
rfo
rma
nc
e
mea
su
re
men
ts
ea
ch
bu
ild
-
Daily
mid
-
siz
e
per
for
manc
e
tes
ts
-
Peri
od
ic
la
rg
e
-
sc
al
e
/ upti
me
tes
ts
ou
ts
id
e CI
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
34
Automation: Limitati
ons
•
Works
gre
at
to
fin
d re
gre
ss
ion
s
an
d che
ck
ag
ain
st
req
uire
me
nts
•
Doesn’
t
cover:
-
Explo
rat
ory
tes
ts
-
La
rg
e s
ca
le
/ sco
pe
/ dur
ati
on
/ volu
me
•
“Ful
l A
ut
om
at
io
n”
is no
t
a
rea
l o
pt
io
n,
sho
ul
d
be
a
comb
ina
tio
n
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
35
Myth or Reality?
•
In the middle
-
Depe
nd
s
on
c
on
tex
t
•
Realit
y in some
cas
es
-
Usu
al
ly
s
tab
le
s
ys
tems
/ simpl
er
tes
t
ca
se
s
-
Often
si
ng
le
-
user
-
Strong
CI cu
ltu
re
/ C
I ex
per
tis
e in ho
us
e
•
Still
r
athe
r
myth
gene
rically
-
Not mu
ch
too
l sup
po
rt
for
ge
ne
ric
us
e
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
36
Agenda
•
Agil
e Dev
elo
pme
nt
& Perfo
rman
ce
Testing
•
Contin
uous
Performance
Testing
•
Performance
Engin
eering
Puzz
le:
Changi
ng
Dy
nam
ics
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
37
Performance Risk
Miti
gation
•
Single
-
user
perform
ance
engineer
ing
-
Profiling,
WPO,
single
-
use
r
perfor
mance
•
Soft
war
e
Performanc
e Engineering
-
Modeling,
P
erformance
Pat
terns
•
Instrumentation
/
AP
M
/ Monitoring
-
Produc
tion
sy
stem
insigh
ts
•
Capac
ity
Planning/Management
-
Res
ourc
es
Allocation
•
Continuous
Int
egration /
Deployment
-
Abilit
y to dep
loy
and
remov
e
cha
nges
quic
kly
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
38
But all
of them
don’t
repl
ace
load
testing
:
Lo
ad
t
es
ting
co
mp
le
ments
t
h
em
i
n
se
v
eral
im
po
rtant
wa
y
s
!
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
39
Can Sys
tem
Handle Peak Load?
•
You
can
’t
know
with
ou
t t
estin
g:
Con
tin
u
ous Perf
orman
ce
T
es
tin
g
Myths
and
R
eal
itie
s
40