Mastering a
performance testing tool is the ultimate goal
I thought of writing some articles about performance engineering
old beliefs or misconceptions which drive you off road. So this is my first part of the series, hence
read carefully.
If I ask, what is the most vital part of the performance
testing or engineering domain? I am pretty sure that most of the answers would
be based on mastering a performance
assessment tool. Yes, learning a tool rather mastering a tool is not that
easy and it needs a lot of effort, no doubt about that. But unfortunately,
according to my experience, the answer is too old :)
OK, now all of you are pointing the figures and ask what are you going
to say? In my performance engineering carrier time line, I went
through a lot of real time projects, customers, techniques, phases, various types
of tools, methodologies and industry best practices. In this all situations, I
realized that we have tons of things to know other than dealing with a
tool or a tool suite.
So let’s discuss these things briefly:
Requirements:
When doing a performance assessment task, we need to know
the entire background and characteristics of the application such as customer
background, product domain, product technologies, critical scenarios, user base,
shared resources, batch processes, environments, benchmarks and the competitors.
If performance assessment is a building, this is the first brick of the
construction. But this phase also differs if your application or the product is
a well known one or internal application. Then requirements gathering section
will be quite easy. Anyway, you need to put 100% to this phase because as I said
earlier, this is the foundation of your entire assessment. The foundation is poor,
poor output.
Technology Expertise:
In the above section, I have mentioned that we need to know
things such as product technologies and environments. But this is not just
know, we need to learn things thoroughly. During the last few years, I have realized that product performance
engineering is very difficult and critical area that cannot be matched with
other functional and non-functional
attributes such as Test automation, Security, Usability, etc. Because it is
tightly coupled with application architecture and its technologies.
I will tell you why, say for an instance, if we need to
automate tests for a particular product, main difficultly will be to capture
the product objects (page objects) using our existing tools and add those things
to the CI framework. In this case we need to be a good tool expert and need to
have good knowledge in OOP concepts.
But if you are going to do a performance
assessment of a product, you need to know things such as environment configurations, network and related rules, future user growth
and capacity planning, client side technologies, CDN, compression,
minification, DNS, Javascript rendering, waterfall analysis, server
side components, database technologies and configurations, product architecture, design, load balancing algorithms other than the performance assessment tools suite and CI configurations. So
if you want to become a product performance expert, you should be an overall
guy with a lot of tech knowledge.
Process:
We have always targeted an application or a product. Hence we
need to have a look at the process that the application is getting constructed
and as well as the life cycle milestone that the application is in when we
receive this assignment. Say for an instance, the application is almost completed
and several weeks to be released, then our approach should be different then
the application is only 5~6 sprints old. Or application is in the initial
development stage. And as well as if the application is developed using agile
methodology, then we need to adapt to that process and do the tasks in sprint
wise to make sure that the performance assessment is not a burden or a comet
which falls from the sky.
Check and Proof:
If we decide to take the
assignment, we need to make sure that the requirements are doable and can be
done within the required time slot. In order to do so we need to conduct a Proof of
Concept (POC) first place and assure that the application or the product
technologies can be addressed and critical scenarios can be scripted. And one
more thing to say, as you already know in performance engineering, we have two
areas to be tested. One is Front end and other is Back End. So when we say we
will do a POC and let you know the feasibility, we are targeting back end and
not the front end testing because we do not need a POC to do a Front end
assessment as it is dealing with the client side UI. But
my advice is to do a POC first with the back end technologies and know the
situation even though we execute Front end testing first. Hope you understand
the concept.
Model the user load:
When doing a load test, load
generation is a vital part and I’ll talk about that later. But before the load
generation, do you ever bother about the pattern which real users are accessing
your awesome web application. All the users that access your application has
their own way of doing things. Say for an example, if user A login to the system
and do scenario S and T and logout. But user B login to the system and do
scenario S and U. And user C is not even login to the system and he is just a
new user that browses the site to know some info.
Hence, can we just script
the scenarios and do the test. No, we cannot. So what we need to do is, we need
to create a model to simulate the user behavior in the load generation to mimic
the real situation. In this case we use a concept called ‘User Community Model
(UCM)’ (http://www.perftestplus.com/articles/ucml.pdf). This model is created
using ‘User Community Model Language’ (UCML) which created by Scott Barber who
is a pioneer in this field.
Using this language we
can easily graph the entire user scenario base and create a real user load
environment without much effort. We can user google analytics and server logs
to get the information to draw this UCM.
Load generation types:
Load generation is not a
simple task in a performance test assessment. We need to know the environment
that we are going to be tested. In this case, we can use 3 types of load
generation mechanisms which address different aspects. We can use On-premise
model that generate load inside the same internal environment. This is quite an
isolated environment which suitable for segregating specific issues in the system
as we do not have any network or latency issues.
The second option is to use
the cloud VMs (Ex: Azure, Amazon EC2) to generate the load to the cloud hosted
application. In this type, we can use geographically distributed load
generators to mimic the actual user distribution in different countries. But in
this case, we do not have much idea about the network bandwidth and the latency
as all the components are in the cloud. Hence network issues isolation is not
quite easy.
The final option is the
hybrid version which divides the server environment and load generation into
two locations. System Under Test (SUT) or the application server is in the on-premise
environment and load generators are from the cloud (Ex: Azure, Amazon EC2). In
this option, issues isolation is not more difficult than the cloud version as the
server environment is in our hands. Hence, most performance engineering gurus’ are
recommending this.
Result Presentation:
This is also very
important thing in the performance test assessment, but not conceded as a vital part in
most of the times. When presenting results, we need to make sure that we know
the audience. If the audience is not coming from the tech background (Ex: Business stakeholders), then our report should be simple and contain a lot statistics such
as impact to the business, future user growth, can we deploy this, how current performance impact to the business continuity, etc.
But the audience is technical
oriented people, then we can include some technical backgrounds of the issues we
have encountered. Then those people can benefit out of it. Finally, make sure to add
a lot of tables, charts, graphs and a summary rather than tons of texts as readers not tend
to read a lot.
Hope you learn something.
Next, I'll write an article about another performance engineering misconception soon.
Next, I'll write an article about another performance engineering misconception soon.
