Common Performance Engineering Misconceptions - Part 1

Mastering a performance testing tool is the ultimate goal

I thought of writing some articles about performance engineering old beliefs or misconceptions which drive you off road.  So this is my first part of the series, hence read carefully.

If I ask, what is the most vital part of the performance testing or engineering domain? I am pretty sure that most of the answers would be based on mastering a performance assessment tool. Yes, learning a tool rather mastering a tool is not that easy and it needs a lot of effort, no doubt about that. But unfortunately, according to my experience, the answer is too old :)

 

OK, now all of you are pointing the figures and ask what are you going to say? In my performance engineering carrier time line, I went through a lot of real time projects, customers, techniques, phases, various types of tools, methodologies and industry best practices. In this all situations, I realized that we have tons of things to know other than dealing with a tool or a tool suite.

So let’s discuss these things briefly:

Requirements:

When doing a performance assessment task, we need to know the entire background and characteristics of the application such as customer background, product domain, product technologies, critical scenarios, user base, shared resources, batch processes, environments, benchmarks and the competitors. If performance assessment is a building, this is the first brick of the construction. But this phase also differs if your application or the product is a well known one or internal application. Then requirements gathering section will be quite easy. Anyway, you need to put 100% to this phase because as I said earlier, this is the foundation of your entire assessment. The foundation is poor, poor output.

Technology Expertise:

In the above section, I have mentioned that we need to know things such as product technologies and environments. But this is not just know, we need to learn things thoroughly. During the last few years, I have realized that product performance engineering is very difficult and critical area that cannot be matched with other functional and  non-functional attributes such as Test automation, Security, Usability, etc. Because it is tightly coupled with application architecture and its technologies.

I will tell you why, say for an instance, if we need to automate tests for a particular product, main difficultly will be to capture the product objects (page objects) using our existing tools and add those things to the CI framework. In this case we need to be a good tool expert and need to have good knowledge in OOP concepts. 

But if you are going to do a performance assessment of a product, you need to know things such as environment configurations, network and related rules, future user growth and capacity planning, client side technologies, CDN, compression, minification, DNS, Javascript rendering, waterfall analysis, server side components, database technologies and configurations, product architecture, design, load balancing algorithms other than the performance assessment tools suite and CI configurations. So if you want to become a product performance expert, you should be an overall guy with a lot of tech knowledge.  

Process:

We have always targeted an application or a product. Hence we need to have a look at the process that the application is getting constructed and as well as the life cycle milestone that the application is in when we receive this assignment. Say for an instance, the application is almost completed and several weeks to be released, then our approach should be different then the application is only 5~6 sprints old. Or application is in the initial development stage. And as well as if the application is developed using agile methodology, then we need to adapt to that process and do the tasks in sprint wise to make sure that the performance assessment is not a burden or a comet which falls from the sky.  

        

Check and Proof:

If we decide to take the assignment, we need to make sure that the requirements are doable and can be done within the required time slot. In order to do so we need to conduct a Proof of Concept (POC) first place and assure that the application or the product technologies can be addressed and critical scenarios can be scripted. And one more thing to say, as you already know in performance engineering, we have two areas to be tested. One is Front end and other is Back End. So when we say we will do a POC and let you know the feasibility, we are targeting back end and not the front end testing because we do not need a POC to do a Front end assessment as it is dealing with the client side UI. But my advice is to do a POC first with the back end technologies and know the situation even though we execute Front end testing first. Hope you understand the concept.

Model the user load:

When doing a load test, load generation is a vital part and I’ll talk about that later. But before the load generation, do you ever bother about the pattern which real users are accessing your awesome web application. All the users that access your application has their own way of doing things. Say for an example, if user A login to the system and do scenario S and T and logout. But user B login to the system and do scenario S and U. And user C is not even login to the system and he is just a new user that browses the site to know some info.

Hence, can we just script the scenarios and do the test. No, we cannot. So what we need to do is, we need to create a model to simulate the user behavior in the load generation to mimic the real situation. In this case we use a concept called ‘User Community Model (UCM)’ (http://www.perftestplus.com/articles/ucml.pdf). This model is created using ‘User Community Model Language’ (UCML) which created by Scott Barber who is a pioneer in this field.

Using this language we can easily graph the entire user scenario base and create a real user load environment without much effort. We can user google analytics and server logs to get the information to draw this UCM.

Load generation types:

Load generation is not a simple task in a performance test assessment. We need to know the environment that we are going to be tested. In this case, we can use 3 types of load generation mechanisms which address different aspects. We can use On-premise model that generate load inside the same internal environment. This is quite an isolated environment which suitable for segregating specific issues in the system as we do not have any network or latency issues. 

The second option is to use the cloud VMs (Ex: Azure, Amazon EC2) to generate the load to the cloud hosted application. In this type, we can use geographically distributed load generators to mimic the actual user distribution in different countries. But in this case, we do not have much idea about the network bandwidth and the latency as all the components are in the cloud. Hence network issues isolation is not quite easy.

The final option is the hybrid version which divides the server environment and load generation into two locations. System Under Test (SUT) or the application server is in the on-premise environment and load generators are from the cloud (Ex: Azure, Amazon EC2). In this option, issues isolation is not more difficult than the cloud version as the server environment is in our hands. Hence, most performance engineering gurus’ are recommending this.

Result Presentation:

This is also very important thing in the performance test assessment, but not conceded as a vital part in most of the times. When presenting results, we need to make sure that we know the audience. If the audience is not coming from the tech background (Ex: Business stakeholders), then our report should be simple and contain a lot statistics such as impact to the business, future user growth, can we deploy this, how current performance impact to the business continuity, etc.

But the audience is technical oriented people, then we can include some technical backgrounds of the issues we have encountered. Then those people can benefit out of it. Finally, make sure to add a lot of tables, charts, graphs and a summary rather than tons of texts as readers not tend to read a lot.

Hope you learn something.

Next, I'll write an article about another performance engineering misconception soon.