How to create front-end performance Test Cases and beyond

Front-end performance testing is crucial when considering today’s performance testing and engineering domain. Now a days we are more or less into the web base application implementation and testing. Hence we need to look at client side performance while we are taking care of the server side (Back-end).

As quality assurance engineers, we are more methodical. Whatever things we have to test, we tend to create a test case in order to keep the test results visible and manageable. And also, QAs are using these TCs to get the history of that feature as well as (considering a performance test) we can take the benchmark of the figures which were there before the optimization. So we can compare results and show to our customers as they are keen on the ROI.

In this blog, I am going to describe and show you how to create an initial front-end performance test cases which you can use for performance testing. The main motivation behind these test cases is to avoid the unprofessional way of doing it.

First of all, lets have a look at following cycle which discribe all the areas including mitigation.

1. Info/Data: First of all, you need to read articles, books, blogs to get the data and information to do this. Without the knowledge, you cannot create any valuble scenarios.

2. Learn / R&D: Then, you need to learn from them and do the R&D stuffs to clear things. Get the idea behind it and properly structure the knowledge base.

3. Create TCs: Now you have proper knowledge to create test cases. So create those according to your domain, SLA, etc. But there should be common practices.

4. Execute/Assess: We need to execute the test cases with the SUT (System Under Test) and update the TCs. Make sure to mention all the performance values in the comment area. And also take screen shots if necessary (I prefer it’s good to take screen shots).

5. Log: All the finding should be logged after you had a chat with your developers. So this should be a collaborative approach and not a single decision.

6. Optimize: After you know that there is an issue, and after you log it in your tracking tool (e.g.: Jira), developers can work on that. So developers are fixing this and then QAs are testing with the benchmark to check that is there any improvement.

According to my experience, do not create separate hardening sprints to fix those non-functional issues. What we can do is, if you are fixing 10 functional issues in your sprint, take only 9 and pick one from the non-functional backlog.

7 Re-assess: After all the improvements are fixed, then QAs can do another round of official test to check the results. In here, make sure to update your test cases with new figures so that all the stakeholders can see that.

Likewise, the cycle should execute regularly.

So let’s start describing front end test cases from now on. There are test cases I have created upon my requirements so you can have your own test cases.

1. As the first part of my test cases, I have selected to validate the front end performance rules as it is utmost important. Targeting Web and mobile apps.

2. And then, I am referring to analyze how content size, type, and global rate as well as analyze how user see the performance and timing. In this case I have used 2 very popular tools which freely available.

Pingdom: https://tools.pingdom.com/

GTMetrix: https://gtmetrix.com/

3. As the 3^rd area, I have selected to analyze the Critical Rendering Path (CRP) using manual analysis and using the lighthouse tool in the Chrome browser. In this case we need to manually analyze the CRP and find issues. As well as from the lighthouse tool, there are good options to check the CRP.

Hope you got an idea about Front-end testing, test case writing and beyond. And make sure to create your own TCs according to your requirements. And please share all those with us coz then we also can learn from it.

How Infrastructure Monitoring and Tooling are connected with Performance Engineering

In the last five to six years, I had a chance to work in a few applications which are into CRM and financial closure domains. While working as a QA Lead and the Scrum Master, I was able to work as the performance tester as well. Fortunately, in these applications, our customers are willing to spend some time on performance engineering and monitoring through different Application Performance Management Systems which targeting application performance and infrastructure monitoring. Other than the APM systems, I was able to setup CI environments and dashboards.

From this article, I would like to share my experience with the application performance monitoring and how it is connected with Performance Engineering. First of all, I will start with the main performance approaches and talk about how we can connect tools, etc. Then all can clearly understand what I am going to be discussed.  

Three Performance Engineering Approaches:

In Performance Engineering, we have 3 approaches which we can present the entire life cycle. Those are Reactive, Proactive and Pridictive.

First: Reactive approach

When we are dealing with software applications, there a lot of sudden performance concerns due to various issues such as the requirements are not identified properly (SLA), customer database related issues, problems in the user groups or user personas and issues when the user load getting increased unexpectedly. This approach is needed, but not encourage to categorize as the best way of ensuring the product performance. Hence we are looking for the second approach.

Second: Proactive approach

In this, we are targeting a sustainable way of ensuring the product performance. Nowadays, when we are going to start a software product implementation, we need to think about the non-functional aspects more than ever. Specially performance engineering aspects as it will trigger issues such as we may need to change the entire application architecture at the last stages. Hence, starting the performance assessments in the early stages of the life cycle is a must. In an agile environment, we can say it should be started from Sprint 0. Not only that, it should go throughout all the sprints.

Furthermore, we need to focus more on continuous performance assessments and monitoring (Unit and functional level) rather than doing manual assessments by developers or testers. By using continues approach, we can eliminate the poor cost of quality and its remedies more effectively and efficiently. Using a monitoring dashboard, we can see the release or commit wise performance figures to detect issues ASAP. And it will increase the visibility for the customers who bother about the application performance and improvements. Say for an example, if we can establish a CI environment which triggers in the nightly build, we can find the performance issues easily by spotting the trend graphs rather than waiting till hardening sprint or system tests at the end of the release. As depicted in the following images, we can display figures on the dashboard:

  

Following is the figure I took from the daily performance monitoring dashboard which configured in Jenkins with JMeter plugin. Every morning, it will be triggered and pass all the data to the dashboard plugin.

Some of the useful links to configure the dashboard as follows:

https://www.blazemeter.com/blog/continuous-integration-101-how-run-jmeter-jenkins

https://www.blazemeter.com/blog/how-run-taurus-jenkins-performance-plugin

Third: Predictive approach

Now I am going to talk about the approach that most of the project teams, customers and stakeholders are neglected. However, if you noticed the topic of the third approach, it will give you the whole idea behind this. When our application is in the production, it’s not enough to deal with the development and continues CI environments. Hence, we must work on the production figures and trends to analyze the current user behaviors to identify the usage patterns. In other words, we should be able to predict or forecast the future loads and the impact (Ex: To indentify seasons or dates such as Black Friday, etc).

As I told you in the first paragraph, there are a lot of APM (Application Performance Management) systems in the market to get these valuable data and deliver great digital customer experience. Therefore real user relations and critical transactions can be monitored and analyzed down to code level. So we can categorize those as standalone applications and systems that coming with the cloud platforms.

Following are some standalone APM systems that we can use on the market (Only a few).

I had a chance to work with New Relic tool and it has most of the features which intended to have in such application. Very good visualization and data analysis, including find the most cumbersome SQL queries, etc. Following is a sample dashboard which used New Relic insights:

If we are using cloud platforms, then we can use Application Insights with Azure and CloudWatch with AWS.

    

Both Azure and AWS have a built in solutions for keeping data and tracking of metrics with the online alerting system. These tools are there to see the big picture of what is going on with your platform as a service environment, like standing on the Mount Everest and watching everything below. If we talk about Azure application insights, we can make its own Dashboard or show the data through the PowerBI.

The first thing you see after you log in to the Azure portal is the dashboard. Here you can bring together the charts (Server responce Time, page view Load Time, Errors, etc) that are most important to you and to your stakeholders.

*Following are the areas of the dashboard:

1. Navigate to specific resources such as your app in Application Insights: Use the left bar.
2. Return to the current dashboard, or switch to other recent views: Use the drop-down menu at top left.
3. Switch dashboards: Use the drop-down menu on the dashboard title
4. Create, edit, and share dashboards in the dashboard toolbar.
5. Edit the dashboard: Hover over a tile and then use its top bar to move, customize, or remove it.

*Took from https://docs.microsoft.com

If the Azure dashboard is not good enough for you, as I told you earlier, we can use PowerBI tool easily.

Power BI is a suite of business analytics tools to analyze and share data. Its dashboards provide a wideangle view for business users with their most important metrics in one place.

Some of the useful links to configure PBI:

https://docs.microsoft.com/en-us/azure/application-insights/app-insights-export-power-bi

https://powerbi.microsoft.com/en-us/documentation/powerbi-content-pack-application-insights/

Not only the PowerBI standard reports (adapter), but also we can get the data via Export Analytics queries and Azure Stream Analytics. In this way, we can write any query you want to use and export it to Power BI. In our current project, we are basically working with Azure Stream Analytics as it has good control of data.

https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-power-bi-dashboard

 

Hope you understood the difference between these approaches and what are the places which we can use monitoring systems and dashboards to upgrade the customer visibility, application performance and product quality.

Do not just follow the process, observe, improvise and adapt

During last few years, I worked as a Scrum Master who is responsible for process activities, I learned a lot related to ISO/SCRUM rules and its behaviors. As a starter, I tend to do the same as what process tries to explain and implant. I tried to tally each process item to the project workflow and make an extra effort to maintain the process conformity. Sometimes I have created some documents and graphs which do not have much contribution to the smooth progress of the project. However, I do not want to think differently as I thought it will harm to the process and then the process marks will go down (NB: In our company, there are some marks for the process conformity).

But when there are a lot of issues due to particular project behaviors and stakeholder requirements, we tend to apply different principles, methodologies and exceptions become more realistic with what we are actually doing inside the project. In other words, we try to think out of the box and applied necessary process changes to be more practical, visible and productive while confirming to the process.

In the next few paragraphs, I will try to explain what are the particular project behaviors / stakeholder requirements and how we changed the way we are doing in order to be ISO confirmed and practical within an agile scrum environment.

What are the particular project behaviors and stakeholder requirements?

In that period of time, from the pool of projects, our project is the most prioritized one in their stack and we were pressurized to release features very frequently. And they did not support much for the process things since the mindset was towards release items and time to market. First they gave us 10 big features and set a deadline, which is not that suitable for an agile scrum environment. Initially we tried to explain the situation related to process, etc., but those are not much considered.

In that time we had 2 weeks sprints and we had to create a lot of artifacts in order to confirm the project and process for each sprint. And also we do not have much idea about the sprint outcome and that is not tallied with the deliverables as we cannot give a shippable product within 2 weeks. But as product owners, they are expecting a release after every sprint.

And also, due to the big regression work we have to do before a release, we have asked for test automation suite several times but, those are not accepted as they need features more than else (though that is not the correct way and maybe they were also know that). As well as test automation, we asked to do thorough performance assessment, but the answer was the same.

While these things are happening, as a Scrum Master, I also pressurized due the process and product work. But one day I thought this is not going to work and this will end up in nowhere. Hence I have started to create a process or way of working which is suitable for the current situation and also can maintain the process conformity. In other words, a way to simplify the process/work pattern to cater the project delivery needs and avoid extra work.

What we did to become more realistic in process, improve visibility and avoid extra work?

We did a few things to make the change and those are as follows:

1.  Learn the process well

First of all, what we did was we went to the process head and know about the company process well. Not only the documented points, but the essence one by one, what he said was that if you need any changes, do it and add those to the exception document. And most importantly he said that we do not need to do the exact thing what process doc says. But do things which brings what process hope to add to the project.

2.  Learn different methodologies and applied

In that period, we followed scrum but actually it was “SCRUM BUT”. Hence, we have introduced new but sustainable methodology called Scrumban which is a combination of Scrum and Kanban. Because the application is in the production and it is sort of in the maintenance + adding feature phase and we need to follow some of the best scrum practices such as daily standups, grooming sessions, demos, retrospectives, etc. It gave us more flexibility on ceremonies. And also we have changed the sprint length to one month, which minimize the overhead as we do not need to start, close sprints and no need to create documents every two weeks. Hence, we had more time to do project work while doing the bare minimum to satisfy the process needs and product quality.

Following is the diagram which I was created to depict the full process of the application (Start to Release) 

3.  Learn how to make the proper artifacts and improve customer visibility, which we really need.

Then we come up with proper artifacts such as exception document, release related statistics, project matrixes for all the releases, the progress status of the major features (10 features) and all the bugs added/resolved which all the stakeholders can see. Specially, the stakeholders are like to see the current statistics, hence we should make sure that the visibility is there. If not, they will suddenly come and ask questions which we do not have any clue at all. Glad to say that, the CTO who was there at that time, looked at the progress status page more frequently to understand the progress and releases. He always adding comments to that page so it was a live document. Likewise, proper artifacts will add more value rather than traditional documents.

Finally, hope this blog will help you to understand the real meaning of ‘Do not just follow the process. Observe, improvise and adapted’. Thank you.