Sunday, February 11, 2018

Why Quality Assurance Engineers should be UX Engineers

Software Quality Assurance is a buzz word quite a sometime but with the agile transformation hits the ground, there were some questions like ‘will it die out sooner or do we need a separate gang to assure quality?’ But, still, it stands tall with other areas in the software field with more interesting methodologies and practices. Anyway, as QAs, we kept questioning by various parties about the real requirement of a QA or in other words, what we are actually doing other than just testing functionalities in the application and adding bugs.

I can remember a few years back, one of my colleague developers also questioned us about a designation which we have added to our QA track. He blamed that we took an engineering track which belongs to them.

Whatever the things done in the past, future of the QA depends on the change that we are going to be incorporated and how effective it is. In this article I like to emphasize on one factor which we can gain the boost and differentiation. That is how QAs can be UX Engineers.

In our projects, we are normally working with product owners and stakeholders directly to get the requirements. So this is a better move rather than requirements are coming from a BA or a person who is dealing with the customers. Hence, this is a very good chance to get leverage over traditional mechanisms.

I need to start like this, according to my experience and knowledge, User Experience (UX) has two areas which can depict as follows:

According to the diagram, the first area is basically related to UI/UX engineers on the project. They need to make sure that the web site should be designed according to the user expectations. Say for an example, if we are creating a website for Scandinavian countries, they have a different color set and unique thinking patterns which we need to read.

Other than that, as the second area depicted in the diagram, QAs have a big part in order to ensure the functional and non-functional user experience. Why functional and non-functional areas related to user experience. What is the relationship. I would say, even though it is not highlighted, QA is all about ensuring functional and non-functional user expectations and improve the experience. Now let’s have a look at how exactly this happens.

Functional User Experience:

In an application, first priority should go to the functionality. I will take ‘Login to the system’ as my user story in all the cases. Let say login functionality is not working properly, it is directed to the sub page instead the home page. You can see, obviously, the user experience ruined. User is not satisfied and there will be a lot of customer complaints. To fix this issue in the application, we need to check root cause and the quality assurance process to understand why it is not being tested. Are we covered it or not in the test case suite. If it is covered then why it is not being executed, etc. So the point is, whatever we call, functionality is directly affected to UX.

Nonfunctional User Experience:

Even though first priority should go to the functional areas, a lot of big problems coming from the nonfunctional part of the application. Say for an example, in my project, due to a performance issue, we had to spend 2 weeks to find the problem. Fortunately, we were able to but at a cost. Some of the experts say, nonfunctional requirements or problems are not clear enough and it’s like a hidden part of the iceberg. Hence user experience upon nonfunctional areas is crucial and we need to spend more time on it.


I will take the same example. Let say due to the weak password validation mechanism, unauthorized user entered into the system and deleted some valuable info. Now some of the real users cannot access the site as their login data is missing in the DB. So this triggers a big impact on the user experience (UX) and really bad for the reputation of the application and the brand. Sometimes, an entire business can be collapsed within few days due to such incidents. Hence, all the QA engineers need to concentrate on the Security aspect and need to learn how to apply these tests in the testing process.


In the peck time of the day, if some of the regular customers cannot login the site due to a performance issue, it will lead to a big loss in the reputation and the growth of the business. This is an often situation in the Sri Lankan GOV web sites, since it is free of charge we are not bothered much. But, if it is a service for a certain cost, then we have to face a lot of problems from the users. Therefore, whether it is Performance Engineering or whatever, it is impacting the user experience (UX). Hence, we should look at these nonfunctional areas from the beginning of the production cycle than wait till the end.

I am not going to describe the reliability and maintainability but those are also big pieces of the UX puzzle. Therefor when you get the requirements from the customers or end users, we need to check the SLAs (Service Level Agreements) and Security background (Who are the user group), etc of the application. I encourage you to have a nonfunctional requirement gathering document which we can send to our customers or end users to fill with their requirements. If they are not technical enough to fill that, better we fill it with the past experience and send for the validation. Following are a few sample questions for application performance data gathering:

Finally, What I have to tell you is that, even we have divided into several areas such as Quality Assurance, Security Engineering, Performance Engineering, etc., User Experience (UX) is the umbrella which there in the top. Hence, all the QA Engineers should think from the user perspective first of all before going to the technical separation as I mention above. In other words, we should be UX engineers than QA engineers.

Thursday, December 7, 2017

How to create front-end performance Test Cases and beyond

Front-end performance testing is crucial when considering today’s performance testing and engineering domain. Now a days we are more or less into the web base application implementation and testing. Hence we need to look at client side performance while we are taking care of the server side (Back-end).

As quality assurance engineers, we are more methodical. Whatever things we have to test, we tend to create a test case in order to keep the test results visible and manageable. And also, QAs are using these TCs to get the history of that feature as well as (considering a performance test) we can take the benchmark of the figures which were there before the optimization. So we can compare results and show to our customers as they are keen on the ROI.

In this blog, I am going to describe and show you how to create an initial front-end performance test cases which you can use for performance testing. The main motivation behind these test cases is to avoid the unprofessional way of doing it.

First of all, lets have a look at following cycle which discribe all the areas including mitigation.

1. Info/Data: First of all, you need to read articles, books, blogs to get the data and information to do this. Without the knowledge, you cannot create any valuble scenarios.

2. Learn / R&D: Then, you need to learn from them and do the R&D stuffs to clear things. Get the idea behind it and properly structure the knowledge base.

3. Create TCs: Now you have proper knowledge to create test cases. So create those according to your domain, SLA, etc. But there should be common practices.

4. Execute/Assess: We need to execute the test cases with the SUT (System Under Test) and update the TCs. Make sure to mention all the performance values in the comment area. And also take screen shots if necessary (I prefer it’s good to take screen shots).

5. Log: All the finding should be logged after you had a chat with your developers. So this should be a collaborative approach and not a single decision.

6. Optimize: After you know that there is an issue, and after you log it in your tracking tool (e.g.: Jira), developers can work on that. So developers are fixing this and then QAs are testing with the benchmark to check that is there any improvement.

According to my experience, do not create separate hardening sprints to fix those non-functional issues. What we can do is, if you are fixing 10 functional issues in your sprint, take only 9 and pick one from the non-functional backlog.

7 Re-assess: After all the improvements are fixed, then QAs can do another round of official test to check the results. In here, make sure to update your test cases with new figures so that all the stakeholders can see that.

Likewise, the cycle should execute regularly.

So let’s start describing front end test cases from now on. There are test cases I have created upon my requirements so you can have your own test cases.

1. As the first part of my test cases, I have selected to validate the front end performance rules as it is utmost important. Targeting Web and mobile apps.

2. And then, I am referring to analyze how content size, type, and global rate as well as analyze how user see the performance and timing. In this case I have used 2 very popular tools which freely available.

3. As the 3rd area, I have selected to analyze the Critical Rendering Path (CRP) using manual analysis and using the lighthouse tool in the Chrome browser. In this case we need to manually analyze the CRP and find issues. As well as from the lighthouse tool, there are good options to check the CRP.

Hope you got an idea about Front-end testing, test case writing and beyond. And make sure to create your own TCs according to your requirements. And please share all those with us coz then we also can learn from it.

Friday, September 8, 2017

How Infrastructure Monitoring and Tooling are connected with Performance Engineering

In the last five to six years, I had a chance to work in a few applications which are into CRM and financial closure domains. While working as a QA Lead and the Scrum Master, I was able to work as the performance tester as well. Fortunately, in these applications, our customers are willing to spend some time on performance engineering and monitoring through different Application Performance Management Systems which targeting application performance and infrastructure monitoring. Other than the APM systems, I was able to setup CI environments and dashboards.

From this article, I would like to share my experience with the application performance monitoring and how it is connected with Performance Engineering. First of all, I will start with the main performance approaches and talk about how we can connect tools, etc. Then all can clearly understand what I am going to be discussed.  

Three Performance Engineering Approaches:

In Performance Engineering, we have 3 approaches which we can present the entire life cycle. Those are Reactive, Proactive and Pridictive.

First: Reactive approach

When we are dealing with software applications, there a lot of sudden performance concerns due to various issues such as the requirements are not identified properly (SLA), customer database related issues, problems in the user groups or user personas and issues when the user load getting increased unexpectedly. This approach is needed, but not encourage to categorize as the best way of ensuring the product performance. Hence we are looking for the second approach.

Second: Proactive approach

In this, we are targeting a sustainable way of ensuring the product performance. Nowadays, when we are going to start a software product implementation, we need to think about the non-functional aspects more than ever. Specially performance engineering aspects as it will trigger issues such as we may need to change the entire application architecture at the last stages. Hence, starting the performance assessments in the early stages of the life cycle is a must. In an agile environment, we can say it should be started from Sprint 0. Not only that, it should go throughout all the sprints.

Furthermore, we need to focus more on continuous performance assessments and monitoring (Unit and functional level) rather than doing manual assessments by developers or testers. By using continues approach, we can eliminate the poor cost of quality and its remedies more effectively and efficiently. Using a monitoring dashboard, we can see the release or commit wise performance figures to detect issues ASAP. And it will increase the visibility for the customers who bother about the application performance and improvements. Say for an example, if we can establish a CI environment which triggers in the nightly build, we can find the performance issues easily by spotting the trend graphs rather than waiting till hardening sprint or system tests at the end of the release. As depicted in the following images, we can display figures on the dashboard:

Following is the figure I took from the daily performance monitoring dashboard which configured in Jenkins with JMeter plugin. Every morning, it will be triggered and pass all the data to the dashboard plugin.

Some of the useful links to configure the dashboard as follows:

Third: Predictive approach

Now I am going to talk about the approach that most of the project teams, customers and stakeholders are neglected. However, if you noticed the topic of the third approach, it will give you the whole idea behind this. When our application is in the production, it’s not enough to deal with the development and continues CI environments. Hence, we must work on the production figures and trends to analyze the current user behaviors to identify the usage patterns. In other words, we should be able to predict or forecast the future loads and the impact (Ex: To indentify seasons or dates such as Black Friday, etc).

As I told you in the first paragraph, there are a lot of APM (Application Performance Management) systems in the market to get these valuable data and deliver great digital customer experience. Therefore real user relations and critical transactions can be monitored and analyzed down to code level. So we can categorize those as standalone applications and systems that coming with the cloud platforms.

Following are some standalone APM systems that we can use on the market (Only a few).

I had a chance to work with New Relic tool and it has most of the features which intended to have in such application. Very good visualization and data analysis, including find the most cumbersome SQL queries, etc. Following is a sample dashboard which used New Relic insights:

If we are using cloud platforms, then we can use Application Insights with Azure and CloudWatch with AWS.


Both Azure and AWS have a built in solutions for keeping data and tracking of metrics with the online alerting system. These tools are there to see the big picture of what is going on with your platform as a service environment, like standing on the Mount Everest and watching everything below. If we talk about Azure application insights, we can make its own Dashboard or show the data through the PowerBI.

The first thing you see after you log in to the Azure portal is the dashboard. Here you can bring together the charts (Server responce Time, page view Load Time, Errors, etc) that are most important to you and to your stakeholders.

*Following are the areas of the dashboard:

  1. Navigate to specific resources such as your app in Application Insights: Use the left bar.
  2. Return to the current dashboard, or switch to other recent views: Use the drop-down menu at top left.
  3. Switch dashboards: Use the drop-down menu on the dashboard title
  4. Create, edit, and share dashboards in the dashboard toolbar.
  5. Edit the dashboard: Hover over a tile and then use its top bar to move, customize, or remove it.
*Took from

If the Azure dashboard is not good enough for you, as I told you earlier, we can use PowerBI tool easily.

Power BI is a suite of business analytics tools to analyze and share data. Its dashboards provide a wideangle view for business users with their most important metrics in one place.

Some of the useful links to configure PBI:

Not only the PowerBI standard reports (adapter), but also we can get the data via Export Analytics queries and Azure Stream Analytics. In this way, we can write any query you want to use and export it to Power BI. In our current project, we are basically working with Azure Stream Analytics as it has good control of data.


Hope you understood the difference between these approaches and what are the places which we can use monitoring systems and dashboards to upgrade the customer visibility, application performance and product quality.

Friday, January 6, 2017

Do not just follow the process, observe, improvise and adapt

During last few years, I worked as a Scrum Master who is responsible for process activities, I learned a lot related to ISO/SCRUM rules and its behaviors. As a starter, I tend to do the same as what process tries to explain and implant. I tried to tally each process item to the project workflow and make an extra effort to maintain the process conformity. Sometimes I have created some documents and graphs which do not have much contribution to the smooth progress of the project. However, I do not want to think differently as I thought it will harm to the process and then the process marks will go down (NB: In our company, there are some marks for the process conformity).

But when there are a lot of issues due to particular project behaviors and stakeholder requirements, we tend to apply different principles, methodologies and exceptions become more realistic with what we are actually doing inside the project. In other words, we try to think out of the box and applied necessary process changes to be more practical, visible and productive while confirming to the process.

In the next few paragraphs, I will try to explain what are the particular project behaviors / stakeholder requirements and how we changed the way we are doing in order to be ISO confirmed and practical within an agile scrum environment.

What are the particular project behaviors and stakeholder requirements?

In that period of time, from the pool of projects, our project is the most prioritized one in their stack and we were pressurized to release features very frequently. And they did not support much for the process things since the mindset was towards release items and time to market. First they gave us 10 big features and set a deadline, which is not that suitable for an agile scrum environment. Initially we tried to explain the situation related to process, etc., but those are not much considered.

In that time we had 2 weeks sprints and we had to create a lot of artifacts in order to confirm the project and process for each sprint. And also we do not have much idea about the sprint outcome and that is not tallied with the deliverables as we cannot give a shippable product within 2 weeks. But as product owners, they are expecting a release after every sprint.

And also, due to the big regression work we have to do before a release, we have asked for test automation suite several times but, those are not accepted as they need features more than else (though that is not the correct way and maybe they were also know that). As well as test automation, we asked to do thorough performance assessment, but the answer was the same.

While these things are happening, as a Scrum Master, I also pressurized due the process and product work. But one day I thought this is not going to work and this will end up in nowhere. Hence I have started to create a process or way of working which is suitable for the current situation and also can maintain the process conformity. In other words, a way to simplify the process/work pattern to cater the project delivery needs and avoid extra work.

What we did to become more realistic in process, improve visibility and avoid extra work?

We did a few things to make the change and those are as follows:

1.  Learn the process well

First of all, what we did was we went to the process head and know about the company process well. Not only the documented points, but the essence one by one, what he said was that if you need any changes, do it and add those to the exception document. And most importantly he said that we do not need to do the exact thing what process doc says. But do things which brings what process hope to add to the project.

2.  Learn different methodologies and applied

In that period, we followed scrum but actually it was “SCRUM BUT”. Hence, we have introduced new but sustainable methodology called Scrumban which is a combination of Scrum and Kanban. Because the application is in the production and it is sort of in the maintenance + adding feature phase and we need to follow some of the best scrum practices such as daily standups, grooming sessions, demos, retrospectives, etc. It gave us more flexibility on ceremonies. And also we have changed the sprint length to one month, which minimize the overhead as we do not need to start, close sprints and no need to create documents every two weeks. Hence, we had more time to do project work while doing the bare minimum to satisfy the process needs and product quality.

Following is the diagram which I was created to depict the full process of the application (Start to Release) 

3.  Learn how to make the proper artifacts and improve customer visibility, which we really need.

Then we come up with proper artifacts such as exception document, release related statistics, project matrixes for all the releases, the progress status of the major features (10 features) and all the bugs added/resolved which all the stakeholders can see. Specially, the stakeholders are like to see the current statistics, hence we should make sure that the visibility is there. If not, they will suddenly come and ask questions which we do not have any clue at all. Glad to say that, the CTO who was there at that time, looked at the progress status page more frequently to understand the progress and releases. He always adding comments to that page so it was a live document. Likewise, proper artifacts will add more value rather than traditional documents.

Finally, hope this blog will help you to understand the real meaning of ‘Do not just follow the process. Observe, improvise and adapted’. Thank you.

Tuesday, November 11, 2014

How to create a file with exact amount of bytes to test the boundary value

Once I need to create a file with exact amount of bytes (file size) to test the boundary value of a file upload event in my project. Requirement is to be able to upload a file < or = 100MB. So I tried to create a file to test this scenario.

To create a file less than 100MB is very simple. But, according to the QA principles, we need to check the boundary value of that feature with the file size 99MB, 100MB and 101MB won't be an easy task as we have to create 3 files with exact amount of bytes. We called it as Boundary Value Analysis.

Then I found a nice command line to create such files without wasting our valuable time.

What you need to do is just create a .bat file with following command line and execute it.

fsutil file createnew filename length 

Ex: Let say if you need to create a file with 100 MB (1024 * 1024 * 100)

fsutil file createnew 100MB_File 104857600

Thursday, October 23, 2014

Usability Engineering Basics - Part 3 (First)

In this time, I thought writing some words about usability engineering as I promise you some months back. If we need to highlight real usability issues which customers have, we need to conduct a practical usability test with appropriate infrastructure and tools. Then we can collect data to be analyzed and get the correct actions upon solving any usability issues. So we called it as Moderated Usability Test and let’s discuss how we can plan and do it.

1. Planning a Moderated Usability Test

Planning a test should be the first part and it is very vital when we conducting this kind of complex test. And the testing scope will get differ upon the usability requirements.

What you should test?

First, as usability engineer we need to decide what the functionalities that we are going to take are. In this case, we can collect valuable information from BA, design and development teams and select features that are new, often used, error prone and specially critical. After that we should prioritize them and write scenarios which represents typical user work flow. Scenarios should be:
  • Small. Time is costly during usability testing.
  • To the point. The meaning of the scenario should have a specific DOD.
  • Realistic. The scenario should be a normal activity that the average user do.
  • Scenario should be on user’s context. Selected scenarios should be related to the user’s context and we need to understand the participant’s connection with the system.

Following is an example for a scenario:
You need to add a contact with first name, last name and contact image in your company’s CRM web application. Add those information and click on “Save” button. Make sure to let me know when you are done.

Who is going to evaluate the Application?

Who you choose to evaluate the application will have a massive effect on the outcome of the test. Imagine that you’re creating an application that reconcile accounts. Your customers are people who deal with account reconciliation. That is a huge group of people. Narrow your focus to a particular user profile.

While you’re creating the user profiles, you may realize that you have two or more equally important subgroups like people who control the system (Business Admin) and people who use the system (Normail User).

Test with proper participants – Rule of thumb is testing no more than 5 users and running small tests as you can afford.

According to the industry standards, the common curve for usability test is as follows:

If we try to read the curve, as soon as you collect data from a first 3 users, you have already captured almost 50% of the usability issues. After you have usability insights from 5th user almost all of the usability issues were captured and no need to spend more time and money. In other words, as you add more and more users, you learn less and less because you will keep seeing the same things again and again.

Who is going to Observe and how?

When talk about the observation, we have 2 main observation types. First one is ‘Obtrusive observation’ method and the other one is Unobtrusive observation’ method which concentrate on observing what the test user does and refrain as much as possible from influencing her/him by explaining the design or asking questions. In Unobtrusive observation’ method, we can have following combination as observers:
  •         At least 2 developers.
  •         At least 1 tester
  •        And a BA

Where Are You Going to Test?

Now we know, what we are planning to test and who is going to evaluate the system. Now we have another question about the place which we going to conduct the test. The location of the test can be as simple as a meeting room or as complex as a purpose-built facility. If we need to conduct an advanced usability test we need video or audio recording equipment for analysis. Conduct formal tests in an environment that simulates normal use as much as possible. Usability tool like Morae is a good example for a usability tool which we can use to do such tests with advanced data analysis and presentation of results.

In coming articles, I’ll continue this and hope to address topics such as “How we going to create a script”, “Run the tests” and “How we gonna analyze results and prepare the report”.