Testing requires a particular mindset. If you possess the skill of identifying flaws or weaknesses in systems, then you possess the essence of testing! Although this idea has been around for a long time, it’s not as straightforward as it seems, right? Throughout my extensive 20+ year journey in the tech industry, I’ve encountered numerous QA professionals, and one consistent trait I’ve observed is their conscientiousness—a blend of organization, meticulousness, and a natural inclination to pay attention to details. Another set of qualities includes curiosity and the resulting ingenuity. These traits foster creative thinking and a willingness to explore unconventional approaches, like repeatedly pressing the login button a thousand times to observe the outcome or attempting to input alphabets into a numeric field. Apart from being extremely thorough and inquisitive, the true value a tester brings lies in effective communication and collaboration. Testers often engage in discussions with product managers, program managers, developers, and fellow testers to establish the what, why, how, and when of the project requirements, leveraging this knowledge to determine the testing approach.
Now, here’s the million-dollar question: Can defects be found without even touching the keyboard? Absolutely! By asking pertinent questions to product managers and developers, you can uncover potential issues. This is where it becomes crucial to break away from established routines and habits.
Test Outside the Box
“It’s easier to think outside the box when you break the shackles in your mind”
Our minds are often conditioned to seek out what is obvious. Have you ever experienced this? Personally, I enjoy cooking occasionally, especially on weekends, and I have a penchant for using a variety of Indian spices. However, there have been instances when I struggled to locate a particular chili sauce or strands of saffron. It was only when my wife entered the kitchen and pointed out that they were right in front of me, that I realized I had been staring at them the whole time. Why did this happen? It occurred because subconsciously I had convinced myself that I wouldn’t be able to find them.
Applying a similar line of thinking, in the realm of testing, when we are presented with a set of requirements, why do we tend to assume that they are flawless? With this mindset, our attention becomes limited to merely confirming whether the product or its functionalities operate as stated. However, we forget to consider other aspects. The burning question is, how can we train our subconscious mind to overcome this limitation?
This is where I prefer to redefine testing as a learning process that involves exploring and experimenting with the product, in addition to validating its adherence to the requirements. The former helps train our subconscious mind to think beyond the obvious, while the latter allows our conscious mind to verify the specified requirements. The exploration and experimentation exercises will allow our brain to ask questions like –
“What else? What if? How and when can this be used? What happens when? Who are the users? What value does it provide? To whom? How many users? How many concurrent users? How long? Who are the bad actors? How can they harm? “
From my perspective, if we approach a project with the belief that our sole responsibility is to verify and validate the software based on the given requirements, we will overlook significant defects that may prove costly. The key is to actively challenge the product or system under test by fostering creativity and examining its flows from various perspectives. This approach enables us to uncover missed requirements or identify gaps in the requirements as a whole. One such interesting concept is the Time Shift Testing.
Time Shift Testing
Often times we are so engrossed in testing the functionality, that we forget to ask questions such as
“What happens next? Is there a wait period? Is this really the end of the user journey? Could something else happen?
I would like to share a recent real-life example that highlights the importance of this. After the implementation of PSA Japan, an incident occurred where several orders remained stuck even after grading and payment processing, resulting in shipping delays. Upon investigation, it was discovered that the system was referencing the order date instead of the payment posting date. This issue could have been easily missed during testing if we had processed all orders on the same day, as the order date and payment date would have been identical, concealing the problem. However, by simulating the production environment with a time gap between order initiation and payment processing, ranging from 1 to 90 days, we significantly increase the likelihood of detecting such issues. As it turns out, our QA team was already knowledgeable about this situation and had previously tested this specific scenario. The incident, therefore, was determined to have been caused by other factors.
The 80:20 Rule
The principle of the 80:20 rule also applies to testing. Personally, I am a strong advocate of this rule, which can be applied to various concepts in life. As a non-testing example, it is known that over 80% of the world’s income is controlled by just 20% of the population. In the testing domain, this rule can be a smart approach to prioritize testing efforts on areas of the application that are likely to have a higher number of defects. Let’s consider the following example for Windows users:
“Approximately 80% of errors and crashes in Windows and Office are caused by just 20% of all detected bugs.”
Typically, around 80% of defects arise from testing only 20% of the modules. This means that once we discover a defect, there’s a good chance we will find more in the same area, making it a prime location for testing. Conversely, if we run tests in an area and don’t find any defects, it is likely that this module is relatively bug-free. This insight can greatly assist in planning testing efforts by focusing on these key areas for automation or exploratory testing. While this approach effectively directs testing focus to the right areas of the system, it is somewhat reactive. We need to start testing to uncover the weaker areas of the application and then allocate the majority of our efforts there. However, is there a proactive way to identify these areas using this principle?
“Today’s risks are tomorrow’s problems.”
Taking a proactive approach involves identifying potential risk areas of the application. If we prioritize building tests around these areas first, we increase the likelihood of discovering defects more quickly and achieve the desired left-shift in testing, as advocated by many. But how do we identify these risk areas? One example of a risk area could be high-usage sections of the application. Here’s an analogy that everyone can relate to:
“20% of the carpet in your office will be used 80% of the time.”
If we know that our customers primarily spend their time in specific areas of the application, it makes sense to invest more testing time and effort in these areas to ensure they are free of bugs. For instance, in our PSA software, the “online submission” and the “my orders” screen are high-usage areas for customers, whereas the PSA store may not experience heavy traffic.
Another risk area to consider is critical functionality, such as the login screen or payment and checkout screens. Focused testing in these high-risk areas can effectively minimize the exposure to defects for our customers.
Old Habits Die Hard
While I strongly advocate for shifting testing as early as possible in the development process and recognize the many advantages it brings, such as quicker feedback, early bug identification leading to cost reduction, and overall quality improvement, I observe that many organizations struggle to make this transition. Culturally, we need to shift our mindset away from considering testing as the final stage of a process and instead embrace the concept of incorporating testing from the very beginning. Whether it involves testing a product idea or reviewing business requirements, by identifying ambiguities or potential issues early on, the team can prevent rework and address gaps that may arise later in the development process.
Let me share an example from a recent project. During a recent project, we encountered numerous bugs during integration testing when the actual systems started communicating with each other. It raised the question of why we were discovering so many issues at such a late stage in the project. Didn’t we test the individual components using mocks and stubs to simulate the external components? Well, the test harnesses and stubs that were implemented served their purpose up to a certain level, but there were several scenarios that couldn’t be adequately replicated with those approaches. This isn’t to suggest that we can’t build more robust harnesses that closely mimic the real systems, which could be a separate topic to explore. However, limitations will still exist. It’s also not to say that we should avoid building any test harnesses altogether; they are essential for testing code that is ready without having to wait for other systems. This way, we can identify some unit and component-level bugs early on.
Pertinent questions raised by the team during integration testing shouldn’t have to wait until that phase begins. These discussions should happen before a developer writes a single line of code. They should take place during PRD (Product Requirements Document) reviews, design discussions, and scoping discussions. Ideally, these discussions should involve domain experts from different teams, including QA. This presents a valuable opportunity to uncover missed requirements and underscores the need for shift-left testing.
In Summary….
Regardless of whether you adhere to conventional testing methodologies as outlined in established guidelines or leverage innovative thinking through techniques such as Test-outside-the-box, Time-shift, 80-20 rule, or the Left-shift strategy mentioned earlier, it remains crucial to challenge the existing norms and break free from habitual practices. Breaking free from habits involves consciously stepping outside of comfort zones and exploring new ideas, techniques, and tools. It encourages testers to embrace change, experiment with different approaches, and be open to learning from failures and successes. I will leave you with this story I came across some time back.
“There was a man who noticed some elephants in a circus, seemingly held in place by a small rope tied to the front leg. It was clear that the elephants had the physical strength to break free from their restraints, but they made no attempt to do so.
Curious, the man approached a nearby trainer and asked why the elephants remained stationary without trying to escape. The trainer explained, ‘When these elephants were very young and much smaller, we used the same rope to tie them. At that age, the rope was strong enough to keep them in place. As they grew up, they became conditioned to believe that they couldn’t break free. They still think the rope can hold them, so they never attempt to break away.”
In a similar way, how many of us come to work with the belief that we cannot accomplish something, simply because we fear failure?
