When we test software, the quality of test ideas determines the quality of the tests and the information discovered. Many testers get stuck after a few test ideas. They feel exhausted and bored. If one expects to achieve a broad and deep test coverage, the test ideas must also be broad and deep. Here is an attempt to highlight a repository of test ideas. All of these might not be required for all the applications. Based on your application and the type of tests you need to focus on, pick the relevant test ideas.
Use case of the Feature
Every feature is built to serve one or more important use
cases. For example, a registration feature’s main use case could be to onboard
new users and create accounts for them in this particular product. The next
time they want to login, they could use the credentials used to register an
account. If the feature registers users but doesn’t save the credentials in the
database or doesn’t allow them to login with the same credentials, then the
purpose of registration as a feature is lost.
As soon as a feature is up for testing, many testers jump to
test the field validations and edge cases. Sometimes, the main use case is not
tested and comes as a surprise after release. I am also guilty of this mistake.
There was a feature which let users register via email or Active Directory. I
had tested for all combinations of email and had never tested the Active
Directory mode of registration. During production testing, it was discovered
and I was asked the question – when was the last time it was tested? When I
answered in the negative, I had learned an important lesson and started testing
for the use case as the first test.
Is the problem
solved?
Before the feature came into existence, there must have been
a problem and this feature was planned to solve that problem. Do you as a
tester understand the problem? If you don’t understand the problem, it will be
difficult to appreciate the solution. One would also not be confident if the
actual problem is solved or not. I learned this when I had to test the guided
workflow to generate APNS (Apple Push Notification Service) certificate. Before
testing the new workflow, I experienced the existing way of generating a
certificate which was easily 10-12 steps and error prone at each step. This problem
was to be solved via the workflow. I could now easily relate to the pain in the
old flow and highlight if the new workflow did not remove them.
Known constraints
Multiple validations and messages are highlighted beforehand
to the users in most of the applications. Password policy, maximum file size
limit, minimum members required for a group chat, who can perform certain
actions and so on. These constraints are highlighted to the user. Test those.
Also, look out for constraints that are not documented but
are present in the system.
Test if those constraints are necessary or not.
If necessary, get them documented unless it is a security
issue.
If they are not necessary, get them removed from the
application.
Client side vs Server side validation: Also check if there
is server side validation too or just the client side validation is fooling
you.
Users
Then I start thinking about the users – both intended and unintended users. Are the intended users able to carry out their day to day use cases with the application? What would be their top five use cases? If you don’t understand the users, you might be spending time on features which the user would never use. Even if the features are the same, the way they are used will definitely change. In one of the releases, we had changed the way users could reset their password. We had introduced the security questions and removed the phone number. The application worked well for new users who were asked the security questions. The old users who were onboarded before the addition of security questions couldn’t reset their passwords. It is a good idea to make a list of users and think how would they interact with the product. A sample list could be
- new users
- existing users
- migrating from competition
- blacklisted users
- hibernating users
- premium users
- loyal users
- hackers
- competition trying to learn about your product
Field Validations
What kind of validations are present for each field? Are they reasonable? Are they highlighted before a user can violate them? I have encountered multiple applications where the password policy is not mentioned beforehand and the user has to discover the allowed length, valid characters, password rules, repetition policy all by experimentation. It is useful to display the validation message beforehand. The application should help users save time. If the application makes the user feel helpless and frustrated, the chances of them shifting to the competition increases manifold.
Heuristics to the rescue
Testers get blocked because they are not able to think of alternative ways of approaching the same application. Heuristics can be of great help here. A good start can be ‘FEW HICCUPPS’ heuristic where testers can get started on testing any application by applying this heuristic. You can find a good collection of mnemonics and heuristics here. It is not necessary to follow every heuristic every time. It is helpful to know multiple heuristics and apply the right one as applicable.
Another useful list of heuristics can be found by Elisabeth
Hendrickson here. http://testobsessed.com/2007/02/test-heuristics-cheat-sheet/
Feature Interaction
Once the individual feature has been tested for use case, the validations are taken care of, it is now time to think of feature’s impact on other features and the interaction across features.
For example, if the file description is to be passed in the
script, due care has to be taken for the characters which might indicate the
start of comments. A feature might work individually well but when integrated
with other features too, it should continue to work. Many times, testers take
care of individual features and the integration is untested till it’s too late.
Thinking in terms of feature interaction also helps in prioritizing cases
during regression.
Think in terms of the data flow. Does this feature take data
from other features as input? Does the output of this feature act as input for
other features? This understanding will help finetune your cases in terms of
the data flow.
Standards and user expectations
Outside of the application, there are certain standards one
expects from any application. This is similar to the S – Standards in the FEW
HICCUPPS heuristic. As a user, what would you expect from the feature? Is it
met? Are there questions? Are you confused? Is the feature irritating? These
are some of the questions I am thinking during this section.
Probable bugs for the
feature:
One of the interesting tasks I do is to list out the probable bugs for a
feature well before its handed over to testing. Most of us can come up with a
list of “What can go wrong” with a particular feature. If we brainstorm as a
team and prepare this list beforehand and even share this with the development
team, it will save a lot of time for everyone. This list acts as a checklist to
quickly report the bugs, spend time exploring the unknowns and discovering more
useful information about the feature.
Code, Database, Logs
Going through the code gives you another perspective of what
to expect and what has been implemented. One can always expect new test ideas
after looking at the code. It is also beneficial to discuss with the
development and product team members about the product and their expectations.
I have had the best discussions with the development and product teams
understanding the why and how behind the feature. Most of the user actions
would leave a trace either in the database or in the logs. Check for the
consistency and what’s being saved at both these places.
Error scenarios and
the messages for the feature
What if a user gets into an error state? Is it easy to
recover from there? Is the system capable of handling the error scenarios? What
messages have been coded to be displayed in different scenarios? This can also
be used as a checklist to ensure that all those scenarios are covered at least
once. As a tester, I would be very worried if I see a message in code and
unaware of its presence in the user interface.
Other quality
criteria
With most of the above points covered, I would also move on
to other quality criteria like Usability, Security, Accessibility, Performance
and Compatibility. This would ensure that the majority of the scenarios are
covered and most of the stakeholders have important questions answered through
the tests.
To wrap up, use this checklist as a guide:
- Understand the Use case of the feature
- Ask yourself: Is the problem solved by this feature
- Test for the known constraints
- Think about the users
- Do validate the field validations
- Take help of heuristics
- Map out the feature interaction
- Beware of the Standards and user expectations
- Hunt for the probable bugs
- Look in the code, database and logs
- Review error messages
- Think of other quality criteria
What is your order of tasks when it comes to test ideas?
Written by: Ajay Balamurugadas, AVP Delivery, Qapitol QA.
Reach him at: [email protected]