Pop-ups come in different shapes and sizes! Pop-up definition: anything that suddenly appears (text and/or window) and you don’t need to hover over it in order to see it.
What this rule means for you: The rule exists to help clients and testers discern between pop-ups that could represent a bug or a defect, and something that is a harmless browser or system notification, or sign of things working as intended in their app. It’s important to differentiate between the types of pop-ups that are out there, as described below.
What clients want:
- Clients are looking for bugs that need fixing: things that prevent their application from working properly. Bugs usually cause the app to crash or give errors.
- Some may be testing explicit pop-up behavior, case in which they will usually let you know or it will be obvious from the instructions and you should follow their guidance.
- In many cases, they may have more harmless, unrelated pop-ups than they remember to mention in instructions (usually marketing pop-ups). In these cases we do advise them to issue a blanket ‘ignore all pop-ups’ message just to be sure; if you see that, you should only focus on those that may indicate a bug or crash.
- In rare cases, they may ask you to ignore a specific kind of error or crash pop-up, e.g. due to a testing environment not behaving exactly in the same way as a real, production environment would. As described below, you should follow the test author’s instructions and ignore specific pop-ups called out but let them know about any others that may indicate their app is not working.
- In exceptionally rare cases, they may ask you to ignore all pop-ups, including those that may indicate errors or crashes, and keep answering the questions. If you can still answer the question asked with a YES based on the instructions, keep going. As mentioned in Rule 1, you should be concerned if the client forces you to answer yes when the answer should be no.
How to Apply This Rule:
You should classify each pop-up you see into one of the 3 groups below and follow the instructions based on the relevant group that the pop-up belongs to:
- Pop-ups which indicate there is a bug
- Pop-ups which don’t indicate a bug
- Pop-ups which may indicate a bug, but you aren't sure if this is the case
Here is what you should do in each situation:
1. Pop-up which indicates a Bug - This is a pop-up that includes an ERROR/CRASH notification. Basically, it indicates that something went wrong.
- Always fail a test by reporting No when you encounter an ERROR/CRASH notification that you weren’t specifically instructed to ignore.
- If the test author instructed you to ignore a specific type of ERROR/CRASH pop-up, then you should follow the test author’s instructions and ignore only this specific pop-up. That being said, make sure to report all other ERROR/CRASH pop-ups that weren’t mentioned by the test author.
- If a test author tells you to ignore all pop-ups, you should ignore all pop-ups, except ERROR/CRASH notifications. You must always answer No to report ERROR/CRASH notifications not specifically mentioned in the instructions.
- If a test author tells you to ignore all pop-ups including error pop-ups, then you should follow the test author’s instructions and ignore all pop-ups, including ERROR/CRASH notifications. In other words, continue with the test and questions until you can’t answer yes anymore. If you do encounter an ERROR/CRASH notification you think the customer really should know about, you should alert them using the “Suggest Improvement” category “There was an unexpected popup in this step”, and then continue with the test if possible.
2. Pop-up which doesn't indicate a bug - this pop-up indicates an expected system/browser/application behavior. for example - a side chat-box, marketing ad, standard browser question such as “do you want to save this password?”, “do you want to run this file?”, etc.
- Ignore these pop-ups. If they don't disappear and don’t interfere with the flow of the test - take the necessary action to close them so you can move on with your test. Note: This could mean clicking ‘accept’ on something, more often than not, unless specified otherwise by the client (e.g. “Do you want to allow the application to use your location?” - both yes and no would make it go away, unless the author is testing something else and advises you differently. See our educational articles if you are confused).
3. Pop-ups which may indicate a bug, but you aren't sure if this is the case -
- If the pop-up doesn’t fall into one of the two other categories (i.e. it’s not an ERROR/CRASH pop-up, and it’s not an expected behavior by the system/browser/application), and you think that there is a slight chance that this pop-up may indicate a bug - then take the most reasonable action to close it, notify the user that you closed it via the “Suggest Improvement” option (click "Suggest Improvement" and choose “There was an unexpected popup in this step" category) and move on with your test.
- If the test author instructed you to ignore/close a specific type of pop-up, then you should follow the instructions and ignore/close it without mentioning it in the "Suggest Improvement”.
Using Suggest Improvement
While we recommend the use of Suggest Improvement in some of the above cases, it is not required. If you choose not to use SI, follow the instruction to the best of your ability and answer Yes or No accordingly.