Push Notification QA Checklist

Background

Running QA on push notifications has its own unique challenges. The iOS and Android platforms have very specific requirements to make sure that delivery is successful and that pushes are being delivered to the right devices. It is because of this that the QA of push is unique from that of our A/B testing and Analytics products.

This guide will provide some best practices, as well as a full checklist you and your team can go through to make running QA on your implementation of push, as quick and easy as possible.

Finally, as you read this guide, just remember that these are general recommendations. We believe they will make your life better, but of course, there is no one-size fits all solution, so feel free to modify as necessary for your specific needs. And if you need help building a custom QA process, our Customer Success team would be happy to help!

 

Helpful Tools

There are some tools that can also make your life easier in understanding what’s happening “under the hood” so to speak. If you can be armed with these tools ahead of running through QA, you can save yourself a lot of time. This guide will not require the use of anything more than the core Taplytics Push platform and your QA devices, but if you would like help with how to use other tools in your QA process our Customer Success team would be happy to help get you setup on these convenient tools.

• Access to the Taplytics Interface
• I know this should be a no-brainer, but sometimes QA users don’t get access to the tool that is generating the push notifications. Making sure you or other QA users have access to your app’s Taplytics project should be the first thing you do.
• Charles (or some other proxying tool)
• Charles is a network monitoring that shows you the network requests that your device is making. This tool is amazingly useful for letting you know what your device is getting from Taplytics.
• Taplytics BigQuery (if you aren’t using this yet, ask your Account Manager)
• Taplytics BigQuery gives you and your team access to the real-time feed of all of your raw data within Taplytics. If you know SQL, you can query the data on the fly to see what events and data Taplytics is logging.

 

Key Reminders:

1. Pushes are generally delivered based on user-level data.
2. Taplytics associates static device push tokens with users when you set a user ID.
3. The deletion of apps cannot be detected by Taplytics. As far as Taplytics knows you are still logged in to the app.
4. Because of this, if you delete an app without logging out and then re-install a new build, we will have two active user accounts for you.
5. If you are running QA on multiple devices with multiple user IDs, it can become unpredictable as to which devices and push tokens are associated with which user IDs at any given point in time.

The end result is that push notification delivery can become unpredictable in an uncontrolled QA process. To combat this we suggest a few simple overarching rules for your QA process.

 

Process Rules:

• Require each QA user to have their own user account for QA.
• Ensure that there is only one build of the app running on a QA device at a time.
• Log in to your app at the beginning of each session.
• Log out of your app at the end of each session.
• Always log out of your account before deleting a build of the app.

 

Pre-steps: (if you need to do some spring cleaning)

• Log out of your user account on your app
• On Android, delete all app data from App Info screen
• Delete all builds of the app on your device
• On iOS, reset your advertising identifier in Settings > Privacy > Advertising
• Generate a new, never before used, user ID to test with

 

Core Steps:

• Install the most recent build of your app onto your test device
• Open your app and log in (if applicable)
• Background your app
• Force close your app
• Open your app
• Background your app again
• Search for your device in Taplytics User Insights by using your user ID
• Confirm in User Insights that we have registered a push token for your device (this may take up to 5 minutes to register)
• If at this point no push token has registered for the environment you are trying to QA, please consult with your engineering team or your Taplytics Customer Success team to confirm push token registration is set up appropriately.
• Once you have confirmed that you have a push token, go to the push notification area of Taplytics
• Create a new notification
• Target the environment that matches the push token that was registered for your device, Development or Production
• Set up a segmentation that limits to your device or a set of QA devices (this can be your new user ID)
• Give your push a name
• Provide some test text, images, custom data, or anything else
• Select the “Send it Now” delivery option
• Double check all of your settings and setup
• Double check that your app has been backgrounded
• Triple check all of your push settings, particularly your segmentation if Taplytics is live in your app already
• Check how many users will be sent the push, by confirming the estimate of how many will be sent next to the “Send Now” button
• If that number looks good, click the “Send Now” button
• Wait a few seconds
• Confirm that the push is received on your device