First Things First
Before we dive in, to make things easier on you during QA, we highly recommend that you set a user_id via user attributes. Why? If you have an issue, you can literally look up that user_id in the user insights section of the platform and see what’s going on with that specific user.
For a team’s first experiment
While we do not suggest pushing an experiment live to QA, you may want to do so anyway. We understand some teams may make this decision. The QA guide below will give best practices for QA’ing in draft mode and will make suggestions on how to QA with an experiment that is live where necessary. If you choose the route of running QA on live experiments please do reach out directly for custom guidance for your specific needs.
QA for Visual Editor Tests
Our expectation for visual editor tests is that you will be running QA to verify that:
- your experiment looks as expected
- the app / experiment behaves as expected
- events are being tracked appropriately
- distribution / segmentation work the way you expect them to
The Tools at your disposal to run QA are:
- The Taplytics Shake Menu
- Previewing selector in visual editor
- User Insights
- Taplytics BigQuery
Whatever devices you’ll be using for your QA process, you will need to make sure if you want to switch between variations your device needs to be paired.
For visual changes you need to make sure you’re at the highest level (root) activity to invoke the change.
Why do I need to be at the root activity?
With the way layouts are created on Android, if you invoke a change while on the activity that is changing any changes might be applied inconsistently, if at all, because the code to draw that activity or fragment has already run.
Android Code-based Experiments
Code experiments are one of the most powerful features of Taplytics. To be able to QA code experiments properly, there are a few best practices that you need to follow and tools that we’ve built specifically for the purpose of QA.
In order to QA for Android code based events you need to leverage two Taplytics features together. One is the Taplytics Shake Menu and the other is the TaplyticsExperimentUpdatedListener.
By wrapping the dynamic variables in the TaplyticsExperimentUpdatedListener, you will enable use of the Shake Menu so that after the Shake Menu is used you can grab the new values for the dynamic variable and test the dynamic variable that way.
When leveraging the Shake Menu with code experiments, because default code will load on app launch you need to ensure that you can re-run the relevant code.
Ideally you should force the change using the shake menu after navigating to another screen. Once the change has been loaded to the app you can then navigate back to the screen where the code experiment is supposed to take place.
How to Test Specific Experiments and Variation combinations from app launch
If you need to test experiment variation combinations right from app launch we have provided a client-side option flag to enable this.This is a map you can add to Taplytics start options with the experiments and the corresponding variations you want to test. This allows you to get around having to use the Shake Menu to invoke a test in your app.
Please see our docs on how to do this.
How to QA Event Tracking
There are three types of events to keep in mind during QA:
- Dynamic events (button clicks, views appearing, time on view)
- Taplytics code events
- Third party analytics events
In order to QA dynamic or third party events, you would need to start an experiment as opposed to leaving an experiment in draft mode for visual related experiments This is to see real live event data flowing.
To limit it to testing devices only, you should set your distribution to an internal build only before choosing a build that’s publicly available.
If you plan to use dynamic events or third party events for your goals
Open up the app after you push the experiment live. You would then trigger the event (button events, view appeared goals etc) and perform the actions that trigger the events that you’re looking for.
Once you’ve triggered the events, background your app (hit the home button). Why? This makes sure that the events are batched and delivered to Taplytics’ servers. There is a potential for a longer delay loading data while waiting for the batch to happen without backgrounding the app.
While we try to populate event data in real-time it might take up to 30 minutes for the first events to populate, so sit back, relax and make yourself a coffee while you wait. ☕️
If you plan to use Taplytics events for your goals
With Taplytics events you can QA that they are being fired correctly without having to set your experiment live. The way to verify for Taplytics code events is through user insights. As soon as the events are triggered they will show up in the timeline view of the user insights page for your specific device.
This is where setting a user ID comes in very handy, because you can use that ID to pinpoint your device in Taplytics and verify your setup right away.
TESTING DISTRIBUTION / SEGMENTATION
While testing distribution and segmentation would normally require you to set your experiment live, Taplytics has functionality built in where you can test distribution and keep your experiment in draft.
This feature is a setting in the Project Settings page. Scroll down to the bottom of the page. Once you're there, you can enable the option to distribute draft experiments to Taplytics’ connected development builds.
If you’re using the getRunningExperimentsAndVariations method, the great thing about this is that even if they haven’t pushed the experiment live, you can see what variation a user is assigned to.
You should be able to see console logs for Xcode or in device instruments, and verify that the distribution is applied correctly as well. Finally you’ll also be able to verify experiment and variation distribution for devices on the user insights page
How do I verify what variation my device is in?
You can determine if a device has been bucketed into a certain variation by logging into user insights and verifying manually.
You’ll be able to see distribution in the experiments page, specifically how many people were bucketed into each variation, it’s important to note that the way we handle distribution is we effectively distribute based off your weighted chosen probabilities independently for each device.
Even if you set your distribution to 50/50 and you try with 10 different devices, there’s still a possibility, however small, all 10 devices will be in one variation. The bigger the sample you’re distributing the experiment to, the closer the distribution will get to 50/50.
In order to QA properly, our recommended best practices is to have specific QA user accounts. You should always use the same user account on the same device for QA.
DELETING DEV BUILDS / LOADING NEW VERSIONS
When changing builds, deleting an app, loading a new app, or just changing from a Test Flight build to an App Store build - there are possible repercussions in terms of the way in which a device is tracked in Taplytics.
What that means is that we assign our own unique identifier to a device. We log that device and set it up and attach it to our model of a user. The device and user have their own unique identifiers, and we do our very best to track those users and those devices between sessions, between updates and even between deletion and re-installation.
However, when it comes to QA, when your team may be loading builds that are provisioned in different ways (App Store, TestFlight, HockeyApp, Crashlytics etc) or even have completely different bundle identifiers. It is possible that Taplytics may not be able to effectively track the upgrade or change as the exact same device, for the exact same user.
The consequence of this is that we may maintain old devices as active or multiple instances of the same device. This mainly manifests itself when testing push notifications and the possibility of having multiple push tokens set for a single device.
To ensure a proper QA environment and management of users and devices within Taplytics we believe that when deleting a QA app or changing between build types you should first ensure that you are resetting the Taplytics appUser through a logout and then have all QA users log out of the app before making a change to the app on their device. The docs on how to reset the Taplytics appUser can be found here.
The key is to reset the app user. If you are running tests in the future, we’re not trying to send to that app user because they deleted it and we think that app user is still affected.
MULTIPLE USER ACCOUNTS PER DEVICE
This is more a caution to be aware of. In some QA environments, you’ll have multiple users per device (sharing QA devices between users). The best practice is having one user ID per QA device that is kept consistent. In the real world, you're going to have 1 user device per person and not changing user accounts.
The other thing is to be aware of is that if you don’t do those things properly (reset users, log in / log out), and you have multiple user accounts on the device, you will have no way of keeping track what devices might be getting messages. We won’t be able to determine know if that device is no longer active unless you tell us.