First Things First
Before we dive in, to make things easier on you during QA, we highly recommend that you set a user_id via user attributes. Why? If you have an issue, you can easily look up that user_id in the User Insights section of the platform and see what’s going on with that specific user.
For a team’s first experiment
While we do not suggest pushing an experiment live to QA, you may want to do so anyway. We understand some teams may make this decision. The QA guide below will give best practices for QA’ing in draft mode and will make suggestions on how to QA with an experiment that is live where necessary. If you choose the route of running QA on live experiments, please do reach out directly for custom guidance for your specific needs.
QA for Visual Editor Tests
Our expectation for visual editor tests is that you will be running QA to verify that:
- your experiment looks as expected
- the app/experiment behaves as expected
- events are being tracked appropriately
- distribution/segmentation works the way you expect it to
Tools at your disposal are
- Variation Switcher
- Shake Menu
- Previewing selector in the Visual Editor
- User Insights
- Advanced pairing - to test experiments on production builds (Test Flight, Hockey App, App Store, Fabric, enterprise distribution builds, etc)
Whatever devices you’ll be using for your QA process, you will need to make sure if you want to switch between variations your device needs to be paired.
For visual changes, you need to make sure you’re at the highest level (root) activity to invoke the change.
Why do I need to be at the root activity?
With the way layouts are created in Android, if you invoke a change while on the activity that is changing, any changes might be applied inconsistently, if at all, because the code to draw that activity or fragment has already run.
Android Code-based Experiments
Code experiments are one of the most powerful features of Taplytics. To be able to QA code experiments properly, there are a few best practices that you need to follow and tools that we’ve built specifically for the purpose of QA.
In order to QA for Android code based events, you need to leverage two Taplytics features together. One is the Taplytics Shake Menu and the other is the TaplyticsExperimentUpdatedListener.
By wrapping the dynamic variables in the TaplyticsExperimentUpdatedListener, you will enable the use of the Shake Menu so that after the Shake Menu is used you can grab the new values for the dynamic variable and test the dynamic variable that way.
When leveraging the Shake Menu with code experiments, because default code will load on app launch you need to ensure that you can re-run the relevant code.
Ideally, you should force the change using the shake menu after navigating to another screen. Once the change has been loaded to the app you can then navigate back to the screen where the code experiment is supposed to take place.
Once you find your device using User Insights or through Device Settings (if you've paired your device), you can force our device into a variation for the duration of the experiment. Once assigned a variation, that device will continue to get that variation until you have removed it from the experiment.
It's important to keep in mind that this variation will still continue to be delivered to the device even if it's in paused or draft mode until you remove it from the experiment. Other ways that will stop the delivery of the experiment to the device is if you have archived or deleted the experiment.
How to Test Specific Experiments and Variation combinations from app launch
If you need to test experiment variation combinations right from app launch, we have provided a client-side option flag to enable this. This is a map you can add to Taplytics start options with the experiments and the corresponding variations you want to test. This allows you to get around having to use the Shake Menu to invoke a test in your app.
Please see our docs on how to do this.
How to QA Event Tracking
There are three types of events to keep in mind during QA:
- Dynamic events (button clicks, views appearing, time on view)
- Taplytics code events
- Third party analytics events
In order to QA dynamic or third party events, you would need to start an experiment as opposed to leaving an experiment in draft mode for visual-related experiments. This is to see real live event data flowing.
To limit it to testing devices only, you should set your distribution to an internal build only before choosing a build that’s publicly available.
If you plan to use dynamic events or third party events for your goals
Open up the app after you push the experiment live. You would then trigger the event (button events, view appeared goals etc) and perform the actions that trigger the events that you’re looking for.
Once you’ve triggered the events, background your app (hit the home button). Why? This makes sure that the events are batched and delivered to Taplytics’ servers. There is a potential for a longer delay loading data while waiting for the batch to happen without backgrounding the app.
While we try to populate event data in real-time it might take up to 30 minutes for the first events to populate, so sit back, relax, and make yourself a coffee while you wait. ☕️
If you plan to use Taplytics events for your goals
With Taplytics events, you can QA that they are being fired correctly without having to set your experiment live. The way to verify for Taplytics code events is through User Insights. As soon as the events are triggered they will show up in the timeline view of the User Insights page for your specific device.
This is where setting a user ID comes in very handy, because you can use that ID to pinpoint your device in Taplytics and verify your setup right away.
TESTING DISTRIBUTION / SEGMENTATION
While testing distribution and segmentation would normally require you to set your experiment live, Taplytics has functionality built in where you can test distribution and keep your experiment in a draft.
This feature is a setting in the Project Settings page. Scroll down to the bottom of the page. Once you're there, you can enable the option to distribute draft experiments to Taplytics’ connected development builds.
If you’re using the getRunningExperimentsAndVariations method, the great thing about this is that even if they haven’t pushed the experiment live, you can see what variation a user is assigned to.
You should be able to see console logs for Xcode or in device instruments and verify that the distribution is applied correctly as well. Finally, you’ll also be able to verify experiment and variation distribution for devices on the User Insights page
How do I verify what variation my device is in?
You can determine if a device has been bucketed into a certain variation by logging into User Insights and verifying manually.
You’ll be able to see distribution in the experiments page, specifically how many people were bucketed into each variation, it’s important to note that the way we handle distribution is we effectively distribute based off your weighted chosen probabilities independently for each device.
Even if you set your distribution to 50/50 and you try with 10 different devices, there’s still a possibility, however small, all 10 devices will be in one variation. The bigger the sample you’re distributing the experiment to, the closer the distribution will get to 50/50.
In order to QA properly, our recommended best practices are to have specific QA user accounts. You should always use the same user account on the same device for QA.
DELETING DEV BUILDS / LOADING NEW VERSIONS
When changing builds, deleting an app, loading a new app, or just changing from a Test Flight build to an App Store build - there are possible repercussions in terms of the way in which a device is tracked in Taplytics.
What that means is that we assign our own unique identifier to a device. We log that device and set it up and attach it to our model of a user. The device and user have their own unique identifiers, and we do our very best to track those users and those devices between sessions, between updates and even between deletion and re-installation.
However, when it comes to QA, when your team may be loading builds that are provisioned in different ways (App Store, TestFlight, HockeyApp, Crashlytics etc) or even have completely different bundle identifiers, it is possible that Taplytics may not be able to effectively track the upgrade or change the exact same device for the exact same user.
The consequence of this is that we may maintain old devices as active or multiple instances of the same device. This mainly manifests itself when testing push notifications and the possibility of having multiple push tokens set for a single device.
To ensure a proper QA environment and management of users and devices within Taplytics, we believe that when deleting a QA app or changing between build types, you should first ensure that you are resetting the Taplytics appUser through a logout and then have all QA users log out of the app before making a change to the app on their device. The docs on how to reset the Taplytics appUser can be found here.
The key is to reset the app user. If you are running tests in the future, we’re not trying to send to that app user because they deleted it and we think that app user is still affected.
MULTIPLE USER ACCOUNTS PER DEVICE
This is more of a caution to be aware of. In some QA environments, you’ll have multiple users per device (sharing QA devices between users). The best practice is having one user ID per QA device that is kept consistent. In the real world, you're going to have 1 user device per person and not changing user accounts.
The other thing is to be aware of is that if you don’t do those things properly (reset users, log in / log out), and you have multiple user accounts on the device, you will have no way of keeping track of what devices might be getting messages. We won’t be able to determine if that device is no longer active unless you tell us.
BEST PRACTICES FOR DOGFOODING NEW FEATURES
When testing new features before release to the general public, it's generally a good idea to release your feature internally and "dogfood" your product before it goes live.
QA for a small number of devices
If you're QA'ing for a manageable number of devices (10-20 device), using the Variation Switcher to force a user's device into an experiment is the easiest way to go without having to set any custom user attribute flags via code.
QA for entire organizations
If you're looking to distribute something only internally before a feature goes public, it all comes down to the segmentation options.
The easiest way to achieve this is to use appVersion as a segmentation option that isn't publically available yet. If you have a version 2.0 that's only been released internally and isn't slated to be released in the App Store or Play store, you can start an experiment releasing the unreleased appVersion so that only internal users who have access to the unreleased version will receive a feature flag or experiment.
Another way to achieve the desired segmentation so only internal users receive the feature flag or experiment would be to set a custom user attribute that would flag a user as someone who is internal to the company such as "companyEmployee".
Beware of Conflicting Experiments
If you're running multiple experiments concurrently that involve using code variables with audiences that are mutually exclusive, this might lead to weird behavior where your device can be delivered a feature when it's not supposed to or vice versa.
When there are multiple experiments running using the same code variable and conflicting segmentation that you're not aware of, which are receiving (or not receiving) an experiment or feature flag would lead to some false positives in failing QA tests and a frustrating QA experience.
To ensure everyone has visibility on what code variables are being used in live experiments, we can send an export of what code variables are being used in currently running experiments. Just reach out to us in Slack or email, and we can help you get more visibility on code variables.