QA Best Practices for iOS

First Things First

Before we dive in, to make things easier on you during QA, we highly recommend that you set a user_id via user attributes. Why? if you have an issue, you can easily look up that user_id in the User Insights section of the platform and see what’s going on with that specific user.

QA for Appearance-related tests

The best practice for QA is while you’re in draft mode in order to ensure:

• your experiment looks as expected.
• the app/experiment behaves as expected.
• events are being tracked appropriately.
• distribution/segmentation works the way you expect it to.

Tools at your disposal are

• Variation Switcher
• Shake Menu
• Previewing selector in the Visual Editor
• User Insights
• Advanced pairing - to test experiments on production builds (Test Flight, Hockey App, App Store, Fabric, enterprise distribution builds, etc)

Whatever devices you’ll be using for your QA process, you will need to make sure if you want to switch between variations your device needs to be paired.

How to QA in draft mode

If you want to pair with a release build - you’ll have to push the experiment live, which is never recommended. Why? Because any edits you make to a live experiment may introduce some bias and a possibility of human error.

What if some QA people don’t have an account for Taplytics?

In the event a QA person doesn’t have an account with Taplytics, they may just verify the look and feel behaviour, all they need is a dev build.

However, if you want to verify distribution and event tracking, that QA person would need Taplytics, and the designated Taplytics Project Owner or Project Manager should be able to invite that person via email. If you’re having any trouble, just reach out to us at support@taplytics.com or if you have a private Slack channel with Taplytics, send us a message and we'll be happy to help you out.

Yes, our QA people have Taplytics accounts

The first step to get a QA user up and running is signing them onto a Taplytics account and pairing their device

If your QA team is going to QA a production release or enterprise distributed build, they’ll need to set up advanced pairing and pair via email or text message. Please see our Advanced pairing docs for iOS and Android.

In terms of an actual process for QA, to verify look and feel, you would:

1. Open up the app
2. Shake the device to invoke the shake menu
3. Select the experiment, the variation, and do this throughout the app.

If you QA for iOS, you can invoke shake menu from anywhere in a dev build. For production builds, the shake menu can be invoked when you enable advanced pairing.

How to QA iOS code based experiments

In order to QA for iOS code based events, you need to invoke the shake menu, and force the change on some sort of “pre-screen”.

To make iOS code based experiments work, you can’t make the change on the page and just see it happen, because the code has already run. Instead, what you need to do is invoke the change on some other page (“pre-screen”), handle that change so you can apply it then navigate to the page where you’re expecting the code based experiment to occur. When using the shake menu, keep in mind that it does not assign the user/device the variation you're QA'ing - you're only previewing the variation.

Another method to QA code based experiments is the Variation Switcher. This works best when you've set a userID or userEmail. 

Once you find your device using User Insights or through Device Settings (if you've paired your device), you can force our device into a variation for the duration of the experiment. Once assigned a variation, that device will continue to get that variation until you have removed it from the experiment.

It's important to keep in mind that this variation will still continue to be delivered to the device even if it's in paused or draft mode until you remove it from the experiment. Other ways that will stop the delivery of the experiment to the device is if you have archived or deleted the experiment. 

How to test dynamic variables

if you want to test the variables that you’re changing, what you need to do is use the TaplyticsPropertiesLoaded callback and wrap the dynamic variable in that callback.

[Taplytics propertiesLoadedCallback:^(BOOL loaded) {

   TaplyticsVar* 3PageOnboardingFlow = [TaplyticsVar                      taplyticsVarWithName:@"3 Page Onboarding Flow" defaultValue:@NO         updatedBlock:^(NSObject* value){

            // Update your interface/functionality with new value.

    }];

}];

Once you wrap the dynamic variables in the TaplyticsPropertiesLoadedCallback, you can then grab the value of the dynamic variables after that callback.

How to Test Specific Experiments

To test/QA specific experiment and variation combinations, use the TaplyticsOptionTestExperiments start option with an NSDictionary containing keys of the experiment names, and values of variation names (or baseline).

Deleting Dev Builds / Loading New Versions

When changing builds, deleting an app, loading a new app, or just changing from a Test Flight build to an App Store build - there are possible repercussions in terms of the way in which a device is tracked in Taplytics.

What that means is that we assign our own unique identifier to a device. We log that device, set it up, and attach it to our model of a user. The device and user have their own unique identifiers, and we do our very best to track those users and those devices between sessions, between updates, and even between deletion and re-installation.

However, when it comes to QA, when your team may be loading builds that are provisioned in different ways (App Store, TestFlight, HockeyApp, Crashlytics etc) or even have completely different bundle identifiers, it is possible that Taplytics may not be able to effectively track the upgrade or change as the exact same device, for the exact same user.

The consequence of this is that we may maintain old devices as active or multiple instances of the same device. This mainly manifests itself when testing push and the possibility of having multiple push tokens set for a single device.

To ensure a proper QA environment and management of users and devices within Taplytics, we believe that when deleting a QA app or changing between build types you should first ensure that you are resetting the Taplytics appUser logout and then have all QA users log out of the app before making a change to the app on their device. The docs on how to reset the Taplytics appUser can be found here.

The real key is to reset the app user. If you are running tests in the future, we’re not trying to send to that app user because they deleted it and we think that app user is still affected.

How to Deal with Multiple User Accounts Per Device

This is more of a caution to be aware of.  In some QA environments, they’ll have multiple users per device (sharing QA devices between users). The best practice is having one user ID per QA device that is kept consistent.  In the real world, you're going to have 1 user device per person and not changing user accounts.

The other thing is that if you don’t do those things properly (resetting users, logging in, logging out), and you do multiple user accounts on the device, you literally have no way of keeping track what devices might be getting messages. We don’t know if that device is no longer active unless you tell us.

BEST PRACTICES FOR DOGFOODING NEW FEATURES

When testing new features before release to the general public, it's generally a good idea to release your feature internally and "dogfood" your product before it goes live.

QA for a small number of devices

If you're QA'ing for a manageable number of devices (10-20 device), using the Variation Switcher to force a user's device into an experiment is the easiest way to go without having to set any custom user attribute flags via code.

QA for entire organizations

If you're looking to distribute something only internally before a feature goes public, it all comes down to the segmentation options.

The easiest way to achieve this is to use appVersion as a segmentation option that isn't publically available yet. If you have a version 2.0 that's only been released internally and isn't slated to be released in the App Store or Play store, you can start an experiment releasing the unreleased appVersion so that only internal users who have access to the unreleased version will receive a feature flag or experiment.

Another way to achieve the desired segmentation so only internal users receive the feature flag or experiment would be to set a custom user attribute that would flag a user as someone who is internal to the company such as "companyEmployee".

Beware of Conflicting Experiments

If you're running multiple experiments concurrently that involve using code variables with audiences that are mutually exclusive, this might lead to weird behavior where your device can be delivered a feature when it's not supposed to or vice versa.

When there are multiple experiments running using the same code variable and conflicting segmentation that you're not aware of, which are receiving (or not receiving) an experiment or feature flag would lead to some false positives in failing QA tests and a frustrating QA experience.

To ensure everyone has visibility on what code variables are being used in live experiments, we can send an export of what code variables are being used in currently running experiments. Just reach out to us in Slack or email, and we can help you get more visibility on code variables.