QA Best Practices for iOS

First Things First

Before we dive in, to make things easier on you during QA, we highly recommend that you set a user_id via user attributes. Why? if you have an issue, you can easily look up that user_id in the User Insights section of the platform and see what’s going on with that specific user.

QA for Appearance-related tests

The best practice for QA is while you’re in draft mode in order to ensure:

• your experiment looks as expected.
• the app/experiment behaves as expected.
• events are being tracked appropriately.
• distribution/segmentation works the way you expect it to.

Tools at your disposal are

• Shake Menu
• Previewing selector in the Visual Editor
• User Insights
• Advanced pairing - to test experiments on production builds (Test Flight, Hockey App, App Store, Fabric, enterprise distribution builds, etc)

Whatever devices you’ll be using for your QA process, you will need to make sure if you want to switch between variations your device needs to be paired.

How to QA in draft mode

If you want to pair with a release build - you’ll have to push the experiment live, which is never recommended. Why? Because any edits you make to a live experiment may introduce some bias and a possibility of human error.

What if some QA people don’t have an account for Taplytics?

In the event a QA person doesn’t have an account with Taplytics, they may just verify the look and feel behaviour, all they need is a dev build.

However, if you want to verify distribution and event tracking, that QA person would need Taplytics, and the designated Taplytics Project Owner or Project Manager should be able to invite that person via email. If you’re having any trouble, just reach out to us at support@taplytics.com or if you have a private Slack channel with Taplytics, send us a message and we'll be happy to help you out.

Yes, our QA people have Taplytics accounts

The first step to get a QA user up and running is signing them onto a Taplytics account and pairing their device

If your QA team is going to QA a production release or enterprise distributed build, they’ll need to set up advanced pairing and pair via email or text message. Please see our Advanced pairing docs for iOS and Android.

In terms of an actual process for QA, to verify look and feel, you would:

1. Open up the app
2. Shake the device to invoke the shake menu
3. Select the experiment, the variation, and do this throughout the app.

If you QA for iOS, you can invoke shake menu from anywhere in a dev build. For production builds, the shake menu can be invoked when you enable advanced pairing.

How to QA iOS code based experiments

In order to QA for iOS code based events, you need to invoke the shake menu, and force the change on some sort of “pre-screen”.

To make iOS code based experiments work, you can’t make the change on the page and just see it happen, because the code has already run. Instead, what you need to do is invoke the change on some other page (“pre-screen”), handle that change so you can apply it then navigate to the page where you’re expecting the code based experiment to occur.

How to test dynamic variables

if you want to test the variables that you’re changing, what you need to do is use the TaplyticsPropertiesLoaded callback and wrap the dynamic variable in that callback.

[Taplytics propertiesLoadedCallback:^(BOOL loaded) {

   TaplyticsVar* 3PageOnboardingFlow = [TaplyticsVar                      taplyticsVarWithName:@"3 Page Onboarding Flow" defaultValue:@NO         updatedBlock:^(NSObject* value){

            // Update your interface/functionality with new value.

    }];

}];

Once you wrap the dynamic variables in the TaplyticsPropertiesLoadedCallback, you can then grab the value of the dynamic variables after that callback.

How to Test Specific Experiments

To test/QA specific experiment and variation combinations, use the TaplyticsOptionTestExperiments start option with an NSDictionary containing keys of the experiment names, and values of variation names (or baseline).

Deleting Dev Builds / Loading New Versions

When changing builds, deleting an app, loading a new app, or just changing from a Test Flight build to an App Store build - there are possible repercussions in terms of the way in which a device is tracked in Taplytics.

What that means is that we assign our own unique identifier to a device. We log that device, set it up, and attach it to our model of a user. The device and user have their own unique identifiers, and we do our very best to track those users and those devices between sessions, between updates, and even between deletion and re-installation.

However, when it comes to QA, when your team may be loading builds that are provisioned in different ways (App Store, TestFlight, HockeyApp, Crashlytics etc) or even have completely different bundle identifiers, it is possible that Taplytics may not be able to effectively track the upgrade or change as the exact same device, for the exact same user.

The consequence of this is that we may maintain old devices as active or multiple instances of the same device. This mainly manifests itself when testing push and the possibility of having multiple push tokens set for a single device.

To ensure a proper QA environment and management of users and devices within Taplytics, we believe that when deleting a QA app or changing between build types you should first ensure that you are resetting the Taplytics appUser logout and then have all QA users log out of the app before making a change to the app on their device. The docs on how to reset the Taplytics appUser can be found here.

The real key is to reset the app user. If you are running tests in the future, we’re not trying to send to that app user because they deleted it and we think that app user is still affected.

How to Deal with Multiple User Accounts Per Device

This is more of a caution to be aware of.  In some QA environments, they’ll have multiple users per device (sharing QA devices between users). The best practice is having one user ID per QA device that is kept consistent.  In the real world, you're going to have 1 user device per person and not changing user accounts.

The other thing is that if you don’t do those things properly (resetting users, logging in, logging out), and you do multiple user accounts on the device, you literally have no way of keeping track what devices might be getting messages. We don’t know if that device is no longer active unless you tell us.