What are the best ways to QA JS experiments when using Fast Mode?

UPDATE: Fast Mode is deprecated. We highly advise switching to Synchronous script loading if you're still using Fast Mode. More info can be found here!

QA Considerations with Fast-Mode

Taplytics’ expectation for running experiments in Fast-Mode is that an experiment is started, maintained without editing until statistical significance has been reached, and a winning variation is set and/or the experiment is then archived. Any changes to distribution, variables, or more during the course of an experiment will negatively affect the data that is being tracked. As such, we do not suggest edits be made while an experiment is running. Instead, if an experiment needs to be modified for any reason, it should be recreated and started fresh to keep the data clean.

Without Fast-Mode, you have a number of different tools available to you for QA purposes. The first of which is our Variation Switcher which can be found in User Insights. At the top of a User Profile page, you’ll have an available list of Draft and Active experiments which you can force devices into or out of as desired. Once you have set up your desired configuration, all you have to do is refresh the browser and the config will be delivered. This tool relies on server delivery of experiments so it is not available with Fast-Mode.

The next QA tool is the Test Experiments start option. The documentation for this can be found here. This allows you to set the exact combination of experiments and variations in client-side code, allowing you complete control on which experiments to deliver. This tool relies on server delivery to support draft/paused experiments, so if you enable this start option, it will automatically disable Fast-Mode to deliver the desired experiment/variation combinations.

QA Best Practices

Given these considerations, I have two recommendations for you for QA’ing experiments, one if you want Fast-mode on during QA, the other if it can be turned off while testing.

1) QA’ing with Fast-Mode On

If Fast-Mode must be left on during QA, my main suggestion would be to leverage the redistribution of an experiment after Archiving or Pausing. I would suggest running all QA in the PreProd project before going live with a test and setting distribution of the test to 100% on the variation you want to QA. When you want to switch between variations, just pause the test, adjust the distribution to 100% of the next variation you would like to test, start the experiment, and then refresh your browser. On refresh, the client will be re-bucketed because of the experiment having been paused.

Once you have validated all of your variations, you can copy the code over to the production project, set up the final distribution you would like for the test, and run it with clean data.

2) QA’ing with Fast-Mode Off

If you do not need to leave Fast-Mode running while testing, I would suggest leveraging either the Test Experiments start option or the Variation Switcher in User Insights. With either of these methods, you can leave your experiments in draft while QA’ing, which is always ideal. This will allow you to QA tests in either a PreProd or Production environment, while keeping data clean and consistent.