QA Considerations with Fast-Mode
Taplytics’ expectation for running experiments in Fast-Mode is that an experiment is started, maintained without editing until statistical significance has been reached and then a winning variation will be set and/or the experiment will be archived. Any changes to distribution, variables, or more during the course of an experiment will negatively affect the data that is being tracked, as such we do not suggest edits be made while an experiment is running. Instead, if an experiment needs to be modified for any reason it should be recreated and started fresh to keep the data clean.
Without Fast-Mode you have a number of different tools available to you for QA purposes. The first of which is our Variation Switcher that can be found in User Insights. At the top of a User Profile page you’ll have an available list of Draft and Active experiments which you can force devices into or out of as desired. Once you have set up your desired configuration all you have to do is refresh the browser and the config will be delivered. This tool relies on server delivery of experiments so it is not available with Fast-Mode.
The next QA tool is the “Test Experiments” start option. The documentation for this can be found here (https://github.com/taplytics/Taplytics-js/blob/master/EXPERIMENTS.md#testing-experiments). This allows you to set the exact combination of experiments and variations in client-side code, allowing you complete control on which experiments to deliver. This tool relies on server delivery to support draft/paused experiments, so if you enable this start option it will automatically disable Fast-Mode to deliver the desired experiment/variation combinations.
QA Best Practices
Given these considerations I have two recommendations for you for QA’ing experiments, one if you want Fast-mode on during QA, the other if it can be turned off while testing.
1) QA’ing with Fast-Mode On
If Fast-Mode must be left on during QA my main suggestion would be to leverage the redistribution of an experiment after Archiving or Pausing. I would suggest running all QA in the PreProd project before going live with a test and setting distribution of the test to 100% on the variation you want to QA. When you want to switch between variations, just pause the test, adjust the distribution to 100% of the next variation you would like to test, start the experiment and then refresh your browser. On refresh the client will be re-bucketed because of the experiment having been paused
Once you have validated all of your variations you can copy the code over to the production project, set up the final distribution you would like for the test and run it with clean data.
2) QA’ing with Fast-Mode Off
If you do not need to leave Fast-Mode running while testing, I would suggest leveraging either the “Test Experiments” start option or the Variation Switcher in user insights. With either of these methods you can leave your experiments in draft while QA’ing, which is always ideal. This will allow you to QA tests in either a PreProd or Production environment while keeping data clean and consistent.