🛤️ Handling A/B testing in Waldo

Does Waldo support A/B Testing?


A/B Testing is a tried and true tenet of end user experience testing. It allows you to test how simple UI changes can help achieve desired user behavior faster.

The way Waldo is built (and the way good automation works) is that the “thing” you are testing must be repeatable. A/B test makes behavior not consistently repeatable by presenting an element of randomization into your testing flow.

However, there are several ways to accommodate your A/B testing needs in Waldo.

A/B Testing in Waldo

When selecting between the A or B variation of your application, the decision is made two ways:

  1. By the app itself (rare)
  2. By the determined tie to a user, via the backend (much more common)

Depending on what you want to test and how you have the A/B variation selection built in your application, we offer a few options:

If it is fine to test either variation A or variation B…you have no preference:

All you need to do is make sure that the app always choses a single version of the build (A or B) consistently.

  • If the selection of the variation is made by the app:

  • If the selection is determined by a user state:

    • Simple Solution: whitelist Waldo’s IP address to always get A or B ⇒ https://docs.waldo.com/docs/connecting-to-your-firewall
    • Good Solution: log in with a specific account (for instance featureA{pattern}@testing.waldo.io) and get the A or B version
    • Best Solution: implement a deep-link route to specify which group you want to be in.

If you want to test both variations in Waldo:

This can be solved in Waldo by maintaining separate tests: creating an explicit test for each variation. For example: “Signup – Variation A” and “Signup – Variation B”.

That way, you can run your required tests on each variation without introducing an element of randomness: allowing each suite to be repeatable and deterministic.


Before building out too many tests...

A/B Testing is generally a short lived exercise. Before building out a long term solution in Waldo (like multiple test suites), consider what you are testing for.