Testing on Multiple Device Configurations

One of the key values of Waldo is that we support running your tests, across multiple device configurations, concurrently.

👍

Value of Multi-Concurrency Model

This ensures that you maintain total device coverage while also taking significantly less time to complete the necessary test runs.

For instance, you likely want to verify that your application behaves correctly on both the latest OS version and on the previous OS version.

To accomplish this, your application must declare one or more device configurations. A device configuration comprises three properties:

• OS version (for example, iOS 13.7)
• Device model (for example, iPhone 8)
• System language (for example, English)

📘

Declaring device configurations

Your application always declares at least one device configuration: it's default configuration. This configuration will be set by Waldo (to the device you request) during initial account setup.

Whenever you launch a recorder session, Waldo uses your application’s default configuration automatically. This is what creates your baseline recording. However, you can run against other configurations if you need or want.

Note: When you validate a test, that validation run will include all device configurations added to your account.

📘

When to run tests on certain configurations

You can select which device configurations will be used for a test (post-validation) by going to your Rules page in Waldo. For more information, click here.

Device configurations that are available to select must be configured for your account by Waldo. Reach out to us to add additional device configurations to your account.

Do I need to record a baseline for each device configuration?

Not necessarily. When you record your initial baseline on your default device, Waldo will attempt to play that same test on any subsequent device configuration you try to test. So long as the test is able to pass, the baseline can be reused across configurations.

❗️

When you may need to re-record...

Significant changes to the UI that are not optimized for older configurations may impact Waldo's ability to progress a test, prompting the need to re-record.

The image below shows a successfully translated baseline: while the baseline recording of the test (screen on the left) was captured on an iPhone 13, the test still replays and passes on an iPhone 6 (screen on the right). It was not necessary to record the baseline for an iPhone 6 separately in this instance.

📘

Multiple configurations + automated runs

If your application declares more than one device configuration, every automatically triggered run evaluates your tests against all of them concurrently.

What if there is a significant change between device configurations that impacts Waldo’s ability to progress a test?

In cases where there are nuances between device configurations (for example, if your test interacts with a system modal that displays differently between iOS 13.1 and iOS 6.2), you are able to update the flow for that specific device configuration on the “Runs” page of the Waldo dashboard.

📘

Re-recording the baseline for specific configurations

You can also elect to re-record a baseline for a specific device configuration that does not impact the baseline on other tests/device configurations. This can be done within the Step Modal Viewer within a test run result, pictured below.