Terminology

Application

An application (sometimes abbreviated to app) is the primary entity in Waldo. It corresponds to a mobile app on either iOS or Android. Typically, an app in Waldo is distributed on the App Store (iOS) or Google Play (Android), or is intended to be so distributed. (Enterprise distribution is not supported.) An application in Waldo is specific to a single OS platform. Therefore, a mobile app that supports both iOS and Android requires two distinct applications in Waldo.

Assertion

An assertion is a logical expression that captures an expectation about the state of an application screen. An assertion is what turns a flow into a test. Waldo supports several kinds of assertions:

  • A text assertion verifies that a specific element on the screen displays a specific text.
  • A UI assertion verifies that a specific section of the screen renders correctly.
  • A screen assertion verifies that the overall screen meets certain criteria.

During replay, assertions serve two important purposes:

  • They enforce that the application is behaving correctly and allow tests to have both functional and UI coverage of the app.
  • They can help determine when a screen has properly loaded, by determining what elements of the screen should be present.

Build

A build is a specific instance of an application created at a specific point in time. A build can be an “.app” bundle (iOS) or an APK (Android). Ideally, a build should be uniquely identified by version number (and build number on iOS), but Waldo does not require it.

Device Configuration

A device configuration is a static description that comprises three properties:

  • OS version (for example, “iOS 13.7”)
  • device model (for example, “iPhone 8”)
  • system language (for example, “English”)

Flow

A flow is a sequence of interactions (tap, swipe, etc.) that defines a user journey in an application. A flow can start either at application launch from a pristine state (fresh install) or from the last step of another flow. In the first case, the application is launched in a pristine state from a fresh install. In the second case, the other (preceding) flow is called a dependent flow.

For example, an application might have a sign up flow that includes the following interactions:

  1. Launch app.
  2. Tap the Sign Up button.
  3. Type “[email protected]” into the “email” field.
  4. Type “pass1234” into the “password” field.
  5. Tap the Register button.
  6. Display the home screen.

A flow is can be replayed against any device configuration.

Replay

A replay is the evaluation of a test in a particular context:

  • A specific device configuration (for example, “iPhone X on iOS 13.5 in French”).
  • A specific application build.
  • A specific point in time.

A replay always completes with a status:

  • success — All interactions in the flow replayed successfully and all assertions evaluated successfully.
  • assertions failed — All interactions in the flow replayed successfully, but one or more assertions failed on evaluation.
  • error — An interaction in the flow failed to replay.
  • crash — An interaction in the flow caused the app to crash.

Run

A run is a set of replays performed concurrently. Given a set of one or more tests and a set of one or more device configurations, each test is evaluated against each device configuration using a specific application build.

For instance, given a set of two tests (for example, “Sign Up” and “Log In”) and a set of three device configurations (for example, “iPhone 8 on iOS 14.2 in English”, “iPhone 8 on iOS 14.2 in French”, and “iPhone X on iOS 13.5 in English”) and a specific build of your application (for example, “4.1.6-3”), a run would perform six replays:

  1. “Sign Up” using iPhone 8 on iOS 14.2 in English
  2. “Log In” using iPhone 8 on iOS 14.2 in English
  3. “Sign Up” using iPhone 8 on iOS 14.2 in French
  4. “Log In” using iPhone 8 on iOS 14.2 in French
  5. “Sign Up” using iPhone X on iOS 13.5 in English
  6. “Log In” using iPhone X on iOS 13.5 in English

All six replays would use the same build of your application.

Waldo triggers a run automatically upon upload of a new application build. You can also trigger a run manually from the Waldo dashboard or from your CI with the Waldo API.

A run has a final status that is determined by combining the statuses of all its replays:

  • success — All replays succeeded.
  • error — One or more replays did not succeed.

Test

A test is a flow combined with a set of assertions that define the criteria for acceptance.

For example, you can require that the replay of your “Sign Up” flow be considered successful only if every interaction in the flow can be performed in order and on the third screen the Register button is disabled and on the fifth screen the text “Welcome ${email}” is displayed in the top bar.