Usage

The usage on Waldo is calculated based on Interactions.

In essence, Waldo calculates how many user actions have been triggered on your behalf over a month.

The below describes how Waldo actually counts.

A test without any dependencies

When you replay a single test, it is very straighforward. The number of interactions used is simply the number of screens that you can see in your results page, minus 1 (there is no interaction happening on the last screen).

Under some circumstances, when Waldo suspects a test to be flaky, it might decide to retry it one or several times. This does not count against your interaction quota. If you launch a run for a single test, no matter how many times it is effectively retried, it counts as if it is replayed once.

In other words, Waldo never uses your interaction quota when it retries.

A test with dependencies

When a test has dependencies, Waldo has to replay those as well as the interactions from the test itself.
Imagine you have Sign Up -> Log Out. When Waldo replays Log Out, it must replay Sign Up first.

So the number of interactions used to replay Log Out is the number of interactions in Sign Up plus the number of interactions in Log Out.

A full run: the power of de-duplication

Now let's assume there are 4 tests:

Sign Up (12 interactions)
   > Log Out (4 interactions)
      > Log In (6 interactions)
   > Go to Settings (4 interactions)

When you launch a run on Waldo, with those 4 tests, you are essentially requesting Waldo to give results for each of those 4 tests.

• you want to run Sign Up
• you want to run Log Out, which requires to run Sign Up
• you want to run Log In, which requires to run Log Out and Sign Up
• you want to run Go to Settings, which requires to run Sign Up.

So in this specific case, Waldo is going to launch only Log In and Go to Settings. It knows that by launching only those 2 tests, it's actually going to get results for all 4 tests.
In theoretical terms, it only needs to launch each of the leaves in the dependency tree to explore the whole tree.

Therefore, the interaction count for such a run follows the same logic. The total interaction usage for the run is:

totalInteractions('Login') + totalInteractions('Go to Settings') 
  = 2 x interactions('Sign Up') 
    + interactions('Log Out') 
    + interactions('Login') 
    + interactions('Go to Settings')
  = 38

Note: If you launched a run with Login and Go to Settings only, you would get exactly the same results and exactly the same interaction usage, because you still need to replay Sign Up and Log Out to get there.

The importance of keeping dependencies short

There are many reasons why you want to keep dependencies short. Obviously, they are going to make your test take longer to run. Also, this is increasing the likelihood of something going wrong on the way to your test (the more tests in your dependency chain, and the more likely one is to fail).

And, quite logically, this is also going to use much more of your interaction quota.

Let's add another test to the previous suite, to test that you can reset your password. This new test only has 4 interactions.
To test this functionality, you need to be logged out of the app. So you could potentially chain that test after Log Out (at the end of that stage, you're indeed expected to be logged out).

Sign Up (12 interactions)
   > Log Out (4 interactions)
      > Log In (6 interactions)
      > Forgot Password (4 interactions)
   > Go to Settings (4 interactions)

But you could also simply test that functionality directly after an app open.

Sign Up (12 interactions)
   > Log Out (4 interactions)
      > Log In (6 interactions)
   > Go to Settings (4 interactions)
Forgot Password (4 interactions)

In both cases, the total usage is:

totalInteractions('Log In') 
  + totalInteractions('Go to Settings')
  + totalInteractions('Forgot Password')

However, in the first case,

totalInteractions('Forgot Password') 
  = interactions('Sign Up')
    + interactions('Log Out')
    + interactions('Forgot Password')
  = 20

whereas in the second case,

totalInteractions('Forgot Password') 
  = interactions('Forgot Password')
  = 4

So by adding the test Forgot Password chained after Log Out, you increase your interaction usage from 38 to 58 (more than 50% increase for just one test). However, if you simply add it after an app open, you just increase your interaction usage from 38 to 42.

The importance of short cuts

The previous section illustrates that the least amount of interactions you have to do in order to set up your test, the better it is: faster, more reliable, less costly.

A very good approach towards this goal is to use seeding for your app.