Queuing Tests
If you send Sauce Labs more tests at once than your concurrency allows (for example, you have a concurrency of 100 but send 200 tests at once), the "extra" tests will be queued and run as the first tests finish. Using the previous example, when the number of running tests drops to 99, 1 test from the queue will start running, and when the next test finishes, another test from the queue will start running. This continues until all the queued tests have been run. However, there are some caveats to this queuing process worth knowing.
- We have a limit to the number of jobs that can be queued up. The number of queued tests must be less then <your concurrency limit> + 150. For example, if your concurrency limit is 100, there can be no more than 250 tests (100 + 150) in the queue. Once this queue limit is reached, further session requests are rejected right away. Thus, if you send 351 tests, 100 will run right away, 250 will be queued, and 1 will be rejected immediately.
- A test is only allowed to remain in the queue for 10 minutes. If it has not been moved out of the queue to run before 10 minutes elapse, the test is removed from the queue and not run at all.
- If a test is queued and run later, there is a chance that, by the time the test is run, your test runner will have already timed-out the test. In this case, Sauce Labs will start a virtual machine to run the test and then wait for commands from the test runner. If the test runner has already timed-out, our VM will receive no commands. After 90 seconds, Sauce will end the session and shut down the VM. The error message of those tests will be "Test did not see a new command for 90 seconds. Timing out." and no commands will have been run (the Commands tab for your test will be empty). To accommodate this, you will need to adjust the timeout of your test runner to take into account the time that tests are queued before they are able to start.
CCY: Concurrency Limit. This is the number of concurrent test sessions you are able to run on Sauce Labs, defined by your Sauce Labs organization or team. See https://docs.saucelabs.com/basics/acct-team-mgmt/concurrency-limits/ for more information.
Client-side Timeouts
When tests go into the queue they will stay there for up to a max of 10 mins waiting to get a CCY slot. Let’s say it takes 5 mins for a test to get a slot to start to run. If your test runner times out after 4 mins, the test session will start after 5 minutes, receive no commands from the test runner and then will time out due to inactivity, resulting in a “Test did not see a new command for x seconds. Timing out.” error recorded on the test. Locally you’ll see some sort of timeout error in your test runner.
To avoid that, whilst still using the queuing mechanism, assess the average and maximum test durations within your build/suite of tests. Any timeouts in your test runner or test framework pertaining to obtaining a session need to be adjusted to be above the time that tests will sit in the queue to minimize queuing related timeout errors.
Also bear in mind that 10 minute queue limit - that’s probably the best value to use for such timeouts. The exact timeout settings to adjust depend on the language and framework you are using to execute your tests.
How many tests can be executed at once?
Let’s try an example. Assume a CCY of 100, and an average test duration of 3 minutes.
- Up to 250 tests can be queued (CCY plus 150), so a max of 350 tests can be launched at once. If more than 350 tests are launched at once, any tests beyond that limit of 350 will return the error “You've Exceeded Your Sauce Labs Concurrency Limit”
- 350 tests are launched.
- 100 tests will run immediately, taking ~3 minutes each.
- After ~3 minutes, the first 100 tests should complete and another 100 tests should be able to run, leaving 150 tests in the queue.
- After another ~3 minutes (~6 total), the second 100 tests should complete (200 total) and another 100 can run.
- After another ~3 minutes (~9 total), the third 100 tests should complete (300 total) and the last 50 can run.
In this example, all 350 tests are able to run. However, a batch will live in the queue and won’t start running for ~3 minutes, another batch for ~6 minutes, and another batch for ~9 minutes. Any timeout relating to session creation on your test runner/test framework would need to be set to over 9 minutes, otherwise they’ll get “Test did not see a new command for x seconds. Timing out.” errors.
Let’s look at a second example. Assume a CCY of 100, and an average test duration of 5 minutes.
- Up to 250 tests can be queued (CCY plus 150), so a max of 350 tests can be launched at once.
- 350 tests are launched.
- 100 tests will run immediately, taking ~5 minutes each.
- After ~5 minutes, the first 100 tests should complete and another 100 tests should be able to run, leaving 150 tests in the queue.
- After another ~5 minutes (~10 total), the second 100 tests should complete (200 total) but anything left in the queue at this point (150 tests) won’t start, as the queue time limit has been reached. Those will all error out with “You've exceeded your Sauce Labs concurrency limit. This test was throttled and ultimately timed out waiting for a free slot to run.”
Formula for “How many tests can I launch that will get to start before the queue time limit is reached, without errors?”
IF(((ROUNDDOWN(600/Avg Test Duration in seconds)*CCY)+CCY)>((CCY*2)+150),((CCY*2)+150),((ROUNDDOWN(600/Avg Test Duration in seconds)*CCY)+CCY))
Essentially this is:
- Take the queue time limit (600 seconds), divide by the average test duration. Round that down to a whole number, multiply by CCY and then add CCY. If this is lower than the queue limit (CCY * 2 +150), that’s your answer.
- If it’s greater than the queue limit (CCY * 2 +150), the answer is the queue limit (CCY * 2 +150) + CCY.
Of course, the above assumes all test durations are close to the average. Running a suite of tests with wildly varying test durations will skew this somewhat and could result in some tests being unable to run despite following the above formula.
In Summary
- Test duration: Shorter tests allow you to utilize the queuing mechanism to the fullest extent. Shorter test durations also make troubleshooting test failures much easier.
- Client-side timeouts: Ensure any session creation timeout your test runner/test framework has is set to at least 10 minutes to allow for the max time a test can be queued.
- Limit the number of tests launched: Use the above calculation as a way to determine how many tests to launch at once.