If you send Sauce Labs more tests at once than your concurrency allows (for example, you have a concurrency of 100 but send 200 tests at once), the "extra" tests will be queued and run as the first tests finish. Using the previous example, when the number of running tests drops to 99, 1 test from the queue will start running, and when the next test finishes, another test from the queue will start running. This continues until all the queued tests have been run. However, there are some caveats to this queuing process worth knowing.
- We have a limit to the number of jobs that can be queued up. The number of queued tests must be less then <your concurrency limit> + 150. For example, if your concurrency limit is 100, there can be no more than 250 tests (100 + 150) in the queue. Once this queue limit is reached, further session requests are rejected right away. Thus, if you send 351 tests, 100 will run right away, 250 will be queued, and 1 will be rejected immediately.
- A test is only allowed to remain in the queue for 10 minutes. If it has not been moved out of the queue to run before 10 minutes elapse, the test is removed from the queue and not run at all.
- If a test is queued and run later, there is a chance that, by the time the test is run, your test runner will have already timed-out the test. In this case, Sauce Labs will start a virtual machine to run the test and then wait for commands from the test runner. If the test runner has already timed-out, our VM will receive no commands. After 90 seconds, Sauce will end the session and shut down the VM. The error message of those tests will be "Test did not see a new command for 90 seconds. Timing out." and no commands will have been run (the Commands tab for your test will be empty). To accommodate this, you will need to adjust the timeout of your test runner to take into account the time that tests are queued before they are able to start.
