Lately I am writing a lot of tests for a new PHP framework: at the moment we passed 2800 tests, but the number will grow, since we're at about 50%. Tests were running fine, however I was wondering if there were any hidden issues due the large amount of mocked object.

The next thing to do was to run them in a random order: sadly the results were unexpected.

Why random order is so important?

Unit tests are The Ultimate Truth™: if the system under test does not respect the expectations, it means that there is a problem in your code, most likely the last changes you did introduced a bug.
Now, image if there is a false positive: the code is acting correctly, but there is a bug in your tests. You will spend countless hours chasing an inexistent bug until you give up or, even worse, change the code to satisfy the tests, introducing a real bug.

One of the hardest bug to detect are the so-called "entangled tests".

Let's say you have TEST A and TEST B (in a real world you will have TEST A - some hundred more tests - then TEST B).
The first one sets a persistent variable (maybe a session token or a configuration option) because you need it, then the second one (or more probably one of its parents) uses it automatically.
If you keep testing in the same order you will never notice it, because TEST A is setting some information that will be used by TEST B, "corrupting" it.

One solution would be to run tests using separate processes, however they are really slow and almost impossible to debug.
So the other solution is to run them in a random order: if TEST A injects some info required by TEST B, inverting their order would break the code, unveiling hidden issues.

OK, got it. Let's do random stuff tests

After a quick research, I found a very interesting library that does exactly what I needed: PHPUnit Randomizer. The usage is pretty straightforward, since it supports the same arguments of the main PHPUnit library:

vendor/bin/phpunit-randomizer -c ../phpunit.xml

 However the output is quite different:

............................................................. 2806 / 2839 ( 98%)
.................................
Time: 1.85 minutes, Memory: 71.00Mb

OK, but incomplete, skipped, or risky tests!             
Tests: 2839, Assertions: 5736, Incomplete: 2, Skipped: 3.

Randomized with seed 9511

As you can see there is a seed reference; given the same seed, the order would be the same. This is a crucial detail: if for any reason you found a failing test, you can reply the exact order over and over until you fix the issue.

vendor/bin/phpunit-randomizer --seed 9511 -c .phpunit.xml tests/

This seems cool, how can I run my tests in random order?

The usage is pretty straightforward: first of all you have to require the library inside your Composer file

"require-dev":
{
  "phpunit/phpunit": "4.2.*",
  "phpunit/dbunit": "1.3.*",
  "fiunchinho/phpunit-randomizer": "1.0.*"
}

Please note that PHPUnit-Randomizer requires you to load PHPUnit library using Composer (so no .phar packages are allowed), moreover at the moment it's tied to version 4.2.6.

Another limit is that the @depends annotation is not supported. Using that annotation, you can run a test before another one and pass its result to the second one, so that it can setup the object that will be used later. For example you can add some items to a collection, then pass that collection to another test that will remove them.
Personally I'm not a big fan of them, but sometimes they are useful and they could save you some time.

As you can easily image, this is exactly what we want to avoid: coupled tests that should run in a specific order. PHPUnit-Randomizer will skip them, so you'll have to refactor them and remove the dependencies.

Once you did that, you'll be ready to run your tests in a random order. Happy testing!