Day 15 - Integrating with Unit Tests

Unit tests rarely come to mind when talking about performance management. This is for a good reason: tests depending on time lead to transient failures. Blackfire lets developers write tests on stable metrics that are related to the root causes of performance issues instead of time. Using such metrics in a unit test suite is a powerful option.

To make things more concrete, let's write some unit tests for the Gitter Gitter\Client::getVersion() method we optimized previously. The test suite already has a test covering the output of the method:

1
2
3
4
5
public function testIsParsingGitVersion()
{
    $version = $this->client->getVersion();
    $this->assertNotEmpty($version);
}

But how can we test that git --version only runs the very first time getVersion() is called? Have a look at the getVersion() method implementation and note the usage of the Process class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
public function getVersion()
{
    static $version;

    if (null !== $version) {
        return $version;
    }

    $process = new Process($this->getPath() . ' --version');
    $process->run();

    if (!$process->isSuccessful()) {
        throw new \RuntimeException($process->getErrorOutput());
    }

    $version = trim(substr($process->getOutput(), 12));
    return $version;
}

As PHPUnit cannot easily mock the Process instance, we need to find another way:

  • We can re-define the Process class (extending the real one and wrapping the start() method) before running any tests and register it before Composer's autoloader kicks in. That's rather ugly, fragile, and confusing to do it by hand but some mocking libraries can help you achieve that.
  • Use Blackfire!

Using Blackfire in PHPUnit

Let's see how Blackfire integrates with PHPUnit. The easiest way is to include the Blackfire\Bridge\Phpunit\TestCaseTrait trait.

1
2
3
4
5
6
use Blackfire\Bridge\Phpunit\TestCaseTrait as BlackfireTrait;

class ClientTest extends \PHPUnit_Framework_TestCase
{
    use BlackfireTrait;
}

Note

Trait support was added in PHP 5.4. For older PHP versions, read the PHPUnit integration documentation for a workaround.

Then, create a PHPUnit test like this one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
use Blackfire\Profile\Configuration;

public function testIsGitVersionCached()
{
    $config = new ProfileConfiguration();
    $config->assert('metrics.symfony.processes.count == 1', 'One Process call only');

    $this->assertBlackfire($config, function () {
        $this->client->getVersion();
        $this->client->getVersion();
        $this->client->getVersion();
    });
}

The assertBlackfire() call is where the magic happens:

  • It takes a configuration where some Blackfire assertions are defined (metrics.symfony.processes.count == 1);
  • It instruments the code in the anonymous function before running it;
  • It converts the Blackfire assertions to PHPUnit ones so that errors and failures are injected into PHPUnit's report, as any other assertion.

Caution

Be sure to add this test before any other tests; if not, the cache could be already warmed and the test would fail. To make the test more robust, move the $version variable as a static class variable and use reflection to reset its state at the beginning of the test:

1
2
3
4
$p = new \ReflectionProperty($this->client, 'version');
$p->setAccessible(true);

$p->setValue($this->client, null);

Debugging Failures and Errors

If the test fails, PHPUnit displays a nice error message with all the information you need to debug it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
F

Time: 3.03 seconds, Memory: 15.00Mb

There was 1 failure:

1) Gitter\Tests\ClientTest::testIsGitVersionCached
Failed asserting that Blackfire tests pass.
1 tests failures out of 1.

    failed: One Process only
      - metrics.process.count 3 == 1

More information at https://blackfire.io/profiles/4427f830-62fe-469f-b6aa-8ad8f6dcdff7/graph.

/.../gitter/vendor/blackfire/php-sdk/src/Blackfire/Bridge/PhpUnit/TestConstraint.php:60
/.../gitter/vendor/blackfire/php-sdk/src/Blackfire/Bridge/Phpunit/TestCaseTrait.php:48
/.../gitter/tests/Gitter/Tests/ClientTest.php:60

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

If cache is commented out, the assertion fails as 3 processes are run. The call graph URL points to the profile where you can debug problems more easily.

If the assertion has a syntax error, the profile "Assertions" tab helps you understand the problem:

/docs/phpunit-assertion-error.png

Defining your own Metrics

This test was easy to write because we were able to use a built-in metric, but this is not always the case. Luckily, it is possible to create custom metrics with the PHP SDK:

1
2
3
4
5
6
7
use Blackfire\Profile\Configuration as ProfileConfiguration;
use Blackfire\Profile\Metric;

$config = new ProfileConfiguration();

$config->defineMetric(new Metric('process', '=Symfony\Component\Process\Process::start'));
$config->assert('metrics.process.count == 1', 'One Process only');

Conclusion

Read our PHPUnit integration documentation for more tips on how to leverage Blackfire in PHPUnit test suites.

Blackfire lets you test your code's behavior without the need for mocks. This is a very useful technique when you need to test the behavior of a third-party library you cannot easily modify. With Blackfire there is no need to change third-party code or write an ugly hack. Simple define custom metrics and write assertions against them.

Using Blackfire in a PHPUnit test has one great side-effect: your performance tests automatically benefit from your continuous integration setup (if you have one of course). Automation is key for continuously managing performance, and this is the topic for the next few chapters.