Writing Tests Premium/Enterprise

Blackfire gathers a lot of data about how your code behaves at runtime. Tests allow to write assertions on this data. You can write performance tests with Blackfire by writing assertions on time dimensions (wall-clock time, I/O time, and CPU time), but whenever possible, we recommend you to write assertions that do not depend on time. The main reason is that time is always a consequence, a symptom of a deeper issue. In addition, time is not stable from one profile to another, which makes your tests volatile. Instead, find what is slowing down your application and write assertions on it, i.e. the number of SQL queries, the number of times a function is called, or the memory usage.

Moreover, Blackfire tests are a great way to test legacy applications.

The .blackfire.yaml File

To get started, create a .blackfire.yaml file in the root directory of an application:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
tests:
    "Pages should be fast enough":
        path: "/.*" # run the assertions for all HTTP requests
        assertions:
            - "main.wall_time < 100ms" # wall clock time is less than 100ms

    "Commands should be fast enough":
        command: ".*" # run the assertions for all CLI commands
        assertions:
            - "main.wall_time < 2s" # wall clock time is less than 2s

Note

You can bootstrap a new .blackfire.yaml file with the following command: blackfire bootstrap-tests

.blackfire.yaml is a YAML file where tests, metrics, and scenarios can be defined.

Tests Basics

A test must located under the tests key and is composed of the following required items:

  • A name (e.g. “Pages should be fast enough”);
  • A context. It can be either an HTTP request or a CLI command.
    • An HTTP request is identified with a regular expression on its path, under the path key, where you may specify one path or a collection of paths. You may specify HTTP methods under the methods key. It is possible to exclude one or several paths by specifying them under the exclude key;
    • A CLI command is identified with a regular expression on the command itself, under the command key.
  • A set of assertions. In the example above, we want to ensure that the wall clock time, main.wall_time (the time it takes for your application to render the HTTP response) is less than 100 milliseconds, 100ms.

Here is another example with several assertions limited to some API calls of the application:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
tests:
    "Homepage should not hit the DB":
        path:
            - '^/api/article/\d+$'
            - '^/api/category/[^/]+/details$'
        exclude:
            - '^/api/category/all/details'
        methods: [GET, HEAD]
        assertions:
            - "metrics.sql.queries.count == 0"  # no SQL statements executed
            - "main.peak_memory < 10mb"         # memory does not exceed 10mb
            - "metrics.output.network_out < 100kb"  # the response size is less than 100kb

When a profile is made on a project that contains a .blackfire.yaml file, Blackfire automatically runs all tests matching the HTTP request path. The result of the tests is displayed as a green or red icon in the dashboard and the full report is available on the profile page. The same goes when profiling a CLI script via blackfire run.

Note

Note that assertions in the report contain the actual metric and variable values so that you know if you are close to the target or not (metrics.sql.queries.count 5 == 0; 0 is the target, 5 is the actual number of SQL statements executed).

Assertions support profile comparison as well to assert the performance evolution of your code:

1
2
3
4
5
6
tests:
    "Pages should not become slower":
        path: "/.*"
        assertions:
            - "percent(main.wall_time) < 10%"       # time does not increase by more than 10%
            - "diff(metrics.sql.queries.count) < 2" # less than 2 additional SQL statements

Note

Comparison assertions are only evaluated when running builds. Values are compared between the current build and a reference build. The reference can be either the last successful build, or a build that has been referenced in the webhook command with –external_parent_id.

Read the assertion reference guide to learn more about the Blackfire assertion syntax.

Conditions

A condition is an expression similar to an assertion. If the condition is fulfilled, the test is evaluated.

There are two kinds of conditions: when and unless. They can be used separately or combined for a more complex conditions.

when Expression

A when expression acts like an if condition. The corresponding test is evaluated if the expression returns true.

1
2
3
4
5
6
tests:
    'A database connection should be opened only when queries are made':
        path: '/.*'
        when: "metrics.sql.connections.count > 0"
        assertions:
            - 'metrics.sql.queries.count > 0'

The example above means “For any HTTP request, if a database connection is used, at least 1 SQL query must be run”.

unless Expression

An unless expression acts like an if not condition. The corresponding test is evaluated unless the expression returns true.

1
2
3
4
5
6
tests:
    'Twig template cache should be enabled in production':
        path: '/.*'
        unless: 'is_dev()'
        assertions:
            - 'metrics.twig.compile.count == 0'

The example above means “For any HTTP request, unless the profile is run in a development environment, Twig template path should be enabled”.

Full Tests DSL

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
tests:

    assertion_label:
        # A path or an array of paths, when in web context;
        path:
            # Regular expression the web path must match to trigger the assertions.
            - '^/api/article/\d+$'
            - '^/api/category/[^/]+/details$'

        # A path or an array of paths to exclude, when in web context;
        exclude:
            - '^/foo/bar'

        # A command, when in CLI context;
        # Regular expression the command must match to trigger the assertions.
        command: bin/console project:cron

        # A collection of HTTP Methods.
        # Use ANY for accepting anything (it's the default value).
        methods: # Example: [ GET, POST ]
            # Default:
            - ANY

        # A condition. If not valid, the test will not be used.
        when:                 ~ # Example: is_dev() == true

        # A condition. If valid, the test will not be used.
        unless:               ~ # Example: is_dev() == true

        # A collection of assertions
        assertions:
            -
                label:                ~ # Example: No more than 5 SQL query
                expression:           ~ # Required, Example: metrics.sql.queries.count < 5