Assertions Premium/Enterprise

An assertion is an expression that must return a Boolean (true if the assertion succeeds, false otherwise). It always relies on metric values, operators, units, functions, and variables and must be written in the tests section of the .blackfire.yaml file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
tests:
    'All pages are fast':
        path: '/.*'
        assertions:
            - 'main.wall_time < 50ms'
            - 'main.peak_memory < 10mb * var(memory_coeff)'
            - 'main.network_out < 10kb'

    'Not too many SQL queries on the homepage':
        path: '/(en|fr)/blog/'
        assertions:
            - 'metrics.sql.queries.count < 5'

In the above example, main.network_out, metrics.sql.queries.count, and main.peak_memory are metric values. 10mb uses a unit to make the expression more readable, and var(memory_coeff) refers to a variable for which the value has been defined in the environment.

Metric Values

Blackfire exposes metrics that are associated with the current profile. The value of one dimension of a metric can be used in assertions; for instance, the count value of the sql.queries metric is stored in metrics.sql.queries.count.

Learn more about all built-in metrics exposed in assertions by default and how to create your own.

The available dimensions for metrics are the following ones:

  • count
  • wall_time
  • cpu_time
  • memory
  • peak_memory
  • network_in
  • network_out
  • io

Units

To make assertions more readable, you can add a unit to numbers. A unit is a multiplier applied to the raw number associated with it.

The default time unit is the millisecond; when using 10 in an assertion for a time value, that evaluates to 10 milliseconds. But using a unit (like ms or s) makes assertions more explicit; 10s is interpreted as 10 seconds.

The default memory unit is the byte. The following units are also available: kb, kib, mb, mib, gb, gib.

Blackfire also supports the following generic units: k, ki, m, mi, g, gi.

Operators

The following operators are supported in assertions:

Comparison

  • == equals;
  • != not equals;
  • < less than;
  • > greater than;
  • <= less than or equal to;
  • >= greater than or equal to.

Logic

  • not or !;
  • and or &&;
  • or or ||.

Math

  • + addition;
  • - subtraction;
  • * multiplication;
  • / division;
  • % modulus;
  • ** pow.

Variables

Variables are useful when you configure several Blackfire environments to run performance tests on various machines hosting the same application. Variables make it possible to use the same .blackfire.yaml configuration on several servers that have different purposes (e.g. development, staging, production).

Variables can be defined in the environment configuration and used in an expression by passing the metric key to the var() function. Variable values may also use units.

Example: You have configured two different environments in Blackfire:

  • Integration hits your integration server, where your application runs with debug mode on;
  • Production hits your production servers, where your application runs with debug mode off.

The same application runs on all servers, but the debug mode increases memory usage a lot. As such, you cannot set the same maximum value for your assertions on memory.

Your .blackfire.yaml would look like this when using variables:

1
2
3
4
5
tests:
    "Pages shouldn't use too much memory":
        path: "/.*"
        assertions:
            - "main.peak_memory < 10mb * var('memory_coeff')"

Now in both your Integration and Production Blackfire environments, you have to create the memory_coeff variable:

  • For Integration, memory_coeff == 2
  • For Production, memory_coeff == 1

The var() function may also receive a default value as a second argument. This default value is used whenever the variable is not defined in one of your environments.

1
2
3
4
5
tests:
    "Pages shouldn't use too much memory":
        path: "/.*"
        assertions:
            - "main.peak_memory < 10mb * var('memory_coeff', 1)"

Caution

The vars.xxx notation, supported in older versions of Blackfire, is now deprecated. We strongly advise you to migrate your expression to use the var() function instead.

Development vs Production

When an assertion is run from an environment, the is_dev() function returns false when the environment is configured for production usage.

Builds Comparison Premium/Enterprise

When using builds, it is possible to compare one build to another. This is useful when you want to validate a code merge (e.g. a pull-request) by triggering a build webhook or a Blackfire Player collection of scenarios.

The comparison is made between the current build and a reference build. The reference build is determined with the following rules:

  • If the build is periodic, the reference build is the last successful build in the same environment.

  • If the build is triggered by a webhook with Blackfire CLI, the reference build is identified with the value passed to the --external-parent-id option, in the blackfire build-trigger command. Note that --external-id option must be set.

    1
    2
    3
    4
    5
    6
    blackfire build-trigger \
        http://example.com/ \
        --env=<ENV-UUID> \
        --title="Build PR 1234" \
        --external-id=<some_unique_id_for_the_build> \
        --external-parent-id=<some_unique_id_for_the_reference_build>
    
  • If the build is triggered by a webhook with cURL, the reference build is identified with the value passed to the external_parent_id POST parameter. Note that external_id parameter must be set.

    1
    2
    3
    4
    5
    6
    curl -X POST https://blackfire.io/api/v2/builds/env/<ENV-UUID>/webhook \
        --user "CLIENT-ID:CLIENT-TOKEN" \
        -d "endpoint=http://example.com/" \
        -d "title=Build PR 1234" \
        -d "external_id=<some_unique_id_for_the_build>"
        -d "external_parent_id=<some_unique_id_for_the_reference_build>"
    
  • If the build is triggered by Blackfire Player, the reference build is identified with the value passed to the BLACKFIRE_EXTERNAL_PARENT_ID environment variable. Note that BLACKFIRE_EXTERNAL_ID environment variable must be set.

    1
    2
    3
    4
    5
    6
    BLACKFIRE_EXTERNAL_ID="<some_unique_id_for_the_build>" \
    BLACKFIRE_EXTERNAL_PARENT_ID="<some_unique_id_for_the_reference_build>" \
    blackfire-player run \
        --endpoint=http://example.com \
        --blackfire-env=<ENV-UUID> \
        my_scenario.bkf
    

Comparison Functions

Comparison can be made within assertion expressions thanks to the percent and diff functions:

If you want to assert that a metric value does not increase by more than a given percentage between two builds, use the percent function:

1
"percent(main.wall_time) < 10%"

You can also test the evolution of a metric value in absolute terms with the ``diff`` function:

1
"diff(metrics.sql.queries.count) < 2"

In this example, the assertion checks that the profile from the newer build has less than 2 additional SQL statements compared to the previous one.