Overview

Module eqc

• Description
• Data Types
• Function Index
• Function Details

Copyright © Quviq AB, 2004-2023

Version: 1.46.3

Description

?FORALL(X,Gen,Prop)

 prop_reverse() ->
   ?FORALL(Xs,list(int()),
      lists:reverse(lists:reverse(Xs)) == Xs).
 

?IMPLIES(Pre,Prop)

?WHENFAIL(Action,Prop)

?TRAPEXIT(Prop)

Note that actions in the ?WHENFAIL macro are not executed if they are inside a TRAPEXIT, thus one may have less printed information in the failing test case. When using eqc_component specifications there is an alternative to ?TRAPEXIT by using a worker process instead.

?TIMEOUT(Limit,Prop)

Example:

 prop_sleep() ->
   ?FORALL(N,choose(50,150),
     ?TIMEOUT(100,
       timer:sleep(N) == ok)).
 

 1> eqc:quickcheck(foo:prop_sleep()).
 ....Failed! Reason:
 {'EXIT',{timeout,100}}
 After 5 tests.
 118
 Shrinking....(4 times)
 Reason:
 {'EXIT',{timeout,100}}
 101
 false
 

?SETUP(Fun,Prop)

   prop_ok() ->
      ?SETUP(fun() -> setup_mocking(),
                  fun() -> teardown_mocking() end
             end,
         ?FORALL(Cmds, commands(), ...)).
 

?ALWAYS(N,Prop)

?FORALL(X,...,?ALWAYS(N,?FORALL(Y,...,...)))

• Generating X, or changing X from test to test, is expensive--for example, if X is a configuration which is expensive to install, so that we want to run many tests for each configuration.
• We are trying to shrink X, but Y is generated from X, and most choices of Y make the test pass. In this situation, then shrinking is likely to fail, because whenever we try to shrink X to a smaller value, then we are likely to choose a Y that happens to make the test pass. Trying many values of Y makes it much more likely that one of them will fail, and so shrinking will succeed.

?SOMETIMES(N,Prop)

?ONCEONLY(Prop)

?FORALL(N,nat(),?ONCEONLY(N*N >= 0))

?ONCEONLY saves each generated test, and skips any tests which are generated for a second time---it is equivalent to

?IMPLIES(...is a new test case...,Prop)

Note that duplicate test cases are generated as usual, but ?ONCEONLY prevents them from being run. This can save testing time if running tests is much more expensive than generating them. Of course, there is a space and time penalty for saving old tests and comparing new tests against them, so using ?ONCEONLY will not necessarily speed up testing.

Test cases are saved only during one run of quickcheck/1; this does not prevent repetition of tests in different runs of quickcheck/1.

Data Types

aggregated_data()

aggregated_data() = [{term(), integer()}]

Aggregated data, in principle a map from terms to number of occurences.

counterexample()

abstract datatype: counterexample()

A counter-example to a QuickCheck property, which can be obtained using counterexample/0 or counterexample/1, and used to repeat a test, or test a different property in the same case. Counterexamples are represented by the values bound by ?FORALL--for the counterexample to make sense independently, it's important that these were generated without side-effects.

eqc_info_map()

eqc_info_map() = #{result => boolean() | counterexample(), statistics => #{outcome => passed | failed | failed_as_expected | passed_unexpectedly | gaveup | bad_distribution, numtests => integer(), discards => integer()}, aggregated_data => [{atom(), aggregated_data()}], measurements => [{atom(), measure_map()}], user_info => eqc_user_info_map()}

eqc_user_info_map()

eqc_user_info_map() = #{}

Map containing user defined data. Data is stored using user_info/3, and the result is a map with all keys defined by the user.

measure_map()

measure_map() = #{count => integer(), min => number(), max => number(), sum => number(), avg => number(), stddev => number()}

Map containing the returned data collected by measure/3.

print_method()

print_method() = print_method_fun() | {atom() | string(), none | string(), print_method_fun()}

Instructions on how to print statistics; either a pure print_method_fun (the old way), or a triple {Tag, Title, PrintFun}. Where Tag is used to tag the returned statistics see counterexample/2, Title is printed as a header for the statistics printed by PrintFun. Used by collect/3 and aggregate/3.

print_method_fun()

print_method_fun() = fun((aggregated_data()) -> false | any())

Printing statistics, the function is passed a list of samples and is expected to print statistical information about them. Print functions are used by collect/3 and aggregate/3.

property()

abstract datatype: property()

QuickCheck properties, which can either be boolean expressions, or constructed using the functions in this module. QuickCheck properties are tested using quickcheck/1.

Function Index

Function Details

active_users/0

Fetch a list of the currently active users of your licence. The users are sorted by the expiry times of their current sessions; thus the user whose licence may first become free is first in the list. Of course, other users who continue running QuickCheck will automatically extend their sessions, so there is no guarantee that a licence will actually be released at the time indicated.

aggregate/2

A property logically equivalent to Prop, but which collects a list of values in each test, and displays the distribution of these values once testing is complete. A typical use would be to aggregate the list of command names generated by eqc_statem:commands/1, in order to see how often each individual command appeared in generated tests:

aggregate(command_names(Cmds), ...) 

See also aggregate/3.

aggregate/3

Like aggregate/2, but allows the user to specify how the collected values should be printed. The PrintMethod parameter is called with a list of the collected data and the number of occurrences of each datum as an argument, and is expected to print some statistics. If the print method returns false the property fails. This is used in the implementation of check_distribution/4. A predefined printing methods is provided to add a title to the statistics:

aggregate(with_title(T),L,Prop).

backtrace/0

Displays a stack backtrace from the last exception QuickCheck caught. Note that this is only possible if the exception is raised in the process in which the test case starts. If a test case fails because of an exception in another, linked, process, then no backtrace is available. Calls to functions in the implementation of QuickCheck itself are not included in the backtrace.

If you really need to see a backtrace from a linked process, then you can do so by catching the exception yourself in that process, using try-catch to obtain the backtrace, and printing it yourself.

check/1

Equivalent to check(Prop, eqc:current_counterexample()).

check/2

Tests the property in the case given. Counterexamples are generated by testing a property using counterexample/1 or counterexample/0, and contain a list of the values bound by ?FORALL. A property tested by check should begin with the same sequence of ?FORALL s as the property from which the counterexample was generated, otherwise the results will be unpredictable. In particular, there is no check that the values in the counterexample could actually have been generated by the ?FORALL s in the property under test.

check/2 can be used without a QuickCheck licence, allowing anyone to run tests that a licenced user has generated.

check/3

Tests the property in the case given. Counterexamples are generated by testing a property using counterexample/1 or counterexample/0, and contain a list of the values bound by ?FORALL. A property tested by check should begin with the same sequence of ?FORALL s as the property from which the counterexample was generated, otherwise the results will be unpredictable. In particular, there is no check that the values in the counterexample could actually have been generated by the ?FORALL s in the property under test. If {with_info, true} is passed as an Option, a richer structure is returned, see counterexample/2.

check/3 can be used without a QuickCheck licence, allowing anyone to run tests that a licenced user has generated.

check_distribution/4

Check that at least a given fraction of test cases satisfy a condition. Unless Ok is true in at least Fraction of test cases the property fails. For instance,

check_distribution(non_empty, 0.75, Xs /= [], Prop)

classify/3

Property which is logically equivalent to Prop, but also classifies test cases and displays the distribution of test case classes when testing is complete. If the boolean is true then the current test case is labelled with the term S, and, after testing is complete, QuickCheck prints out the percentage of test cases carrying each label. This can be used to check that the space of possible test cases has been covered reasonably well. For example, classifying test cases according to the length of a list enables one to see whether unreasonably many lists were short. Classifying test cases is a way to discover skewed distributions, such as can arise from using ?IMPLIES. It is good practice to check the distribution of test data using classify or collect/2, at least while properties are being developed.

Each test case can be labelled with any number of labels: QuickCheck then displays the percentage of each label in the generated test data.

Calls of classify or collect can be nested, in which case each call generates its own table of distributions.

collect/2

Equivalent to aggregate([S], Prop).

collect/3

Equivalent to aggregate(PrintMethod, [S], Prop).

conjunction/1

conjunction([{Tag1,Prop1},...,{TagN,PropN}]) is a property which is true if all of the properties Prop1...PropN are true. The tags are used when reporting counterexamples: the test case generated for each property is labelled with its tag in the output, together with an indication of whether that property passed or failed. Shrinking discards tags and properties which do not contribute to the test failure.

count_values/0

A print method that displays the number of occurrences for each value in the collected data. It is intended to be passed to collect/3 or aggregate/3.

counterexample/0

Returns the last counter-example found. See counterexample/1.

counterexample/1

Equivalent to counterexample(P, []).

counterexample/2

Tests the property in the same way as quickcheck/1, but if a test fails, then the failing test case is returned as a counterexample.

• result - what counterexample/1 would return
• statistics - A map containing, outcome of the test as well as the number of executed tests (numtests) and discarded tests (discards). If the test fails it also contains successful shrink steps (shrinksteps) and failed shrink steps (failed_shrinksteps).
• aggregated_data - all aggregated data (collect/3/aggregate/3/classify/3) that has a tag is put into this field. Aggregated data is tagged by using with_tag/1 or with_title as its print method.
• measurements - Data collected by measure/3, formatted as a measure_map.
• user_info - Data collected by ?USER_INFO in the property, format is a eqc_user_info_map.

counterexamples/0

Returns a list of the counterexamples found by the last call of eqc:module, paired with the name of the property that failed.

current_counterexample/0

Returns the most recent counterexample found by QuickCheck. This can be used while QuickCheck is shrinking a failed test case to follow progress, or if shrinking must be interrupted, to recover the last failed test case that QuickCheck had found. The counterexample is fetched from a file in the current directory.

disjunction/1

disjunction([Prop1,...,PropN]) is a property which is true if any of the properties Prop1...PropN is true. disjunction tests the properties from left-to-right, and stops as soon as one of them passes.

dont_print_counterexample/1

Equivalent to eqc_gen:with_parameter(print_counterexample, false, Prop).

Suppress the normal printing of counterexamples. Custom counterexample printing can be added using ?WHENFAIL.

equals/2

A property which holds if X and Y are equal... and displays their values when a test fails.

fails/1

A property which succeeds when its argument fails. Sometimes it is useful to write down properties which do not hold (even though one might expect them to). This can help prevent misconceptions. fails(P) is tested in the same way as P, but fails only if P succeeds for every test. Thus fails(P) declares that QuickCheck should be able to find a counter-example to property P.

features/2

Attach a list of features to a test case, with the interpretation "this test case tests these features". For example, the features might be lines of code covered by the test case, or argument values that were tested, or success/failure of individual functions. Features can be used to generate test suites that cover all of the features encountered. See eqc_suite:feature_based/1 for details.

force_registration/1

Register a QuickCheck licence without asking questions. This function is useful when scripting user registration, but dangerous since licences may only be registered once! Note that it takes a list of exactly once licence to enable use with erl -run force_registration LicenceId

format/2

Can be used in place of io:format/2 inside ?WHENFAIL to allow the output to be captured by on_output/2.

in_parallel/1

A property which is tested in parallel.

in_sequence/1

A property which is tested sequentially. Takes priority over in_parallel/1.

less_or_equal/2

A property which holds if X =< Y, and displays their values when a test fails.

licences_installed/0

Returns a list of the installed licences.

measure/3

Collects the values of X while testing Prop, and if all tests pass, displays statistics such as the minimum, average, and maximum values, identified by the name Name. X can also be a list of values, in which case all of them are included in the measurements.

module/1

Tests all the properties exported from a module, given the module name. Any function with arity zero whose name begins with "prop_" is treated as a property. The result is a list of the names of the properties that failed. See also module/2.

module/2

Tests all the properties exported from a module, given options and the module name. Any function with arity zero whose name begins with "prop_" is treated as a property. The result is a list of the names of the properties that failed.

The Options parameter can be a single option, or a list of options. Each option represents a function to be applied to each property before it is tested, and can either be a fun value, an atom (representing a function exported from module eqc), or a tuple of an atom and some arguments (representing a function exported by module eqc taking additional arguments before the property).

Example
To test every property in a module 1,000 times, use

 eqc:module({numtests,1000},ModuleName).
 

Testing budget

Using the testing_time option (thus calling eqc:testing_time/2) one can set a fixed testing time for each property. This is quite inflexible, however, and often you want different testing times for different properties and you don't want the total time to depend on the number of properties in the module. For this purpose there is a special option {testing_budget, Seconds} which controls the total time available for testing. By default the testing budget is distributed uniformly among the properties, but the distribution can be controlled by defining the property_weight/2 callback. This takes a testing profile and a property name as arguments and returns the relative weight for the property. The testing profile can be set with the {testing_profile, Profile} option and can be any Erlang term.

Example
Here is a simple example with two properties and two testing profiles.

 -module(example_eqc).
 -compile(export_all).

 prop_a() -> ...
 prop_b() -> ...

 property_weight(more_a, prop_a) -> 3;
 property_weight(more_b, prop_b) -> 3;
 property_weight(_, _) -> 1.
 

 1> eqc:module({testing_budget, 60}, example_eqc).
 prop_a: .....
 Time limit reached: 30.0 seconds.
 OK, passed 204 tests
 prop_b: .....
 Time limit reached: 30.0 seconds.
 OK, passed 213 tests
 []
 2> eqc:module([{testing_budget, 60}, {testing_profile, more_a}], example_eqc).
 prop_a: .....
 Time limit reached: 45.0 seconds.
 OK, passed 298 tests
 prop_b: .....
 Time limit reached: 15.0 seconds.
 OK, passed 102 tests
 []
 3> eqc:module([{testing_budget, 60}, {testing_profile, more_b}], example_eqc).
 prop_a: .....
 Time limit reached: 15.0 seconds.
 OK, passed 96 tests
 prop_b: .....
 Time limit reached: 45.0 seconds.
 OK, passed 311 tests
 []
 

numtests/2

Property which is logically equivalent to Prop, but is tested N times rather than 100.

It is also possible to specify {min,N} or {max,N} as the number of tests, in which case we run at least N tests, or at most N tests, but otherwise according to the time limits or numtests/2 specified in the rest of the property.

on_output/2

Supplies an output function to be used instead of io:format when QuickCheck generates output. All output generated by QuickCheck is passed to Fun, in the form of a format string and a list of terms--the same arguments expected by io:format. By supplying a function which does nothing, QuickCheck can be run silently. By supplying a function which writes to a file, all QuickCheck output can be saved.

Note that output generated by user code is not passed to this output function. For example, calls to io:format in the property, or in the code under test, will generate output in the shell as usual. This applies even to calls inside a ?WHENFAIL, although this output can be redirected by replacing calls to io:format by calls to eqc:format/2. If you want to redirect output outside of ?WHENFAIL, then you need to modify your own code appropriately.

The reason that Fun is passed a format string and arguments, rather than an already formatted string, is to make it easier to extract information from the output without parsing it. However, there is no guarantee that different versions of QuickCheck will use the same format strings and term lists--you use this information at your own risk, in other words.

on_test/2

Attaches a function to a property which is called every time a test passes or fails. The arguments are the test case (a list of values), and a boolean indicating whether or not the test passed. Tests which are skipped (because of an ?IMPLIES(false,...)) are not included.

only_top/1

A printing method for discarding all but the top N collected values. Discarded values are grouped together into an entry labelled '...'. The resulting print method is intended to be passed to collect/3 or aggregate/3.

only_top/2

A printing method for discarding all but the top N collected values. Discarded values are grouped together into an entry labelled '...'. The second argument is a print method for the pruned data, such as with_title/1. The resulting print method is intended to be passed to collect/3 or aggregate/3.

quickcheck/1

Tests the property in 100 random cases, printing a counter-example if one is found. Initially small test cases are generated, then the size increases as testing progresses (see eqc_gen, ?SIZED, eqc_gen:resize/2 for the way size affects test data generation). The result is true if all tests succeeded (or if one failed, and failure was expected). On success, quickcheck analyses the distribution of test case labels. On failure, quickcheck tries to simplify the counter-example found as far as possible (see shrinking, described in eqc_gen).

recheck/1

Tests the property with the same random number seed as the last failing call of quickcheck/1. If the property is the same as in that last call, then the same test case will be generated. Note that recheck repeats the test and its shrinking. This can be used to adjust the shrinking strategy in the property, then reshrink the same counterexample, perhaps to a better result. If you just want to repeat the shrunk test, then use

eqc:check(Prop,eqc:counterexample())

recheck/2

Same as recheck/1, but if {with_info, true} is passed as an option a richer structure is returned. See counterexample/2 for a description of the return format.

registration/1

Create a QuickCheck licence for the current user, using a one-time use registration identifier. Note that using the same registration identifier twice will invalidate it---if you do this, ask your licence administrator to reset your licence for you.

Each registration identifier is associated with a licence name, usually the name of the licence owner, such as "Quviq AB". It is possible to register several identifiers with different licence names, which makes it possible to use different licences on different occasions. If you are registered as a user of more than one licence, then you will need to create a preferences file called .eqc_licence to specify which licence you want to use. This file can be placed in your home directory, or locally in the directory in which you are running QuickCheck, or one of its parent directories. A local file takes precedence over the one in your home directory.

The .eqc_licence file can just contain the name of the licence you want to use, or the names of several licences, one per line. In the latter case, QuickCheck tries each licence in turn, using the first one that enables a successful start. A warning is generated for each licence name (mentioned in this file) for which no registration identifier is installed. The warning can be disabled by appending a '?' to the licence name.

reserve/1

Reserve a QuickCheck licence for this machine. Reservations can be made for up to seven days; once a licence is reserved, it cannot be used on another machine until the reservation has expired. Reserving a floating licence can be useful to ensure that another user does not begin using it at a critical time, or, since a reserved licence can be used without internet access, to prepare a laptop for a trip during which internet access will not be available.

Examples: eqc:reserve({3,days}), eqc:reserve({6,hours}), eqc:reserve(5). (If the unit, days or hours, is not specified, then it defaults to days).

reserved_until/0

The local time that the currently active QuickCheck licence is reserved until.

start/0

Starts the QuickCheck server. If it is already running on this node, nothing is done.

start/1

Equivalent to start().

stop/0

Stops the QuickCheck server. QuickCheck properties are tested in the QuickCheck server process, which is spawned automatically when quickcheck is first called. Usually there is no need to stop the QuickCheck server explicitly, but if a need does arise then this function can be used.

testing_time/2

Property which is logically equivalent to Prop, but is tested continuously for T time units (by default, seconds), rather than a fixed number of times. This can be used to apportion available testing time among a set of properties. When the time limit is reached, then no more tests are started, but any currently running test is completed.

The time unit can be specified using with_testing_time_unit/2.

It is also possible to specify {min,Time} or {max,Time} as the time limit, in which case tests are run for at least Time time units, or at most Time units, but otherwise according to the time limits or numtests/2 specified in the rest of the property.

user_info/3

Sometimes it is useful to record data deep inside a property, thus we provide a basic key-value store. Either some static piece of data can be stored for Key, if DataOrFun is not a function. Otherwise the function (with arity 1) in DataOrFun is applied to whatever data is stored for Key, if no data is yet stored the function is applied to undefined.

   prop_ok() ->
     ?FORALL(X, nat(),
       user_info(sum, fun(undefined) -> X; (Sum) -> Sum + X end, ...)).
 

version/0

Returns the version number of this version of QuickCheck.

watch_shrinking/1

Repeats the most recent failed test, but displays all the test cases explored during shrinking. The test cases are reported in the order they are tried, so each Failed test is followed by attempts to shrink it, while each OK test (or test skipped because a precondition failed) is followed by an alternative way to shrink the last Failed test. The last Failed test displayed is the final result of shrinking.

with_tag/1

A printing method for collected data, which tags the collected data so that it is returned in the rich result structure from counterexample/2. It is intended to be passed to collect/3 or aggregate/3.

with_testing_time_unit/2

Property which is logically equivalent to Prop, but in which calls of testing_time/2 measure testing time in units of Unit milliseconds--by default 1000. If testing_time/2 is used to apportion available testing time between properties, then with_testing_time_unit/2 can be used as a property wrapper to increase or decrease the total time spent testing.

with_title/1

A printing method for collected data, which displays a title before the percentages of each value in the data. It is intended to be passed to collect/3 or aggregate/3.

with_title/2

Add a title to a print method. The title is printed before the print method is called.

Overview

Generated by EDoc