Module eqc

This module defines functions for writing and testing QuickCheck properties.

Copyright © Quviq AB, 2004-2022

Version: 2.02.0

Description

This module defines functions for writing and testing QuickCheck properties. Much of the interface is provided via macros (defined in eqc.hrl). These are documented below:

?FORALL(X,Gen,Prop)

Property that holds if Prop holds for all values X that can be generated by Gen. For example,
 prop_reverse() ->
   ?FORALL(Xs,list(int()),
      lists:reverse(lists:reverse(Xs)) == Xs).
 
Generators are defined using the module eqc_gen.

?IMPLIES(Pre,Prop)

Property that holds if Prop holds whenever the precondition Pre is true. The precondition must be a boolean, but Prop can be any QuickCheck property. An implication is tested by discarding test cases which do not satisfy the precondition. This can make testing slow, since many more test cases may need to be generated to find 100 which satisfy the precondition. In the worst case, QuickCheck may not be able to find enough test cases that do satisfy the precondition, in which case the number actually found is reported. Some preconditions may also skew the test data badly--for example, a precondition that a list is sorted skews the test data towards short lists, since random longer lists are extremely unlikely to be sorted just by chance. ?IMPLIES works well for preconditions which are true with a high probability, but if the precondition is unlikely to hold, then it is better to write a custom generator which generates test cases where the precondition is true.

?WHENFAIL(Action,Prop)

Property that is equivalent to Prop, but performs Action (for its side effects) when Prop fails. This can be used to print additional information when a test case fails.

Data Types

aggregated_data()

aggregated_data() = [{term(), integer()}]

Aggregated data, in principle a map from terms to number of occurences.

counterexample()

abstract datatype: counterexample()

A counter-example to a QuickCheck property, which can be obtained using counterexample/0 or counterexample/1, and used to repeat a test, or test a different property in the same case. Counterexamples are represented by the values bound by ?FORALL--for the counterexample to make sense independently, it's important that these were generated without side-effects.

eqc_info_map()

eqc_info_map() = #{result => boolean() | counterexample(), statistics => #{outcome => passed | failed | failed_as_expected | passed_unexpectedly | gaveup | bad_distribution, numtests => integer(), discards => integer()}, aggregated_data => [{atom(), aggregated_data()}], measurements => [{atom(), measure_map()}], user_info => eqc_user_info_map()}

eqc_user_info_map()

eqc_user_info_map() = #{}

Map containing user defined data. Data is stored using user_info/3, and the result is a map with all keys defined by the user.

I.e. user_info(key1, data1, ...) will result in a map #{ key1 => data1, ... }.

measure_map()

measure_map() = #{count => integer(), min => number(), max => number(), sum => number(), avg => number(), stddev => number()}

Map containing the returned data collected by measure/3.

print_method()

print_method() = print_method_fun() | {atom() | string(), none | string(), print_method_fun()}

Instructions on how to print statistics; either a pure print_method_fun (the old way), or a triple {Tag, Title, PrintFun}. Where Tag is used to tag the returned statistics see counterexample/2, Title is printed as a header for the statistics printed by PrintFun. Used by collect/3 and aggregate/3.

print_method_fun()

print_method_fun() = fun((aggregated_data()) -> false | any())

Printing statistics, the function is passed a list of samples and is expected to print statistical information about them. Print functions are used by collect/3 and aggregate/3.

property()

abstract datatype: property()

QuickCheck properties, which can either be boolean expressions, or constructed using the functions in this module. QuickCheck properties are tested using quickcheck/1.

Function Index

aggregate/2A property logically equivalent to Prop, but which collects a list of values in each test, and displays the distribution of these values once testing is complete.
aggregate/3Like aggregate/2, but allows the user to specify how the collected values should be printed.
check/1Equivalent to check(Prop, eqc:current_counterexample()).
check/2Tests the property in the case given.
check/3Tests the property in the case given.
check_distribution/4Check that at least a given fraction of test cases satisfy a condition.
classify/3Property which is logically equivalent to Prop, but also classifies test cases and displays the distribution of test case classes when testing is complete.
collect/2Equivalent to aggregate([S], Prop).
collect/3Equivalent to aggregate(PrintMethod, [S], Prop).
counterexample/0Returns the last counter-example found.
counterexample/1Equivalent to counterexample(P, []).
counterexample/2Tests the property in the same way as quickcheck/1, but if a test fails, then the failing test case is returned as a counterexample.
counterexamples/0Returns a list of the counterexamples found by the last call of eqc:module, paired with the name of the property that failed.
equals/2A property which holds if X and Y are equal...
fails/1A property which succeeds when its argument fails.
format/2Can be used in place of io:format/2 inside ?WHENFAIL to allow the output to be captured by on_output/2.
in_parallel/1A property which is tested in parallel.
in_sequence/1A property which is tested sequentially.
less_or_equal/2A property which holds if X =< Y, and displays their values when a test fails.
measure/3Collects the values of X while testing Prop, and if all tests pass, displays statistics such as the minimum, average, and maximum values, identified by the name Name.
module/1Tests all the properties exported from a module, given the module name.
numtests/2Property which is logically equivalent to Prop, but is tested N times rather than 100.
on_output/2Supplies an output function to be used instead of io:format when QuickCheck generates output.
on_test/2Attaches a function to a property which is called every time a test passes or fails.
only_top/1A printing method for discarding all but the top N collected values.
only_top/2A printing method for discarding all but the top N collected values.
quickcheck/1Tests the property in 100 random cases, printing a counter-example if one is found.
recheck/1Tests the property with the same random number seed as the last failing call of quickcheck/1.
recheck/2Same as recheck/1, but if {with_info, true} is passed as an option a richer structure is returned.
start/0Starts the QuickCheck server.
start/1Equivalent to start().
stop/0Stops the QuickCheck server.
user_info/3Sometimes it is useful to record data deep inside a property, thus we provide a basic key-value store.
version/0Returns the version number of this version of QuickCheck.
with_tag/1A printing method for collected data, which tags the collected data so that it is returned in the rich result structure from counterexample/2.
with_title/1A printing method for collected data, which displays a title before the percentages of each value in the data.
with_title/2Add a title to a print method.

Function Details

aggregate/2

aggregate(L::[term()], Prop::property()) -> property()

A property logically equivalent to Prop, but which collects a list of values in each test, and displays the distribution of these values once testing is complete. A typical use would be to aggregate the list of command names generated by eqc_statem:commands/1, in order to see how often each individual command appeared in generated tests:

aggregate(command_names(Cmds), ...) 

See also aggregate/3.

aggregate/3

aggregate(PrintMethod::fun(([{term(), int()}]) -> any()), L::[term()], Prop::property()) -> property()

Like aggregate/2, but allows the user to specify how the collected values should be printed. The PrintMethod parameter is called with a list of the collected data and the number of occurrences of each datum as an argument, and is expected to print some statistics. If the print method returns false the property fails. This is used in the implementation of check_distribution/4. A predefined printing methods is provided to add a title to the statistics:

aggregate(with_title(T),L,Prop).
This is useful when a property contains several calls to aggregate or collect.

check/1

check(Prop) -> any()

Equivalent to check(Prop, eqc:current_counterexample()).

check/2

check(Prop::property(), Values::counterexample()) -> bool()

Tests the property in the case given. Counterexamples are generated by testing a property using counterexample/1 or counterexample/0, and contain a list of the values bound by ?FORALL. A property tested by check should begin with the same sequence of ?FORALL s as the property from which the counterexample was generated, otherwise the results will be unpredictable. In particular, there is no check that the values in the counterexample could actually have been generated by the ?FORALL s in the property under test.

check/2 can be used without a QuickCheck licence, allowing anyone to run tests that a licenced user has generated.

check/3

check(Prop::property(), Values::counterexample(), Options::[term()]) -> bool() | eqc_info_map()

Tests the property in the case given. Counterexamples are generated by testing a property using counterexample/1 or counterexample/0, and contain a list of the values bound by ?FORALL. A property tested by check should begin with the same sequence of ?FORALL s as the property from which the counterexample was generated, otherwise the results will be unpredictable. In particular, there is no check that the values in the counterexample could actually have been generated by the ?FORALL s in the property under test. If {with_info, true} is passed as an Option, a richer structure is returned, see counterexample/2.

check/3 can be used without a QuickCheck licence, allowing anyone to run tests that a licenced user has generated.

check_distribution/4

check_distribution(Tag::term(), Fraction::float(), Ok::boolean() | [boolean()], Prop::property()) -> property()

Check that at least a given fraction of test cases satisfy a condition. Unless Ok is true in at least Fraction of test cases the property fails. For instance,

check_distribution(non_empty, 0.75, Xs /= [], Prop)
checks that in at least 75% of test cases the list Xs is non empty. If Ok is a list, the distribution of the aggregated boolean values is checked (see aggregate/2).

classify/3

classify(B::bool(), S::term(), Prop::property()) -> property()

Property which is logically equivalent to Prop, but also classifies test cases and displays the distribution of test case classes when testing is complete. If the boolean is true then the current test case is labelled with the term S, and, after testing is complete, QuickCheck prints out the percentage of test cases carrying each label. This can be used to check that the space of possible test cases has been covered reasonably well. For example, classifying test cases according to the length of a list enables one to see whether unreasonably many lists were short. Classifying test cases is a way to discover skewed distributions, such as can arise from using ?IMPLIES. It is good practice to check the distribution of test data using classify or collect/2, at least while properties are being developed.

Each test case can be labelled with any number of labels: QuickCheck then displays the percentage of each label in the generated test data.

Calls of classify or collect can be nested, in which case each call generates its own table of distributions.

collect/2

collect(S::term(), Prop::property()) -> property()

Equivalent to aggregate([S], Prop).

collect/3

collect(PrintMethod::fun(([{term(), int()}]) -> any()), S::term(), Prop::property()) -> property()

Equivalent to aggregate(PrintMethod, [S], Prop).

counterexample/0

counterexample() -> undefined | counterexample()

Returns the last counter-example found. See counterexample/1.

counterexample/1

counterexample(P::property()) -> true | counterexample()

Equivalent to counterexample(P, []).

counterexample/2

counterexample(P::property(), Option::[term()]) -> true | counterexample() | eqc_info_map()

Tests the property in the same way as quickcheck/1, but if a test fails, then the failing test case is returned as a counterexample.

If {with_info, true} is passed as an option, an even richer structure is returned, namely an eqc_info_map. The returned map contains the following fields:

counterexamples/0

counterexamples() -> [{atom(), counterexample()}]

Returns a list of the counterexamples found by the last call of eqc:module, paired with the name of the property that failed.

equals/2

equals(X::any(), Y::any()) -> property()

A property which holds if X and Y are equal... and displays their values when a test fails.

fails/1

fails(Prop::property()) -> property()

A property which succeeds when its argument fails. Sometimes it is useful to write down properties which do not hold (even though one might expect them to). This can help prevent misconceptions. fails(P) is tested in the same way as P, but fails only if P succeeds for every test. Thus fails(P) declares that QuickCheck should be able to find a counter-example to property P.

format/2

format(Fmt::string(), Args::[term()]) -> ok

Can be used in place of io:format/2 inside ?WHENFAIL to allow the output to be captured by on_output/2.

in_parallel/1

in_parallel(Prop::property()) -> property()

A property which is tested in parallel.

in_sequence/1

in_sequence(Prop::property()) -> property()

A property which is tested sequentially. Takes priority over in_parallel/1.

less_or_equal/2

less_or_equal(X::any(), Y::any()) -> property()

A property which holds if X =< Y, and displays their values when a test fails.

measure/3

measure(Name::atom() | string(), X::number() | [number()], Prop::property()) -> property()

Collects the values of X while testing Prop, and if all tests pass, displays statistics such as the minimum, average, and maximum values, identified by the name Name. X can also be a list of values, in which case all of them are included in the measurements.

module/1

module(Mod::atom()) -> [atom()]

Tests all the properties exported from a module, given the module name. Any function with arity zero whose name begins with "prop_" is treated as a property. The result is a list of the names of the properties that failed. See also module/2.

numtests/2

numtests(N::nat(), Prop::property()) -> property()

Property which is logically equivalent to Prop, but is tested N times rather than 100.

It is also possible to specify {min,N} or {max,N} as the number of tests, in which case we run at least N tests, or at most N tests, but otherwise according to the time limits or numtests/2 specified in the rest of the property.

on_output/2

on_output(Fun::fun((string(), [term()]) -> any()), Prop::property()) -> property()

Supplies an output function to be used instead of io:format when QuickCheck generates output. All output generated by QuickCheck is passed to Fun, in the form of a format string and a list of terms--the same arguments expected by io:format. By supplying a function which does nothing, QuickCheck can be run silently. By supplying a function which writes to a file, all QuickCheck output can be saved.

Note that output generated by user code is not passed to this output function. For example, calls to io:format in the property, or in the code under test, will generate output in the shell as usual. This applies even to calls inside a ?WHENFAIL, although this output can be redirected by replacing calls to io:format by calls to eqc:format/2. If you want to redirect output outside of ?WHENFAIL, then you need to modify your own code appropriately.

The reason that Fun is passed a format string and arguments, rather than an already formatted string, is to make it easier to extract information from the output without parsing it. However, there is no guarantee that different versions of QuickCheck will use the same format strings and term lists--you use this information at your own risk, in other words.

on_test/2

on_test(Fun::fun((counterexample(), bool()) -> any()), Prop::property()) -> property()

Attaches a function to a property which is called every time a test passes or fails. The arguments are the test case (a list of values), and a boolean indicating whether or not the test passed. Tests which are skipped (because of an ?IMPLIES(false,...)) are not included.

only_top/1

only_top(N::integer()) -> print_method()

A printing method for discarding all but the top N collected values. Discarded values are grouped together into an entry labelled '...'. The resulting print method is intended to be passed to collect/3 or aggregate/3.

only_top/2

only_top(N::integer(), F::print_method()) -> print_method()

A printing method for discarding all but the top N collected values. Discarded values are grouped together into an entry labelled '...'. The second argument is a print method for the pruned data, such as with_title/1. The resulting print method is intended to be passed to collect/3 or aggregate/3.

quickcheck/1

quickcheck(P::property()) -> bool()

Tests the property in 100 random cases, printing a counter-example if one is found. Initially small test cases are generated, then the size increases as testing progresses (see eqc_gen, ?SIZED, eqc_gen:resize/2 for the way size affects test data generation). The result is true if all tests succeeded (or if one failed, and failure was expected). On success, quickcheck analyses the distribution of test case labels. On failure, quickcheck tries to simplify the counter-example found as far as possible (see shrinking, described in eqc_gen).

recheck/1

recheck(Prop::property()) -> bool()

Tests the property with the same random number seed as the last failing call of quickcheck/1. If the property is the same as in that last call, then the same test case will be generated. Note that recheck repeats the test and its shrinking. This can be used to adjust the shrinking strategy in the property, then reshrink the same counterexample, perhaps to a better result. If you just want to repeat the shrunk test, then use

eqc:check(Prop,eqc:counterexample())
instead.

recheck/2

recheck(Prop::property(), Options::[term()]) -> boolean() | eqc_info_map()

Same as recheck/1, but if {with_info, true} is passed as an option a richer structure is returned. See counterexample/2 for a description of the return format.

start/0

start() -> pid()

Starts the QuickCheck server. If it is already running on this node, nothing is done.

Each user can run only one instance of the QuickCheck server at a time. If the server is already running on another Erlang node, it will be terminated automatically.

start/1

start(Dummy) -> any()

Equivalent to start().

stop/0

stop() -> any()

Stops the QuickCheck server. QuickCheck properties are tested in the QuickCheck server process, which is spawned automatically when quickcheck is first called. Usually there is no need to stop the QuickCheck server explicitly, but if a need does arise then this function can be used.

user_info/3

user_info(Key::atom(), FunOrData::fun((any()) -> any()), Prop::property()) -> property()

Sometimes it is useful to record data deep inside a property, thus we provide a basic key-value store. Either some static piece of data can be stored for Key, if DataOrFun is not a function. Otherwise the function (with arity 1) in DataOrFun is applied to whatever data is stored for Key, if no data is yet stored the function is applied to undefined.

   prop_ok() ->
     ?FORALL(X, nat(),
       user_info(sum, fun(undefined) -> X; (Sum) -> Sum + X end, ...)).
 
If this property is tested with counterexample/2 passing {with_info, true} as an option it will result in a result map where user_info is a map containing the key sum and the value will be the total sum of all tested values (including shrunk values if the property fails!). The rich result is further described for counterexample/2.

version/0

version() -> any()

Returns the version number of this version of QuickCheck.

with_tag/1

with_tag(Tag::atom() | string()) -> print_method()

A printing method for collected data, which tags the collected data so that it is returned in the rich result structure from counterexample/2. It is intended to be passed to collect/3 or aggregate/3.

with_title/1

with_title(Title::atom() | string()) -> print_method()

A printing method for collected data, which displays a title before the percentages of each value in the data. It is intended to be passed to collect/3 or aggregate/3.

The function also tags the collected data (with the title) so that it is returned in the rich result structure from counterexample/2.

with_title/2

with_title(Title::atom() | string(), Fun::print_method()) -> print_method()

Add a title to a print method. The title is printed before the print method is called.


Generated by EDoc