Module runner

This module allows executing test cases. The test runner executes the test cases and sum-marizes the results. The test cases and the results are certain facts in the knowledge base. The test runner can be invoked via the predicate runner_batch/0. The coverage depends on how well the test cases are designed in respect of the probed execution paths.

The test cases are stored by as facts and rules in the following form:
:- multifile(test_ref/4).
:- discontiguous(test_ref/4).
% test_ref(Fun, Arity, Suite, Ref). :- multifile(test_case/4). :- discontiguous(test_case/4). % test_case(Fun, Arity, Suite, Number) :- Body.

Note: Fun/Arity is used to denote predicates, and Fun/-Arity-1 is used to denote evaluable functions. The test steps and the test validation points need to be implemented in the body of the predicate test_case/4. The body is assumed to terminate, the test runner doesn't impose some timeout currently. The body can be used to check a multitude of scenarios:

Examples:
% Check whether the goal succeeds:
    Goal
% Check whether the determinitic goal succeeds with result Value:
    Goal, Var==Value
% Check whether the non-determinitic goal first succeeds with result Value:
    Goal, !, Var==Value
% Check whether the goal fails:
    \+ Goal:
% Check whether the goal succeeds on redo:
    findall([],Goal,[_,_|_])
% Check whether the goal succeeds on redo with result Value:
    findall(Var,Goal,[_,Var|_]), Var==Value
% Check whether the goal fails on redo:
    findall([],Goal,[_])
% Check whether the goal throws the exception Value:
    catch(Goal,error(Var,_),true), Var==Value. 

The testing is not limited to the above example scenarios. A particular application domain might need additional test helper predicates to express the desired test steps and test validation points. Examples are the CLP(FD) test cases which use the predicate call_residue/2. The test results are ok and not-ok counts, a non-conclusive count is currently not provided.

The results are stored by the following facts:
:- public result_summary/1.
:- dynamic result_summary/1.
% result_summary(OkNok).

:- public result_suite/2.
:- dynamic result_suite/2.
% result_suite(Suite, OkNok).

:- public result_predicate/4.
:- dynamic result_predicate/4.
% result_predicate(Fun, Arity, Suite, OkNok).

:- public result/5.
:- dynamic result/5.
% result(Fun, Arity, Suite, Number, OkNok).

The results are accessible by the diagnose module or the report module from the Jekejeke Prolog development environment. Or the tester might code its own analysis based on these facts. In the future we might as well provide additional tools, such as a coverage analysis tool or similar.

The following runner predicates are provided:

runner_batch:
The predicate executes the test cases, collects and summarizes the results.

Comments