Test Results

In this section we will present the figures that were obtained from our test runs. Figures for various versions of the Jekejeke Prolog system will be given. We were running our test harness on the following machine:

Operating System: Windows 7 Professional
Processor: Intel Core i7-2620M @ 2.70 GHz
Memory: 4.00 GB
Energy Settings: HP Optimized

We used the following Jekejeke Prolog system version. The test harness was started from the runtime library via the Java Class Harness. We did not use the development environment to run the test harness:

Java Version: JDK 1.8.0 (64-bit)
VM Parameters: -mx512m -Duser.language=en
Library Version: Jekejeke Prolog Runtime Library 1.0.1
Concurrently Idle Applications: Word, Excel, Tomcat, SQL Server

We were then comparing the Jekejeke Prolog system under various optimization settings. In particular we were running our test harness for the following different settings, gradually increasing the number of optimizations:

Table 2: Compared Optimization Settings
Setting Choice
Point
Clause
Index-
ing
Body
Varia-
ble
Stack
Frame
Head
Varia-
ble
1) None off off off off off
2) Discussion Choice Point
on off off off off
3) Discussion Clause Indexing
on on off off off
4) Discussion Body Variable
on on on off off
5) Discussion Stack Frame
on on on on off
6) Discussion Head Variable
on on on on on

We were running the test suite two times in a row, and were only measuring the second run. The first run would be a cold start with incomplete caching of the predicate references. The second run is a warm start which we wanted to measure.

Measuring warm start means also that we will not see some costs incurred by class loading or by un-optimized Java code. The Java runtime had enough time to load classes and to apply just in time compilation and as well adaptive optimizations.

On the other hand the memory might already be fragmented. To avoid seeing a severe degradation because of constant garbage collection we use a memory allocation that is larger than the default of the Java virtual machine.

The absolute raw results measured in milliseconds are displayed in the table below:

Table 3: Absolute Detailed Strategies Results (ms)
Test 1) 2) 3) 4) 5) 6)
nrev 995
989
942
911
748
720
crypt 667
667
621
643
620
624
deriv 6'634
5'506
5'924
4'107
509
391
poly 662
671
594
556
554
502
qsort 1'092
1'095
10965
1'046
870
779
tictac 951
930
968
985
994
969
queens 823
812
806
796
784
739
query 2'276
2'339
1'039
1'001
968
985
mtak 1'006
1'012
1'035
836
815
676
perfect
1'271
1'159
1'205
1'238
740
527
calc 2'659
2'948
2'904
2'237
641
492
Total 19'036 18'128 17'103 14'356 8'243 7'404

The picture below shows the total results relative to their previous discussion:


Picture 2: Incremental Relative Strategies Results

We did not subtract the execution time for a dummy test from the results. Our measuring of a dummy test revealed that its execution time is in the range of 2 milliseconds. Therefore all our test results also measure the slight overhead of the test harness itself.

Kommentare