Skip to content

tests

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

tests

README.md

Testing APE

Author: Kaarel Kaljurand

Version: 2016-04-02

Introduction

This document explains how to test APE including the various DRS translators that APE contains.

APE Regression test

To regression test APE, just run:

bash rtest.bash

or if you want to download the latest regression testset first then run:

bash rtest.bash d

Note that for the above command to work you need an internet connection, and curl installed. (If you edit the script rtest.bash, then you can replace the need for curl with the need for wget.)

A file with a timestamp containing the regression test results along with APE's output is stored into the testruns-directory.

In order to get a digest of the testrun, grep the file for regression tester messages, e.g.:

cat testruns/rtest_050601-1310.txt | grep "^0"

or cat an already filtered file:

cat tmp/now.txt

In order to explore the erronous DRSes, open the file and search for the strings

  • 0000 (Correctly generated an empty DRS)
  • ---- (Correctly generated a DRS which is not empty)
  • 0-># (Failed to generate an empty DRS)
  • #->0 (Incorrectly generated an empty DRS)
  • #### (DRSs which are not empty do not match.)

To get a listing of all the regressions:

cat tmp/now.txt | grep "[.#.]"

Note: the regression testing is only tested with SWI-Prolog. It is assumed that SWI-Prolog is called swipl, if it is not the case then modify rtest.bash accordingly or set a symbolic link, e.g.:

ln -s which pl swipl

Runtime of the complete test run on different machines:

  • Mac OS X G4: ~25 seconds (~3000 testcases)
  • Intel i3, 2011-06-20: ~11 seconds (3613 test cases, excluding bug reports)
  • Intel i7, 2016-04-02: ~9 seconds (3733 test cases, excluding bug reports, including loading the large lexicon from a flat file)

Testing Drace Core and Drace NP

Running the test:

echo "[test_drace]. test_drace(core)." | swipl > testruns/drace_test_results.txt

echo "[test_drace]. test_drace(np)." | swipl > testruns/dracenp_test_results.txt

Getting an overview of the problems by filtering out certain testcases that point to pseudo-problems.

cat testruns/drace_test_results.txt | grep "FAIL" | grep -v ":" | grep -v "ach of" | wc

Running all the tests

time bash test_everything.sh > out.txt 2> err.txt

This takes about

  • 55 seconds on an i3 Linux laptop with SWI-Prolog v6.
  • 45 seconds on an i7 Linux laptop with SWI-Prolog v7.