Testing

Tests are an important part on preventing regressions and covering new feature functionality. There are multiple ways to run CIR tests.

Unit tests

The more aggresive (slower) one:

$ ninja check-all 

CIR specific test targets using ninja:

$ ninja check-clang-cir $ ninja check-clang-cir-codegen 

Using lit from build directory:

$ cd build-release $ ./bin/llvm-lit -a ../../clang/test/CIR 

LLVM Test Suite

We often evaluate and monitor ClangIR support against singlesource, multisource and spec benchmarks. See the benchmark results page for more information.

How to build and run

• Build a release version of ClangIR with assertions enabled (prevents some false positives).

• Cherry-pick this git commit for extra CMake flags that enable ClangIR.

• Create a build directory and build the tests with the release version of ClangIR:

  cd <path-to>/test-suite rm -rf ./build && mkdir build && cd build cmake --no-warn-unused-cli ../ \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER=<path-to-clangir-build>/bin/clang \ -DCMAKE_CXX_COMPILER=<path-to-clangir-build>/bin/clang++ \ -C=../cmake/caches/O3.cmake \ -DTEST_SUITE_SUBDIRS=SingleSource \ # MultiSource, External/SPEC/CINT2017rate, External/SPEC/CFP2017rate -DTEST_SUITE_CLANGIR_ENABLE=ON 

• For SPEC extra setup is also needed.

• Build tests (-k ensures it won’t stop if a test fails to compile):

  make -j -k 

• In the build directory, run the tests with:

  lit -v . 

Generating a comparison Table

• Patch the utils/compare.py tool with:
  002-generate-clangir-comparison-table.patch

• Build the binaries and collect results for 15 runs of both baseline and ClangIR tests:

   # Set TEST_SUITE_CLANGIR_ENABLE=OFF to get the baseline tests. for i in {1..15}; do lit -v -o "baseline_$i.json" .; done; for i in {1..15}; do lit -v -o "clangir_$i.json" .; done; 

• Create the comparison table using the patched utils/compare.py:

  utils/compare.py \ --lhs-name baseline --rhs-name clangir --minimal-names \ --merge-average --all -m compile_time -m exec_time \ --csv-output results.csv \ baseline_1.json ... baseline_15.json vs clangir_1.json ... clangir_15.json