This repository contains tests for benchmarking ROP compilers. ROP Benchmark is intended to compare ROP compilers. ROP Benchmark was used to evaluate existing open source tools in "Survey of Methods for Automated Code-Reuse Exploit Generation" paper [english] [russian].

binutils gcc gcc-multilib nasm make docker 

$ sudo apt install build-essential nasm gcc-multilib $ sudo snap install docker 

ROP Benchmark is supposed to run in docker container. It provides configured environment with all tools installed and /bin/sh replaced by script reporting success status of ROP chain execution.

Dockerfile is placed inside docker folder. To build docker image:

$ cd docker $ sudo docker build -t rop-benchmark . 

Entry point to run benchmark is run.sh script.

$ ./run.sh --help	usage: run.py [-h] [-s] [-t TOOL] [-r REAL_LIFE] [-n CORES] [-a ARCH] [-c] [-g] [-b BINARY] [--timeout TIMEOUT] [--clean]	Rop-benchmark entry point. By default it runs all tests.	optional arguments: -h, --help show this help message and exit -s, --synthetic Run only synthetic test-suite -t TOOL, --tool TOOL Run only tool -r REAL_LIFE, --real-life REAL_LIFE Run only specified real life binary test-suite. -n CORES, --cores CORES The number of parallel instances to run. -a ARCH, --arch ARCH The target architecture of framework. -c, --check-only Only check chains generated previously -g, --generate-only Only generate chains. Do not try to run and check them. -b BINARY, --binary BINARY Run particular binary e.g. openbsd-62/ac.bin -d BADCHARS, --badchars BADCHARS	Bytes banned for use as part of chain --timeout TIMEOUT The timeout in seconds for each binary --clean Clean rop-benchmark working tree from any artifacts of previous runs 

$ sudo ./bash.sh 

There are many different types of ROP chain payloads. At the moment we test only one type of payload - system call of execve to "/bin/sh". It is most general type of payload supported by all tools.

The benchmark provides target binaries for ROP compilers. These binaries supply two things: exploitable vulnerability and a set of ROP gadgets. A set of ROP gadgets can be of two different types: synthetically created or just taken from real life binaries. To use both of them we created the simple vulnerable program vul.c which reads an input file into the buffer placed on the stack (without boundary checking, of course). Then we compiled this program and inserted each target binary as code section inside a separate ELF file. So we get exploitable vulnerability and target binary code all together in one address space.

Synthetic tests are written in nasm and placed in binaries/x86/synthetic/source directory. Every file contains a small set of ROP gadgets and checks the ability to chain particular combination of gadgets.

To run only synthetic tests:

$ ./run.sh -s 

In synthetic tests, a trick is used to get rid off extra gadgets emitted by C runtime. We want a tool to construct chains only from gadgets written in nasm source files. Two custom linker scripts were created to achieve this. Using them a linker creates two binaries .gdt64 and .vuln64 inside binaries/x86/synthetic/vuln directory. The first one is used only for a chain creation, whereas the second one is used only for checking generated chains. Both files contain two executable segments: one contains code from vul.c and C runtime, the another contains only sections with actual code of gadgets named .gadgets.text. The only difference between .gdt64 and .vuln64 files is the first segment flags. File .gdt64 has only read permissions that have to restrict tools taking gadgets from them. So you should name section with gadgets in nasm files as .gadgets.text.

Real life binaries are placed in binaries/x86/reallife/orig. It contains several set of binaries from different Linux distributions:

1. CentOS 7.1810
2. Debian 10 cloud
3. OpenBSD 6.2
4. OpenBSD 6.4

It is just almost all ELF files (both binaries and shared libraries) of default installation.

To run only real life set of binaries, e.g. openbsd-62:

$ ./run.sh -r openbsd-62 

Note: we tested binaries both of OpenBSD 6.2 and OpenBSD 6.4 because their developers intentionally try to reduce the amount of ROP gadgets.

There are many tools to automatically create ROP chains. Supporting all of them is not a easy task; so we pick these ones as most popular and easier to support.

1. ROPgadget

2. Ropper

3. ROPium

4. angrop

5. Exrop

To run all tests only with e.g. ropper

$ ./run.sh -t ropper 

Benchmark print results in terminal like this:

=== Tool 'angrop' === Exp. type 'execve' 1:rop-benchmark:angrop:binaries/x86/reallife/vuln/centos-7.1810/ld.bfd.bin - INFO - OK 2:rop-benchmark:angrop:binaries/x86/reallife/vuln/centos-7.1810/ld.gold.bin - CRITICAL - FAIL TIMEOUT 3:rop-benchmark:angrop:binaries/x86/reallife/vuln/centos-7.1810/libBrokenLocale-2.17.so.bin - ERROR - Compilation ERROR with 1 (angrop) 4:rop-benchmark:angrop:binaries/x86/reallife/vuln/centos-7.1810/libasound.so.2.0.0.bin - CRITICAL - FAIL HIJACK --- Test suite --- binaries/x86/reallife/vuln/centos-7.1810 : 53 / 649 (passed/all) 

There are 4 states of tests:

1. ERROR - tool didn't generate a ROP chain.
2. FAIL TIMEOUT (TL) - tool exceeds the time limit (300 s as default).
3. FAIL HIJACK (F) - tool generated a ROP chain but it didn't run /bin/sh.
4. OK - tool generated a ROP chain and it ran /bin/sh.

Each tool was run single-threaded with 1 hour time limit.

At least one OK means at least one tool produced OK.

Note: Ropper almost always generates a ROP chain script file, so FAIL HIJACKs were not evaluated. results.pdf contains a hyperlink for each tool repository with a specific commit hash.

MAJORCA: Multi-Architecture JOP and ROP Chain Assembler paper results [slides].

Prints results table. Takes rop-benchmark output as input. --latex for LaTeX table.

The script helps to identify two things:

1. the common subset of binaries that are OK at least for one instrument,
2. the difference between the particular tool result and the common subset.

To print the common subset:

$ ./common.py rop-benchmark.output 

To print the difference:

$ ./common.py rop-benchmark.output --diff ropper 

If you want to contribute then you may:

1. Support new ROP chain generating tool.
2. Add new type of payload: memory write, direct call of linked function, indirect call of linked function, something with bad characters.
3. Add more synthetic tests (any kind of jop call ending gadgets also).
4. Add more real life tests.
5. Support Windows.
6. Support x86 32-bit tests.

Everything related to a particular tool should be placed under folder with corresponding name. This directory should contain job runners for every supported payload type with names job_{payload_type}.py:

from roptest import get_class_name job_class = get_class_name() class ExecveToolNameJob(job_class): def __init__(self): super().__init__() self.rop_tool = "ToolName" def run_rop_tool(self): # Implement here commands to run tool ... ExecveToolNameJob().run()

New payload type can be supported by adding new job runners inside every tool directory job_{exploit_type}.py.

To add a new synthetic test one may just write new .nasm64 file in binaries/x86/synthetic/source and then compile them:

$ cd binaries/x86/synthetic $ make 

To add a new test suite of real life binaries one may create directory under binaries/x86/reallife/orig and place original binaries there. Then compile them to target test programs with vulnerabilities:

$ cd binaries/x86/reallife/ $ make 

To support windows you should implement platform specific functions in roptest/windows_job.py and create environment suitable for testing workability of exploits like docker container on Linux.

Vishnyakov, A.V. and Nurmukhametov, A.R., 2021. Survey of Methods for Automated Code-Reuse Exploit Generation. Programming and Computer Software 47(4), pp. 271-297. DOI: 10.1134/S0361768821040071

@article{vishnyakov21, title = {Survey of Methods for Automated Code-Reuse Exploit Generation}, author = {Vishnyakov, A.~V. and Nurmukhametov, A.~R.}, journal = {Programming and Computer Software}, volume = {47}, number = {4}, pages = {271--297}, year = {2021}, doi = {10.1134/S0361768821040071}, } 

Nurmukhametov, A., Vishnyakov, A., Logunova V., Kurmangaleev Sh. MAJORCA: Multi-Architecture JOP and ROP Chain Assembler. 2021 Ivannikov ISPRAS Open Conference (ISPRAS), IEEE, 2021, pp. 37-46. DOI: 10.1109/ISPRAS53967.2021.00011

@inproceedings{nurmukhametov21, title = {{{MAJORCA}}: Multi-Architecture JOP and ROP Chain Assembler}, author = {Nurmukhametov, Alexey and Vishnyakov, Alexey and Logunova, Vlada and Kurmangaleev, Shamil}, booktitle = {2021 Ivannikov ISPRAS Open Conference (ISPRAS)}, pages = {37--46}, year = {2021}, publisher = {IEEE}, doi = {10.1109/ISPRAS53967.2021.00011}, }