10

I'm struggling getting perf_events to give me stack traces with symbols, despite reading many tutorials on the subject and doing (I think) all the necessary things. It's possible that my local install of perf (details on that below) is somehow botched? Anyway, here's what I did:

main.cpp is a simple C++ program that calls a few functions defined in the same file, allocates some memory and frees it, and prints a few things out.

compilation command:

gcc -std=c++11 -lstdc++ main.cpp -Og -fno-omit-frame-pointer -fno-inline -o arr_test

profile command:

perf record -a -g -- ./arr_test && perf report --stdio

I do get the following warnings about kernel symbols, but I don't think this should matter given that I only care about symbols in my application for now:

[ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.052 MB perf.data (~2285 samples) ] [kernel.kallsyms] with build id e22966849c48748782a1be4fe0ce94db6838b806 not found, continuing without symbols [kernel.kallsyms] with build id e22966849c48748782a1be4fe0ce94db6838b806 not found, continuing without symbols Warning: Kernel address maps (/proc/{kallsyms,modules}) were restricted. Check /proc/sys/kernel/kptr_restrict before running 'perf record'. As no suitable kallsyms nor vmlinux was found, kernel samples can't be resolved. Samples in kernel modules can't be resolved as well.

Here's a snippet of the output:

# Overhead Command Shared Object # ........ ........ ................. # 83.27% arr_test arr_test | |--34.12%-- 0x400908 | 0x7fe72b381ec5 | |--10.48%-- 0x400903 | 0x7fe72b381ec5 | |--10.08%-- 0x4008b8 | 0x7fe72b381ec5 | |--9.22%-- 0x4008e5 | 0x7fe72b381ec5 | |--9.05%-- 0x4008da | 0x7fe72b381ec5 | |--8.49%-- 0x4008f0 | 0x7fe72b381ec5 | |--6.87%-- 0x4008d5 | 0x7fe72b381ec5 | |--6.23%-- 0x4008c2 | 0x7fe72b381ec5 | |--4.76%-- 0x4008fd | 0x7fe72b381ec5 --0.70%-- [...] 8.02% arr_test [kernel.kallsyms] | |--4.87%-- 0xffffffff81140b64 | 0xffffffff81146646 | 0xffffffff81182751 | 0xffffffff811829eb | 0xffffffff8173317d | 0x7fe72bab86a7 | 0x7fe72baa7e00

file info (shows "not stripped"):

$ file arr_test arr_test: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, not stripped

Details on my perf install (do any of these warnings prevent me from seeing symbols in stacks?)

Auto-detecting system features: ... backtrace: [ on ] ... dwarf: [ OFF ] ... fortify-source: [ on ] ... glibc: [ on ] ... gtk2: [ on ] ... gtk2-infobar: [ on ] ... libaudit: [ OFF ] ... libbfd: [ OFF ] ... libelf: [ OFF ] ... libelf-getphdrnum: [ OFF ] ... libelf-mmap: [ OFF ] ... libnuma: [ on ] ... libperl: [ on ] ... libpython: [ on ] ... libpython-version: [ on ] ... libslang: [ on ] ... libunwind: [ OFF ] ... on-exit: [ on ] ... stackprotector: [ on ] ... stackprotector-all: [ on ] ... timerfd: [ on ] config/Makefile:264: No libelf found, disables 'probe' tool, please install elfutils-libelf-devel/libelf-dev config/Makefile:329: No libunwind found, disabling post unwind support. Please install libunwind-dev[el] >= 1.1 config/Makefile:354: No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev

How can I find my symbols in perf?

Improve this question
0

2 Answers 2

Reset to default
8

I'm compiling using more debugging options:

-Og -ggdb3 -fno-omit-frame-pointer

Then, when I record I'm not using -a option (that should monitor all the system processes), I'm using

perf record -e cycles -g --call-graph fp -- ./your_app your_args

Finally, to show the result I'm using

perf report -g graph

and the output looks like expected (note, I'm using debian 9 and the perf report output is ncurses based)

- 92.18% 0.00% stsm stsm [.] main ◆ - main ▒ - 91.77% STSM::run ▒ + 56.86% STSM::generateCandidates ▒ - 25.22% STSM::detectBlocksOfAllSolidSequences ▒ + 23.42% STSM::detectSolidSequenceBlocksFromSolidSequence ▒ 0.81% Segment::unify ▒ + 5.25% STSM::updateKernelsOfAllCandidates ▒ 1.80% RangedSequence::range ▒ + 1.45% STSM::updateMatchingPositions ▒ 0.99% Segment::intersects ▒ + 92.18% 0.00% stsm libc-2.24.so [.] __libc_start_main ▒ + 92.18% 0.00% stsm [unknown] [k] 0x4d96258d4c544155 ▒ + 91.77% 0.00% stsm stsm [.] STSM::run ▒ + 56.86% 6.74% stsm stsm [.] STSM::generateCandidates ▒ + 49.99% 49.99% stsm stsm [.] Segment::intersects ▒ + 25.22% 0.00% stsm stsm [.] STSM::detectBlocksOfAllSolidSequences
Improve this answer
4
  • 1
    In case someone is like me and hasn't realized that -0g will keep less information than -O0, here you go: stackoverflow.com/questions/12970596/gcc-4-8-does-og-imply-g/… Commented Apr 5, 2022 at 6:40
  • 1
    Nice comment @lol, follow your link I just learn that if you specify more than one -g* options (like in my original answer) than "the last such option is the one that is effective" (gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html) so I updated the answer to use just -ggdb3 Commented Apr 6, 2022 at 10:08
  • Yes, unfortunately (or fortunately) gcc wouldn't warn you about that since it allows you to specify different arguments for different parts of the input parameters. Like the whole-archive flag for specific ar files. Commented Apr 6, 2022 at 16:43
  • 1
    This is still not working for me. Compiling with -O0 -ggdb3 -fno-omit-frame-pointer -std=c++20 (also tried -Og) and doing the rest as posted here; there are no function names whatsoever. Only "[unknown]". Commented Jun 4, 2022 at 16:57
5

This is an old topic but am sure it still happens to hundreds of people and because this StackExchange question still shows up high on Google results, am sharing answer found on StackOverflow that worked for me: https://stackoverflow.com/questions/33137543/linux-perf-top-kernel-symbol-not-found

Basically:

Before starting perf record

echo 0 > /proc/sys/kernel/kptr_restrict

Also it might be necessary to setup these

For RHEL/CentOS/Fedora/etc

yum install -y elfutils-libelf-devel libunwind-devel audit-libs-devel slang-devel

Or for Debian/Ubuntu/etc

apt-get install libelf-dev libunwind-dev libaudit-dev libslang-dev

Improve this answer
1
  • 1
    If Linux says "/proc/sys/kernel/kptr_restrict: Permission denied" then try sudo sh -c "echo 0 > /proc/sys/kernel/kptr_restrict" Commented Jul 6, 2021 at 14:03

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.