ptools is a collection of Linux utilities for inspecting the state of processes, inspired by the tools of the same name on Solaris/illumos.
Linux already has a number of mechanisms which can be used to inspect the state of processes (the /proc filesystem, ps(1), lsof(1), etc.). Why add a new set of tools?
The main advantage of ptools is consistency. The utilities provided by ptools are consistently named and have a consistent interface. Also, significantly, they can be run against core dumps where applicable, providing a uniform way to examine live processes and core dumps. This is very useful for those who rely heavily on core dumps to do postmortem debugging. The goal of this project is to make this same consistent debugging experience available on Linux.
For more information, see Dave Pacheco’s article “illumos tools for observing processes”.
ptools requires libdw (from elfutils) and libsystemd at run time.
Ubuntu/Debian:
sudo apt-get install libdw1 libsystemd0Fedora:
sudo dnf install elfutils-libs systemd-libsInstall from crates.io:
cargo install ptoolsAlternatively, download the latest .deb or .rpm package from the GitHub Releases page.
pstack(1) shows you thread stack traces for the processes you give it. This is incredibly useful as a first step for figuring out what a program is doing when it’s slow or not responsive.
$ pstack $$ 31903: /bin/zsh 0x00007fc764b24c5e __internal_syscall_cancel+0x7e 0x00007fc764b24c84 __syscall_cancel+0x14 0x00007fc764ad0525 __sigsuspend+0x25 0x000055bec2e64dad zwaitjob+0x5cd 0x000055bec2e64e8f waitjobs+0x2f 0x000055bec2e39682 execpline.lto_priv.0+0xba2 0x000055bec2e3abfe execlist+0x52e 0x000055bec2e3b4ce execode+0xae 0x000055bec2e5a532 loop+0x792 0x000055bec2e620e6 zsh_main+0x5a6 0x000055bec2e0ed0d main+0xd 0x00007fc764ab95b5 __libc_start_call_main+0x75 0x00007fc764ab9668 __libc_start_main@@GLIBC_2.34+0x88 0x000055bec2e0ed35 _start+0x25 If DWARF debug information is installed, you can use -v to show source code locations (file and line number) and inlined function frames.
$ pstack -v $(pgrep firefox) 27981: /usr/lib64/firefox/firefox 0x00007fa051e879a2 __syscall_cancel_arch+0x32 (syscall_cancel.S:56) 0x00007fa051e7bc3c __internal_syscall_cancel+0x5c (cancellation.c:49) 0x00007fa051e7c2ac __futex_abstimed_wait_common+0x7c (futex-internal.c:57) 0x00007fa051e7e97e pthread_cond_wait@@GLIBC_2.3.2+0x14e (pthread_cond_wait.c:421) 0x000055894a58616b mozilla::detail::ConditionVariableImpl::wait+0x1b (ConditionVariable_posix.cpp:104) 0x00007fa03b98067a mozilla::ThreadEventQueue::GetEvent+0xea (CondVar.h:58) 0x00007fa03b995d5d nsThread::ProcessNextEvent+0x18d (nsThread.cpp:1125) 0x00007fa03b9964a1 NS_ProcessNextEvent+0x41 (nsThreadUtils.cpp:461) 0x00007fa03c1cb39b mozilla::ipc::MessagePumpForNonMainThreads::Run+0xcb (MessagePump.cpp:329) 0x00007fa03c179b94 MessageLoop::RunInternal+0x54 (message_loop.cc:368) 0x00007fa03c179b94 MessageLoop::RunHandler [inlined] (message_loop.cc:361) 0x00007fa03c179b94 MessageLoop::Run [inlined] (message_loop.cc:343) 0x00007fa03b99cfd6 nsThread::ThreadFunc+0xd6 (nsThread.cpp:373) 0x00007fa0522fee63 _pt_root+0x103 0x000055894a5558ec set_alt_signal_stack_and_start+0xec (pthread_create_interposer.cpp:81) 0x00007fa051e7f464 start_thread+0x2e4 (pthread_create.c:448) 0x00007fa051f025ec __clone3+0x2c (clone3.S:78) pfiles(1) shows you every file descriptor a process has open (similar to lsof(1), but for a specific process). This includes details on regular files (including offset, which is great for checking on programs that scan through large files) and sockets. For more information, see Chris Siebenmann’s article “In praise of Solaris’s pfiles command”.
$ pfiles $(pgrep example) 2785: sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups Current rlimit: 1024 file descriptors Current umask: 022 0: S_IFCHR mode:0600 dev:0,27 ino:4 uid:1000 gid:5 rdev:136,1 O_RDWR /dev/pts/1 offset: 0 1: S_IFCHR mode:0600 dev:0,27 ino:4 uid:1000 gid:5 rdev:136,1 O_RDWR /dev/pts/1 offset: 0 2: S_IFCHR mode:0600 dev:0,27 ino:4 uid:1000 gid:5 rdev:136,1 O_RDWR /dev/pts/1 offset: 0 3: S_IFREG mode:0644 dev:252,0 ino:58197644 uid:0 gid:0 size:19 O_RDONLY|O_CLOEXEC /etc/locale.conf offset: 19 4: S_IFSOCK mode:0777 dev:0,9 ino:25978 uid:0 gid:0 size:0 O_RDWR|O_CLOEXEC|O_NONBLOCK sockname: AF_INET 0.0.0.0 port: 22 SOCK_STREAM SO_REUSEADDR,SO_ACCEPTCONN,SO_SNDBUF(16384),SO_RCVBUF(131072) congestion control: cubic state: TCP_LISTEN 5: S_IFSOCK mode:0777 dev:0,9 ino:25980 uid:0 gid:0 size:0 O_RDWR|O_CLOEXEC|O_NONBLOCK sockname: AF_INET6 :: port: 22 SOCK_STREAM SO_REUSEADDR,SO_ACCEPTCONN,SO_SNDBUF(16384),SO_RCVBUF(131072) congestion control: cubic state: TCP_LISTEN ptree(1) shows you a process tree for the whole system or for a given process or user:
$ ptree 1 1 /usr/lib/systemd/systemd --switched-root --system --deserialize=48 1427 /usr/lib/systemd/systemd-journald 1451 /usr/lib/systemd/systemd-userdbd 32574 systemd-userwork: waiting... 32873 systemd-userwork: waiting... 34264 systemd-userwork: waiting... 1465 /usr/lib/systemd/systemd-udevd 2439 /usr/lib/systemd/systemd-oomd 2440 /usr/lib/systemd/systemd-resolved 2441 /usr/bin/auditd pargs(1) prints the arguments a process was started with:
$ pargs $(pgrep Xwayland) 3978: Xwayland :0 -rootless -core -listenfd 55 -listenfd 56 -displayfd 98 -wm 95 argv[0]: Xwayland argv[1]: :0 argv[2]: -rootless argv[3]: -core argv[4]: -listenfd argv[5]: 55 argv[6]: -listenfd argv[7]: 56 argv[8]: -displayfd argv[9]: 98 argv[10]: -wm argv[11]: 95 penv(1) prints the environment variables of a process. Unlike cat /proc/[pid]/environ or ps e, which only show a static snapshot captured at process start, penv(1) reads live process memory (via the environ symbol) to show the current environment, including any variables added or modified at runtime with setenv(3) or putenv(3). No other Linux command-line tool provides this capability; on stock Linux the only alternative is attaching a full debugger. This makes penv(1) invaluable for debugging processes whose environment changes after startup.
$ penv $(pgrep Xwayland) 3978: Xwayland :0 -rootless -core -listenfd 55 -listenfd 56 -displayfd 98 -wm 95 envp[0]: SHELL=/bin/zsh envp[1]: GTK_IM_MODULE=wayland envp[2]: XDG_BACKEND=wayland envp[3]: XDG_CONFIG_DIRS=/etc:/etc/xdg:/usr/share envp[4]: XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session1 pauxv(1) prints the process’s auxiliary vector:
$ pauxv $(pgrep sshd) 2887: sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups AT_SYSINFO_EHDR 0x00007fca98be9000 AT_MINSIGSTKSZ 0x0000000000000d30 AT_HWCAP 0x00000000178bfbff FPU | VME | DE | PSE | TSC | MSR | PAE | MCE | CX8 | APIC | SEP | MTRR | PGE | MCA | CMOV | PAT | PSE36 | CLFSH | MMX | FXSR | SSE | SSE2 | HTT AT_PAGESZ 0x0000000000001000 AT_CLKTCK 0x0000000000000064 AT_PHDR 0x00005626e0d9a040 AT_PHENT 0x0000000000000038 AT_PHNUM 0x000000000000000d AT_BASE 0x00007fca98beb000 AT_FLAGS 0x0000000000000000 AT_ENTRY 0x00005626e0d9fc90 AT_UID 0x0000000000000000 0(root) AT_EUID 0x0000000000000000 0(root) AT_GID 0x0000000000000000 0(root) AT_EGID 0x0000000000000000 0(root) AT_SECURE 0x0000000000000000 AT_RANDOM 0x00007ffd97c8b079 AT_HWCAP2 0x0000000000000002 FSGSBASE AT_EXECFN 0x00007ffd97c8cfe9 /usr/bin/sshd AT_PLATFORM 0x00007ffd97c8b089 AT_RSEQ_FEATURE_SIZE 0x000000000000001c AT_RSEQ_ALIGN 0x0000000000000020 psig(1) shows what signals a process is catching:
$ psig 1 1: /usr/lib/systemd/systemd --switched-root --system --deserialize=48 HUP blocked,default INT blocked,default QUIT caught ILL caught TRAP default ABRT caught BUS caught FPE caught KILL default USR1 blocked,default SEGV caught USR2 blocked,default PIPE ignored ALRM default TERM blocked,default STKFLT default CLD blocked,default CONT default STOP default TSTP default TTIN default TTOU default URG default XCPU default XFSZ default VTALRM default PROF default WINCH blocked,default POLL default PWR blocked,default SYS default SIG32 default SIG33 default pstack(1) attaches to target processes using the ptrace(2) system call. Some distributions, notably Ubuntu, ship with the Yama Linux Security Module enabled and kernel.yama.ptrace_scope set to 1 by default. This restricts ptrace(2) to parent-child relationships only, meaning pstack cannot attach to arbitrary same-user processes unless it is run as root.
You can check the current setting with:
cat /proc/sys/kernel/yama/ptrace_scopeThe values are:
| Value | Meaning |
|---|---|
| 0 | Classic ptrace(2) permissions (any process can trace same-uid processes) |
| 1 | Restricted to parent-child relationships only |
| 2 | Admin-only (CAP_SYS_PTRACE required) |
| 3 | No ptrace(2) allowed at all |
To allow pstack(1) to trace same-user processes without root, set the classic behavior:
sudo sysctl kernel.yama.ptrace_scope=0Alternatively, simply run pstack(1) with sudo(1).
Fedora ships with ptrace_scope set to 0 by default, so pstack(1) works against same-user processes without any additional configuration.
Core dump support is available for pargs(1), pauxv(1), pcred(1), penv(1), pfiles(1), plgrp(1), plimit(1), psig(1), and pstack(1). Process metadata is retrieved from systemd-coredump(8) extended attributes and journal metadata; pstack(1) additionally reads the ELF core image to recover thread backtraces. Even when a core file has been removed by systemd-tmpfiles(8) or by storage limits, process metadata can often still be retrieved from the systemd-coredump(8) journal entry; use coredumpctl list <name> -F COREDUMP_FILENAME to find the path and pass it to any of these tools.
Ubuntu/Debian apport .crash files are also supported. These files use Debian control syntax with base64-encoded gzip core dumps. Text fields from the crash file are mapped to the corresponding COREDUMP_* fields and the core dump is extracted lazily on first access. Pass the .crash file path as the core operand to any of the tools listed above.
The following table lists all Solaris/illumos ptools and their status in this project. Tools provided by procps-ng, glibc, or python-linux-procfs are not reimplemented here, as these packages are widely available on Linux distributions and already provide equivalent functionality. There are a number of commands available on Solaris/illumos which have not been implemented here yet.
| Command | Description | Status |
|---|---|---|
pargs(1) | Print process arguments | ✅ Implemented |
pauxv(1) | Print process auxiliary vector | ✅ Implemented |
pcred(1) | Print process credentials | ✅ Implemented |
penv(1) | Print process environment variables | ✅ Implemented |
pfiles(1) | Print information for all open files in each process | ✅ Implemented |
pflags(1) | Print process status flags | ➡️ See python-linux-procfs |
pgrep(1) | Find processes by name | ➡️ See procps-ng |
pkill(1) | Signal processes by name | ➡️ See procps-ng |
pldd(1) | Print process dynamic libraries | ➡️ See glibc |
plgrp(1) | Print current NUMA node and thread CPU affinities | ✅ Implemented |
plimit(1) | Get or set process resource limits | ✅ Implemented |
plockstat(1) | Print lock statistics | 🔲 Not yet implemented |
pmadvise(1) | Apply advice about memory to a process | 🔲 Not yet implemented |
pmap(1) | Print process address maps | ➡️ See procps-ng |
ppgsz(1) | Set preferred page size | 🔲 Not yet implemented |
ppriv(1) | Print or modify process privilege sets and attributes | 🔲 Not yet implemented |
preap(1) | Force a defunct process to be reaped | 🔲 Not yet implemented |
prun(1) | Set stopped processes running with SIGCONT | ✅ Implemented |
psecflags(1) | Print or modify process security flags | 🔲 Not yet implemented |
psig(1) | Print process signal actions | ✅ Implemented |
pstack(1) | Print process call stack | ✅ Implemented |
pstop(1) | Stop processes with SIGSTOP | ✅ Implemented |
ptime(1) | Time a process using microstate accounting | ✅ Implemented |
ptree(1) | Print process trees | ✅ Implemented |
pwait(1) | Wait for processes to terminate | ✅ Implemented |
pwdx(1) | Print the current working directory of the process | ➡️ See procps-ng |
See CONTRIBUTING.md for details on contributing.
This project is licensed under the Apache License, Version 2.0. See the LICENSE file for details.