git clone https://github.com/cirosantilli/linux-kernel-module-cheat cd linux-kernel-module-cheat ./configure && ./build && ./run

root insmod /hello.ko insmod /hello2.ko rmmod hello rmmod hello2

hello init hello2 init hello cleanup hello2 cleanup

Many of the modules have userland test scripts / executables with the same name as the module, e.g. form inside the guest:

The sources of those tests will further clarify what the corresponding kernel modules does. To find them on the host, do a quick:

If you make changes to the kernel modules or most configurations tracked on this repository, you can just use again:

and the modified files will be rebuilt.

If you change any package besides kernel_module, you must also request those packages to be reconfigured or rebuilt with extra targets, e.g.:

Those aren’t turned on by default because they take quite a few seconds.

Linux and QEMU rebuilds are so common that we have dedicated shortcut flags for them:

You did something crazy, and nothing seems to work anymore?

All builds are stored under buildroot/,

The most coarse thing you can do is:

To only nuke one architecture, do:

Only nuke one one package:

This is sometimes necessary when changing the version of the submodules, and then builds fail. We should try to understand why and report bugs.

The root filesystem is persistent across:

then:

This is particularly useful to re-run shell commands from the history of a previous session with Ctrl + R.

However, when you do:

the disk image gets overwritten by a fresh filesystem and you lose all changes.

Remember that if you forcibly turn QEMU off without sync or poweroff from inside the VM, e.g. by closing the QEMU window, disk changes may not be saved.

We use printk a lot, and it shows on the QEMU terminal by default. If that annoys you (e.g. you want to see stdout separately), do:

See also: https://superuser.com/questions/351387/how-to-stop-kernel-messages-from-flooding-my-console

You can scroll up a bit on the default TTY with:

but I never managed to increase that buffer:

• https://askubuntu.com/questions/709697/how-to-increase-scrollback-lines-in-ubuntu14-04-2-server-edition

• https://unix.stackexchange.com/questions/346018/how-to-increase-the-scrollback-buffer-size-for-tty

The superior alternative is to use text mode or a telnet connection.

Show serial console directly on the current terminal, without opening a QEMU window:

To exit, just do a regular:

This mode is very useful to:

• get full panic traces when you start making the kernel crash :-) See also: https://unix.stackexchange.com/questions/208260/how-to-scroll-up-after-a-kernel-panic

• copy and paste commands and stdout output to / from host

• have a large scroll buffer, and be able to search it, e.g. by using GNU screen on host

If the system crashes and you can’t can quit QEMU with poweroff, or if poweroff is just too slow for your patience, you can hard kill the VM with

or:

or on host:

or:

See also:

• http://stackoverflow.com/questions/14165158/how-to-switch-to-qemu-monitor-console-when-running-with-curses

• https://superuser.com/questions/1087859/how-to-quit-qemu-monitor

• https://superuser.com/questions/488263/problems-switching-to-qemu-control-panel-with-nographics

• https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui/1211516#1211516

Limitations:

• TODO: Ctrl + C kills the emulator for some setups (TODO which what exactly?), and not sent to guest processes. See:

  • cloudius-systems/osv#49

  • https://unix.stackexchange.com/questions/167165/how-to-pass-ctrl-c-in-qemu

    This is however fortunate when running QEMU with GDB, as the Ctrl + C reaches GDB and breaks.

• Very early kernel messages such as early console in extract_kernel only show on the GUI, since at such early stages, not even the serial has been setup.

When debugging a module, it becomes tedious to wait for build and re-type:

every time.

Instead, you can either run them from a minimal init:

or run them at the end of the BusyBox init, which does things like setting up networking:

or add them to a new init.d entry:

and they will be run automatically before the login prompt.

S99 is a git tracked convenience symlink to the gitignored rootfs_overlay/etc/init.d/S99

Scripts under /etc/init.d are run by /etc/init.d/rcS, which gets called by the line ::sysinit:/etc/init.d/rcS in /etc/inittab.

We try to use the latest possible kernel major release version.

In QEMU:

or in the source:

Build configuration can be observed in guest with:

or on host:

Bootloaders can pass a string as input to the Linux kernel when it is booting to control its behaviour, much like the execve system call does to userland processes.

This allows us to control the behaviour of the kernel without rebuilding anything.

With QEMU, QEMU itself acts as the bootloader, and provides the -append option and we expose it through ./run -e, e.g.:

Then inside the host, you can check which options were given with:

They are also printed at the beginning of the boot message:

See also:

• https://unix.stackexchange.com/questions/48601/how-to-display-the-linux-kernel-command-line-parameters-given-for-the-current-bo

• https://askubuntu.com/questions/32654/how-do-i-find-the-boot-parameters-used-by-the-running-kernel

The arguments are documented in the kernel documentation: https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html

When dealing with real boards, extra command line options are provided on some magic bootloader configuration file, e.g.:

• GRUB configuration files: https://askubuntu.com/questions/19486/how-do-i-add-a-kernel-boot-parameter

• Raspberry pi /boot/cmdline.txt on a magic partition: https://raspberrypi.stackexchange.com/questions/14839/how-to-change-the-kernel-commandline-for-archlinuxarm-on-raspberry-pi-effectly

Sometimes in Ubuntu 14.04, after the QEMU SDL GUI starts, it does not get updated after keyboard strokes, and there are artifacts like disappearing text.

We have not managed to track this problem down yet, but the following workaround always works:

This started happening when we switched to building QEMU through Buildroot, and has not been observed on later Ubuntu.

Using text mode is another workaround if you don’t need GUI features.

When you start interacting with QEMU hardware, it is useful to see what is going on inside of QEMU itself.

This is of course trivial since QEMU is just an userland program on the host, but we make it a bit easier with:

Then you could:

And in QEMU:

Just make sure that you never click inside the QEMU window when doing that, otherwise you mouse gets captured forever, and the only solution I can find is to go to a TTY with Ctrl + Alt + F1 and kill QEMU.

You can still send key presses to QEMU however even without the mouse capture, just either click on the title bar, or alt tab to give it focus.

./run -d

./rungdb start_kernel

l n c

./rungdb

/count.sh

Ctrl + C break sys_write continue continue continue

Loadable kernel modules are a bit trickier since the kernel can place them at different memory locations depending on load order.

So we cannot set the breakpoints before insmod.

However, the Linux kernel GDB scripts offer the lx-symbols command, which takes care of that beautifully for us:

In QEMU:

In GDB, hit Ctrl + C, and note how it says:

That’s lx-symbols working! Now simply:

In QEMU:

and GDB now breaks at our fop_write function!

Just don’t forget to remove your breakpoints after rmmod, or they will point to stale memory locations.

TODO: why does break work_func for insmod kthread.ko not break the first time I insmod, but breaks the second time?

See also: http://stackoverflow.com/questions/28607538/how-to-debug-linux-kernel-modules-with-qemu/44095831#44095831

TODO: why can’t we break at early startup stuff such as:

See also: https://stackoverflow.com/questions/2589845/what-are-the-first-operations-that-the-linux-kernel-executes-on-boot

GDB can call functions as explained at: https://stackoverflow.com/questions/1354731/how-to-evaluate-functions-in-gdb

However this is failing for us:

• some symbols are not visible to call even though b sees them

• for those that are, call fails with an E14 error

E.g.: if we break on sys_write on /count.sh:

even though fdget_pos is the first thing sys_write does:

See also: #19

./run -k ./rungdb -k

c

/count.sh & /kgdb.sh

b sys_write c c c c

In QEMU:

In GDB:

and you now control the count.

TODO: if I -ex lx-symbols to the gdb command, just like done for QEMU -gdb, the kernel oops. How to automate this step?

If you modify runqemu to use:

instead of kgdboc=ttyS0,115200, you enter a different debugging mode called KDB.

Usage: in QEMU:

Boot finishes, then:

And you are back in KDB. Now you can:

And you will break whenever sys_write is hit.

The other KDB commands allow you to instruction steps, view memory, registers and some higher level kernel runtime data.

But TODO I don’t think you can see where you are in the kernel source code and line step as from GDB, since the kernel source is not available on guest (ah, if only debugging information supported full source).

/gdbserver.sh /myinsmod.out /hello.ko

./rungdbserver kernel_module-1.0/user/myinsmod.out

find buildroot/output.x86_64~/build -name myinsmod.out

As usual, different archs work with:

BusyBox executables are all symlinks, so if you do on guest:

on host you need:

Our setup gives you the rare opportunity to step debug libc and other system libraries e.g. with:

Or simply by stepping into calls:

This is made possible by the GDB command:

which automatically finds unstripped shared libraries on the host for us.

See also: https://stackoverflow.com/questions/8611194/debugging-shared-libraries-with-gdbserver/45252113#45252113

QEMU -gdb GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well.

• https://stackoverflow.com/questions/26271901/is-it-possible-to-use-gdb-and-qemu-to-debug-linux-user-space-programs-and-kernel

• https://stackoverflow.com/questions/16273614/debug-init-on-qemu-using-gdb

The only use case I can see for this is to debug the init process (and have fun), otherwise, why wouldn’t you just use gdbserver? Known limitations of direct userland debugging:

• the kernel might switch context to another process, and you would enter "garbage"

• TODO step into shared libraries. If I attempt to load them explicitly:

  (gdb) sharedlibrary ../../staging/lib/libc.so.0 No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.

  since GDB does not know that libc is loaded.

Custom init process:

• Shell 1:

  ./run -d -e 'init=/sleep_forever.out' -n

• Shell 2:

  ./rungdb-user kernel_module-1.0/user/sleep_forever.out main

BusyBox custom init process:

• Shell 1:

  ./run -d -e 'init=/bin/ls' -n

• Shell 2:

  ./rungdb-user -h busybox-1.26.2/busybox ls_main

This follows BusyBox' convention of calling the main for each executable as <exec>_main since the busybox executable has many "mains".

BusyBox default init process:

• Shell 1:

  ./run -d -n

• Shell 2:

  ./rungdb-user -h busybox-1.26.2/busybox init_main

This cannot be debugged in another way without modifying the source, or /sbin/init exits early with:

Non-init process:

• Shell 1

  ./run -d -n

• Shell 2

  ./rungdb-user kernel_module-1.0/user/sleep_forever.out Ctrl + C b main continue

• Shell 1

  /sleep_forever.out

This is of least reliable setup as there might be other processes that use the given virtual address.

First build:

Debug:

TODOs:

• only managed to run in the terminal interface (but weirdly a blank QEMU window is still opened)

• GDB not connecting to KGDB. Possibly linked to -serial stdio. See also: https://stackoverflow.com/questions/14155577/how-to-use-kgdb-on-arm

As usual, we use Buildroot’s recommended QEMU setup QEMU aarch64 setup:

• https://github.com/buildroot/buildroot/blob/2017.08/board/qemu/aarch64-virt/readme.txt

• https://github.com/buildroot/buildroot/blob/2017.08/configs/qemu_aarch64_virt_defconfig

This makes aarch64 a bit different from arm:

• uses -M virt. https://wiki.qemu.org/Documentation/Platforms/ARM explains:

  Most of the machines QEMU supports have annoying limitations (small amount of RAM, no PCI or other hard disk, etc) which are there because that’s what the real hardware is like. If you don’t care about reproducing the idiosyncrasies of a particular bit of hardware, the best choice today is the "virt" machine.

  -M virt has some limitations, e.g. I could not pass -drive if=scsi as for arm, and so Snapshot fails.

• uses initramfs, so thre is no filesystem persistency.

So, as long as you keep those points in mind, our -a aarch64 offers an interesting different setup to play with.

TODOs:

• GDB step debugging appears to be stuck on an infinite loop:

  no module object found for ''

Is the default BusyBox /init too bloated for you, minimalism freak?

No problem, just use the init kernel boot parameter:

Remember that shell scripts can also be used for init https://unix.stackexchange.com/questions/174062/init-as-a-shell-script/395375#395375:

Also remember that if your init returns, the kernel will panic, there are just two non-panic possibilities:

• run forever in a loop or long sleep

• poweroff the machine

The default BusyBox init scripts enable networking, and there is a 15 second timeout in case your network is down or if your kernel / emulator setup does not support it.

To disable networking, use:

To restore it, run:

The docs make it clear https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html

The kernel parses parameters from the kernel command line up to “–”; if it doesn’t recognize a parameter and it doesn’t contain a ‘.’, the parameter gets passed to init: parameters with ‘=’ go into init’s environment, others are passed as command line arguments to init. Everything after “–” is passed as an argument to init.

And you can try it out with:

modprobe dep2 lsmod # dep and dep2

modprobe -r dep2 lsmod # Nothing.

modprobe dep modprobe dep2 modprobe -r dep2 # Nothing.

ls /lib/modules/*/extra/

modprobe dummy-irq

./build -x ./run

startx

On ARM, startx hangs at a message:

and nothing shows on the screen, and:

says:

A friend told me this but I haven’t tried it yet:

• xf86-video-modesetting is likely the missing ingredient, but it does not seem possible to activate it from Buildroot currently without patching things.

• xf86-video-fbdev should work as well, but we need to make sure fbdev is enabled, and maybe add some line to the Xorg.conf

time ./run -n -e 'init=/poweroff.out' -- -trace exec_tb,file=trace && \ time ./qemu/scripts/simpletrace.py qemu/trace-events trace >trace.txt && \ wc -l trace.txt && \ sed '/0x1000000/q' trace.txt >trace-boot.txt && \ wc -l trace-boot.txt

GEM5 is a system simulator, much like QEMU: http://gem5.org/

Vs QEMU:

• advantage: simulates a generic more realistic pipelined and optionally out of order CPU cycle by cycle, including a realistic DRAM memory access model with latencies, caches and page table manipulations. This allows us to:

  • do much more realistic performance benchmarking with it, which makes absolutely no sense in QEMU, which is purely functional

  • make functional cache observations, e.g. to use Linux kernel APIs that flush memory like DMA, which are crucial for driver development. In QEMU, the driver would still work even if we forget to flush caches.

    It is not of course truly cycle accurate, as that would require exposing proprietary information of the CPU designs: https://stackoverflow.com/questions/17454955/can-you-check-performance-of-a-program-running-with-qemu-simulator/33580850#33580850, but the approximation is reasonable.

    It is used mostly for research purposes: when you are making a new chip technology, you don’t really need to specialize enormously to an existing microarchitecture, but rather develop something that will work with a wide range of future architectures.

• disadvantage: slower than QEMU by TODO 10x?

  This also implies that the user base is much smaller, since no Android devs.

  Instead, we have only chip makers, who keep everything that really works closed, and researchers, who can’t version track or document code properly >:-) And this implies that: the documentation is more scarce it takes longer to support new hardware features

On another shell:

TODO didn’t get it working yet.

Related threads:

• https://www.mail-archive.com/gem5-users@gem5.org/msg11384.html

• https://stackoverflow.com/questions/37906425/booting-gem5-x86-ubuntu-full-system-simulation

• http://www.lowepower.com/jason/creating-disk-images-for-gem5.html claims to have a working config for x86_64 kernel 4.8.13

./run -i

date >f poweroff cat f # can't open 'f': No such file or directory

initramfs is just like initrd, but you also glue the image directly to the kernel image itself.

So the only argument that QEMU needs is the -kernel, no -drive not even -initrd! Pretty cool.

Try it out with:

since our aarch64 setup uses it by default.

In the background, it uses BR2_TARGET_ROOTFS_INITRAMFS, and this makes the kernel config option CONFIG_INITRAMFS_SOURCE point to the CPIO that will be embedded in the kernel image.

http://nairobi-embedded.org/initramfs_tutorial.html shows a full manual setup.

cd /sys/kernel/debug/tracing/ # Stop tracing. echo 0 > tracing_on # Clear previous trace. echo '' > trace # List the available tracers, and pick one. cat available_tracers echo function > current_tracer # List all functions that can be traced # cat available_filter_functions # Choose one. echo __kmalloc >set_ftrace_filter # Confirm that only __kmalloc is enabled. cat enabled_functions echo 1 > tracing_on # Latest events. head trace # Observe trace continously, and drain seen events out. cat trace_pipe &

# tracer: function # # entries-in-buffer/entries-written: 97/97 #P:1 # # _-----=> irqs-off # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / delay # TASK-PID CPU# |||| TIMESTAMP FUNCTION # | | | |||| | | head-228 [000] .... 825.534637: __kmalloc <-load_elf_phdrs head-228 [000] .... 825.534692: __kmalloc <-load_elf_binary head-228 [000] .... 825.534815: __kmalloc <-load_elf_phdrs head-228 [000] .... 825.550917: __kmalloc <-__seq_open_private head-228 [000] .... 825.550953: __kmalloc <-tracing_open head-229 [000] .... 826.756585: __kmalloc <-load_elf_phdrs head-229 [000] .... 826.756627: __kmalloc <-load_elf_binary head-229 [000] .... 826.756719: __kmalloc <-load_elf_phdrs head-229 [000] .... 826.773796: __kmalloc <-__seq_open_private head-229 [000] .... 826.773835: __kmalloc <-tracing_open head-230 [000] .... 827.174988: __kmalloc <-load_elf_phdrs head-230 [000] .... 827.175046: __kmalloc <-load_elf_binary head-230 [000] .... 827.175171: __kmalloc <-load_elf_phdrs

echo 1 > max_graph_depth echo 1 > events/enable echo function_graph > current_tracer

# CPU DURATION FUNCTION CALLS # | | | | | | | 0) 2.173 us | } /* ntp_tick_length */ 0) | timekeeping_update() { 0) 4.176 us | ntp_get_next_leap(); 0) 5.016 us | update_vsyscall(); 0) | raw_notifier_call_chain() { 0) 2.241 us | notifier_call_chain(); 0) + 19.879 us | } 0) 3.144 us | update_fast_timekeeper(); 0) 2.738 us | update_fast_timekeeper(); 0) ! 117.147 us | } 0) | _raw_spin_unlock_irqrestore() { 0) 4.045 us | _raw_write_unlock_irqrestore(); 0) + 22.066 us | } 0) ! 265.278 us | } /* update_wall_time */

sudo apt-get install qemu-user ./build -a arm cd buildroot/output.arm~/target qemu-arm -L . bin/ls

arm-linux-gnueabi-gcc -o hello -static hello.c qemu-arm hello

package/qemu/qemu.mk:110: *** "Refusing to build qemu-user: target Linux version newer than host's.". Stop.

qemu-arm -g 1234 -L . bin/ls ../host/usr/bin/arm-buildroot-linux-uclibcgnueabi-gdb -ex 'target remote localhost:1234'

/count.sh

echo 'savevm my_snap_id' | ./qemumonitor

echo 'loadvm my_snap_id' | ./qemumonitor

echo 0 >f

echo 'savevm my_snap_id' | ./qemumonitor

echo 1 >f

echo 'loadvm my_snap_id' | ./qemumonitor

cat f

cd kernel_module ./make-host.sh

mv broken.c broken.c~ ./build_host

sudo insmod hello.ko # Our module is there. sudo lsmod | grep hello # Last message should be: hello init dmest -T sudo rmmod hello # Last message should be: hello exit dmesg -T # Not present anymore sudo lsmod | grep hello

cd hello_host make insmod hello.ko dmesg rmmod hello.ko dmesg

ls -l /bin/lsmod

lrwxrwxrwx 1 root root 4 Jul 25 15:35 /bin/lsmod -> kmod

dpkg -l | grep -Ei

ii kmod 22-1ubuntu5 amd64 tools for managing Linux kernel modules

Name of a predecessor set of tools.

Load module under different name to avoid conflicts:

/dts-v1/; / { a; };

dtc a.dts

/dts-v1/; / { a; }; / { b; };

/dts-v1/; / { a; b; };

and update the README!

During update all you kernel modules may break since the kernel API is not stable.

They are usually trivial breaks of things moving around headers or to sub-structs.

The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.

This backwards compatibility is just awesome, it makes getting and running the latest master painless.

This also makes this repo the perfect setup to develop the Linux kernel.

The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.

The default Linux kernel version is bumped in Buildroot with commit messages of type:

So you can try:

Those commits change BR2_LINUX_KERNEL_LATEST_VERSION in /linux/Config.in.

You should then look up if there is a branch that supports that kernel. Staying on branches is a good idea as they will get backports, in particular ones that fix the build as newer host versions come out.

Hit / and search for the settings.

Save and quit.

Copy and paste the diff additions to buildroot_config_fragment.