Dirty scripts way
There are multiple ways to reboot. One way I see to investigate is to replace shutdown, halt, reboot and init by scripts:
for cmd in shutdown halt reboot init do mv /usr/bin/$cmd /usr/bin/${cmd}.ori ln -s /usr/bin/${cmd}.ori /usr/bin/$cmd done
Then create a shell script logging the information:
#!/bin/bash echo "$0 called with $@ by $USER at `time` - Processes: `ps axjf`" >> /var/log/reboot.log ${0}.ori $@
And copy or link it as /usr/bin/{shutdown,halt,reboot,init}. You also want to create the /var/log/reboot.log and ensure anybody will be able to log in it, although only root should be able to reboot.
The ps axjf should allow you to see which process called your script. You may want to add more information in the log but this should be a good start.
Things that may be missed
Other commands could reboot the system by calling the syscall reboot. It could be "official" commands or "rogue" ones.
X server is able to shutdown and with some settings, any user can do it. If remote access and shutdown are allowed, other people can require a shutdown at distance (however this wouldn't explain the every other 3 reboot behaviour).
Init target
You could:
- add a step at the beginning of the target in runlevels 0 and 6 that wait for some time (e.g:
sleep 60 or a bit more) - have a script running
ps axjf every 60 seconds and logging in a file
This should provide info on running processes at the time of the shutdown. Of course, you can add other commands that could help you to investigate to the stuff executed every 60 seconds
Kernel auditing
Another way to investigate this would be to use the kernel auditing feature, but it needs to be enabled in your kernel (not the case in many distribs).
Then use:
auditctl -a entry,always -S reboot
to record every call to reboot, and use ausearch to find the calls that were done.
which reboots every 1 in 3 reboots<--- really ? :-)