6

I have a Pi running Raspbian in a headless configuration that has crashed a few times, once for unknown reasons, and once due to a power outage. In both cases, I had to connect a monitor and keyboard and "help" it boot up (once it required an extra fsck execution, and once it wouldn't boot because the clock was off and the timestamp in the superblock was in the future; setting the correct date/time fixed it).

I'd like to make this system as robust as possible so that it is less likely to require intervention. What steps can help me achive this?

Improve this question
4
  • Sounds like ext2 and not ext3. A journaling file system is much more robust against unexpected shutdowns. Commented Jun 8, 2013 at 8:26
  • Parted reports it as ext4; when I ran fsck one of the times it mentioned that it was applying journaled changes (don't remember exact language). Commented Jun 8, 2013 at 11:51
  • The errors you mention sounds like it should be fixed upstream. I suggest you report it as bugs against the raspbian distribution. Commented Jun 8, 2013 at 15:37
  • I've opened a bug with Raspbian for this issue: bugs.launchpad.net/raspbian/+bug/1212020 Commented Aug 13, 2013 at 23:06

4 Answers 4

Reset to default
4

you might think about separating your system to the read-only partition and your data files somewhere else. read-only system is unlikely to get corrupted and after you have it up and running, you may start recovery scripts and/or login over SSH to fix the possible problems.

Improve this answer
2
  • Although not mounted ro, it's pretty close to read only. As far as I can determine, there's only one file being written to on my data partition, and it has a log entry added once or twice a day; I'll be removing that soon, so mounting ro would certainly be a possibility. Not sure if I can get away with that for the root partition, though. Commented Jun 8, 2013 at 11:53
  • Accepting, as I think that a RO system partition is the only sure solution, any other solution will have some scenario where it won't boot. Commented Mar 4, 2014 at 2:36
14

What happens is that fsck gets run after so many reboots or when it suspects the disk is in trouble. The number of reboots can be changed with tune2fs, but that won't fix your problem. Working backwards to the solution, here's the correct option for fsck:

 -y For some filesystem-specific checkers, the -y option will cause the fs-specific fsck to always attempt to fix any detected filesystem corruption automatically. Sometimes an expert may be able to do better driving the fsck manually. Note that not all filesys‐ tem-specific checkers implement this option. In particular fsck.minix(8) and fsck.cramfs(8) do not support the -y option as of this writing.

What causes the fsck to happen? Well, there's an init file at /etc/init.d/checkfs.sh. Look at line 17:

[ "$FSCKFIX" ] || FSCKFIX=no

Later, line 67:

 if [ "$FSCKFIX" = yes ] then fix="-y" else fix="-a"

So FSCKFIX controls if fsck is called with -y. We want -y so you can have it check automagically while headless. The easiest way is to modify /etc/default/rcS around line 24 and enable FSCKFIX, like this:

Original: #FSCKFIX-no

Modify to: FSCKFIX=yes

After that, you're done!

Improve this answer
2
  • +1, edit made. Waiting to see if any other suggestions come along before accepting. Commented Jun 8, 2013 at 2:17
  • checkfs.sh is not involved if using systemd, at least in jessie. Analog for systemd-fsck is to add parameter to kernel cmdline: fsck.repair=yes Commented Mar 15, 2017 at 11:18
3

If you feel like file corruption is one of your major problems, consider moving the entire OS away from the SD-card. On a Raspberry Pi 1 / 2 / Zero (W) you need at least some boot files on the SD-card but you can run the rest from a USB device. With a freshly downloaded image:

  • Copy the entire Raspbian image to a USB-device.
  • Copy (only!) the /boot/ partition to an SD-card.

Attention for the next 2 steps / edits: these paths are followed by a number. Don't edit or remove the number! (mmcbkl0p1 becomes sda1, mmcbkl0p2 becomes sda2, etc. ...).

  • Edit cmdline.txt in the /boot/ partition on the SD-card and change /dev/mmcblk0p to /dev/sda. Consider editing the same file in the /boot/ partition on the USB device so you have a ready-to-use backup of your /boot/ partition.
  • Edit /etc/fstab and change /dev/mmcblk0p to /dev/sda on the USB device.

Done! You can make the /boot/ partition your SD-card read-only (but there wouldn't be any writes anyway). Also, if the SD card would ever fail again, you have a backup of the /boot/ partition on your USB device.

Optional

If you have a Raspberry Pi 3 B, perform these extra steps (on top of the previous steps) to not need an SD-card at all.

  • Boot Raspbian (with the SD-card)
  • Update/upgrade your Raspbian OS
  • Edit /boot/config.txt on the SD-card, add a new line at the end with: program_usb_boot_mode=1
  • Reboot (Raspbian will enable USB boot mode during this boot cycle)

Done! Shut down, remove the SD-card from the Raspberry Pi 3 B and boot again. This will take longer than usual. Mine did a strange sequence on it's own: I put the power in, then it showed a color image, it rebooted on it's own, showed the color image again, rebooted on it's own again, showed the color image again and suddenly booted into Raspbian. The next reboots were much faster.

Make sure to remove the program_usb_boot_mode=1 line in /boot/config.txt on the SD-card if you plan on reusing the SD-card in other Raspberry Pi's that don't need USB boot mode enabled.

Improve this answer
1

Power-related crashes are better handled with some sort of UPS solution like juice4halt.

For cases where unexpected shutdown cannot be prevented, I suggest you try data=journal option for your root partition. It will make writes slower (but as you said in comments your root is virtually rea-only so there will be little impact), and fsck will be able to fix most issues automatically upon restart.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.