34

Never thought this would happen to me, but there you go. ¯\_(ツ)_/¯

I ran a build script from a repository inside the wrong directory without looking at the source first. Here's the script Scripts/BuildLocalWheelLinux.sh:

cd ../Dependencies/cpython mkdir debug cd debug ../configure --with-pydebug --enable-shared make cd ../../.. cd .. mkdir -p cmake-build-local cd cmake-build-local rm -rf * cmake .. -DMVDIST_ONLY=True -DMVPY_VERSION=0 -DMVDPG_VERSION=local_build make -j cd .. cd Distribution python3 BuildPythonWheel.py ../cmake-build-local/[redacted]/core.so 0 python3 -m ensurepip python3 -m pip install --upgrade pip [more pip install stuff] python3 -m setup bdist_wheel --plat-name manylinux1_x86_64 --dist-dir ../dist cd .. cd Scripts

The dangerous part seems to be

mkdir -p cmake-build-local cd cmake-build-local rm -rf *

But thinking about it, it actually seems like it couldn't possibly go wrong.

The way you're supposed to run this script is cd Scripts; ./BuildLocalWheelLinux.sh. When I ran it the first time, it showed an error on the very last line (as I learned afterwards). I was in a hurry, so I though "maybe the docs are outdated, I'll try running from the project root instead. So I ran ./Scripts/BuildLocalWheelLinux.sh. Suddenly, vscodes theme and zoom level changed, my zsh terminal config was reset, terminal fonts were set to default, and I Ctrl+C'd once I realized what was happening.

There are some files remaining, but there's no obvious pattern to them:

$ ls -la total 216 drwx------ 27 felix felix 4096 May 12 18:08 . drwxr-xr-x 3 root root 4096 Apr 15 16:39 .. -rw------- 1 felix felix 12752 Apr 19 11:07 .bash_history -rw-r--r-- 1 felix felix 3980 Apr 15 13:40 .bashrc drwxrwxrwx 7 felix felix 4096 May 12 18:25 .cache drwx------ 8 felix felix 4096 May 12 18:26 .config drwx------ 3 root root 4096 Apr 13 21:40 .dbus drwx------ 2 felix felix 4096 Apr 30 12:18 .docker drwxr-xr-x 8 felix felix 4096 Apr 15 13:40 .dotfiles -rw------- 1 felix felix 8980 Apr 13 18:10 examples.desktop -rw-r--r-- 1 felix felix 196 Apr 19 15:19 .gitconfig -rw-r--r-- 1 felix felix 55 Apr 16 13:56 .gitconfig.old -rw-r--r-- 1 felix felix 1040 Apr 15 13:40 .gitmodules drwx------ 3 felix felix 4096 May 6 10:10 .gnupg -rw-r--r-- 1 felix felix 1848 May 5 14:24 heartbeat.tcl -rw------- 1 felix felix 1610 Apr 13 20:36 .ICEauthority drwxr-xr-x 5 felix felix 4096 Apr 21 16:39 .ipython drwxr-xr-x 2 felix felix 4096 May 4 09:35 .jupyter -rw------- 1 felix felix 161 Apr 27 14:23 .lesshst drwx------ 3 felix felix 4096 May 12 18:08 .local -rw-r--r-- 1 felix felix 140 Apr 29 17:54 minicom.log drwx------ 5 felix felix 4096 Apr 13 18:25 .mozilla drwxr-xr-x 2 felix felix 4096 Apr 13 18:10 Music drwxr-xr-x 6 felix felix 4096 May 12 17:16 Nextcloud -rw-r--r-- 1 felix felix 52 Apr 16 11:43 .nix-channels -rw------- 1 felix felix 1681 Apr 20 10:33 nohup.out drwx------ 3 felix felix 4096 Apr 15 11:16 .pki -rw------- 1 felix felix 946 Apr 16 11:43 .profile drwxr-xr-x 2 felix felix 4096 Apr 13 18:10 Public drwxr-xr-x 2 felix felix 4096 May 12 18:08 .pylint.d -rw------- 1 felix felix 1984 May 12 18:06 .pythonhist -rw-r--r-- 1 felix felix 2443 Apr 19 13:40 README.md drwxr-xr-x 13 felix felix 4096 May 12 18:08 repos drwxr-xr-x 6 felix felix 4096 Apr 19 11:08 snap drwx------ 3 felix felix 4096 May 5 15:33 .ssh drwxr-xr-x 5 felix felix 4096 Apr 26 17:39 .stm32cubeide drwxr-xr-x 5 felix felix 4096 May 5 15:52 .stm32cubemx drwxr-xr-x 2 felix felix 4096 Apr 23 11:44 .stmcube drwxr-xr-x 2 felix felix 4096 Apr 13 18:10 Templates drwxr-xr-x 3 felix felix 4096 Apr 19 11:57 test drwxr-xr-x 2 felix felix 4096 Apr 13 18:10 Videos -rw------- 1 felix felix 14313 May 12 10:45 .viminfo -rw-r--r-- 1 felix felix 816 Apr 15 13:40 .vimrc drwxr-xr-x 3 felix felix 4096 Apr 16 12:08 .vscode -rw-r--r-- 1 felix felix 2321 Apr 19 18:47 weird_bug.txt -rw-r--r-- 1 felix felix 162 Apr 15 13:40 .xprofile

.config is gone, as well as some standard XDG dirs like Pictures and Desktop, but .bashrc is still there. .nix-channels is still there, but .nix-defexpr was nuked.

So, this leads me to two questions:

  1. What went wrong? I'd like to fix this build script and make a PR to prevent this from happening in the future.
  2. What order were the files deleted in? Obviously not in alphabetical order, but * expands in alphabetical order, so something else is going on here, it seems.
Improve this question
9
  • 4
    As for your second question. rm would use the directory order. More details here: unix.stackexchange.com/questions/13451 . It would be like the default ordering of find's output. Commented May 12, 2021 at 17:27
  • 14
    If the mkdir fails and the directory didn't already exist, the cd will fail, and it's game over. Commented May 12, 2021 at 17:30
  • 3
    As a best practice, this shell scripts should stop executing as soon as the first error occurs, which probably would have been at the very first line cd ../Dependencies/cpython. That can be done by putting a && at the end of every command. There is more to it because sometimes it's ok for a command to fail, but that can be handled as well. Commented May 13, 2021 at 10:13
  • 5
    @mars I think set -e at the beginning is a more effective way of doing this. Commented May 13, 2021 at 10:36
  • 7
    set -e has its own issues. cd foo || exit 1 is more explicit with fewer surprises. Commented May 13, 2021 at 12:14

3 Answers 3

Reset to default
70

Ouch. You aren't the first victim.

What went wrong?

Starting in your home directory, e.g. /home/felix, or even in /home/felix/src or /home/felix/Downloads/src.

cd ../Dependencies/cpython

Failed because there is no ../Dependencies.

mkdir debug cd debug

You're now in the subdirectory debug of the directory you started from.

../configure --with-pydebug --enable-shared make

Does nothing because there's no ../configure or make.

cd ../../.. cd ..

If you started out no more than three directory levels deep, with cd debug reaching a fourth level, the current directory is now the root directory. If you started out four directory levels deep the current directory is now /home.

mkdir -p cmake-build-local

This fails since you don't have permission to write in / or /home.

cd cmake-build-local

This fails since there is no directory cmake-build-local.

We now get to…

What order were the files deleted in?

rm -rf *

This tries to recursively delete every file in the current directory, which is / or /home. The home directories are enumerated in alphabetical order, but the files underneath are enumerated in the arbitrary order of directory traversal. It's the same order as ls --sort=none (unless rm decides to use a different order for some reason). Note that this order is generally not preserved in backups, and can change when a file is created or removed in the directory.

How to fix the script

First, almost any shell script should have set -e near the top. set -e causes the script to abort if a command fails. (A command fails if its exit status is nonzero.) set -e is not a panacea, because there are circumstances where it doesn't go into effect. But it's the bare minimum you can expect and it would have done the right thing here.

(Also the script should start with a shebang line to indicate which shell to use, e.g. #!/bin/sh or #!/bin/bash. But that wouldn't help with this problem.)

rm -rf *, or variants like rm -rf $foo.* (what if $foo turns out to be empty?), are fragile. Here, instead of

mkdir -p cmake-build-local cd cmake-build-local rm -rf *

it would be more robust to just remove and re-create the directory. (This would not preserve the permissions on the directory, but here this is not a concern.)

rm -rf cmake-build-local mkdir cmake-build-local cd cmake-build-local

Another way is more robust against deleting the wrong files, but more fragile against missing files to delete: delete only files that are known to have been built, by running make clean which has rm commands for known build targets and for known extensions (e.g. rm *.o is ok).

Improve this answer
9
  • 11
    Perfect answer! I already submitted a PR. Commented May 12, 2021 at 17:56
  • 11
    And the PR got approved and merged about 5 hours later. Gotta love open source. Commented May 13, 2021 at 10:39
  • 2
    set -eo pipefail fixes most of the shortcomings of set -e. Commented May 13, 2021 at 20:31
  • 2
    @KonradRudolph No, a failure in the function is ignored! It doesn't make the conditional false. Here's a simple case without a function sh -c 'set -e; if false; true; then echo oops; fi' (in the real world, the failing command usually buried in a function, which makes it hard to find.). As for pipefail, the reason not to use it is that not all shells have it. Commented May 13, 2021 at 22:19
  • 4
    @iFreilicht "gotta love open source". Except when a botched script deletes your home folder. Commented May 14, 2021 at 14:02
11

Tracing your cd calls, assuming we are running the script in ~/Distribution/Scripts and assuming that every cd succeeds:

cd ../Dependencies/cpython

We are now in ~/Distribution/Dependencies/cpython.

mkdir debug cd debug

We are now in ~/Distribution/Dependencies/cpython/debug.

cd ../../..

We are now in ~/Distribution.

cd ..

We are now in your home directory.

mkdir -p cmake-build-local cd cmake-build-local

We are now in ~/cmake-build-local. This is where you run rm -rf *.

cd ..

We are now back in your home directory

cd Distribution

We are now in ~/Distribution.

cd .. cd Scripts

We are now in ~/Scripts (you get an error from this because you are one level further up than what you expect).

Now then. You tried running the same script, but from ~/Distribution.

cd ../Dependencies/cpython

This fails. This leaves you still in ~/Distribution.

mkdir debug cd debug

You are now in ~/Distribution/debug.

cd ../../..

You are now in ~/.. (probably in /home).

cd ..

You are most likely in / now.

cd Distribution mkdir -p cmake-build-local cd cmake-build-local

These probably fail due to "no such file or directory" and "permission denied".

rm -rf *

You are still in the / directory, and your rm command would try to delete every single file in the whole filesystem. Permissions only allows for deleting files that are located in directories that you have write permissions to, so you likely only loose files in /tmp and in your home directory.

The arguments listed on the command line of rm would have been processed the order that * expanded them (lexicographical, i.e., bin, boot, cdrom, dev, etc, etc.) Each directory listed would then have been processed recursively, and in "directory order" (unsorted).

What you should have done:

#!/bin/sh topdir=$HOME/Distribution mkdir -p "$topdir/Dependencies/cpython/debug" ( cd "$topdir/Dependencies/cpython/debug" || exit 1 ../configure --with-pydebug --enable-shared make ) rm -rf "$topdir/cmake-build-local" mkdir -p "$topdir/cmake-build-local" ( cd "$topdir/cmake-build-local" || exit 1 cmake .. -DMVDIST_ONLY=True -DMVPY_VERSION=0 -DMVDPG_VERSION=local_build make -j ) ( cd "$topdir" || exit 1 python3 BuildPythonWheel.py ../cmake-build-local/[redacted]/core.so 0 python3 -m ensurepip python3 -m pip install --upgrade pip python3 -m setup bdist_wheel --plat-name manylinux1_x86_64 --dist-dir ../dist )

The working directory within each separate ( ... ) sub-shell is local to that sub-shell. The initial cd in a sub-shell will not affect the working directory of the "outer" environment. The rest of the code uses absolute pathnames that do not depend on the initial working directory of the user's shell session. Note particularly that the rm command does not blindly expand * but rather deletes a specific directory specified via an absolute path (this would not go horribly wrong if the directory does not exist).

Improve this answer
2
  • Subshells seem like a much more robust solution, thanks! Commented May 12, 2021 at 17:58
  • 1
    Ah, and cmake-build-local is located in the project root folder Commented May 14, 2021 at 15:40
5

If the script assumes that it is run from inside Scripts, then none of these will do what the script author intended:

cd ../Dependencies/cpython mkdir debug cd debug cd ../../.. cd .. mkdir -p cmake-build-local cd cmake-build-local

The first cd will fail, the second cd moves into a folder, then the cd ../../.. and cd .. probably put you somewhere above your home directory (maybe in the /home directory, where you usually don't have rights to make anything, so the mkdir and subsequent cd fails). Then the * probably expanded to your home directory, so rm -rf operated on it, recursing into the contents, which would explain the random order of deletes (directory entries aren't sorted in any particular order).

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.