Oneliner bash built-in:

 DIR=/dir/to/remove
 PATH=${PATH/${PATH/#$DIR:*/$DIR:}/}${PATH/${PATH/*:$DIR*/:$DIR}/}

- Uses recursive [shell parameter expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html) to remove colon before or after removed directory
- Pattern has two halves. One half is always empty, other provide reduced path.
- Removes just one instance of $DIR from $PATH