Why do electrical engineers prefer "phase shift" instead of "time shift"?

Depends on your realm of study. In some industries "time delay" is distinctly different than "phase delay." But to your question, I have 3 points (and they're really all related to the last one).

1. Mathematically, phase is usually easier to fit in a formula. \$\exp(-j\phi)\$ is much simpler than \$\exp(-2\pi jt/f)\$ because now I need to know frequency.
2. Practically, phase is bounded \$[0, 360]\$ or \$[0, 2\pi]\$ and repeats. Time is neither bounded nor repeating.
3. Finally, and fundamentally, most of the our mathematics is in Frequency Domain (because it's drastically simpler to solve). Time offsets are replaced with phase offsets because the notion of "time delta" in a cyclic domain doesn't make sense. You're tracing the same circle(s) over-and-over into infinity and eventually the notion of time is lost and replaced with just phase (e.g. location on that circle).

It may be easier to understand time but it's easier to measure and perform mathematics with phase.

Phase-shift is a frequency-domain characteristic, whereas time is a time-domain measurement. When working with things like filters, it's generally of more interest to know the phase-shift of the signal at a certain point in the response curve than the time-delay.
Phase-shift is also critical when determining the stability of system with negative feedback, as the time-delay of the loop does not directly relate to its stability, whereas the phase-shift does, such as in a Bode plot where the gain and phase margin is shown.

Why then are most material in electrical engineering characterized in terms of phase shift as opposed to time delay?

Consider a 1st order RC low-pass filter of arbitrary cut-off frequency. At exactly 45° phase shift on the output compared to the input, the signal power is halved: -

It's also the point where both resistance and capacitive reactance are equal in magnitude. Image above is from electronics-tutorials with my additions in red.

So, consider the lack of clarity if instead of phase angle, we used time delay (dependent on the cut-off frequency). So many important relationships in all filters are only recognizably useful (and self-evident) when we use phase angle and not time delay.

Another example is the phase shift of a 2nd order low-pass filter. At exactly 90° is the pole frequency. This is also the point where the Q of the filter defines the output amplitude. Fundamentally linked to phase shift irrespective of frequency: -

Image modified from my basic website.

Anything other than phase shift is obscuring the fundamental working of the filter. Nothing useful to be gained by using time delay.

Stability for control

All closed-loop control needs negative feedback for stability. That is, if you get some outside push on your system, then your control action has to push in the opposite direction. If you push in the direction of travel, as you do with a swing, then like a swing the movements away from the "settled" position get bigger and bigger, which is the definition of instability.

Phase shift gives you a mechanism for describing how all these signals line up, for pushing in exactly the same or opposite directions (zero degrees or 180 degrees respectively), or as in reality, somewhere in between. And using phase shift, you can more easily tell where the system is going to start going unstable.

With that information, you can then work out what's the maximum frequency which the system can handle without going unstable. You then know where to limit your inputs to make sure it doesn't happen. Or if you need to work at a frequency, you know that you need to do something else to address this - perhaps adding mechanical damping instead, or perhaps figuring out a way to reduce the time delays.

There is a term for time delay, it's called group delay.

A simple delay corresponds to a constant group delay for all frequencies or a linear phase shift. This only describes a subset of systems.

More generally, systems can have different delays for different signals. That is why the more detailed phase response is needed.

Why the signal delay time is called the phase shift is because it makes sense for input signals that are sine waves.

The bode plot that describes how the system changes the input sine wave specifes the output amplitude and phase in regard to the original input.

A lot of the time we don't know and/or care about the actual time delay, only the phase delay (mod \$2\pi\$) associated with it. Imagine I have a high-power microwave or rf system, and I need to know whether the field strengths resulting from combining two signals are going to be so large that they ionize the air inside the waveguides and cause arcing and damage. If they're in phase, the field strengths will add together. If they're out of phase, they'll subtract. If I ask you to solve this problem and you come back and tell me that one signal has 112.1 ns of delay at 2.4 GHz and the other has 97.4 ns at the combining point, then technically the relevant information is there but I will tell you to go back and find the phase difference this corresponds to. If you then come back and tell me it is 221.7 radians I will ask if you are being annoying on purpose, and tell you to give it mod \$2\pi\$. If you come back and say it's 1.76 rad then at that point it's probably on me for not specifying that you should also give it as a fraction of pi, \$\Delta\phi = 0.56\pi\$. Now I know that they're somewhere between constructive and destructive, and can go straight to working out resultant maximum fields for given input amplitudes.

But if you were to tell me that the signal going through the filter will be delayed for 2 seconds, then I could appreciate it better.

I suspect you're thinking of a delay line rather than a filter.

As you point out, it takes some time for a signal to propagate over some distance, and that would best be described as a time-shift or delay. In that case, each frequency component (ideally) is delayed by the same amount of time which means that the phase-shift is linear in frequency (a sinusoid with twice the frequency of another will be phase-shifted by twice the phase).

But, as other answers pointed out, filtering is different. For example, we can say with certainty that a 1st order low-pass filter has a phase shift of \$-45^\circ\$ at the corner frequency \$f_c\$ independent of \$f_c\$.

However, the time delay \$\tau_c\$ at \$f_c\$ depends on \$f_c\$:

$$\tau_c = \frac{1}{8\cdot f_c}$$

So, if I told you that the signal going through this filter will be delayed by \$\tau_c\$ seconds and nothing else, I'm quite certain you wouldn't appreciate it better.

What hasn't been said is that your concept of delay is much like that of an echo. You say something, then wait, then the echo repeats your speech. If you play music through an echo, it makes each note seem to happen twice. In these cases, the echo and the original are completely separated.

Where things are different is if this delayed signal is occurring while the original is still going on. The original and the delayed one will mix together and produce some very interesting effects. You get a single result that is different from either part. When this is happening, you are less interested in how much time separates the two, and rather what part of the original signal is combining with which part of its replica.

If we know, for example, that we can take a sound and cancel it out by adding in a replica that is "upside down", what we are really saying is that as the waves go up and down we want to take a copy where it is at the lowest part of the wave, and combine it with the original right where it is at the highest part. Where do these occur? We can find out once we know the frequency. The frequency tells us how much time passes between the top and the bottom of the wave. And that means that if we try this at a different frequency, we will have to do the calculation again to get the new time delay. It's more convenient to say, "Find the high a low points. They are halfway through the cycle." That statement works for any frequency, and the actual time delay needed can be calculated as necessary.

That "half a cycle" is a phase determination, and so how much of the wave has passed becomes described as a fraction of a cycle, and that's what they end up calling the phase angle.