Did DOS on the early, slow machines ever fail to handle interrupts timely, and what happened if it did?

There are many false assumptions how it worked.

First of all, if an interrupt is currently executing, it cannot be normally interrupted by another interrupt, for two reasons. When an interrupt is entered, it clears the CPU interrupt flag so further interrupts are disabled even if the interrupt controller signals that new interrupts are pending, and the interrupt controller will not even signal about new interrupts until software has acknowledged the current interrupt as being handled, so other pending interrupts can be signaled to CPU.

So, if a higher-priority interrupt did occur during execution of lower priority interrupt, the current handler was most definitely not pushed anywhere and the higher-priority interrupt was not serviced, it was kept pending until current interrupt is handled.

Also, there were few interrupts that DOS needed to handle. In fact none, as it was the BIOS that handled interrupts, and DOS simply used the BIOS via standard API. A 8088 DOS machine such as IBM 5150 basically had three hardware interrupts in use by BIOS, the system tick timer (IRQ0), keyboard (IRQ1) and floppy controller (IRQ6). There were possibility for interrupts like UARTs and parallel port, but they were not used by BIOS and DOS used BIOS for accessing these.

Now, if you run any DOS program, it can do whatever it wants such as install IRQ handlers and enable them, so what happened when programs did that is up to how correctly their programmers implemented these things.

One thing for sure, video at that time did not generate any interrupts, it was not wired to do so on CGA or MDA. EGA and VGA allowed for that, but it was extremely rarely used because it was not mandatory and different implementations from different companies may differ and people might have other adapter cards already installed on the same IRQ line so it might not be possible to use it.

What happens if an interrupt is missed depends on what it is.

So, basically, if you had a 8250 UART communications program that installed a custom interrupt handler for receiving data from a serial port, if some other interrupt takes too long to execute, you will get an interrupt pending about character being received, and if that's not handled in time, another character may come in and when the interrupt finally gets executed, the code needs to ask UART what is the reason for the interrupt. OK, there's a status flag for character being received. But there's also a status flag for overrun error, which means that one or more bytes received is lost. The program can then do anything it wants with that info.

For DOS, and BIOS, it's much simpler. If a timer interrupt occurs, and they are not handled in time so multiple timer interrupts occur, then the interrupt will run only once it has time and tick count increases by one, so time ticks can be lost. If a keyboard interrupt occurs, keyboard interface is kept on hold so keyboards cannot send more data until interrupt is handled (on a IBM 5150). If a floppy operation is being executed, basically the PC is anyway waiting for the interrupt to be signalled about finishing execution, and there's a timeout so the operation will fail and is cancelled/ignored and considered as an error (maybe there was no floppy in the drive).

So if a buffer overflows, or stack overflows, or whatever, it's up to how the program should work with the hardware and what happens if something goes wrong.

Also, CPU does not see serial transmission at bit level, but byte level, so a 1000 bps transmission will have a byte interrupt at 100 times per second (assuming 8N1 framing). And the CPU running at 4.77 MHz does not mean it runs 4.77 million instructions per second, as most instructions take several clock cycles to execute.

Given that any program can execute CLI and thus disable interrupts for as long as it likes -- yes, it is entirely possible to fail to respond to interrupts in a timely manner.

The effect depends on how the particular hardware reacts to not being able to get its interrupt serviced.

In 1992 I created a pair of DOS TSRs (Terminate and Stay Resident programs) to share modems across a Novell network. One TSR was installed on the PC that had a modem, and it handled the 16550 UART interrupts. The other went on to a PC that wished to use one of the shared modems. Then a terminal program could be used to request an available modem and transparently use it as if it was connected to that PC. I got it to work well, which was an achievement over 30 years ago, with basically no tools, no way to debug, and only two books from the local bookstore to give me any info.

Both TSRs had to be able to stream bidirectionally through IPX, the Novell packet interface. It is basically equivalent to UDP. I constructed a simple streaming protocol with sliding windows, retransmission and send credits, a lot like TCP. I didn't use SPX because the additional driver took up too much memory on the anemic PCs that the customers used. But I didn't have any problems with lost interrupts in the networking code, as best I know. A lost interrupt would mean a chunk of data missing, and the streaming method would request a retransmission.

The UART handling code was less forgiving. The 16550 has a 16 byte buffer, and lost characters could not reasonably be requested again. Some characters would just go missing, and in the middle of an image, that was pretty damaging. The TSRs overall seemed reliable except in one particular situation, and I was not able to determine the cause. Still it was a useful system and also easy to use. I thought about modifying it to allow sharing printers, but I think Novell introduced that at some point, or Windows For Workgroups did or something.

The only case where DOS would hang was when I was developing the IPX handling and there was a missing line of code in the book I was using. After rebooting a hundred times and having no conceivable way to debug a memory-resident driver with no user interface, I went back to the bookstore to get another book. Added one line, and it worked.