Is there was any microprocessor older than Intel 8086 (e.g., Z80, Intel 8080) which can run The Microsoft Windows 1.0 and at least display the GUI without glitches?
No, as 8086 introduced a new instruction set - the very one Windows written for. There was no previous CPU with the same ISA. Sure, large parts of windows were already back then written in HLL, which would help to port it to some other (earlier) CPU. But I guess that's not what you had in mind.
Also, using software mods,
Patching won't do any good, as there is no prior CPU with that instruction set to execute it, so changing some instructions wouldn't do it.
Binary Modifying means translating opcodes to CPU's correct opcodes,
Binary translation is a wide field. A static translator might be possible, but virtually impossible as it would have to be extreme tailored to Windows code base - almost more work than rewriting.
A dynamic translator is of course as well viable. But that's in the end much like running an emulator - which of course can be done with any CPU.
or changing hardware settings to match correct CPU, if there was no non-supported instruction in code.
As said, the whole instruction set was new, so there is no partitial one that can be used and extended via some trapping.
Any modification of hardware to be compatible with new software, but no modifications to CPU, while adding customized hardware to cover features of new hardwares which is essential to run Windows is not allowed.
Somehow this makes no sense. Is additional hardware allowed or not?
In the end all of this only leaves an Emulation setup.
Also, all those somewhat over specific restrictions blur a bit the point that it's not just the CPU that defines an execution environment.
Last but not least, what exactly is the timeframe asked? Really pre 8086? Building an emulation setup providing PC like hardware and speed will be tight but possible.
Then again, there was no Windows until 1985. So a realistic base date would be (pre-)1985, which would provide speeds more than enough to match a 4.77 MHz 8088 :))