The PCF8574 has what the data sheet calls a "quasi-bidirectional" output.
It basically means, it's an open-drain output with weak pull-up (and stronger momentary pull up for rise time acceleration, but that can be ignored for now).
So since it is not a push-pull output like on MCUs or other GPIO expanders, the 10k pull-down as DC load resistance is too much for it to drive.
What you could do is to change the IO expander to something that suits the purpose for best results. Or as a temporary solution, remove all the pull-downs, but the PCF857 will still initialize at power-up to all pins high.
And I just realized the MAX3004 is there between doing level conversion.
It is also intended for data application, instead of driving a capacitive FET gate and DC pull-down resistor. It too has strong momentary rise/fall time accelerator, which then turn off and the state is held very weakly. It is also not a suitable component for driving FETs and the pull-down resistor.
You would need a simpler level shifter, any logic IC that can see 3.3V inputs and drive 5V from supply will work.