Use a lower-threshold PMOS. The FDN306P is a decent one that can work with 3.3V logic. See here: https://www.onsemi.com/pdf/datasheet/fdn306p-d.pdf

You can control it directly with a 3.3V GPIO, or keep the n-FET if you need the inversion.