Since users are pretty good at finding the weak point in any design, I'd recommend just using a buffer for each channel (yes that's 6 total opamps but they're cheap). For example, what if a user wants to temporarily disconnect your high impedance equipment to check the sound using their nice low impedance headphones?
High output impedance (compared to a low impedance buffered output) will make your transmission line more susceptible to noise, so if you're using high end equipment you're not doing yourself any favors by skimping on the output stage as long as you choose a decent quality opamp.
My recommendation would be to move your master potentiometer to the output of the summing stage (instead of R9 and R10), and splitting that into three different paths to feed each buffer opamp circuit. I think it would be a good idea to put a medium sized resistor (10k??) between the master potentiometer wiper lead and each buffer input too, that way the buffers don't interact with each other. Currently it looks like your master volume knob can only go from (all inverted) unity gain to a gain of 3 which doesn't seem like enough on the low end.