You can't; it is linked with the X and Gtk+ client libraries, and I don't think it's even possible to build it without them.
Even in the Firefox headless version, GTK+ is still needed
see: https://bugzilla.mozilla.org/show_bug.cgi?id=1372998GTK+ is still needed.