You want faster? You have to make a custom kernel. Ort you can just one that somebody made.

Basically all the junk is ripped out and you are left with basiscs. THen you add what you need.

[Look at this site][1] for a Raspberry Mini IMage

Also one of the first ones I ever saw is [Damn Small Linux][2] but I don't know if its compatible with ARM. 


 [1]: https://www.linuxsystems.it/raspbian-wheezy-armhf-raspberry-pi-minimal-image/
 [2]: http://www.damnsmalllinux.org/