Like @steve-robillard@jacob suggested pip install --no-cache-dir matplotlib helped to avoid the out of memory error.
However, matplotlib did not build because 'libpng' is not available on the raspberry and cannot be fetched by pip. In this case sudo apt-get install python-matplotlib worked just fine.