I think the problem is not multiple layouts, but the image processing you are doing in the different layouts.
If you set large bitmap to smaller imageView it would take too much memory. So it is better to put image tasks on different thread, for that you can use some good image library like "Picasso" or "Universal Image Loader".
For More optimization tips about recycler view, you can find hereClick here