First: For spam, you can cast a flag instead, there is a specific flag for 'spam - Exists only to promote a product or service'. 

For downvotes, the user being downvoted gets -2 reps, the person voting down gets -1. If the answer / post is then deleted, all reputation is given back.

The reasoning behind the person voting down losing a point is somewhat given for example [here][1]. This is designed to make people think twice before downvoting, to not just do it on reflex or as a "revenge".

Also, from [here][2]:

> The motivation behind it is to put emphasis on up-voting or not voting at all. This way, down votes will carry more weight and it will also prevent users from abusing the system by down-voting excessively.

>According to what Jeff/Joel discussed on the SO podcast, they wanted to find a way to discourage users from down-voting for less legitimate reasons (say a pro-Java developer down-voting everything remotely related to .NET or the like). 


 [1]: https://meta.stackexchange.com/questions/7322/should-the-weight-of-downvotes-be-increased
 [2]: https://meta.stackexchange.com/questions/15863/why-do-you-lose-reputation-for-down-voting