At some level, you implicitly invest trust When you make the decision to use any system which is connected to the Internet, your implicitly trusting that system's operating system to do the 'right thing'. After all, it has full access to anything you do on that device. Likewise, when you decide to use an application on that system, you are implicitly trusting that application to do the right thing. Even if you don't use the address book and manually enter contact details, like a phone number, each time, you are trusting it doesn't send details to an unauthorised 3rd party.
The challenge is in determining what level of trust to assign to an operating system or an application. This may seem challenging, but it is really just an extension of something we do all the time and the same general guidelines can be used. We make decisions about trust every day. When you send your car for maintenance, you are trusting that the mechanic will do the 'right thing', when you visit your doctor, you are trusting they will do the right thing, each time you drive on the road, your trusting that other drivers will do the right thing. The techniques we use to determine what level of trust we assign in these activities use the same principals we should apply when making a decision about an application. In general terms, this comes down to reputation, legislation and what could be referred to as intuition.
Reputation. When you select a mechanic or a doctor, you might ask around and see what others think of them. You might check their reputation with a reputable industry body, accreditation authority or ask others whose opinion you trust. The same approach can be used for applications - see what the industry reviews have to say, talk to other users, review what their privacy statement is. The extent you go to will depend on a combination of how you assess the risk i.e. what is the likelihood that those who wrote the app will do the right thing and what would be the consequence to you if they don't. You will also mediate this by your own level of risk comfort or appetite - some people are very risk adverse and are uncomfortable with relatively low levels of risk, others have a greater appetite and are willing to accept more risk. One of the reasons that things like the Apple app store has been so popular is that supposedly Apple does some vetting of the apps in the store, giving you some increased confidence that at least Apple thinks the application is moderately OK. This in itself is not a guarantee, but it is a higher guarantee than you get downloading and installing an app from some random web site you know nothing about.
Legislation. In many cases, there is varying levels of legislative protection which might increase your level of trust. You might verify a mechanic, a doctor etc has obtained minimum levels of qualification to practice, has mandatory insurance or meets minimum legislative compliance requirements, is licensed to operate or in the case of driving, has a valid license. Sometimes, as is the case with driving, you can assume the other drivers on the road are licensed and know the road rules on the basis that law enforcement monitors the situation and has penalties for those who don't which are sufficient to minimise the risk you will come across an unlicensed driver who does not know the road rules. You have no guarantee, but you can be reasonably confident that the likelihood is low.
Intuition. This is your overall gut feeling. You might reject a mechanic or a doctor despite having verified they have a good reputation, have al the necessary accreditation, licenses, insurance etc, simply because you don't feel comfortable with them, don't like their manner or just feel somehow uneasy about them. Trust this intuition. The same goes with applications. You have to trust them at some level and you need to feel comfortable about that trust.
Are you being overly paranoid? Yes and no. On the whole, we probably should all be a little more paranoid. We shouldn't just download some app and start using it. We should investigate it first - find out what sort of reputation the app and the vendor have, talk to other users, check their website and look at their privacy statement and terms of use, think about the likelihood and consequences of the app doing the wrong thing. Depending on the app, there may be legislation or accreditation which may help (for example, some countries have legislation which require service providers to disclose details of security incidents, guarantees about how data is manged, shared and disposed etc. There are various associations which require their members to obey certain ethical codes in order to be a member which might have a baring on the level of trust your willing to hand over etc.
Where you may be getting a bit paranoid is in thinking about encryption and other mechanisms to try an improve your level of privacy protection. Most of these types od ideas are not vary practical and the level of inconvenience they impose is usually unacceptable (though this will depend on individual circumstances). For example, encrypting your address book is is unlikely to help. At some point, the application will need to know the decryption key in order to use the address book. Once you provide that key, the app has full access. Yes, you could encrypt every entry with a different key, but then you would have to provide a unique key to the app for every entry and you would need to remember all of those keys.
Currently, for most people, it is still too hard to easily assess an arbitrary application. This is especially true for new applications which have not yet developed any real reputation you can use. Likewise, there are as many bad, difficult to verify accreditation schemes out there as there are good ones (look at the Madison Ashely site - it had banners all over the place claiming accreditation from various security and other 'bodies' claiming it was secure and safe).
I think one of the best assessment rules is still "there is no such thing as a free lunch". Basically, if it sounds too good to be true, then it probably is. Software tends to take a lot of time to develop, which means a lot of money. People who do this are generally looking for some sort of reward. Sometimes, it can be reputation, but more often than not, it is financial. If you cannot see what the financial basis or operating model is i.e. how can they afford to do this?, then you should be suspicious that perhaps the app is making its money some other way, such as selling your data to a 3rd party. Avoid installing applications from random web sites you know nothing about, especially if the software is free. Be wary of applications which provide a service that could have questionable ethical standards - for example, any application which claims to help avoid paying a fee, provide free access to a service which you normally need to pay for, etc. If the stated objective of the app sounds a little unethical, it is likely those who created it have similar ethics (or lack of them) and you probably don't want to trust them with your data.
Privacy is not a binary proposition - it isn't a simple 'it is' or 'it isn't'. It has to be considered in context. What we really need is increased disclosure, not increased encryption. Encryption has an important role, but far more important is having sufficient information to allow us to make an informed choice. We should know what a vendor does with our data, how they collect it, who they share it with, how they store it and how/when they destroy it. Vendors need to treat our data with more respect and tell us what they are doing. With this information, we can then make the decision as to whether the convenience is warranted and within what each individual is willing to accept.