This may also help:
When you are configuring your cloud front distribution and you look at what your origin is set to (click your distribution and then click "Origins" tab) it will give you a dropdown of resources pointing to a s3 bucket(s).
When you select a bucket, if that bucket has static site hosting enabled it will recommend you use the bucket site url instead. Don't do this. 
What I have found works better is if you stick with the original bucket arn, select the option that says "Origin access control settings (recommended) Bucket can restrict access to only CloudFront.", and then click "Create a Control Setting" and stick with the default configuration it provides. 
When you goto save this new origin, AWS will prompt you to copy a policy that you can just copy into your bucket ACL and everything will work.
Note: This is just an addition to Kokaubeam answer which is still a necessary evil. The above just helps with keeping from opening up the static site url to public. Additionally, this is for the case where you want to keep your static site url available as well instead of just accessing through cloud front. Ideally, you should disable static site hosting and then follow these same steps.