One of the groups is not setup as a security group in Active Directory.

Answer: One of the groups is not setup as a security group in Active Directory.

This still applies to all versions of SharePoint and is most common for nested sub-groups.

To verify: Contact the team which maintains AD or check yourself using a tool like ldp: Ldp Overview.

Then, change the group to be a security group!

Other issues:

If you cannot find the group in the people picker at all

1. Check it is a security group
2. Check that you can find a different group in the same domain
3. If not able to do #1 and #2 above, you will need to verify you have configured the server to search the domain you are expecting. See Configure People Picker in SharePoint 2013.