I did more digging, and it looks like this has been a pretty typical problem throughout the versions of CyanogenMod. Not sure if it's a bug or I just lack understanding of the problem.
I was able to connect using ssh -i and specify the address of the key. When I had set the -v flag, it checked the key I was using and said it didn't match. So I'm thinking at best ssh ships so it's not set up correctly, and at worst something is broken.
I made an alias so that I don't have to type out -i, then the address, then the other options I want, then the user and hostname every time I want to connect to my home server.
