I did more digging, and it looks like this has been a pretty typical problem throughout the versions of cyanogenmod. Not sure if it's a bug or I just lack understanding of the problem.
I was able to connect using ssh -i and specifying the address of the key. When I had set the -v flag, it checked the key I was using and said it didn't match. So I'm thinking at best ssh ships so it's not set up correctly, and at worst, something is broken.
I made an alias so that I don't have to type out -i then the address then the other options I want then the user and host name every time I want to connect to my home server.