I did more digging, and it looks like this has been a pretty typical problem throughout the versions of cyanogenmod. Not sure if it's a bug or I just lack understanding of the problem.

I was able to connect using ssh -i and specifying the address of the key. When I had set the -v flag, it checked the key I was using and said it didn't match. So I'm thinking at best ssh ships so it's not set up correctly, and at worst, something is broken.

I made an alias so that I don't have to type out -i then the address then the other options I want then the user and host name every time I want to connect to my home server.

I did more digging, and it looks like this has been a pretty typical problem throughout the versions of CyanogenMod. Not sure if it's a bug or I just lack understanding of the problem.

I was able to connect using ssh -i and specify the address of the key. When I had set the -v flag, it checked the key I was using and said it didn't match. So I'm thinking at best ssh ships so it's not set up correctly, and at worst something is broken.

I made an alias so that I don't have to type out -i, then the address, then the other options I want, then the user and hostname every time I want to connect to my home server.