I used to use a spreadsheet/text file per project (ToDo comments in the code don't scale well for the reasons you list; they're local to the code, and if there's an issue that's not, it tends to slip through the cracks).
Recently I've set up a Redmine server on my home network. It's a bit heavyweight for a "team" of one, but I'm working on quite a few projects on my own time, and tend to just use the Issue Tracker + Repository options with maybe the odd wiki page in more complex places.
A friend of mine swears by Pivotal Tracker for the same purposes, but my current employer uses Redmine internally, so I figured this would give me some practice. It's not bad.
For open source projects, I would probably just use GitHub's issue tracking.