Am I being just being paranoid? Is this a legitimate concern?
I think you probably being paranoid. Or mercenary. Or both.
You are worrying about something that is highly unlikely to have any significant inherent value1.
You commented thus:
Does that really matter? The inherent value is the same if it's four minutes or four days.
Erm ... this is not a realistic. A 2 line "hello world" program is not the same as something you spent days, weeks, months working on. But either way, the real value of this code (i.e. the potential for making money) to YOU is most likely zero. If you make a fuss about the IP issues, a prospective employer is likely take this as a sign of things to come.
And on the ethics side, of course they have a legitimate need to see how you implemented the code. How else are they going to assess the quality of your work? They also have a moral obligation to treat your code as your property. But you don't need legal agreements and stuff for something like this. It is simply a waste of everyone's time for a trivial amount of effort.
Footnote 1 - The exception would be is that if your solution to the "interview homework" problem included a large body of private code that you spent months / years developing, and that you have realistic plans to make money out of it. If that is the case you should not be using it in "interview homework" questions.
No employer wants to employ a programmer only to get into fights about who owns the IP rights for critical code that may-or-may-not have been written by said programmer outside of work hours. And if your effectiveness as a programmer depends on your use of a bunch of your own private code, then your employer will have major problems if / when you "move on".