This is an unfortunate situation, but your failure was correct.
First and foremost:
I didn't have enough knowledge on the subject to decide whether it deserved an upvote or not, so what should I do in this situation in the future?
The emphasis is mine and is the key point here. This means you should skip, skip, skip and skip some more.
When judging questions, you don't always need to be well versed in the topic to decide whether or not to upvote as voting on questions in part is based on the apparent quality, effort, research, and conformance with the question asking guidelines. But when reviewing answers, it is much more difficult to vote appropriately when you don't understand the topic. How do you know the answers the question or contains technically accurate information?
From your commentcomment on RobW's answer:
Isn't determining if an answer is valid is separate from determining if it is good? For instance, if I don't know anything about language Z, but someone posts a detailed answer to a question about language Z, I can at least asses that it worthy of being on SO. I should have to pass on it because I don't know about language Z, and I shouldn't have to vote on it
The tooltips on the voting buttons I think answer this.
The tooltip on the question voting buttons say:
- Upvote: This question shows research effort; it is useful and clear
- Downvote: This question does not shows any research effort; it is unclear or not useful
Whereas the tooltop on the answer voting buttons say:
- Upvote: This answer is useful
- Downvote: This answer is not useful
Just based on the tooltips, there is a significant difference in the voting guidelines for questions and answers. For questions, you are encouraged to judge the quality on clarity and research effort, in addition to usefulness. However, for answers, you are encouraged to vote on usefulness only.
From this idea, you should not be voting on apparent quality when reviewing answers, nor are you supposed to be judging research effort or clarity. You aren't assessing appropriateness for SO, especially in the First Posts and Late Answers queue. You are assessing the quality of the answer based on the usefulness. And how is a detailed but factually incorrect answer useful?
Now if the post was clearly inappropriate (spam, offensive, a "Thanks" or "Me too" post, or another question), then you should flag it for moderator attention, but just because it is not inappropriate doesn't mean you should take no action.
By voting "No Action Needed" you were indicating the answer was of sufficient usefulness not to downvote, but was not good enough to upvote. Or if you encounter a decent quality post that you don't think deserves more upvotes. Neither situation appears1 to be the case here (the reason for the latter is if it were a review, you probably would have seen the post with just a 0/+1 score and not it's real score.
1 - I can't say that the post was actually useful myself, but judging by the quantity of upvotes, it would appear to be quite useful.