Vietnam suspends its search as Malaysia's air force chief denies saying the flight was tracked hundreds of miles off course.
In that sentence, saying is a verb.
George denies saying "Rumpelstiltskin"
simply means:
George claims that he didn't say "Rumpelstiltskin"
This could be used to explain a context where the subject is claiming that he said something else:
George: "I never said Rumpelstiltskin! I said Abracadabra!"
or it could mean the subject is saying that someone else said "Rumpelstiltskin":
George: "I never said Rumpelstiltskin! Linda said it!"
or it could mean the subject is simply refuting the fact that he said it.
George: "I never said Rumpelstiltskin!"
In the sentence from the news story, the air force chief is denying that he said that the airplane had flown off course. This indicates that, at one time, people believed that the plane had flown off course, and that this information had come from the air force chief. The official is trying to disassociate himself from that rumor.
The verb deny often takes an -ing verb:
The cyclist denied taking steroids. ("I have never taken performance enhancing drugs.")
The customer denied ordering a pizza. ("I never ordered a pizza! You must have the wrong address.")
The mayor denied having an affair. ("I don't even know that woman.")
but it doesn't have to:
The cyclist denied the rumor, and the mayor denied the allegation.
Had the word saying been functioning as a noun instead of a verb, chances are it would have been preceded by an article.
Mary denied the saying, "An apple a day keeps the doctor away." (this means she is disputing the truth of the saying)
Mary denied saying, "An apple a day keeps the doctor away." (this means she denied saying that out loud)
As for how to solve these mysteries on your own, try replacing the word in question with both a noun and a verb, and see which sentence makes more sense. That might help.