From the National Center for Biotechnology Information:
... while simply overhearing a language during childhood could help adult learners speak it with a more native-like phonology, speaking a language regularly during childhood could help re-learners use it with more native-like grammar as well as phonology.
The study was done on fourteen-year-old students of Spanish who had varying degrees of exposure to the language as children; some learnt it, others merely heard it.
So it would seem that the amount of prior exposure does make a difference, at least, for children.
A study in the Psychological Science magazine suggested the same for adults where we can read:
As it turned out, even though the volunteers showed no memory of the second language in the vocabulary test, they were able to quickly relearn and correctly identify phonemes that were spoken in the neglected language.
This was done with prior learners of Hindi or Urdu who had stopped learning the language entirely.