The documentation for gpg-agent says "gpg-agent uses an environment variable to inform clients about the communication parameters". It doesn't give the actual environment variable name, but some testing indicates that the relevant variable is GPG_AGENT_INFO. I would suggest changing your mutt config to the following:
source "unset GPG_AGENT_INFO; /usr/bin/gpg -d --quiet ~/.mutt/.passwd.gpg |" A perhaps simpler solution would be to pass --no-use-agent to gpg.
Note that this doesn't prevent an attacker from calling gpg with the agent enabled, if your credentials have been cached (perhaps because you let gpg-agent cache them from outside of Mutt). So, even if gpg-agent knew to discriminate against mutt specifically, it wouldn't be a very strong security measure.