In sshd_config 'MaxAuthTries' limits the number of auth failures per connection. What is a connection?

In the case of SSH, a connection is one established connection to the sshd's TCP port (usually port 22). Once sshd stops accepting further authentication attempts, it closes the connection, and at this point the connection is done.

Before a user gets to make an authentication attempt, the SSH protocol requires the negotiation of encryption and other protocol options, establishment of session keys and exchange of host keys. So each new connection requires a non-trivial bit of work: a storm of SSH connection attempts from multiple sources could certainly be used to DoS a server.

An authentication attempt is one attempt of any authentication method that is currently enabled in sshd configuration. For example:

• if client offers a SSH key for authentication, each offered key counts as one attempt.
• if Kerberos/GSSAPI authentication method is enabled, seeing if the client can be authenticated with it counts as one attempt
• each password typed into the password authentication prompt obviously counts as one.

The first two can cause the situation you're experiencing: if you set MaxAuthTries to one and Kerberos/GSSAPI authentication is enabled, it may eat up the single attempt before you even get to try password authentication. Likewise, if your SSH client has an authentication key available, but you haven't added your public key to the destination system's ~/.ssh/authorized_keys for the destination user, the public key authentication attempt will eat up your single attempt and you won't even get to try password authentication.

pam_unix, the PAM library that normally handles password authentication, enforces a delay of two seconds after a failed authentication attempt by default.

If your primary threat is password-guessing worms and bots on other compromised systems in the internet, reducing MaxAuthTries may be a bad move: since a bot won't tire, it will always reconnect and try again. Each attempt requires you to spend some CPU capacity for SSH protocol negotiations. You'll want to primarily ensure that the bot won't succeed, and secondarily that the bot will waste as much of its time as possible on that one existing connection, with minimum cost to you. Allowing multiple authentication attempts over one connection but answering... very... slowly... will do exactly that.

This is also why sshd will request a password from the client even if password authentication is completely disabled: the prompt is completely fake, and the client will be rejected no matter what password is entered. But the client has no way to know that for sure.

Of course, if you allow too many authentication attempts over one connection, the bot may eventually terminate the connection from its side, if the bot programmer has implemented a timeout to limit the effectiveness of such a "tar-pit defense".

To be broad, a connection is defined as any attempt to connect as a certain user.

The pam_tally or pam_tally2 command is useful here depending on which distribution you are using. The following command will show you how many failures there have been for a certain user:

pam_tally --user=username 

To reset it for a certain user:

pam_tally --reset --user=username 

For all users:

pam_tally --reset 

To verify that it has been reset:

pam_tally --user=username 

Executing the command pam_tally or pam_tally2 with no options will show all users which have any failed attempts or who are locked out. Be sure to change to pam_tally2 in case you have that one.

This is a bit crystal ball gazing, but a connection is exactly that, and unless you explicitly set options around connection multiplexing and lingering (ControlMaster, ControlPersist) on the client, it should terminate immediately.

What I suppose is happening (and ssh -v can help you verify, look out for "Trying private key") is that you want to use password-based auth to that machine. However, your client likely has a key pair for passwordless authentication in ~/.ssh. Standard configs give preference to keys over password, so your ssh offers your key, the server doesn't like it, and that's the one failed try you're allowed. (Variation of the theme: maybe you have multiple keypairs setup and only one of those is authorized for your server.) You can use -oPreferredAuthentications=keyboard-interactive,password to override what auth mechanisms to offer: if the client stops suggesting it can do key-based, you should the password prompt, so you can log in to set the value on the server higher again.

As explained in @telcoM's answer, every public key that ssh offers is counted as an "auth retry" on the remote side. If you have multiple public keys ("identities"), ssh will offer all of them in turn, and if multiple keys are rejected, the remotely configured MaxAuthRetries value can be exceeded during a single connection attempt.

This happens in particular if you use an authentication agent like ssh-agent or gpg-agent. Unless configured otherwise, ssh will try every identity provided by the agent besides all Identity values from ssh_config.

The directive IdentitiesOnly yes can be used to avoid this behavior and try only public keys that are explicitly set for the given host with an Identity directive. This doesn't disable the agent entirely—for explicitly configured identities, ssh will still try to obtain the private key from the authentication agent.

If you have lots of different keys for different remote hosts, using IdentitiesOnly yes and explicitly configuring the Identity for every remote host is advisable. For hosts that don't accept public keys at all, you should set PubkeyAuthentication no.