Any method that relies on the crawler's good behaviour, may fail, so the best option is to use the strongest force/authority available, in this case, the web server itself. If you have access to the main web server configuration or at least to the `.htaccess` file, you should use a method that involves those elements.

The best way is using http password, but if you really don't want to use that, then you still have another option.

If you know the IPs of your clients, you can restrict/allow that in your .htaccess with a simple code like this

 Order deny,allow
 Deny from all
 Allow from x.x.x.x
 Allow from y.y.y.y

The IPs can be in the form `x.x.x` instead of `x.x.x.x`, which means that you will be allowing the whole block that is missing.

You can combine that with some HTTP headers. 403 tells the bot to not go there, they usually try a few times, just in case, but it should work quickly if combined with the deny directive.

You can use the HTTP response code even if you don't know your client's IPs.