I run a small webserver that hosts a handful of domains which I use for personal and testing purposes. I want to set it up as a mail server as well, and I'm trying to understand the rDNS lookup that checks against the SMTP HELO header.
The problem is that I have no control over the PTR records for my server's IP address, and there's no chance the ISP would delegate that to me. Let's say my IP address is 123.45.67.89. Then the PTR record for my IP resolves to a hostname like 123-045-067-089.customers.my-isp.net.
TL;DR -- I'm trying to understand what rDNS domain is being looked for in the SMTP HELO header? Is it the domain name of the email's sender (e.g. [email protected]) or the hostname of the mail exchange server even IF the server has a completely different domain name than the sender?
If I were to use the ISP's hostname of my IP address (e.g. 123-045-067-089.customers.my-isp.net) as the HELO header** for emails sent from mydomain.com, would that validate emails from [email protected] sent by my mail server at IP address 123.45.67.89, or does the PTR record for 123.45.67.89 need to resolve to mydomain.com?
**I could also obtain a TLS cert for 123-045-067-089.customers.my-isp.net, and have the MX record and TXT-spf record for mydomain.com point to 123-045-067-089.customers.my-isp.net
Long version:
Here's my very novice understanding of how the SMTP HELO check against rDNS works. Let's say my domain was mydomain.com, and I wanted to send an email from [email protected] to [email protected]. I'd first connect my desktop client to my mail server. The mail server (at IP address 123.45.67.89) would then connect to the mail server at gmail.com. In this scenario, let's say the HELO header listed mydomain.com as its domain name. So the gmail server would then do rDNS lookup on my server's IP address, only to discover that 123.45.67.89 resolves to hostname 123-045-067-089.customers.my-isp.net. Because this does not match the domain provided in the HELO header, the gmail server assumes this is spam and rejects it.
So far so good? Or no?
Now, let's suppose instead of putting mydomain.com as the SMTP HELO header, my mail server instead puts 123-045-067-089.customers.my-isp.net, and furthermore has a TLS certificate for that domain signed by an established CA. Also, the MX and TXT-spf records for mydomain.com point to 123-045-067-089.customers.my-isp.net as the established mail server.
(Note: It's not clear to me whether the SMTP HELO also dictates the domain of the sender's email???)
In this second scenario, would the email being sent from [email protected] to [email protected] be validated by the gmail server and recognized as a legitimate email? Or would it still fail because 123-045-067-089.customers.my-isp.net provided in HELO does not match the email sender's mydomain.com. (again... it's not clear to me whether this is even possible with the SMTP protocol... I'm very new to email servers)
Of course (this should go without saying...) I DO have full control over all of the DNS forward zone records for the domains I own. Also, I do have a stable long-term IP address, and all of my domains (and sub-domains) are configured to resolve back to my established IP address.