A variant of this problem was addressed recently in Kubernetes, so it’s worth looking at what was done there. (The variant is whether to use iptables-legacy or iptables-nft and their IPv6 variants to drive the host’s rules.)

The approach taken in Kubernetes is to look at the number of lines output by the respective “save” commands, iptables-legacy-save and iptables-nft-save (and their IPv6 variants). If the former produces ten lines or more of output, or produces more output than the latter, then it’s assumed that iptables-legacy should be used; otherwise, that iptables-nft should be used.

In your case, the decision tree could be as follows:

• if iptables isn’t installed, use nft;
• if nft isn’t installed, use iptables;
• if iptables-save doesn’t produce any rule-defining output, use nft;
• if nft list tables and nft list ruleset don’t produce any output, use iptables.

If iptables-save and nft list ... both produce output, and iptables isn’t iptables-nft, I’m not sure an automated process can decide.

A variant of this problem was addressed recently in Kubernetes, so it’s worth looking at what was done there. (The variant is whether to use iptables-legacy or iptables-nft and their IPv6 variants to drive the host’s rules.)

The approach taken in Kubernetes is to look at the number of lines output by the respective “save” commands, iptables-legacy-save and iptables-nft-save (and their IPv6 variants). If the former produces ten lines or more of output, or produces more output than the latter, then it’s assumed that iptables-legacy should be used; otherwise, that iptables-nft should be used.

In your case, the decision tree could be as follows:

• if iptables isn’t installed, use nft;
• if nft isn’t installed, use iptables;
• if iptables-save doesn’t produce any rule-defining output, use nft;
• if nft list tables and nft list ruleset don’t produce any output, use iptables.

If iptables-save and nft list ... both produce output, and iptables isn’t iptables-nft, I’m not sure an automated process can decide.