This is not something that varies by operating system. It varies by FTP server software.
What counts as a real person user account in Unix is non-trivial to determine programmatically. Impersonal user accounts can both have non-empty shell fields in the system account database that point to real executables (e.g. uucico) and valid, existing, home directories (e.g. /var/spool/news).
The convention employed by at least two non-anonymous FTP servers is that a user account is a real person for FTP purposes, and can thus perform FTP login, if the shell field in its record in the system accounts database is non-empty and has a value that can be found in the system "user shells" table (/etc/shells for one of those two, per the getusershell() library function in the case of the other).
Clearly, as one can tell from the name alone, the /usr/local/bin/ftponly shell is taking advantage of this in order to have user accounts that execute what is probably a very simple emit-a-message-and-log-off program when the user attempts a terminal log-on, but that work for FTP log-on.
nologin will do the same job if it is in the user shells table. But this is specific to the FTP server software that you happen to be using. Another FTP server software on the same operating system will have quite different rules.
Bruce Guenter's twoftpd, for example, tests the value of the shell field against two configuration settings. If it matches one, the client gets read-only FTP service with the twoftpd-anon program. If it matches the other, the client gets file-upload-only FTP service with the twoftpd-drop program.
Even the software that you may be using might have complex configuration options that affect this. In vsftpd, for example, the check_shell option, the local_enable option, the pam_service_name option, and the PAM configuration for that service (including the presence and configuration of the pam_shells module) are all involved.
Further reading
