Trying to remount a set of nfs folders onto a server that was restarted, am now getting "access denied by server" errors. On client server (clientserver.co.local) I ran:

[root@clientserver ~]# mount -t nfs -vvvv 172.18.4.97:/datalake/raw/org /datalake/org/raw/ mount.nfs: timeout set for Wed Dec 30 19:41:35 2020 mount.nfs: trying text-based options 'vers=4.1,addr=172.18.4.97,clientaddr=172.18.4.98' mount.nfs: mount(2): Permission denied mount.nfs: trying text-based options 'vers=4.0,addr=172.18.4.97,clientaddr=172.18.4.98' mount.nfs: mount(2): Permission denied mount.nfs: trying text-based options 'addr=172.18.4.97' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 172.18.4.97 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: trying 172.18.4.97 prog 100005 vers 3 prot UDP port 20048 mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting 172.18.4.97:/datalake/raw/org 

and got the error you see above. (Notice it tries all the different versions of nfs and still fails).

I ran tcmpdump to monitor the packet traffic during the mount command (based on the advice here), but have no idea how to interpret the logs (could post something like last 10 lines if that would help).

Checking the mounts on the network from the hosting nfsserver.co.local server on the client, I saw:

[root@clientserver ~]# showmount -e Export list for clientserver.co.local: [root@clientserver ~]# showmount -e 172.18.4.97 Export list for 172.18.4.97: /datalake/raw/org/HI_BRFSS clientserver.co.local,otherclient.co.local /datalake/raw/org clientserver.co.local,otherclient.co.local /datalake/analytics/org clientserver.co.local,otherclient.co.local [root@clientserver ~]# service nfs status Redirecting to /bin/systemctl status nfs.service ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled) Active: active (exited) since Wed 2020-12-30 18:32:09 HST; 11min ago Process: 93274 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS) Process: 93271 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS) Process: 93266 ExecStop=/usr/sbin/rpc.nfsd 0 (code=exited, status=0/SUCCESS) Process: 93307 ExecStartPost=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi (code=exited, status=0/SUCCESS) Process: 93290 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS) Process: 93288 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Main PID: 93290 (code=exited, status=0/SUCCESS) Tasks: 0 CGroup: /system.slice/nfs-server.service Dec 30 18:32:09 clientserver.co.local systemd[1]: Starting NFS server and services... Dec 30 18:32:09 clientserver.co.local systemd[1]: Started NFS server and services. 

So everything looks like how I would think it's supposed to (showmount does show the nfs folder I am trying to mount).

After running the mount command, the what I see in the /var/log/messages stream file is just a bunch of messages like

Jan  4 18:37:12 clientserver gssproxy: gssproxy[2557]: (OID: { 1 2 840 113554 1 2 2 }) Unspecified GSS failure.  Minor code may provide more information, Client 'host/[email protected]' not found in Kerberos database 

Not sure what this means, but did check the gssproxy.conf file and it shows

[root@mclientserver ~]# cat /etc/gssproxy/gssproxy.conf [gssproxy] 

Not sure what this means either, since I don't recall ever interacting with this in the past (when the nfs mount still worked).

We do use SSSD (did not set this up) to link our Windows AD accounts to the machine, but IDK if that would even be related here or if this is just something else. In any case, the sssd.conf is shown below

[root@clientserver ~]# cat /etc/sssd/sssd.conf [sssd] domains = co.local config_file_version = 2 services = nss, pam [domain/co.local] ad_domain = co.local ad_server = adserver.CO.local ad_server_backup = adserverbackup.CO.local krb5_realm = CO.LOCAL realmd_tags = manages-system joined-with-samba cache_credentials = False enumerate = true id_provider = ad krb5_store_password_if_offline = True default_shell = /bin/bash ldap_id_mapping = False use_fully_qualified_names = False override_homedir = /home/%u access_provider = ad 

But other than that nothing in the logs seems to be more info than what I was already seeing from the command error...

[root@clientserver ~]# grep mount -rnw /var/log/messages* -e "nfs" grep: mount: No such file or directory /var/log/messages:2782:Jan 4 17:21:23 clientserver kernel: FS-Cache: Netfs 'nfs' registered for caching /var/log/messages:2844:Jan 4 17:21:24 clientserver mount: mount.nfs: access denied by server while mounting nfsserver.co.local:/datalake/analytics/org /var/log/messages:2845:Jan 4 17:21:24 clientserver mount: mount.nfs: access denied by server while mounting nfsserver.co.local:/datalake/raw/org /var/log/messages-20201227:3590:Dec 23 17:46:04 clientserver kernel: FS-Cache: Netfs 'nfs' registered for caching [root@clientserver ~]# [root@clientserver ~]# [root@clientserver ~]# [root@clientserver ~]# grep mount -rnw /var/log/messages* -e "mount" grep: mount: No such file or directory /var/log/messages:2380:Jan 4 17:20:55 clientserver kernel: XFS (dm-3): Ending clean mount /var/log/messages:2530:Jan 4 17:21:07 clientserver kernel: XFS (sda1): Ending clean mount /var/log/messages:2537:Jan 4 17:21:07 clientserver kernel: XFS (dm-5): Ending clean mount /var/log/messages:2844:Jan 4 17:21:24 clientserver mount: mount.nfs: access denied by server while mounting nfsserver.co.local:/datalake/analytics/org /var/log/messages:2845:Jan 4 17:21:24 clientserver mount: mount.nfs: access denied by server while mounting nfsserver.co.local:/datalake/raw/org /var/log/messages:2846:Jan 4 17:21:24 clientserver systemd: datalake-org-analytics.mount mount process exited, code=exited status=32 /var/log/messages:2847:Jan 4 17:21:24 clientserver systemd: Failed to mount /datalake/org/analytics. /var/log/messages:2850:Jan 4 17:21:24 clientserver systemd: Unit datalake-org-analytics.mount entered failed state. /var/log/messages:2851:Jan 4 17:21:24 clientserver systemd: datalake-org-raw.mount mount process exited, code=exited status=32 /var/log/messages:2852:Jan 4 17:21:24 clientserver systemd: Failed to mount /datalake/org/raw. /var/log/messages:2853:Jan 4 17:21:24 clientserver systemd: Unit datalake-org-raw.mount entered failed state. /var/log/messages:3014:Jan 4 17:21:27 clientserver dracut: Executing: /usr/sbin/dracut --hostonly --hostonly-cmdline --hostonly-i18n -o "plymouth dash resume ifcfg" --mount "/dev/mapper/centos_mapr001-root /sysroot xfs defaults,x-systemd.device-timeout=0" --no-hostonly-default-device -f /boot/initramfs-3.10.0-862.6.3.el7.x86_64kdump.img 3.10.0-862.6.3.el7.x86_64 /var/log/messages-20201227:3669:Dec 23 17:47:35 clientserver systemd: mapr.mount mounting timed out. Stopping. /var/log/messages-20201227:3823:Dec 23 17:47:37 clientserver systemd: Unit mapr.mount entered failed state. 

I can ping the nfsserver machine by both name and IP address from the client (and vice versa from the nfsserver machine).

Looking on the nfsserver server, I see:

[root@nfsserver ~]# cat /etc/exports /datalake/analytics/org otherclient(rw,no_root_squash,sync) clientserver(rw,root_squash,sync) /datalake/raw/org otherclient(rw,no_root_squash,sync) clientserver(ro,root_squash,sync) /datalake/raw/org/HI_BRFSS otherclient(ro,no_root_squash,sync) clientserver(ro,root_squash,sync) [root@nfsserver ~]# exportfs -rav exporting otherclient.co.local:/datalake/raw/org/HI_BRFSS exporting clientserver.co.local:/datalake/raw/org/HI_BRFSS exporting otherclient.co.local:/datalake/raw/org exporting clientserver.co.local:/datalake/raw/org exporting otherclient.co.local:/datalake/analytics/org exporting clientserver.co.local:/datalake/analytics/org [root@nfsserver ~]# systemctl status nfs ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled) Active: active (exited) since Wed 2020-12-30 18:38:00 HST; 22min ago Process: 135417 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS) Process: 135414 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS) Process: 135412 ExecStop=/usr/sbin/rpc.nfsd 0 (code=exited, status=0/SUCCESS) Process: 135447 ExecStartPost=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi (code=exited, status= 0/SUCCESS) Process: 135430 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS) Process: 135428 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Main PID: 135430 (code=exited, status=0/SUCCESS) CGroup: /system.slice/nfs-server.service Dec 30 18:38:00 nfsserver.co.local systemd[1]: Starting NFS server and services... Dec 30 18:38:00 nfsserver.co.local systemd[1]: Started NFS server and services. 

So again, everything seems to be configured as needed. (I have seen some answers that recommend a specific configuration for /etc/exports but I'd prefer to keep as is and this configuration was working up until now).

Anyone with more experience know what could be going wrong here? Any further debugging advice / info to add that would make this question better?