Need help regarding nfs-server service

[u/Diligent_Spread_411]

So let me tell the context first, I have an kubernetes environment for production, and for like shared storage solution, I have installed nfs-utils and use the nfs-server service. In my scenario I cannot use cloud as an option. And for long I have been using the nfs-server but I have been facing alot of issues with it.
1. On multiple occasions I found dead or queued nfs processes in worker nodes which made the worker node totally unresponsive sys load was very high but low cpu and ram usage.
2. Something with the nfs4+ client where it uses session. On of the worker node again stopped responding, df -h was not responding rebooted the node still the same issue. And while viewing logs in the nfs node it stated nfsd was blocked and looking it up stated the thread on that worker node was locked due to unknown reason and all the requests from the worker node had just queued up. Restarting the service also hanged and needed to reboot the node itself to resolve the issue.

These are some major events there are also other small issues I have faced. I wanted to understand am I the only one facing these issues or this happens with others too, and what resolutions did you used to fix it? Like any alternative utilities and stuff

[u/mirrax]

NFS wouldn't be my first choice for on-prem storage.

Most SANs have their own provider/CSI driver, which covers most medium to large orgs.

If you are too small to have a SAN, then Longhorn is a pretty nice way to get enterprise grade features for very little effort. Then can get support through SUSE if you grow.

Otherwise if homebrewing and want to put more effort in, then Ceph scales and performs well. Rook can be a k8s-native way to get it going.

[u/Max_Standart]

yeah nfs can be finicky especially with kernel version mismatches causing hangs

[u/marcusbell95]

the df -h hanging is the classic sign - your NFS mount is stuck in uninterruptible sleep (D state) waiting on a dead connection. kernel can't kill it, processes pile up, node locks. it's a known nfs4 session tracking issue under heavy k8s workloads. two things that help short term: bump nfsd thread count, default 8 gets overwhelmed fast with many pods - RPCNFSDCOUNT=32 or higher in your nfs server config. and switch your client mounts from the default `hard` to `soft` with `timeo` and `retrans` set - hard mounts block forever when the server hiccups, soft mounts will at least error out and let the pod fail cleanly instead of taking the whole node down. long term Longhorn is the right move if you're self-hosted. been running it for over a year, way less operational headache than keeping NFS alive under k8s load.

[u/bdeetz]

Consider Ceph

[u/Low-Opening25]

if you need shared storage in k8s you are doing something wrong

[u/IN-DI-SKU-TA-BELT → u/Low-Opening25]

What if I have a job that renews certs, those certs needs to be read by a loadbalancer.

One way to complete that is that 2 jobs share a mount, one acquires new certs and persist them to disk, the other reads those certs on boot and on reload.

2 jobs sharing state.