Background info first. The server that this occurs on has only been on for less than a year and includes an external SAS JBOD. This server was intended to take on the duty of an old web server, running Debian 8, and its dedicated storage server.
A standard Debian Buster installation was completed and after a basic set-up was done most of a LAMP stack, minus the DB portion due to a dedicated DB server, was installed from the repositories. Then ZFS was installed, also from the repositories, a pool set up to use all drives in the JBOD, and a ZFS filesystem set up. So far so good. Lastly I installed all software that was also on the old web server to keep any issues with the website from occurring.
Next came the configuration and after all but mirroring the OS configuration on the old server I copied all customized Apache configuration files from the old server to the new. Then the SSL and SSH certificates and keys were copied with refreshes planned shortly after everything was verified as working. Then an initial copy of the website contents, including the stuff on the data server, was completed. Lastly some minor tweaking of the apache configs had to be done, mostly because the data server stuff was now in a new home, which boiled down to replacing /mnt/xxx with /zfsroot/xxx. After all was done a reboot was performed and the site was live, at least within the local network.
All of the website people checked the site over and found no issues within a browser. It all seemed to work great but before I made the switch an issue was brought up. The old data server was within a secondary DMZ while the web servers all lived in the primary DMZ. Access to all primary DMZ servers are via a single non-root account on any one server with SSH keys and passwords changed monthly. Opening up the main web server to base user access to add data and modify the site on the fly was considered too much of a security risk so the old data server had to be used for now. This server uses NFSv3 over a dedicated link, literally just a CAT6 cable between two interfaces, to provide the website with the necessary data while keeping the users off the main server. So I prepped the interface and the mount points and changed the apache configs to point back to them. Lastly I did a final rsync between the old web server and the new, changed the IP to use the old servers IP(for firewall and NAT reasons), powered down both web servers, moved the data server cable, and powered the new one up. Then the problems started.
First it was an apparent issue with the number of apache processes hitting the upper limit. The new server was more powerful than the old so I calculated out a new upper limit and set it. Then that problem happened again and my first step was to restart the apache2.service which gave the error shown in the title, "Failed to get properties: Transport endpoint is not connected." I tried to reboot the server, using both systemd reboot and the good old shutdown -r, and they both failed. So I devised a decent shutdown procedure ending with some sysrq keys and that worked. Looking around the system logs I found a myriad of NFS errors, see this question, but nothing else. Note in that question everything went down after upgrading from Debian 8 to 11. That happened after the new server had been set up as a possible fix and I hyper-focused on it.
Could NFS be causing this? Maybe the Apache2 configs? I don't know and the searching that error message gives me next to nothing. Any assistance is appreciated.
