Tuesday evening the 10Gbe switch that connects the 3 production Proxmox Hosts, the production NAS, and the Proxmox backup server overheated and stopped working. This had the unfortunate side-effect of taking everything down and corrupting much of the VM data on the NAS. This is the story of the 3 hour recovery process.
My production home lab is not fully redundant in the sense that an enterprise cluster would be. I only have a single 10Gbe unmanaged switch and each production machine has only a single 10Gbit NIC with a single port. All of the hardware lives on a shelf in my living room rather than an air purified data-center or closet.
The switch in question is a Netgear Unmanaged 8 Port 10Gbe switch which I’ve had for about 6 years. It along with the fact the NAS isn’t a dual controller NAS, is a singe point of failure for the Production cluster. The Proxmox Hosts rely on it to communicate with the NAS for Virtual Machine storage. When it dies its like ripping the hard disk out of a computer while its powered on.
The switch failed because it was filled with dust and over heated. I spent about 45 minutes to open it up and carefully clean all of the dust off of the components and case. After cleaning it out and re-assembling it, The switch was in working order again.
I tried firing everything up and noticed that the speed I was getting from the NAS to the VM was excruciatingly slow. The NAS throughput can max out the 10Gbe Connection easily which translates to about 1GB/s of transfer. The speed after the switch came back up was 80MB/s on average.
I checked the network speed and it was fine. I noticed that the IO wait on the NAS was 80-90% which is odd. I checked the ZFS status and it was good. I ran a zfs trim command and that did not improve things after it completed. I figured something was corrupt or broken since ZFS was good SMART metrics on the SSDs were good, and the network throughput and latency were good.
I spent a few hours running backups of all of the VMs to the Proxmox backup server in the hopes that I could avoid any data loss between 11PM the previous day and 7PM Tuesday. The backup worked for all but one VM. That VM was an emergency use Virtual Desktop thus less important.
I then re-formated the ZFS pool on the NAS. Created the NFS Shares and started restoring the VMs. The restore process took roughly 20 minutes. All of the VMs restored and started up. Some had to repair filesystem errors but all was well in the end.
All in all the lesson I take from this is that regular maintenance to keep the switch clean is highly important. Future plans to get a second switch and dual port 10Gbe nics for the 3 physical hosts, the NAS, and the Proxmox backup server will be a good investment in improving redundancy and preventing data loss.
Special thanks to Tufts Lynx for streaming while I did all this work. Helped keep me in good spirits during the process.
