gt24-2 4am issue

The gt24-2 node had an issue starting at apx 4:20am Central time. The load started spiking and the node and all vm’s started slowing down, until the point they became unresponsive. We’ve run updates on the OS and are still investigating what happened.

As the /var/log/messages shows nothing out of the ordinary up until it stopped, we’ll also be logged into the node to see if we can see what is occurring in real-time. It also means it the delay until notification will be gone. We’ll also be sending out a replacement server to the DC in case we need to migrate the VMs off of it. If it does happen again we’ll migrate a couple VMs off the server as well, but trying not to as the server they will be migrated to has less RAM, CPU and disk to begin with. If we do it’s a stop gap measure while the replacement is setup.