UPDATE January 21: On January 7 the NetDepot datacenter experienced a power outage on the B feed in their Atlanta data center. This caused a number of servers to abruptly lose power and go offline. Once power was restored all but one of our servers successfully rebooted. The server that failed was one of two […]
UPDATE: The server performed an automatic disk check when rebooted. The disk check is complete and the server should be online shortly. The miami server failed to reboot after a kernel update this morning. We are investigating the cause now.
UPDATE @ 2:05 a.m. EST: Hypervisor 1 is online. UPDATE @ 1:56 a.m. EST: The datacenter has confirmed that this is a network issue and is working on resolving it. Hypervisor 1 is not accessible. It’s being reported as down by the nagios monitor, but some of the sites on the hypervisor are still accessible, […]
One of the RAID drives on the cherokee server is degraded and needs to be replaced. The drive replacement began at 5:30 a.m. EST and is expected to take no longer than one hour.
There is a network issue at the datacenter that’s affecting Internet connectivity to the host1 and superior servers. The servers are up they are just not accessible via the Internet. We are waiting for a response from the technicians at the datacenter regarding the exact issue and ETA on resolving the issue.
UPDATE @ 2:47 p.m.: The /home partition became read-only again before we could begin moving sites. We are moving the backup drive from the old server to the new server and will begin restoring sites from the backup drive. The host2 server has crashed. It was rebooted and is performing an automatic disk check. Once […]
Earlier today we sent an email stating that the mackinac server needed a file system check on the /usr partition. After examining the drive, the technician at the datacenter suggested that we check the entire drive. I have given him the go-ahead for this. The file system check for the entire drive should take about […]
UPDATE @ 3:20 p.m. — The issue has been resolved and access to the servers has been restored. UPDATE @ 2:54 P.M. – One of the core routers failed, they are replacing it. No ETA on how long it will take to replace the router. We are currently experiencing a network outage that’s affecting connectivity […]
One of the drives in the RAID Array in the host2 server has bad blocks on the /var and /home partitions. After investigating we have decided to replace this drive immediately. This is drive is different from the drive that was replaced on March 9. Downtime is expected to be 30 – 45 minutes.
UPDATE: The server will be taken offline to replace the degraded hard drive and the back plane cable that connects the drives to the RAID card. Downtime is not expected to exceed 30 minutes. UPDATE: The file system check completed and the server rebooted. Unfortunately, after 10 minutes the file system became read only again […]