Downtime Report: Feb 6th, 2015
Posted: 07 Feb 2015, 06:33
As you may have noticed we have had several hours of downtime today. Things are now back up and everything looks nominal.
For those that are curious, here's a rundown of tonight's events.
Whenever a TMWC member with access to the TMW server comes or goes, I make it a habit to take a backup snapshot of the TMW machine. Unfortunately today this did not work so well. The virtual machine management software glitched and appeared to have completely wiped TMW's disks. Ouch!
Thankfully after a considerable amount of high-stress work I was able to recover the data. It wasn't actually wiped from the physical disks, the logical volume manager's references to it had just been corrupted due to a kernel bug. Extracted the data from the storage array manually and cloned it back in to a new (unglitched) volume. Rock on.
I'd like to thank Nicotine and Caffeine for helping me through these tough times.
For those that are curious, here's a rundown of tonight's events.
Whenever a TMWC member with access to the TMW server comes or goes, I make it a habit to take a backup snapshot of the TMW machine. Unfortunately today this did not work so well. The virtual machine management software glitched and appeared to have completely wiped TMW's disks. Ouch!
Thankfully after a considerable amount of high-stress work I was able to recover the data. It wasn't actually wiped from the physical disks, the logical volume manager's references to it had just been corrupted due to a kernel bug. Extracted the data from the storage array manually and cloned it back in to a new (unglitched) volume. Rock on.
I'd like to thank Nicotine and Caffeine for helping me through these tough times.