Those were two turbulent days in the history of Name.ly we hope never to experience again. We were scaling the system up to offer better performance to our visitors. When everything was running smoothly and we were ready to hand the job to the marketing guys, suddenly the database RAID failed. We had the option to proceed with the immediate backup recovery however since RAIDs usually offer a solid option of the data recovery we opted in for the latter.

Having spent a day with four specialists trying to rebuild the dead array we did not manage to run the new RAID. For the sake of data consistency we spent another day trying to recover the data on the low level from one disk that seemed to be still alive. The other pair was unfortunately lost during the crash. This only resulted in 2/3 of the files recovered. Hence we had no other option than to proceed with the latest database backup.

The good news is that all uploads were already stored on the other RAID, hence they are completely unaffected (pictures, avatars, site backgrounds). The loss in question is only the resent changes (settings, posts and comments and latest registrations) made during the last few days.

We would like to apologise for the down time and all the inconveniences caused. The server uptime still stays above 99.5%. With new backup mechanism in place we and our users are now better protected.
Should you need and assistance with your blogs, please let us know on info at Name.ly.

Short link: Copy - http://name.ly/~0c59$An

Posted on Wednesday, June 1st, 2011 at 1:19 pm.
Filed under: Announcements.