Broken hard disk in the web server

Locked
User avatar
thenktor
Salix Wizard
Posts: 2426
Joined: 6. Jun 2009, 14:47
Location: Franconia
Contact:

Broken hard disk in the web server

Post by thenktor »

Hi,

you've probably noticed that during the last few days our website (especially forum and wiki) was down. This was caused by a broken hard disk, which also made the RAID unusable. After exchanging the hard disk and recovering the RAID our MySQL databases for forum and wiki still have been corrupted and could not be repaired. So I had to take the last database backup to get things working again.
Of course this means that every entry/change in forum and wiki that was made after the backup is gone. We are talking about a maximum of 1 day.

Hopefully everything is back to normal again.

Sorry for the inconvenience
Image
burnCDDA (burns audio CDs)
geBIERt (German beer blog)
User avatar
witek
Posts: 233
Joined: 16. Nov 2009, 13:41
Location: Poland.Łódź

Re: Broken hard disk in the web server

Post by witek »

I have so far never used RAID even though I have a small network with several computers and servers, as I was afraid if I was able to configure it properly and had no time to try. I just make regular network cross-backup from one machine to another and vice-versa. I wanted to try RAID someday at last but if it did not help you to restore the database that means it is just useless (or you misconfigured something).
User avatar
mimosa
Salix Warrior
Posts: 3311
Joined: 25. May 2010, 17:02
Contact:

Re: Broken hard disk in the web server

Post by mimosa »

Hmm yes I've wondered what the point of raid is, since someone told me it isn't that you use the redundancy to restore your data on total or partial disk failure ... which surely ought to be theoretically possible? Algorithm: where the two remaining disks agree, you have your data, where not, it is corrupt. Actually I can see it would be nice to have four disks, precisely to allow for a major failure. Then you would go with two out of three.
User avatar
thenktor
Salix Wizard
Posts: 2426
Joined: 6. Jun 2009, 14:47
Location: Franconia
Contact:

Re: Broken hard disk in the web server

Post by thenktor »

witek wrote:I wanted to try RAID someday at last but if it did not help you to restore the database that means it is just useless (or you misconfigured something).
Both drives in the RAID1 are connected to a single controller on the main board. In our case the electrical failure of one disk made access to both disks unstable and very slow, but not totally unusable. Even after marking one drive as faulty and removing it from the RAID the access to the remaining good drive was not ok again. The faulty disk had to be completly removed first. So actually it's no surprise that there is data corruption in files that have been written during this period.
Why do we have a RAID then? Because after exchanging the faulty disk I finally could reassemble the RAID without reinstalling the whole system. The RAID is not the solution for everything, there are probably 1000 reasons why a hard disk fails. Our failed drive also was not logged in SMART. That's why you always should have a backup, too ;)

PS: I guess using an own SATA controller for every drive would be better. Or using a more bulletproof hardware RAID controller. But we don't have these options.
Image
burnCDDA (burns audio CDs)
geBIERt (German beer blog)
Locked