MavEtJu's Distorted View of the World - 2005-09

Posted on 2005-09-13 14:08:27, modified on 2006-01-09 16:29:23
Tags: Happiness, Networking, Voice over IP, DHCP

64 bytes from 10.192.14.243: icmp_seq=25 ttl=127 time=0.532 ms
64 bytes from 10.192.14.243: icmp_seq=26 ttl=127 time=0.546 ms
64 bytes from 10.192.14.243: icmp_seq=27 ttl=127 time=0.564 ms
64 bytes from 10.192.14.243: icmp_seq=28 ttl=127 time=0.564 ms
64 bytes from 10.192.14.243: icmp_seq=29 ttl=127 time=0.570 ms
64 bytes from 0.0.0.0: icmp_seq=34 ttl=249 time=0.420 ms
64 bytes from 0.0.0.0: icmp_seq=35 ttl=249 time=1.807 ms
64 bytes from 0.0.0.0: icmp_seq=36 ttl=249 time=1.283 ms
64 bytes from 0.0.0.0: icmp_seq=37 ttl=249 time=1.036 ms
64 bytes from 0.0.0.0: icmp_seq=38 ttl=249 time=2.385 ms
64 bytes from 10.192.14.243: icmp_seq=39 ttl=249 time=0.764 ms
64 bytes from 10.192.14.243: icmp_seq=40 ttl=249 time=2.500 ms
64 bytes from 10.192.14.243: icmp_seq=41 ttl=249 time=0.507 ms
64 bytes from 10.192.14.243: icmp_seq=42 ttl=249 time=2.383 ms
64 bytes from 10.192.14.243: icmp_seq=43 ttl=249 time=0.612 ms
64 bytes from 10.192.14.243: icmp_seq=44 ttl=249 time=2.373 ms

Posted on 2005-09-09 11:06:32, modified on 2006-01-09 16:29:23
Tags: Computers, RAID

Friday morning 07:05, I get a phone call. Problems with the database server. It gets lots of timeout on disks, but it doesn't show up on the controller as bad. Annoying, but nothing we can do about it. The problem is however that timeouts on a database server aren't making the users happy, so we figure out a way to find out what how to determine which disk is broken: Take out one disk of the RAID5 array, and see if it comes back. In theory a good approach. Now reality: Take out first disk, acknowledge this change in the RAID5 controller. Machine is coming back up, fsck fails again with lots of timeouts. Put disk back, take out second disk, acknowledge this in the... in the... oh shit.

By putting back the first disk and taking out the second disk at the same time, we broke the array. We should have put the first disk back, acknowledge it on the RAID5 controller and then take out the second disk. Oh oh oh oh oh...

Let's restore from backups...