Up to now, I always upgraded my FreeBSD computers via the mini-ISO cdroms provided by the FreeBSD team. Last week, due to circumstances, I had to upgrade them via cvsup and to build the new OS from scratch. A small step for me, a huge step for ME!
At BarNet, there are about eight remote computers. On uninhabited places, like in cupboards, behind the copier, three meters above the floor and so on. Not easily reachable :-)
Also, the computers were running a huge range of OS versions, ranging from FreeBSD 4.2 to FreeBSD 4.9. With matching the OS version with the date the OS version was released you could figure out when the machine came into operation.
And the last thing was that we are going to provide ADSL backup features, which means that when the primary link fails, the traffic will go via a slower link. And zebra, the routing daemon we were going to use, doesn't really work with FreeBSD versions lower than 4.6.
Was that now so difficult? No. So why have I struggled in the last seven years with cdroms? Must be because I had never done it before.
Of course things can go wrong. And will go wrong.
During one night I had xterms open to all machines which had to be upgraded (erm... all of them), running the make buildXXXX. With -j4, the load on the machine will swing between 5 and 10. We use nagios to monitor the systems, and it complained that their loads was too high. And since I'm not the only one receiving these alerts, I got phonecalls saying "what's wrong, what's wrong?". Inform your fellow people before you start.
First machine I upgraded went without a glitch. The second machine I did didn't come back up. So far for a foolproof solution. When I went to the console (next day early in the morning): fsck had failed. With other words, the harddisk is broken.
Two machines which had been given more networks cards recently didn't come back properly. It seemed the network cards were configured properly, but and the ifconfig_ statements added to /etc/rc.conf, but they had their (old, obsolete) "network_interfaces" variable in /etc/rc.conf not updated. A manual ifconfig and editing of /etc/rc.conf fixed it. (Don't forget to reboot afterwards to see if it works again, or at least restart the DHCP daemon since that daemon doesn't automaticly grabs new interfaces).
From now on, no more binary upgrades anymore for me, just cvsup and make! Yay!| Share on Facebook | Share on Twitter
|Posted on:||2004-01-31 16:21:51|
|Comment||You could have built the kernel and system binaries on one machine (preferably the fastest), and distributed them across the network.
That would've saved many phone calls I can imagine. :)
Nice story however. I wish I could remember the first time I played with CVSup.