One of the nice things about Linux is that if a certain feature you need isn't available, you can write or create your own. The obvious drawback, of course, is you need to be a programmer with some mad skillz in order to do that. Fortunately, there are a few of them out there, and here are some of the kernel patches I sometimes use and where to get them. I'm thinking of starting a kernel patch repository here because frankly there isn't one anywhere else!
Suspend2
Those of us with laptops know how nice suspending to disk can be. I use it every day. I standby my laptop before I leave the office, and when I plug it in back at home, I'm right back where I was half an hour before. This comes in real handy when you're reading something online and want to finish when you get home. Anyway, this is the patch that emulates Windows hibernation feature. Not a small or easy change to implement, but worth it in the end.
OpenMosix & OpenSSI
These allow you to turn all of the computers in your location into one big supercomputer. Sort of...Beowulf-style clustering is the most well known Linux parallel-processing hack. The problem with Beowulf is you have to write your software to directly support parallel processing. Your software does all the work, Beowulf just lets it do it. OpenMosix & OpenSSI do this at the kernel level, meaning you kernel moves your threads between machines. You don't get the level of performance you do with Beowulf, but for things like render farms, this comes quite in handy. Unfortunately, neither supports the 2.6 kernel yet, but OpenMosix is pretty close.
User Mode Linux
Esentially, this is a console-mode VMWare. It allows you to compile a special kernel that can be run ("booted") just like a regular executable. Good for testing software before you put it into production.
USB/IP Project
Easily one of my favorites. And the fact that it works on recent kernels so I can use it doesn't hurt much, either. :) Basically, this is just a driver that emulates a USB hub. The hub, however, transmits/receives all traffic to/from the USB device connected to over an IP-based network. In other words, connect a USB device to your machine, and any other machine with this driver can use it as if it were a local device. Imagine a USB-enclosed hard drive that everyone can mount and backup their stuff to. Or, a desk in your house with a server that lives underneath. You plug your scanner into it, and to use it you just take your wireless laptop over and start working. No plugging things in just to work. Sweet!
Tuesday, January 31, 2006
Thursday, January 26, 2006
A....P....C....DEAD!
I tell you, I've got to stop writing this damn thing. I've had more serious troubles with my server since I started writing about how to do stuff. It's almost like every time I make some progress, something else comes along to force my hand...
The other night, we came home from work to find the house dark. My house is automated with HomeSeer. I don't have a shitload of tasks setup, but the most important: turn on the outside and living room lights just before sunset apparently hadn't run. There's usually only one reason for that: loss of power to the server. The server's plugged into a huge surge protector* along with the TV and other electronic equipment in the living room, and a few weeks back I moved it from the floor to an inaccessible spot behind the equipment rack. Beeper, the cat, likes to sleep back there 'cause of the heater and it's pretty isolated. Unfortunately, the huge switch on the protector was too easily hit by the fat cat. If I couldn't get to my mail at any point in the day, I knew it was due to her. :)
But, as we moved closer to the house, we could hear the alarm beeping. Uh-oh. I pulled out my handy Husky pocket flashlight, and took a tour around the house, peeking in the windows and such. The house was secure, and I could see the clock flashing on the oven. Power outage. Fucking RG&E. Well, at least it wasn't anything serious.
Now, as the three people who read this blog know, I've got my drives setup in RAID arrays. But, that don't help much when your drives have become corrupted, or you corrupt the array yourself. I'll spare you the details because, frankly, I'm not 100% sure what I did, or why I had to do it. Suffice it to say, two hours later, I'd pretty much had enough with computers for life!
The next night, I ran to CompUOverpay and grabbed a 305va APC Back-UPS ES. I made sure it was supported under Linux before buying, of course. :) It's not a bad little UPS for $40. Considering you're slightly better protected from power surges, I'd recommend it as a good investment.
Anywho, fortunately setting it up is easy as pie. The first thing you need to do is install apcupsd. This is a reasonably simple install on pretty much any distro. On Fedora, it's as easy as "yum install apcupsd". Typically, you'd take the time to verify your UPS was being recognized by hotplug before bothering to setup the daemon, but I figured "fuck it". So far, Fedora's been pretty good at that stuff, so let's barrel on!
Even better than expected, the rpm containing apcupsd was already pre-configured for a USB UPS (prolly 'cause that's the most common kind now. Ya think?). So, for shits and giggles I typed "apcaccess" and was rewarded with tons of useful info!
APC : 001,034,0884
DATE : Thu Jan 26 16:10:35 EST 2006
HOSTNAME : someplace.oranother.com
RELEASE : 3.12.1
VERSION : 3.12.1 (06 January 2006) redhat
UPSNAME : someplace.oranother.com
CABLE : USB Cable
MODEL : Back-UPS ES 350
UPSMODE : Stand Alone
STARTTIME: Wed Jan 25 20:44:09 EST 2006
STATUS : ONLINE
LINEV : 120.0 Volts
LOADPCT : 68.0 Percent Load Capacity
BCHARGE : 100.0 Percent
TIMELEFT : 3.9 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME : 0 Seconds
LOTRANS : 088.0 Volts
HITRANS : 138.0 Volts
ALARMDEL : Always
BATTV : 13.5 Volts
LASTXFER : No transfers since turnon
NUMXFERS : 0
TONBATT : 0 seconds
CUMONBATT: 0 seconds
XOFFBATT : N/A
STATFLAG : 0x07000008 Status Flag
MANDATE : 2005-02-16
SERIALNO : XXXXXXXXXX
BATTDATE : 2000-00-00
NOMBATTV : 12.0
FIRMWARE : 00.e5.D USB FW:e5
APCMODEL : Back-UPS ES 350
END APC : Thu Jan 26 16:11:29 EST 2006
Yaay! (I took this at 4PM the next day, so that's why the battery's so well charged). I see I don't get a whole lot of time before I die, though. The drawbacks of using a dual-proc server. But, 4 minutes is more than enough time to gracefully shutdown the server and hopefully protect my data and such.
The first thing I need to address is the fact that I've got a W2K3/Exchange 2003 virtual machine running. That needs to be shutdown gracefully first to minimize damage to the database. I have a copy of GSX server, but unfortunately, the newest version of GSX doesn't support machines built with the newest version of Workstation. I've tried a couple of times to wedge it in there, but finally decided to wait for a new GSX. (I know, there are plenty of ways to do it, and I've tried a few with no success for various reasons. Don't bother, it's not that important at the moment). Well, here's the problem, once VMware came out with their "server" products, they removed the ability to shutdown machines gracefully at shutdown (you used to be able to put a line in the VMX file telling it to hibernate the machine on SIGHUP). Since it's a GUI app, I can't just script it, so I'd need a tool to do so, and I looked at a couple. None really did easily what I needed it to do (esentially: bring focus to that window, hit ctrl-Z).
Then, I remembered an easier solution: telnet. W2K3 includes a telnet server, and while I have it disabled by default, that's easy enough to change! So, I enabled and started the service and ran this on the Linux host:
autoexpect -f serversdn.exp telnet hostname
Expect is a nifty little scripting language with a specific purpose: automate other console apps. It's perfect for scripting a telnet session because you can tell it "wait for 'ogin:' and then send the username". Autoexpect simplifies this further. You tell it the name of the file to save your tasks to, and then the command you want it to run. When you're done, you have an expect script that needs no more than a tiny bit o' tweaking to get you up and running.
So, I scripted it to telnet into the server, shutdown the Exchange services** and then shutdown the machine:
Does it work? Oh, hell yeah it works! I had to do a little tweaking of the server first, though. On the first few passes, it took two minutes and fourty five seconds to shut down. Since I've got just under four minutes of battery power, that might not leave enough time to shut the box down. Fortunately, I've got a little experience with Winders, too...
Open regedit, and change the following:
The difference? The Exchange VM now shuts down in one minute and ten seconds. That's a whole lot better, huh?
Now, all I need to do is tell apcupsd what to do when the power goes out, and BOOM! everything shuts down easy as pie. This part's easy enough to figure out. Edit /etc/apcupsd/apccontrol and put your shutdown commands in the various case blocks.
I did a test run by pulling the cord on the UPS. Within a couple of seconds, I watched the VM shutdown and turn itself off. The Linux box then followed soon after without too much issue. I had to tweak the timings as the VM didn't entirely shutdown fast enough, but I think I've got it all set now.
Oh, one final step: go into your BIOS and look for a setting called "Restore on AC/Power Loss". Change it to "Full On" or "Power On". ATX-based machines don't automatically power back on, but changing this setting will make it happen. That way, if the power's only out for a short time, your machine'll be back up and running when you come back!
* I don't put a lot of stock in surge protectors. Even the best triacs used to clamp the circuit are generally not fast enough to stop a lightning bolt from killing Stevie and his siblings. However, I WILL generally spend the extra $10-20 and get a good one 'cause they usually come with guarantees that cover zapped equipment. :)
**This is a single machine acting as domain controller and Exchange server. In that combo, it's best to shutdown your Exchange services before you shutdown. If you take the machine down without doing that, it'll enter a race condition where it tries to shut the services down, but it can't query the domain controller properly because that's going down...the short of it is, in this condition, it can take 30-40 minutes for the box to shut itself down. I don't got that kind of time. Oh, and to prevent accidently doing it when I'm in the machine, I've removed the Shutdown command from the start menu via a policy and replaced it with a batch file that does it right. Where possible, always put a cover over the power switch. ;-)
The other night, we came home from work to find the house dark. My house is automated with HomeSeer. I don't have a shitload of tasks setup, but the most important: turn on the outside and living room lights just before sunset apparently hadn't run. There's usually only one reason for that: loss of power to the server. The server's plugged into a huge surge protector* along with the TV and other electronic equipment in the living room, and a few weeks back I moved it from the floor to an inaccessible spot behind the equipment rack. Beeper, the cat, likes to sleep back there 'cause of the heater and it's pretty isolated. Unfortunately, the huge switch on the protector was too easily hit by the fat cat. If I couldn't get to my mail at any point in the day, I knew it was due to her. :)
But, as we moved closer to the house, we could hear the alarm beeping. Uh-oh. I pulled out my handy Husky pocket flashlight, and took a tour around the house, peeking in the windows and such. The house was secure, and I could see the clock flashing on the oven. Power outage. Fucking RG&E. Well, at least it wasn't anything serious.
Now, as the three people who read this blog know, I've got my drives setup in RAID arrays. But, that don't help much when your drives have become corrupted, or you corrupt the array yourself. I'll spare you the details because, frankly, I'm not 100% sure what I did, or why I had to do it. Suffice it to say, two hours later, I'd pretty much had enough with computers for life!
The next night, I ran to CompUOverpay and grabbed a 305va APC Back-UPS ES. I made sure it was supported under Linux before buying, of course. :) It's not a bad little UPS for $40. Considering you're slightly better protected from power surges, I'd recommend it as a good investment.
Anywho, fortunately setting it up is easy as pie. The first thing you need to do is install apcupsd. This is a reasonably simple install on pretty much any distro. On Fedora, it's as easy as "yum install apcupsd". Typically, you'd take the time to verify your UPS was being recognized by hotplug before bothering to setup the daemon, but I figured "fuck it". So far, Fedora's been pretty good at that stuff, so let's barrel on!
Even better than expected, the rpm containing apcupsd was already pre-configured for a USB UPS (prolly 'cause that's the most common kind now. Ya think?). So, for shits and giggles I typed "apcaccess" and was rewarded with tons of useful info!
APC : 001,034,0884
DATE : Thu Jan 26 16:10:35 EST 2006
HOSTNAME : someplace.oranother.com
RELEASE : 3.12.1
VERSION : 3.12.1 (06 January 2006) redhat
UPSNAME : someplace.oranother.com
CABLE : USB Cable
MODEL : Back-UPS ES 350
UPSMODE : Stand Alone
STARTTIME: Wed Jan 25 20:44:09 EST 2006
STATUS : ONLINE
LINEV : 120.0 Volts
LOADPCT : 68.0 Percent Load Capacity
BCHARGE : 100.0 Percent
TIMELEFT : 3.9 Minutes
MBATTCHG : 5 Percent
MINTIMEL : 3 Minutes
MAXTIME : 0 Seconds
LOTRANS : 088.0 Volts
HITRANS : 138.0 Volts
ALARMDEL : Always
BATTV : 13.5 Volts
LASTXFER : No transfers since turnon
NUMXFERS : 0
TONBATT : 0 seconds
CUMONBATT: 0 seconds
XOFFBATT : N/A
STATFLAG : 0x07000008 Status Flag
MANDATE : 2005-02-16
SERIALNO : XXXXXXXXXX
BATTDATE : 2000-00-00
NOMBATTV : 12.0
FIRMWARE : 00.e5.D USB FW:e5
APCMODEL : Back-UPS ES 350
END APC : Thu Jan 26 16:11:29 EST 2006
Yaay! (I took this at 4PM the next day, so that's why the battery's so well charged). I see I don't get a whole lot of time before I die, though. The drawbacks of using a dual-proc server. But, 4 minutes is more than enough time to gracefully shutdown the server and hopefully protect my data and such.
The first thing I need to address is the fact that I've got a W2K3/Exchange 2003 virtual machine running. That needs to be shutdown gracefully first to minimize damage to the database. I have a copy of GSX server, but unfortunately, the newest version of GSX doesn't support machines built with the newest version of Workstation. I've tried a couple of times to wedge it in there, but finally decided to wait for a new GSX. (I know, there are plenty of ways to do it, and I've tried a few with no success for various reasons. Don't bother, it's not that important at the moment). Well, here's the problem, once VMware came out with their "server" products, they removed the ability to shutdown machines gracefully at shutdown (you used to be able to put a line in the VMX file telling it to hibernate the machine on SIGHUP). Since it's a GUI app, I can't just script it, so I'd need a tool to do so, and I looked at a couple. None really did easily what I needed it to do (esentially: bring focus to that window, hit ctrl-Z).
Then, I remembered an easier solution: telnet. W2K3 includes a telnet server, and while I have it disabled by default, that's easy enough to change! So, I enabled and started the service and ran this on the Linux host:
autoexpect -f serversdn.exp telnet hostname
Expect is a nifty little scripting language with a specific purpose: automate other console apps. It's perfect for scripting a telnet session because you can tell it "wait for 'ogin:' and then send the username". Autoexpect simplifies this further. You tell it the name of the file to save your tasks to, and then the command you want it to run. When you're done, you have an expect script that needs no more than a tiny bit o' tweaking to get you up and running.
So, I scripted it to telnet into the server, shutdown the Exchange services** and then shutdown the machine:
set force_conservative 0 ;
if {$force_conservative} {set send_slow {1 .1}
proc send {ignore arg} {sleep .1 exp_send -s -- $arg}# }
set timeout -1
spawn telnet server
match_max 100000
expect "login: "
send -- "administrator\r"
expect "password: "
send -- "easypass\r"
expect "Administrator>"
send "net stop MSExchangeIS /y\r"
expect "Administrator>"
send -- "net stop MSExchangeMTA /y \r"
expect "Administrator>"
send -- "net stop MSExchangeSA /y \r"
expect "Administrator>"
send -- "net stop WinHttpAutoProxySvc /y\r"
expect "Administrator>"
send -- "net stop HomeSeerService /y\r"
expect "Administrator>"
send -- "tsshutdn 0 /powerdown /delay:0\r"
interact
Does it work? Oh, hell yeah it works! I had to do a little tweaking of the server first, though. On the first few passes, it took two minutes and fourty five seconds to shut down. Since I've got just under four minutes of battery power, that might not leave enough time to shut the box down. Fortunately, I've got a little experience with Winders, too...
Open regedit, and change the following:
"HKCU\Control Panel\Desktop\AutoEndTasks" change from "0" to "1"
"HKCU\Control Panel\Desktop\WaitToKillAppTimeout" This one defaults to 20000 milliseconds, I believe. Change it to 2000.
"HKCU\Control Panel\Desktop\HungAppTimeout" Same as above.
Duplicate the above two entries for HKEY_USERS\.DEFAULT so it'll apply to new users as well.
Finally, change "HKLM\System\CurrentControlSet\ControlWaitToKillServiceTimeout" to 2000 as well.
The difference? The Exchange VM now shuts down in one minute and ten seconds. That's a whole lot better, huh?
Now, all I need to do is tell apcupsd what to do when the power goes out, and BOOM! everything shuts down easy as pie. This part's easy enough to figure out. Edit /etc/apcupsd/apccontrol and put your shutdown commands in the various case blocks.
I did a test run by pulling the cord on the UPS. Within a couple of seconds, I watched the VM shutdown and turn itself off. The Linux box then followed soon after without too much issue. I had to tweak the timings as the VM didn't entirely shutdown fast enough, but I think I've got it all set now.
Oh, one final step: go into your BIOS and look for a setting called "Restore on AC/Power Loss". Change it to "Full On" or "Power On". ATX-based machines don't automatically power back on, but changing this setting will make it happen. That way, if the power's only out for a short time, your machine'll be back up and running when you come back!
* I don't put a lot of stock in surge protectors. Even the best triacs used to clamp the circuit are generally not fast enough to stop a lightning bolt from killing Stevie and his siblings. However, I WILL generally spend the extra $10-20 and get a good one 'cause they usually come with guarantees that cover zapped equipment. :)
**This is a single machine acting as domain controller and Exchange server. In that combo, it's best to shutdown your Exchange services before you shutdown. If you take the machine down without doing that, it'll enter a race condition where it tries to shut the services down, but it can't query the domain controller properly because that's going down...the short of it is, in this condition, it can take 30-40 minutes for the box to shut itself down. I don't got that kind of time. Oh, and to prevent accidently doing it when I'm in the machine, I've removed the Shutdown command from the start menu via a policy and replaced it with a batch file that does it right. Where possible, always put a cover over the power switch. ;-)
Monday, January 23, 2006
Rules to entice Open Source adoption
Over the years, I've seen some pretty consistent mistakes done by a large portion of the open source community that have forced me to stay out of it. I think the biggest issue is open sourcers seem to think everyone's a mind reader and just KNOW everything there is to know about their product, and if you don't, you shouldn't be using Linux anyway. Well, if that's your attitude, fuck you and go somewhere else. It never ceases to amaze me that people will put their software out on the web for others to use, and then bitch when people ask them questions. For those that are interested in having people use their software, read on....
Rule #1. Screenshots should be useful and viewable. Screenshots go a long way to telling people about your software. They can see if it's laid out well, if it has the features they need in a way that makes it easy to use them. Sometimes it'll even tell you more about what the software does. Now, I know it's antithetical to the *nix philosophy of "GUI bad, command line GOOD", but too bad. If you don't want to look, don't look. More often than not, I can decide if a particular piece of software will work for me by just looking at it.
Now, that being said, please read this part: SMALLER IS BETTER! This is the other thing that blows my mind. Wanna piss off an open sourcer? Send them a mail in HTML format. You'll get so much crap about sending "bloated" mail. Then, what do they do? They take a screenshot of their entire 1600x1200, 32 million color desktop to show you their new tray widget. Three hours after it finishes downloading the screenshot you can then decide if you want to go further. Seriously, cut 'em down. Resize the app so it's a minimum size necessary to view the functionality, take a screenshot of it, and then put it through the Gimp to cut down on the colorspace, and perhaps compress it. Hell, crop out anything that's not your app while you're at it. It is not necessary to see every icon on your desktop in order to see the new word processor you wrote.
Rule #2. Docs are very useful ways of populating your website with useful information. I used to use LFS, and how often did I find a package that SEEMED to fit my needs, only to find it required hundreds of packages installed, or really didn't fit those needs at all? Too often. You wanna save me some time and just link your INSTALL and README files on your page? They're plain text, so they're not going to kill your space or bandwidth, and will save me a ton of time. I shouldn've have to download a package, untar it, and then go into an editor just to find out the prereqs for it.
Rule #2a. Know your prereqs. In an ideal world, every developer would have an LFS machine around so they know exactly what libs need to be installed in order for their software to work. There's nothing worse than having a long build appear to finish successfully, only to have the executable panic the machine when run. If they fail to test build it that way first...up against the wall!
Rule #3. About first, history second. The first thing on your website should NOT be a changelog. It doesn't help me to know that you "modified mallocs to use less memory" if I don't know what your software does. Tell me what it is, then put the history and changelogs elsewhere. If I'm interested, I'll look. And, a real-world example or two can sometimes go a long way to helping me decide. Occasionally, I'll come across a project and after looking at it for some time, still have no idea what it does or why I'd use it. I have, more than once, come across a project and then dismissed it only to be directed to it by another site that says "try this, you'll love it!" When I look again, I slap my head. I don't like slapping my head. While we're on this topic, put in an "English" changelog, too. Instead of the above entry, simply say "this version uses less memory".
Rule #4. Not everyone is a programmer. For the love of Cthulu, take pride in what you do! Not everyone can develop software. It's a gift you have, don't take it lightly! If someone asks a question, "look at the source" is not an answer unless they specifically hand you a code block and say "how does this work?"
Rule #5. Don't use Sourceforge. I'm sorry, but I hate SF. It wouldn't be so bad if I didn't have to fight my way through thirty different extremelt slow-loading pages just to download the tarball. Besides, no one uses SourceForge right, anyway. Think i'm being a dick? You find me ONE project on there that actually uses the "Docs" link. Most of the "Project Home Page" links point to an empty index, if you're lucky.
Rule #6. Your software really isn't that revolutionary, work with someone else. How many window managers are there now? The apptrove at Freshmeat tells me it's got 130 projects in that category. Is that really necessary? Really? You mean to tell me no one can create a basic, simple fucking window manager and then add functionality in via plugins? Beyond window managers, though, let's look at some others: instant messengers, how 'bout using a the same config file as one of the others? The information in there should be the same, so if I want to install Gaim and a command line IM, I can and not have to worry about how they're each configured. Would it really be that hard to keep my username/password for each network in just ONE location? For a group that goes nuts about some of the choices Microsoft has made with Windows, you dorks have sure made a lot more mistakes in terms of simplicity and flexibility!
See, the problem is too many choices is NOT a strength. You can spout your stale rhetoric about that all you want, but that don't make it true. Too many choices, especially the apples-to-oranges choices offered only make life more difficult. We have all of these tools and such, and that's great, but they all have different purposes and functionality. You can't just choose one over the other easily...especially when you want to choose on functionality, but you're limited in KDE-type apps or something, and what you want are only available in Gnome. Now I gotta install another fucking environment just to use YOUR app. No thanks.
Okay, this rant's over. Seriously, just make it easier on people. Complexity for complexity's sake is a stupid way to put your software out there.
Rule #1. Screenshots should be useful and viewable. Screenshots go a long way to telling people about your software. They can see if it's laid out well, if it has the features they need in a way that makes it easy to use them. Sometimes it'll even tell you more about what the software does. Now, I know it's antithetical to the *nix philosophy of "GUI bad, command line GOOD", but too bad. If you don't want to look, don't look. More often than not, I can decide if a particular piece of software will work for me by just looking at it.
Now, that being said, please read this part: SMALLER IS BETTER! This is the other thing that blows my mind. Wanna piss off an open sourcer? Send them a mail in HTML format. You'll get so much crap about sending "bloated" mail. Then, what do they do? They take a screenshot of their entire 1600x1200, 32 million color desktop to show you their new tray widget. Three hours after it finishes downloading the screenshot you can then decide if you want to go further. Seriously, cut 'em down. Resize the app so it's a minimum size necessary to view the functionality, take a screenshot of it, and then put it through the Gimp to cut down on the colorspace, and perhaps compress it. Hell, crop out anything that's not your app while you're at it. It is not necessary to see every icon on your desktop in order to see the new word processor you wrote.
Rule #2. Docs are very useful ways of populating your website with useful information. I used to use LFS, and how often did I find a package that SEEMED to fit my needs, only to find it required hundreds of packages installed, or really didn't fit those needs at all? Too often. You wanna save me some time and just link your INSTALL and README files on your page? They're plain text, so they're not going to kill your space or bandwidth, and will save me a ton of time. I shouldn've have to download a package, untar it, and then go into an editor just to find out the prereqs for it.
Rule #2a. Know your prereqs. In an ideal world, every developer would have an LFS machine around so they know exactly what libs need to be installed in order for their software to work. There's nothing worse than having a long build appear to finish successfully, only to have the executable panic the machine when run. If they fail to test build it that way first...up against the wall!
Rule #3. About first, history second. The first thing on your website should NOT be a changelog. It doesn't help me to know that you "modified mallocs to use less memory" if I don't know what your software does. Tell me what it is, then put the history and changelogs elsewhere. If I'm interested, I'll look. And, a real-world example or two can sometimes go a long way to helping me decide. Occasionally, I'll come across a project and after looking at it for some time, still have no idea what it does or why I'd use it. I have, more than once, come across a project and then dismissed it only to be directed to it by another site that says "try this, you'll love it!" When I look again, I slap my head. I don't like slapping my head. While we're on this topic, put in an "English" changelog, too. Instead of the above entry, simply say "this version uses less memory".
Rule #4. Not everyone is a programmer. For the love of Cthulu, take pride in what you do! Not everyone can develop software. It's a gift you have, don't take it lightly! If someone asks a question, "look at the source" is not an answer unless they specifically hand you a code block and say "how does this work?"
Rule #5. Don't use Sourceforge. I'm sorry, but I hate SF. It wouldn't be so bad if I didn't have to fight my way through thirty different extremelt slow-loading pages just to download the tarball. Besides, no one uses SourceForge right, anyway. Think i'm being a dick? You find me ONE project on there that actually uses the "Docs" link. Most of the "Project Home Page" links point to an empty index, if you're lucky.
Rule #6. Your software really isn't that revolutionary, work with someone else. How many window managers are there now? The apptrove at Freshmeat tells me it's got 130 projects in that category. Is that really necessary? Really? You mean to tell me no one can create a basic, simple fucking window manager and then add functionality in via plugins? Beyond window managers, though, let's look at some others: instant messengers, how 'bout using a the same config file as one of the others? The information in there should be the same, so if I want to install Gaim and a command line IM, I can and not have to worry about how they're each configured. Would it really be that hard to keep my username/password for each network in just ONE location? For a group that goes nuts about some of the choices Microsoft has made with Windows, you dorks have sure made a lot more mistakes in terms of simplicity and flexibility!
See, the problem is too many choices is NOT a strength. You can spout your stale rhetoric about that all you want, but that don't make it true. Too many choices, especially the apples-to-oranges choices offered only make life more difficult. We have all of these tools and such, and that's great, but they all have different purposes and functionality. You can't just choose one over the other easily...especially when you want to choose on functionality, but you're limited in KDE-type apps or something, and what you want are only available in Gnome. Now I gotta install another fucking environment just to use YOUR app. No thanks.
Okay, this rant's over. Seriously, just make it easier on people. Complexity for complexity's sake is a stupid way to put your software out there.
Wednesday, January 4, 2006
Your drive is dead, you are so screwed
Remember me saying how you really need to protect your data? Boy, do I know how to predict failure or what?! :) Seriously, I had a panic attack over the last couple of days as I started getting mails from mdadm:
A DegradedArray event had been detected on md device /dev/md0.
Yipes! So, I cat /proc/mdstat and found that all three arrays were showing a degraded state, with one drive missing from each. hdg was not showing in any of the arrays. A mdadm -Q /dev/hdg confirmed that it was not part of any array.
A quick inspection of /var/log/messages shows:
Jan 4 16:09:47 alfred kernel: ide: failed opcode was: unknown
Jan 4 16:09:47 alfred kernel: hdg: task_out_intr: status=0x50 { DriveReady SeekComplete }
Jan 4 16:15:47 alfred kernel: hdg: dma_intr: error=0x84 { DriveStatusError BadCRC }
BadCRC?! BADCRC!?!? This hard drive is just over a month old!! It better not be failing. So, I did me some searching, just to be sure I didn't need to go through the trouble of trying to get this drive out and shipped back to the manufacturer (especially since I JUST threw out the damn box!) I figured I'd give it a little test to make sure the drive itself wasn't bad. Since I knew it wasn't part of any array, I could play with it how I wanted. So, I repartitioned it and created a new ext2 filesystem on it. I then copied a large amount of data to and from the drive, with no errors in /var/log/messages. Hmmm...
Unfortunately, you know all that rhetoric about how Linux support is just BUSTLING on the Internet? How it's so much better than commercial support? Yeah, I've heard it, too...After about two hours of searching, I finally came to the conclusion that this error did not indicate a dying drive, but a problem with either the driver, the card, the cable or the drive (as in one of these things was not entirely compatible). In other words, there were lots of opinions out there on what these messages meant, but no real information. Most blamed it on the kernel ("it works fine with a 2.4 kernel"), some blamed it on the drive's manufacturer ("if it can't keep up with DMA requests, you'll get that error. Get a new drive"), others said it was ACPI ("add pci=noapic to your boot option"). Even on the kernel-dev list, a number of people had posted the exact same problem, few found a solution. None of their solutions worked for me.
I'll spare you the exact details on everything I had to do to fix this, but suffice it to say I believe the problem was that I had two different speed drives on the same controller card. It didn't make any sense to me, either since both drives were on their own controller on the card. The real reason for it was more likely a combo of that, and the fact that I'm using these drives in an array (quite a few of the folks with this issue were using arrays). So, I issued the following:
hdparm -X udma3 /dev/hde
hdparm -X udma3 /dev/hdg
This sets the speeds down to 66Mhz. So far, after a half an hour, I haven't seen any errors reoccur in the event log. I'll try moving them up to UDMA4 (100Mhz) at some point, but for now the RAID1 array is rebuilding and I'm not seeing the error, so I'm pretty sure this is a valid fix.
Update: it's the next day, and I had no errors last night or this morning. The arrays are still holding, so I think we're good. I modified /etc/sysconfig/harddisks to use the following command line on each of my drives at boot time:
hdparm -X udma3 -d1 -c3 /dev/hdx
The drives will be a little slower and not running at max efficiency, but I can deal. For the most part, these drives are for storage, and I don't need quick like a bunny access. I think this confirms my theory of it being an md driver issue. If one drive is faster than the other, md should "wait" for the other to catch up. Of course, that would probably introduce many other timing issues in a driver that's designed to minimize data loss.
This situation could have been a LOT worse had I not setup those arrays (ignoring the fact that it probably wouldn't have occured had it not been for the arrays...). The machine just chugged along nicely without a burp, even with one drive missing. It's also a good thing I told mdadm to send me alert e-mails. The machine would have plodded along without me ever knowing the drive had failed. I would have found out the seriously hard way: when one of the others died, too...
A DegradedArray event had been detected on md device /dev/md0.
Yipes! So, I cat /proc/mdstat and found that all three arrays were showing a degraded state, with one drive missing from each. hdg was not showing in any of the arrays. A mdadm -Q /dev/hdg confirmed that it was not part of any array.
A quick inspection of /var/log/messages shows:
Jan 4 16:09:47 alfred kernel: ide: failed opcode was: unknown
Jan 4 16:09:47 alfred kernel: hdg: task_out_intr: status=0x50 { DriveReady SeekComplete }
Jan 4 16:15:47 alfred kernel: hdg: dma_intr: error=0x84 { DriveStatusError BadCRC }
BadCRC?! BADCRC!?!? This hard drive is just over a month old!! It better not be failing. So, I did me some searching, just to be sure I didn't need to go through the trouble of trying to get this drive out and shipped back to the manufacturer (especially since I JUST threw out the damn box!) I figured I'd give it a little test to make sure the drive itself wasn't bad. Since I knew it wasn't part of any array, I could play with it how I wanted. So, I repartitioned it and created a new ext2 filesystem on it. I then copied a large amount of data to and from the drive, with no errors in /var/log/messages. Hmmm...
Unfortunately, you know all that rhetoric about how Linux support is just BUSTLING on the Internet? How it's so much better than commercial support? Yeah, I've heard it, too...After about two hours of searching, I finally came to the conclusion that this error did not indicate a dying drive, but a problem with either the driver, the card, the cable or the drive (as in one of these things was not entirely compatible). In other words, there were lots of opinions out there on what these messages meant, but no real information. Most blamed it on the kernel ("it works fine with a 2.4 kernel"), some blamed it on the drive's manufacturer ("if it can't keep up with DMA requests, you'll get that error. Get a new drive"), others said it was ACPI ("add pci=noapic to your boot option"). Even on the kernel-dev list, a number of people had posted the exact same problem, few found a solution. None of their solutions worked for me.
I'll spare you the exact details on everything I had to do to fix this, but suffice it to say I believe the problem was that I had two different speed drives on the same controller card. It didn't make any sense to me, either since both drives were on their own controller on the card. The real reason for it was more likely a combo of that, and the fact that I'm using these drives in an array (quite a few of the folks with this issue were using arrays). So, I issued the following:
hdparm -X udma3 /dev/hde
hdparm -X udma3 /dev/hdg
This sets the speeds down to 66Mhz. So far, after a half an hour, I haven't seen any errors reoccur in the event log. I'll try moving them up to UDMA4 (100Mhz) at some point, but for now the RAID1 array is rebuilding and I'm not seeing the error, so I'm pretty sure this is a valid fix.
Update: it's the next day, and I had no errors last night or this morning. The arrays are still holding, so I think we're good. I modified /etc/sysconfig/harddisks to use the following command line on each of my drives at boot time:
hdparm -X udma3 -d1 -c3 /dev/hdx
The drives will be a little slower and not running at max efficiency, but I can deal. For the most part, these drives are for storage, and I don't need quick like a bunny access. I think this confirms my theory of it being an md driver issue. If one drive is faster than the other, md should "wait" for the other to catch up. Of course, that would probably introduce many other timing issues in a driver that's designed to minimize data loss.
This situation could have been a LOT worse had I not setup those arrays (ignoring the fact that it probably wouldn't have occured had it not been for the arrays...). The machine just chugged along nicely without a burp, even with one drive missing. It's also a good thing I told mdadm to send me alert e-mails. The machine would have plodded along without me ever knowing the drive had failed. I would have found out the seriously hard way: when one of the others died, too...
Friday, December 30, 2005
How to get yourself canned
No, I didn't get canned again, but using the info in this article can get your ass in serious hot water if you use it and get caught. You've been warned! Before attempting to use any of these tools, you should also invest in an SSL certificate for your webserver. Can't afford one? No? Have you not heard of CACert? You can get a free cert for your machine from there. They also provide their root certificate in an installable form for your browser so you won't get those annoying "this site is encrypted, but we don't trust it" messages.
My POB has blocked pretty much all Internet access. They've closed down ALL ports except 80 & 443, and those go through a highly restrictive proxy which, oddly enough, blocks www.espn.com, but allows access to Yahoo and Google mail (where you could bring in a virus or trojan). Oh, well, I don't set the policy, I just need to circumvent it.
The first tool in my arsenal is Anyterm. Anyterm is an Apache module which presents to you a terminal window in a webpage. This isn't one of those PHP-based "type a command, see the output" things, you get a full and true terminal window that you can do pretty much anything. Edit a file with vi, play games, whatever, it's possible. Phil Endecott, the author, has done a great job on this. It only came out a short while ago, but it quickly reached a highly stable state, and it just continues to improve. But, don't take my word for it, hit the site, check out the demo. Just make sure you put a pillow on your desk edge so when your jaw hits it, you don't get hurt. :)
The install of the above is well-documented, it's here mostly to let you know it's there. Anyterm has been a great assist in getting this new machine setup, as I can now do it at work when I have more time. :) However, it is limited to a terminal session, and you can't cut and paste into it either. For more advanced things, it would be nice if I could do remote X or even VNC.
Well, you can! With GNU httptunnel, you can create TCP-over-HTTP tunnels that allow you to use any port you want by tunneling it through your home web server. httptunnel consists of two parts: a server which you need to install on an unrestricted machine on the Internet and a client which goes on the machine behind the restrictive firewall.
Installation is as simple as doing "configure && make && make install". To start the server, I use this command:
hts --forward-port localhost:22 80
This tells it to forward incoming httptunnel connections on port 80 to port 22 (ssh, for the real newbs). Now, on the client side, start the tunnel:
htc --forward-port 900 --proxy proxy.mycompany.com:80 --proxyauthorization myname:mypass www.myhomemachine.com:80
For simplicity and security, I put the above line in an htstart script and replaced "mypass" with "$1". That way, I can start it without needing to put my password in a text file. For some reason, GNU decided no output was useful output. So, if you run the above command and nothing appears to have happened, it probably did. To try, just fire up your favorite ssh client and make a connection to localhost on port 900. If all went well, you should get a login prompt from your home machine. Yaay!
Now, you can tunnel any app you want through httptunnel, but I recommend keeping it limited to ssh. First, httpt doesn't support SSL, so you need to provide some kind of encryption. Secondly, httpt only supports one port per instance, and ssh'll allow you to do more. This way, the only unencrypted part of your "conversation" with the outside world is that initial connection. As long as you don't press your luck and use the tunnel too often for large amounts of data, your network team shouldn't notice and you can do what you need.
Good luck, be careful and stay employed!
My POB has blocked pretty much all Internet access. They've closed down ALL ports except 80 & 443, and those go through a highly restrictive proxy which, oddly enough, blocks www.espn.com, but allows access to Yahoo and Google mail (where you could bring in a virus or trojan). Oh, well, I don't set the policy, I just need to circumvent it.
The first tool in my arsenal is Anyterm. Anyterm is an Apache module which presents to you a terminal window in a webpage. This isn't one of those PHP-based "type a command, see the output" things, you get a full and true terminal window that you can do pretty much anything. Edit a file with vi, play games, whatever, it's possible. Phil Endecott, the author, has done a great job on this. It only came out a short while ago, but it quickly reached a highly stable state, and it just continues to improve. But, don't take my word for it, hit the site, check out the demo. Just make sure you put a pillow on your desk edge so when your jaw hits it, you don't get hurt. :)
The install of the above is well-documented, it's here mostly to let you know it's there. Anyterm has been a great assist in getting this new machine setup, as I can now do it at work when I have more time. :) However, it is limited to a terminal session, and you can't cut and paste into it either. For more advanced things, it would be nice if I could do remote X or even VNC.
Well, you can! With GNU httptunnel, you can create TCP-over-HTTP tunnels that allow you to use any port you want by tunneling it through your home web server. httptunnel consists of two parts: a server which you need to install on an unrestricted machine on the Internet and a client which goes on the machine behind the restrictive firewall.
Installation is as simple as doing "configure && make && make install". To start the server, I use this command:
hts --forward-port localhost:22 80
This tells it to forward incoming httptunnel connections on port 80 to port 22 (ssh, for the real newbs). Now, on the client side, start the tunnel:
htc --forward-port 900 --proxy proxy.mycompany.com:80 --proxyauthorization myname:mypass www.myhomemachine.com:80
For simplicity and security, I put the above line in an htstart script and replaced "mypass" with "$1". That way, I can start it without needing to put my password in a text file. For some reason, GNU decided no output was useful output. So, if you run the above command and nothing appears to have happened, it probably did. To try, just fire up your favorite ssh client and make a connection to localhost on port 900. If all went well, you should get a login prompt from your home machine. Yaay!
Now, you can tunnel any app you want through httptunnel, but I recommend keeping it limited to ssh. First, httpt doesn't support SSL, so you need to provide some kind of encryption. Secondly, httpt only supports one port per instance, and ssh'll allow you to do more. This way, the only unencrypted part of your "conversation" with the outside world is that initial connection. As long as you don't press your luck and use the tunnel too often for large amounts of data, your network team shouldn't notice and you can do what you need.
Good luck, be careful and stay employed!
Tuesday, December 27, 2005
RAIDing the data
Having worked in the IT filed for pretty much all of my adult life, and some of my non-adult life, there's one lesson I've learned the hardway: hard drives die. If you store your important data on a hard drive, chances are you're going to lose that data somehow someday. Backups are a great way to preserve your data from these inevitible failures (they'll happen, it's just a matter of when!) The problem is, backing up and protecting data needs to be easy enough that you'll remember to do it, otherwise it just don't happen. I've been doing this for a very long time, and even I don't backup my data as often as I should. I've lost enough data in my life you'd think I'd learn, but...
So, one of my primary goals with this new server was to have it protect my data without needing my intervention. I'm going to do this is a multi-layer approach, and using RAID arrays is the first layer. For those not familiar, RAID stands for Redundant Array of Inexpensive Disks. Hard drives are cheap these days. It's almost impossible to find a drive that doesn't come with triple-digit gigabytes anymore. In fact, the most recent addition to the server, a 200G 7200 RPM monster, cost me only $30 after rebate. The basic idea behind RAID is to spread your data across multiple cheap disks in order in such a way that if one fails, you don't lose everything. You can Google for more info. The two levels of RAID I'll be using are RAID-1 and RAID-5.
RAID-1 is commonly known as "drive mirroring". I setup two drives of equal size, and everytime I write data to one, it's written to the other. If one drive fails, I have a duplicate of the data on the other one. The big drawback to this setup is that writing to two drives is typically slower than writing to one. The other is that you "lose" a whole drive. If you take two 200G drives and mirror them, you only get to store 200G of data. I offset the first drawback by putting the two drives on different controllers in the system (also known as drive duplexing since I'm protected by redundant controllers as well). Performance is then not affected as much. The second is offset by the fact that drives are cheap. For $30, I can't afford to not protect my data.
RAID-5 is also known as "striping with parity". RAID-5 requires at least three drives. In a nutshell, let's say you wanted to store the following sequence of numbers:
1 2 3 4 5 6
With a single drive, all numbers are written to the single drive, obviously. In a mirror, all 6 are written to each drive. In a RAID-5, the numbers are spread out across multiple drives, with one of the drives storing "parity data". Parity data is essentially a mathematical formula that describes the data such that if you lost one of the three pieces of data, you can recover the third. So, here's how the data would look on a R5 array:
Drive 1 Drive 2 Drive 3
1 2 P3
P7 3 4
5 P11 6
For simplicity's sake, I used a simple algorithm to calculate the parity: I added the data written to each drive together. Let's say we lose Drive 1. Well, we can figure out that the data missing from row 1 is the number 1 since we know X + 2 = 3. The parity for row 2 is 7 because X = 3 + 4 and so on. In most implementation, if a drive fails, the system will stay up and running until you replace it since it can figure out what's missing. Drawbacks: performance is similar to RAID-1 in that you have to write now to three drives. Also, you lose one drive's worth of space, but not as much as in a mirror.
So, how am I using this? Well, in this machine I have the following drives: 1x120G, 2x200G. The 120 is where the OS is stored, and the 200s typically store my data. I'm going to carve them up as shown in this picture:
My "I can't possibly live without this, so it needs maximum protection" data is stored on the RAID-5 array listed as "personal". My Exchange virtual machine is stored on the one listed as "exchange". I have them separated to minimize corruption issues. My "I'd like to make sure I don't lose this since it's a PITA to replace, but I CAN replace it if necessary" data is stored on the RAID-1 (MP3s, videos, etc). Here's how I did it:
The first problem is the fact that HDE was already setup with a single 200G partition, filled with about 140G of data. I didn't have enough drive space to store it elsewhere, but fortunately, the mdadm tool in Linux gives us a simple workaround
Firstly, I needed to very carefully document what it was I wanted to do. I got ADD, so I have to make absolutely sure I've got a detailed plan of attack or I'll forget stuff. :)
Creat an /etc/mdadm.conf file
echo 'DEVICE /dev/hd* /dev/sd*' > /etc/mdadm.conf
This tells mdadm that any hard drive in the machine could be considered a candidate for creating arrays, and at boot time, find them.
Partition the disks
HDA already has some partitions on it, some of which I didn't need anymore. I created two 10G partitions, and one big one for the last. Since this is going to be a MythTV box, I'll use the free space on the drive for scheduled, temporary recordings. If I want to save something, I can reencode it and put it into the store. On HGE, I also created 2 10G partitions and one remaining bigity-big one. Remember also when in fdisk to set the partition type to "fd" (they're created as "83" by default). "fd" is the type for Linux RAID Autodetect.
A quick reboot later into single user mode (safest way to do this stuff), I created my arrays. Three simple commands:
mdadm --create /dev/md0 --level 5 --raid-devices=3 /dev/hda6 /dev/hdg1 \ missing
mdadm --create /dev/md1 --level 5 --raid-devices=3 /dev/hda7 /dev/hdg2 \ missing
mdadm --create /dev/md2 --level 1 --raid-devices=2 /dev/hdg3 \ missing
The "missing" directive is what allows me to keep my data intact until the last drive is ready to add to the array. It allows me to create the array without all of the partitions. Now, all we do is format the filesystems:
mkfs.jfs /dev/md0
mkfs.jfs /dev/md1
mkfs.jfs /dev/md2
I chose JFS based on recommendations on the MythTV board. I knew I wanted a journaling filesystem for the arrays, primarily due to their size, and JFS seems to be the most "stable" in this configuration.
As a final step in this section, I mounted the arrays (treat them as a regular drive, i.e. "mount /dev/md0 /mnt/personal") and copied all of the data from the HDE partition over to the new arrays.
Once all the data was copied over, I simply fdisked HDE so that it's partition table was similar to HDG's. A note: Linux' RAID support is pretty flexible. You don't have to worry about getting EXACTLY the same number of blocks in each partition. When I created the personal and exchange partitions, I used "+10000M" in each of the "end block" sections. That got them close enough on size.
To finalize the arrays and get them to sync, simply issue the following:
mdadm --add /dev/md0 /dev/hde1
mdadm --add /dev/md1 /dev/hde2
mdadm --add /dev/md2 /dev/hde3
To check the sync status, do "cat /proc/mdstat". You'll see them syncronizing. Go do something for a half hour or so. The system will sync each in turn. When everything's up and running, you'll see "[UUU]" or "[UU]" at the end of each status line (indicating all three or two drives are Up). It's probably not entirely dangerous, but you should wait until the arrays are up fully before trying to use them.
Some final config file changes
Obviously, you'll need to add the arrays to your /etc/fstab. Again, treat them just like any other type of drive. Mine now looks like this:
/dev/hda3 / ext2 defaults 1 1
/dev/md0 /mnt/personal jfs defaults 0 0
And, so on. You should also do the following:
mdadm --detail --scan >> /etc/mdadm.conf
This puts the information on the arrays in the conf file. Mdadm doesn't really need this, but it's good for you to have the info in the future.
As a final step, let's make sure we know when our arrays have degraded. Edit your rc.local (or its equivelant in your distro) and include the following line for each of your arrays:
nohup mdadm --monitor --mail=root@localhost --delay=300 /dev/md0 &
Don't forget the ampersand. This line will send an e-mail to you anytime mdadm detects a degraded array. The mail directive simply acts as a "frontend" to sendmail, mdadm doesn't actually send the mail. So, if sendmail isn't setup properly, you won't get the mail. Which is why I have it going to the root mailbox.. ;-)
I think that's about it. Sorry it was a bit long, but there was a lot of ground to cover. Drop me a comment if you found this info useful!
So, one of my primary goals with this new server was to have it protect my data without needing my intervention. I'm going to do this is a multi-layer approach, and using RAID arrays is the first layer. For those not familiar, RAID stands for Redundant Array of Inexpensive Disks. Hard drives are cheap these days. It's almost impossible to find a drive that doesn't come with triple-digit gigabytes anymore. In fact, the most recent addition to the server, a 200G 7200 RPM monster, cost me only $30 after rebate. The basic idea behind RAID is to spread your data across multiple cheap disks in order in such a way that if one fails, you don't lose everything. You can Google for more info. The two levels of RAID I'll be using are RAID-1 and RAID-5.
RAID-1 is commonly known as "drive mirroring". I setup two drives of equal size, and everytime I write data to one, it's written to the other. If one drive fails, I have a duplicate of the data on the other one. The big drawback to this setup is that writing to two drives is typically slower than writing to one. The other is that you "lose" a whole drive. If you take two 200G drives and mirror them, you only get to store 200G of data. I offset the first drawback by putting the two drives on different controllers in the system (also known as drive duplexing since I'm protected by redundant controllers as well). Performance is then not affected as much. The second is offset by the fact that drives are cheap. For $30, I can't afford to not protect my data.
RAID-5 is also known as "striping with parity". RAID-5 requires at least three drives. In a nutshell, let's say you wanted to store the following sequence of numbers:
1 2 3 4 5 6
With a single drive, all numbers are written to the single drive, obviously. In a mirror, all 6 are written to each drive. In a RAID-5, the numbers are spread out across multiple drives, with one of the drives storing "parity data". Parity data is essentially a mathematical formula that describes the data such that if you lost one of the three pieces of data, you can recover the third. So, here's how the data would look on a R5 array:
Drive 1 Drive 2 Drive 3
1 2 P3
P7 3 4
5 P11 6
For simplicity's sake, I used a simple algorithm to calculate the parity: I added the data written to each drive together. Let's say we lose Drive 1. Well, we can figure out that the data missing from row 1 is the number 1 since we know X + 2 = 3. The parity for row 2 is 7 because X = 3 + 4 and so on. In most implementation, if a drive fails, the system will stay up and running until you replace it since it can figure out what's missing. Drawbacks: performance is similar to RAID-1 in that you have to write now to three drives. Also, you lose one drive's worth of space, but not as much as in a mirror.
So, how am I using this? Well, in this machine I have the following drives: 1x120G, 2x200G. The 120 is where the OS is stored, and the 200s typically store my data. I'm going to carve them up as shown in this picture:
My "I can't possibly live without this, so it needs maximum protection" data is stored on the RAID-5 array listed as "personal". My Exchange virtual machine is stored on the one listed as "exchange". I have them separated to minimize corruption issues. My "I'd like to make sure I don't lose this since it's a PITA to replace, but I CAN replace it if necessary" data is stored on the RAID-1 (MP3s, videos, etc). Here's how I did it:
The first problem is the fact that HDE was already setup with a single 200G partition, filled with about 140G of data. I didn't have enough drive space to store it elsewhere, but fortunately, the mdadm tool in Linux gives us a simple workaround
Firstly, I needed to very carefully document what it was I wanted to do. I got ADD, so I have to make absolutely sure I've got a detailed plan of attack or I'll forget stuff. :)
Creat an /etc/mdadm.conf file
echo 'DEVICE /dev/hd* /dev/sd*' > /etc/mdadm.conf
This tells mdadm that any hard drive in the machine could be considered a candidate for creating arrays, and at boot time, find them.
Partition the disks
HDA already has some partitions on it, some of which I didn't need anymore. I created two 10G partitions, and one big one for the last. Since this is going to be a MythTV box, I'll use the free space on the drive for scheduled, temporary recordings. If I want to save something, I can reencode it and put it into the store. On HGE, I also created 2 10G partitions and one remaining bigity-big one. Remember also when in fdisk to set the partition type to "fd" (they're created as "83" by default). "fd" is the type for Linux RAID Autodetect.
A quick reboot later into single user mode (safest way to do this stuff), I created my arrays. Three simple commands:
mdadm --create /dev/md0 --level 5 --raid-devices=3 /dev/hda6 /dev/hdg1 \ missing
mdadm --create /dev/md1 --level 5 --raid-devices=3 /dev/hda7 /dev/hdg2 \ missing
mdadm --create /dev/md2 --level 1 --raid-devices=2 /dev/hdg3 \ missing
The "missing" directive is what allows me to keep my data intact until the last drive is ready to add to the array. It allows me to create the array without all of the partitions. Now, all we do is format the filesystems:
mkfs.jfs /dev/md0
mkfs.jfs /dev/md1
mkfs.jfs /dev/md2
I chose JFS based on recommendations on the MythTV board. I knew I wanted a journaling filesystem for the arrays, primarily due to their size, and JFS seems to be the most "stable" in this configuration.
As a final step in this section, I mounted the arrays (treat them as a regular drive, i.e. "mount /dev/md0 /mnt/personal") and copied all of the data from the HDE partition over to the new arrays.
Once all the data was copied over, I simply fdisked HDE so that it's partition table was similar to HDG's. A note: Linux' RAID support is pretty flexible. You don't have to worry about getting EXACTLY the same number of blocks in each partition. When I created the personal and exchange partitions, I used "+10000M" in each of the "end block" sections. That got them close enough on size.
To finalize the arrays and get them to sync, simply issue the following:
mdadm --add /dev/md0 /dev/hde1
mdadm --add /dev/md1 /dev/hde2
mdadm --add /dev/md2 /dev/hde3
To check the sync status, do "cat /proc/mdstat". You'll see them syncronizing. Go do something for a half hour or so. The system will sync each in turn. When everything's up and running, you'll see "[UUU]" or "[UU]" at the end of each status line (indicating all three or two drives are Up). It's probably not entirely dangerous, but you should wait until the arrays are up fully before trying to use them.
Some final config file changes
Obviously, you'll need to add the arrays to your /etc/fstab. Again, treat them just like any other type of drive. Mine now looks like this:
/dev/hda3 / ext2 defaults 1 1
/dev/md0 /mnt/personal jfs defaults 0 0
And, so on. You should also do the following:
mdadm --detail --scan >> /etc/mdadm.conf
This puts the information on the arrays in the conf file. Mdadm doesn't really need this, but it's good for you to have the info in the future.
As a final step, let's make sure we know when our arrays have degraded. Edit your rc.local (or its equivelant in your distro) and include the following line for each of your arrays:
nohup mdadm --monitor --mail=root@localhost --delay=300 /dev/md0 &
Don't forget the ampersand. This line will send an e-mail to you anytime mdadm detects a degraded array. The mail directive simply acts as a "frontend" to sendmail, mdadm doesn't actually send the mail. So, if sendmail isn't setup properly, you won't get the mail. Which is why I have it going to the root mailbox.. ;-)
I think that's about it. Sorry it was a bit long, but there was a lot of ground to cover. Drop me a comment if you found this info useful!
Wednesday, December 21, 2005
Tweaking the penguin's nipples
hdparm: nipple clip of yesteryear
The first tweak I'd recommend is starting with hdparm. It can't hurt, and might help. This article is the one I learned how to use hdparm from a long, long time ago. It's a little dated, but the commands are still the same, so you can at least learn how it works. More than likely, it won't make any difference as the features it's supposed to enable are typically enabled by default. My most current machine did not need hdparm.
Too many consoles spoil the soup
Next, remove some extra virtual consoles. On a Linux box, when you're at the console, you can hit Alt-F1-6 and switch between consoles. Kinda useless if you're booting into a GUI, so let's disable them:
vi /etc/inittab
You'll see a bunch of lines like:
1:2345:respawn:/sbin/mingetty tty1
Comment out the lines that begin with 3 through 6 (keep two as an in case). When you next reboot, you'll only have two gettys running and have freed up a tiny bit of RAM. Hey, tiny, but you weren't using it, right?
Clear out those services
Yes, I'll tell you to do it, too. But, I'll tell you which you can prolly lose:
apmd - The Advanced Power Management daemon. If you don't have a laptop, you can more than likely kill this one.
gpm - Essentially the console mouse driver. In a GUI all the time? Kill it, you won't be using it.
identd - Used for IRC to identify you. Some IRC servers require you to run one. If you don't IRC, say buh-bye!
ip6tables & iptables - Ip6tables can go away if you're not using IPv6. Iptables should only be disabled if your machine is behind a firewall. Even then, you should consider keeping it, but that's your call. This is your machine's firewall.
isdn - Useless if you don't have an ISDN line.
nfs & portmapper - if you're not connecting to NFS file shares somewhere, this can go.
Sloppy with your swappi
The 2.6 kernel gave us the ability to determine how likely things will be swapped to disk, rather than kept in main memory. The kernel does these calculations constantly and there is no cut-and-dried guidelines for setting this. Default is 60, but you can set any value you want betweeen 0 and 100. I use 30 and find that works fine for me on the few occasions I might need to swap (my machine has a gig of RAM, and rarely uses more than that). 20 is a good number if you've got a laptop and want to force the kernel to swap only when absolutely necessary. This is good for those slow-ass laptop harddrives.
Two ways you can set this:
echo 30 >/proc/sys/vm/swappiness
This is a temporary method and lasts until your next reboot. But, you can do a lot of testing by modifying the swappiness on the fly and then determining what works best for you regularly. You can use "free -m" to view your RAM and swap usage at any time. At that point you can....
Most distros use /etc/sysctl.conf to control things like this. Set vm.swappiness = 30 in that file and it'll follow with each reboot.
More to come.
The first tweak I'd recommend is starting with hdparm. It can't hurt, and might help. This article is the one I learned how to use hdparm from a long, long time ago. It's a little dated, but the commands are still the same, so you can at least learn how it works. More than likely, it won't make any difference as the features it's supposed to enable are typically enabled by default. My most current machine did not need hdparm.
Too many consoles spoil the soup
Next, remove some extra virtual consoles. On a Linux box, when you're at the console, you can hit Alt-F1-6 and switch between consoles. Kinda useless if you're booting into a GUI, so let's disable them:
vi /etc/inittab
You'll see a bunch of lines like:
1:2345:respawn:/sbin/mingetty tty1
Comment out the lines that begin with 3 through 6 (keep two as an in case). When you next reboot, you'll only have two gettys running and have freed up a tiny bit of RAM. Hey, tiny, but you weren't using it, right?
Clear out those services
Yes, I'll tell you to do it, too. But, I'll tell you which you can prolly lose:
apmd - The Advanced Power Management daemon. If you don't have a laptop, you can more than likely kill this one.
gpm - Essentially the console mouse driver. In a GUI all the time? Kill it, you won't be using it.
identd - Used for IRC to identify you. Some IRC servers require you to run one. If you don't IRC, say buh-bye!
ip6tables & iptables - Ip6tables can go away if you're not using IPv6. Iptables should only be disabled if your machine is behind a firewall. Even then, you should consider keeping it, but that's your call. This is your machine's firewall.
isdn - Useless if you don't have an ISDN line.
nfs & portmapper - if you're not connecting to NFS file shares somewhere, this can go.
Sloppy with your swappi
The 2.6 kernel gave us the ability to determine how likely things will be swapped to disk, rather than kept in main memory. The kernel does these calculations constantly and there is no cut-and-dried guidelines for setting this. Default is 60, but you can set any value you want betweeen 0 and 100. I use 30 and find that works fine for me on the few occasions I might need to swap (my machine has a gig of RAM, and rarely uses more than that). 20 is a good number if you've got a laptop and want to force the kernel to swap only when absolutely necessary. This is good for those slow-ass laptop harddrives.
Two ways you can set this:
echo 30 >/proc/sys/vm/swappiness
This is a temporary method and lasts until your next reboot. But, you can do a lot of testing by modifying the swappiness on the fly and then determining what works best for you regularly. You can use "free -m" to view your RAM and swap usage at any time. At that point you can....
Most distros use /etc/sysctl.conf to control things like this. Set vm.swappiness = 30 in that file and it'll follow with each reboot.
More to come.
Subscribe to:
Posts (Atom)