At my POE, we have a huge VMWare infrastructure. About 30% of our servers are actually VMs. This isn't just test servers, but production as well. Last night, I was performing some routine maintenance on a VM owned by one of our support and test groups. I was adding a second NIC to their VM so they'd be able to get on the TAN and start restoring production environments so they could duplicate issues. A relatively simple task, I powered down the VM, added the NIC and powered back on.
Or, should I say...hit the power button. The machine wouldn't come back up, I kept getting "File not found" error. So, I tried the logical thing, I removed the NIC and tried again. Same results. Hmmm.. So, now, I needed to do some real investigating. Since it was a test machine and getting very late (and I've got a bad cold), I put it off until this morning. Taking a closer look at the event log, I found the problem: "VMware server cannot find the virtual disk /vmfs/blahblahblah". Uh-oh. That disk is the first image in a group of disks that makes up a 450G volume in the VM. Did I mention this machine wasn't on the TAN, and so not backed up?
I fired up Putty and did some poking around. This machine actually has 4 disks in it. And sure enough, all of my flat files were there, but the descriptor files were missing. For those unfamiliar with the specifics, in newer versions of VMWare, virtual disks are actually made up of two files: disk.vmdk which is just a text file called a descriptor containing the specifics of the disk. Size, type, geometry, etc. There's another vmdk file, disk-flat.vmdk that is the actual flat file that holds your data. Personally, I think it would make more sense if they used different extensions, but that's just me.
I did some poking around and found that disappearing descriptors is a relatively common problem these days. The solution is relatively easy, but not something *I* could do easily. I had to create a new virtual disk, copy the descriptor from it and just edit it to point to the right flat file. The problem is, the first disk that was missing its descriptor is 250G and I've got 50G free on this standalone host. I'd have to recreate on another host, copy over, etc. But, at the last moment I found this:
Simply enter the size in bytes of the flat file (copy...paste) and the name of the flat file (copy...paste) and it generates a new stub that you can...you guessed it...copy, paste into VI on the host. I took a moment to try recreating a couple of vmdks that I did have, just to make sure I got exactly the same results using this tool. I'm not sure if it makes a huge difference or not, but esXpress generator always populates with the line:
ddb.toolsVersion = "0"
And, all of my descriptors have this one:
ddb.toolsVersion = "7299"
I don't think it would've made any difference, but I edited all of mine so the version number was 7299. Fingers crossed, I powered up the VM and it came right up. Chkdsk found no errors on the volume, and we're back in business. Thanks so much to esXpress. You saved me a ton of work this morning!
No comments:
Post a Comment