So Tim and Bill and I have been jamming together for thousands of years. We play about half of every song because we get lost or one of us (usually me) doesn’t know the rest of the song, or we get bored because songs that are meant to have vocals sound pretty repetitive without vocals.
Now I’ve known for years that my wife’s friend Jessica has a magnificent voice. She sang at our wedding, as a matter of fact. I’ve been saying for years that I need to get her to come out and sing with us. It was one of those, “we should do this sometime” kinda things that just never seemed to happen. Well, this past Friday I changed that. I called Jess up and invited her to come out and jam with us.
Then, something magical happened! We actually played a song all the way through to the end! It was amazing! Not to mention that her voice just sounds awesome! I’ve always loved the sound of a powerful female vocalist with a hard rock / heavy metal band behind her. Despite her being nervous as hell, she did great and things just started to click.
So without further ado, here’s a taste of awesomeness.
Armed with the knowledge that Talk Like a Pirate Day is sometime this week, and whizzing by at 90+ MPH on a motorcycle in high winds, you would be amazed at how easy it is for your brain to interpret this sign as a hilarious attempt at humor by the state highway department. I swear I thought it said Scurvy Crew Ahead, and I almost went back around on the expressway to get a picture of it, when about a mile down the road I came upon a Survey Crew and it suddenly wasn’t funny anymore.
When VMware says it will support a maximum partition size of 2TB – 512B, they’re not kidding. And when you somehow create a NetApp LUN of exactly 2TB, and somehow get VMware to create a datastore on it, and somehow manage to operate production servers on it for a couple years without a problem, you will likely review the configuration at some point and realize that you’re sitting on a ticking time bomb.
Now, if at this point you cautiously move all your virtual machines off to another datastore, then you will probably suffer no ill consequences. If, however, one were to misguidedly resize the NetApp LUN to 1.99TB while VMware still thinks it’s a 2.00TB LUN, expect to stay up late trying desperately to salvage your servers from the deathly grips of I/O errors.
Luckily in the situation that inspired this post, only two virtual machines were storing data on the far edge of the datastore that was affected by the sudden change in LUN size. Neither of these servers were home to critical applications or data, and all services and data necessary to production were moved to a stable datastore without issue. After struggling with the data recovery attempts from the corrupt datastore for a day, I decided to cut my losses and restore them from backup.
All was well this time, but things could have been much worse. The moral of the story is, never ever reduce the size of your storage LUN while you have data on it that you want to keep.
… and no, I was not the one who resized the LUN. I did, however, get to clean up the resulting mess!
Update: Godaddy announced today (Sep 11, 2012) that it was an internal issue that caused the outage, and the punk hacker was just trying to make himself look cool by saying he did it. He’s still a jackass regardless.
Well it looks like I’m going to have to do a little work to make my DNS structure a little more resilient to failure. Currently, Godaddy is a single point of failure for me. And here I thought I was in good shape because I was having them host it instead of doing it locally. Perhaps I’ll set up my own authoritative server and have Godaddy act as a secondary. Of course I’ll probably have to pay extra to do it that way. I would know for sure except that… well… Godaddy’s own site is down. I’ll bet they’re slightly more pissed off than I am.