So I’ve been having this mystery problem with Server 2012 R2 for a while now. I have a 2TB external drive that I am using for backup storage. I was running out of space on the drive, and I couldn’t figure out why. When I added up the files on the drive, they didn’t come anywhere near 2TB. In fact, I moved everything off of the drive entirely and I still had only 32 GB free! Something was definitely awry, and I had to figure out what.
My first thought was to just re-initialize the drive and go on with my life, but then I would always be left wondering what caused it. So instead I investigated until I found the problem, and hopefully this article will help someone else resolve the issue without having to shuffle tons of data around to re-initialize a drive.
When VMware says it will support a maximum partition size of 2TB – 512B, they’re not kidding. And when you somehow create a NetApp LUN of exactly 2TB, and somehow get VMware to create a datastore on it, and somehow manage to operate production servers on it for a couple years without a problem, you will likely review the configuration at some point and realize that you’re sitting on a ticking time bomb.
Now, if at this point you cautiously move all your virtual machines off to another datastore, then you will probably suffer no ill consequences. If, however, one were to misguidedly resize the NetApp LUN to 1.99TB while VMware still thinks it’s a 2.00TB LUN, expect to stay up late trying desperately to salvage your servers from the deathly grips of I/O errors.
Luckily in the situation that inspired this post, only two virtual machines were storing data on the far edge of the datastore that was affected by the sudden change in LUN size. Neither of these servers were home to critical applications or data, and all services and data necessary to production were moved to a stable datastore without issue. After struggling with the data recovery attempts from the corrupt datastore for a day, I decided to cut my losses and restore them from backup.
All was well this time, but things could have been much worse. The moral of the story is, never ever reduce the size of your storage LUN while you have data on it that you want to keep.
… and no, I was not the one who resized the LUN. I did, however, get to clean up the resulting mess!