Yesterday, I finished a Physical 2 Virtual (P2V) migration of a server, using an evaluation of Microsoft’s System Center Virtual Machine Manager, which, by the way, is pretty decent.
When I brought the newly minted VM online, everything seemed to be kosher, so I went ahead and wiped the physical server that it was on, and installed an evaluation of Microsoft Forefront TMG. After I had Forefront installed and was configuring it, I was realizing that every once in a while my remote desktop connection to it would just up and die. I started a continuous ping, and I was seeing packet loss here and there, but nothing definitive–I thought I had something configured wrong, so I spent a bit of time double-checking settings, but could not figure it out–So I started another continueus ping, and went home for the day.
This morning, I came back to an average ~9% packet loss to the Forefront server–Not only that, but the new VM that was working so great yesterday was virtually unusable, as I saw the ~90% packet loss going on.
So I pulled out my handy-dandy Wireshark, and started sniffing about–Within 30 seconds, I saw the problem–Do you see it?
Check out packets 25 & 26: ARP requests–ARP requests from the same source address (00:1b:21:44:79:39), which is corresponding to two different IPs! Doh!
Come to find out, when I did the P2V migration, I remember seeing an option that asked if I wanted to generate new MAC addresses for the new VM–Like an idiot, I let the default have its way, which means that I kept the MAC addresses on the new VM the same as the physical server I was migrating from.
Because I had disconnected the physical server from the network before I turned on the new VM, it was not an issue. When I rebuilt the physical server and brought it online, then it started causing problems.
It was an easy fix:
Under the VM settings, I just need to change the MAC address from Static to Dynamic.
So, what have I learned today in the School of Hard Knocks?
-Think before you just click yes
-And when you do forget, Wireshark is your friend
-Josh
Defensive Depth 