I bump up this thread for people still having this problem despite updating their ESX/ESXi.
I had this problem too, and found out that it was Kaspersky Anti-Virus for Windows Server that was making my server so slow. I removed it, and switched to NOD32, and magic happened... My server retrieved its initial responsiveness.
I hope it will help someone.
Like Djeross, I am also bumping this thread for others that are still experiencing issues even after ruling out any Anti-Virus software. I have seen that Server 2008 R2 vm Display performance is slow on ESXi 4 and 4.1 with a default installation, even after installing vmware tools. The solution for me was to install the VMware wddm_video video driver. I have included a link to a short article explaining the steps for this at http://thisworldandi.com/slow2008onesxi