Sorry for again bumping an old post.
Replying to my own post in case anyone runs into the same issue I did. Instead of removing drivers my issue was instead that I was missing a USB 3 Enhanced Controller driver. I used DISM and injected the drivers into the WIM and that resolved the issue. (Directions for using DISM in this manner: https://technet.microsoft.com/en-us/library/hh825070.aspx?f=255&MSPPError=-2147217396)