Possibly old news but thought I would pop this here in case anyone else had an issue migrating from SCUP 4.5 -> 2011 and finds that everything breaks. Like I did. And spent the next few hours Googling incorrect information.

The thing to bear in mind that, contrary to MS instructions, you can't continue to use your current WSUS certificate as SCUP 2011 requires a slightly different cert and just regenerating it isn't enough. You need to install this hotfix: System Center Update Publisher does not publish customized updates to a computer if WSUS 3.0 SP2 and the .Net Framework 4.0 are installed then regenerate the WSUS cert from within SCUP 2011. The hotfix changes the purpose of cert that WSUS creates from 'All' to 'Code signing' which makes SCUP happy again. The new cert needs to replace the current certs on your WSUS server and SCUP machine. And of course you need to replace the certs on all the clients using WSUS.