I managed to configure this with Exchange 2010 (after many hours). The only way I found to do it is to add another website in IIS, bind it to a different IP on the local network and mirror the OWA and ECP applications onto this website, enable forms based authentication only (must be done via EMS) for the Default web site (Must be this way round due to outlook anywhere) and configure second website for SSO with NTLM/Kerb Authentication. Set DNS to route internal requests to this second IP.
IISreset /noforce /timeout:240 seemed to be required many times, but I was stumbling in the dark.