I'm going down the same path. I got as far as regedit -> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WDSServer\Providers\WDSMC\Protocol and I can see things like TpMaxBandwidth, TpMulticastTTL, and all that, but I'm not seeing anything about the custom profile or how to set it. I'd also like to set my Multicast in the 600Mb/s range instead of eating up my school district's entire internet or making me wait at 100Mb/s while most of our bandwidth sits unused.
I'm running System Center Configuration Manager 2016 on a sole primary site with WDS working for single PXE transfers. The server (virtual machine) is also running Windows Server 2016 Standard. Perhaps this can get someone a step closer to getting the info needed and posting it here so I can image 200+ computers in a few weeks at a lot faster rate. Being the only IT person here means I don't have enough time to spend researching everything while still keeping all of the staff and students (100+ and 1000+, respectiely) relatively happy with their IT needs.