coderr

Here's a list of keys that LSASS reads before starting to enumerate SAM (and before it ends up on the reading loop). This from 2019, the only difference between 2012R2 and 2019 are the SampPasswordLegacyModeEnabled and PasswordLegacyMode keys which aren't there in the 2012R2 logs. The rest are identical. Anything here strikes a note with you? 2:59:12.0661138 PM lsass.exe 680 RegOpenKey HKLM\SECURITY\Policy\LastPassCompleted SUCCESS Desired Access: Read 2:59:12.0661454 PM lsass.exe 680 RegQueryValue HKLM\SECURITY\Policy\LastPassCompleted\(Default) BUFFER OVERFLOW Length: 12 2:59:12.0661616 PM lsass.exe 680 RegCloseKey HKLM\SECURITY\Policy\LastPassCompleted SUCCESS 2:59:12.0661779 PM lsass.exe 680 RegOpenKey HKLM\SECURITY\Policy\LastPassCompleted SUCCESS Desired Access: Read 2:59:12.0661823 PM lsass.exe 680 RegCreateKey HKLM\System\CurrentControlSet\Control\Lsa\Audit\PerUserAuditing\System REPARSE Desired Access: Query Value, Set Value 2:59:12.0662621 PM lsass.exe 680 RegCreateKey HKLM\System\CurrentControlSet\Control\Lsa\Audit\PerUserAuditing\System SUCCESS Desired Access: Query Value, Set Value, Disposition: REG_OPENED_EXISTING_KEY 2:59:12.0662635 PM lsass.exe 680 RegQueryValue HKLM\SECURITY\Policy\LastPassCompleted\(Default) SUCCESS Type: REG_NONE, Length: 4, Data: 02 00 00 00 2:59:12.0662892 PM lsass.exe 680 RegCloseKey HKLM\SECURITY\Policy\LastPassCompleted SUCCESS 2:59:12.0663144 PM lsass.exe 680 RegEnumValue HKLM\System\CurrentControlSet\Control\Lsa\Audit\PerUserAuditing\System NO MORE ENTRIES Index: 0, Length: 686 2:59:12.0663565 PM lsass.exe 680 RegCreateKey HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy REPARSE Desired Access: Query Value, Set Value 2:59:12.0663709 PM lsass.exe 680 RegCreateKey HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy SUCCESS Desired Access: Query Value, Set Value, Disposition: REG_OPENED_EXISTING_KEY 2:59:12.0663958 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy\AuditPolicySD BUFFER OVERFLOW Length: 12 2:59:12.0664096 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy\AuditPolicySD BUFFER OVERFLOW Length: 144 2:59:12.0664220 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy\AuditPolicySD SUCCESS Type: REG_BINARY, Length: 324, Data: 01 00 00 00 D0 8C 9D DF 01 15 D1 11 8C 7A 00 C0 2:59:12.0665151 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers REPARSE Desired Access: Read 2:59:12.0665332 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS Desired Access: Read 2:59:12.0665594 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS 2:59:12.0665896 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration REPARSE Desired Access: Read 2:59:12.0666161 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS Desired Access: Read 2:59:12.0666512 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS 2:59:12.0667472 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers REPARSE Desired Access: Read 2:59:12.0667756 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS Desired Access: Read 2:59:12.0668072 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS 2:59:12.0668302 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration REPARSE Desired Access: Read 2:59:12.0668534 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS Desired Access: Read 2:59:12.0668739 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS 2:59:12.0669259 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers REPARSE Desired Access: Read 2:59:12.0669440 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS Desired Access: Read 2:59:12.0669749 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS 2:59:12.0670113 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration REPARSE Desired Access: Read 2:59:12.0670454 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS Desired Access: Read 2:59:12.0670852 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS 2:59:12.0672185 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Hvsi REPARSE Desired Access: Read 2:59:12.0672340 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Hvsi SUCCESS Desired Access: Read 2:59:12.0672602 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Hvsi\IsHvsiContainer NAME NOT FOUND Length: 16 2:59:12.0672764 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Hvsi SUCCESS 2:59:12.0683589 PM lsass.exe 680 RegOpenKey HKLM\SYSTEM\CurrentControlSet\Control\Lsa REPARSE Desired Access: Query Value 2:59:12.0683973 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Lsa SUCCESS Desired Access: Query Value 2:59:12.0684279 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LookupLogLevel NAME NOT FOUND Length: 16 2:59:12.0684636 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupReturnSidTypeDeleted NAME NOT FOUND Length: 16 2:59:12.0684966 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupRestrictIsolatedNameLevel NAME NOT FOUND Length: 16 2:59:12.0685116 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsarpcServerAllowRemotedSecretOperations NAME NOT FOUND Length: 16 2:59:12.0685250 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupCacheRefreshTime NAME NOT FOUND Length: 16 2:59:12.0685373 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupCacheExpireTime NAME NOT FOUND Length: 16 2:59:12.0685492 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupCacheMaxSize NAME NOT FOUND Length: 16 2:59:12.0685618 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaAllowReturningUnencryptedSecrets NAME NOT FOUND Length: 16 2:59:12.0685783 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Lsa SUCCESS 2:59:12.0867158 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\NoLmHash SUCCESS Type: REG_DWORD, Length: 4, Data: 1 2:59:12.0867387 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamReplicatePasswordsUrgently NAME NOT FOUND Length: 16 2:59:12.0867539 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\ForceGuest SUCCESS Type: REG_DWORD, Length: 4, Data: 0 2:59:12.0867682 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LimitBlankPasswordUse SUCCESS Type: REG_DWORD, Length: 4, Data: 1 2:59:12.0867821 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamAccountLockoutTestMode NAME NOT FOUND Length: 16 2:59:12.0867951 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableListenOnTCP NAME NOT FOUND Length: 16 2:59:12.0868081 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SampPasswordLegacyModeEnabled NAME NOT FOUND Length: 16 2:59:12.0868228 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\IgnoreGCFailures NAME NOT FOUND Length: 16 2:59:12.0868397 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamNoGcLogonEnforceKerberosIpCheck NAME NOT FOUND Length: 16 2:59:12.0868540 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamNoGcLogonEnforceNTLMCheck NAME NOT FOUND Length: 16 2:59:12.0868671 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableSingleObjectRepl NAME NOT FOUND Length: 16 2:59:12.0868799 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableRSOOnPDCForward NAME NOT FOUND Length: 16 2:59:12.0868935 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableResetBadPwdCountForward NAME NOT FOUND Length: 16 2:59:12.0869071 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamConnectedAccountsExist NAME NOT FOUND Length: 16 2:59:12.0869200 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableOutboundRSO NAME NOT FOUND Length: 16 2:59:12.0869910 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictAnonymous SUCCESS Type: REG_DWORD, Length: 4, Data: 0 2:59:12.0870048 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictAnonymousSam SUCCESS Type: REG_DWORD, Length: 4, Data: 1 2:59:12.0870192 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictRemoteSamEventThrottlingWindow NAME NOT FOUND Length: 16 2:59:12.0870324 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictRemoteSamAuditOnlyMode NAME NOT FOUND Length: 16 2:59:12.0870448 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictRemoteSam BUFFER OVERFLOW Length: 12 2:59:12.0870624 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictRemoteSam SUCCESS Type: REG_SZ, Length: 70, Data: O:BAG:BAD:(A;;RC;;;BA)(A;;RC;;;WD) 2:59:12.0871097 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\ExtendedSidEmulationMode NAME NOT FOUND Length: 16 2:59:12.0871233 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamLogSize NAME NOT FOUND Length: 16 2:59:12.0871360 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamLogLevel NAME NOT FOUND Length: 16 2:59:12.0871488 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamRestrictOwfPasswordChange NAME NOT FOUND Length: 16 2:59:12.0871614 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\MaxSamConnections NAME NOT FOUND Length: 16 2:59:12.0871739 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\dsrmAdminLogonBehavior NAME NOT FOUND Length: 16 2:59:12.0871865 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamMaxQueueLengthForPDCForward NAME NOT FOUND Length: 16 2:59:12.0871987 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\EnableClaimsTransformationEcho NAME NOT FOUND Length: 16 2:59:12.0872110 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\EnumerationCachePurgeInterval NAME NOT FOUND Length: 16 2:59:12.0872236 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\EnumerationCacheEntryLifetime NAME NOT FOUND Length: 16 2:59:12.0872359 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\PasswordLegacyMode NAME NOT FOUND Length: 16 2:59:12.0872517 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Lsa SUCCESS 2:59:12.0872765 PM lsass.exe 680 ReadFile C:\Windows\System32\samsrv.dll SUCCESS Offset: 849,408, Length: 8,192, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal 2:59:12.0878818 PM lsass.exe 680 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0 2:59:12.0879005 PM lsass.exe 680 RegOpenKey HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\SAM NAME NOT FOUND Desired Access: Read 2:59:12.0879238 PM lsass.exe 680 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0 2:59:12.0879354 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\SAM REPARSE Desired Access: Read 2:59:12.0879493 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\SAM SUCCESS Desired Access: Read 2:59:12.0879705 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\SAM\SamDuplicateAccountNamePolicy NAME NOT FOUND Length: 16 2:59:12.0879856 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\SAM SUCCESS 2:59:12.0880707 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Services\SAMSS REPARSE Desired Access: All Access 2:59:12.0880839 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Services\SAMSS SUCCESS Desired Access: All Access 2:59:12.0881091 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Services\SamSs\PostPromoteBoot NAME NOT FOUND Length: 0 2:59:12.0881228 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Services\SamSs SUCCESS

Looking at the procmon logs for the 2019 server more carefully I noticed that it's not that LSASS is taking a long time to parse the 50K users from SAM, it appears to be stuck in a loop!! LSASS keep reading all the SAM entries over and over again in a loop for 12 hours! Here's an excerpt from the logs showing when it ends one loop and then starts another loop. It's the exact same loop everytime. I can't see the end of the loop because procmon timed out after a few hours of logging but for those few hours it ends the last user, then reads the services NTDS and then restart the SAM user loop again. Anyone may any sense of this? 2:59:42.4202794 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 39,993, Name: X 2:59:42.4202969 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS Desired Access: Read 2:59:42.4203111 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\X\(Default) SUCCESS Type: 2:59:42.4203232 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS 2:59:42.4203319 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names NO MORE ENTRIES Index: 39,994, Length: 84 2:59:42.4203506 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS 2:59:42.4203699 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users SUCCESS Index: 9, Name: XX 2:59:42.4203963 PM lsass.exe 680 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0 2:59:42.4204086 PM lsass.exe 680 RegOpenKey HKLM\SYSTEM\CurrentControlSet\Services\NTDS REPARSE Desired Access: Query Value 2:59:42.4204269 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS Desired Access: Query Value 2:59:42.4204447 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Services\NTDS\DirectoryServiceExtPt NAME NOT FOUND Length: 144 2:59:42.4204594 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS 2:59:42.4204755 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\DOMAINS\Account\Users\Names SUCCESS Desired Access: Read 2:59:42.4205331 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 0, Name: XXX 2:59:42.4205462 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\XXX SUCCESS Desired Access: Read 2:59:42.4205604 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\XXX\(Default) SUCCESS Type:

So I'm looking the registry of 2021R2 vs 2019 and I see that in 2012R2 there is no entries for any of the users under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users, however in 2019 there's an entry for each user under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users and Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users\Names In both 2012R2 and 2019 I see all the users are saved under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users The one difference that I'm seeing in the procmon boot logs is that in 2012R2 there are other activities happening along side with the lsass SAM enumeration. For example I see wmiprvse.exe working alongside lsass.exe throughout the process where as the 2019 boot logs are dominated only by lsass.exe for 9 hours straight. One other difference I see is in how LSASS.exe is reading SAM is very different in 2012R2 vs 2019. In 2019 it's a straight out reading the database incrementing the index number one at a time where as in 2012R2 it's using a different pattern to read the SAM database from the registry (but alongside other things) But I think the biggest difference I'm seeing is that there are other activities going on in parallel with LSASS.exe (including logonui, msmpeng etc), so basically in 2012R2 things are moving in parallel with LSASS and once it starts the login process LSASS just seems to stop enumerating the SAM users, where as in 2019 it's a serial process, unless LSASS completes, nothing else happens.

LASS also appears to be doing the same/similar thing in 2012R2 (according to the procmon logs) but it's just doing it a *hit ton faster, it's completes what's it's doing for the 50K users in under 3 minutes where as 2019 and 2022 are both taking between 12-15 hours to do it at boot time. After boot is completed it appears to be okay but when running on AWS you don't get 12 hours of 100% CPU, you'll run out of CPU credits, you only get about 150 minutes of 100% CPU (less than 2 hours) so you can't just wait for it. MS really did quite a number on their newer servers to take such a performance hit and it appears to be registry related. This is an old application and I'm not sure it can rewritten at this time. I will look into AD but I'm; hoping there's a tweak to speed up LASS/SAM. @psydii can you recommend a easy process to setup AD to serve this application? I'm assuming there are similar API/batch commands to automate creation/deletion of users.

I found the culprit, it's LSASS (using ProcMon and boot logs thanks for the tip @chaplic), it's iterating through every user account in the registry HKLM\SAM\Domains\Account\Users and for every user it's making 4 calls to the registry 2:59:15.2752917 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 8,897, Name: XXXX 2:59:15.2753102 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\XXXX SUCCESS Desired Access: Read 2:59:15.2753356 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\XXXX\(Default) SUCCESS Type: 2:59:15.2753501 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names\XXXX SUCCESS So the question is how do I stop LSASS from doing this? For some reason 2012R2 isn't doing it but 2019 and 2022 servers are doing this and it's taking them 12 hours to complete it before logging in. Is there a registry setting to control this?

Thank you all for your suggestions, I do want to try to get the bottom of this. Like it was mentioned it's crazy that it's working on 2012R2 but not a newer version. Infact I tried 2016, 2019 and 2022 servers on AWS just for kicks and with each newer version of the server OS it got slower and slower. It's insane! Okay coming back, so the 2019 server with 50K users takes about 12 hours to complete the boot up and login process. I'm almost convinced it's a SAM issue. When I look at the server system event logs (without any users created, clean install 2019), the Directory-Services-SAM events, there are 3 SAM event logs that complete within about 1 second. On the server with the 50k users, those same 3 events are 12 hours apart, the first one right after Wininit and the the next one 12 hours later (when login takes place) So the first one after Wininit: > Remote calls to the SAM database are being restricted using the default security descriptor: O:SYG:SYD:(A;;RC;;;BA). For more information please see http://go.microsoft.com/fwlink/?LinkId=787651. The second one after 12 hours: > The domain is configured with the following minimum password length-related settings. MinimumPasswordLength: 0 MinimumPasswordLengthAudit: -1 For more information see https://go.microsoft.com/fwlink/?LinkId=2097191. And the third one immediately thereafter: > The security account manager is now logging periodic summary events for remote clients that call legacy password change or set RPC methods. For more information please see https://go.microsoft.com/fwlink/?linkid=2150956. Any thoughts on how to get SAM to speed up or not load all profiles at boot up?

Dont' mind trying it out but can you share your thought on how that might change things? Would that change how the users are stored or processed? After installing the DC component do I need to do something different to migrate the users from local to DC? DC is new to me. My apps just uses the built in Windows authentication for IIS, assuming that won't change with DC. Thanks

Yes, I just kill the old VM and spin up a new one to try the next iteration. I tried limiting the number of users, at 100 users it's loading fine. At 3000 users it takes about 15 minutes, at 50K users it's taking 12 hours to complete the boot login. When it did eventually complete the login after 12 hours I looked the system logs and there was a 3 hour gap between the kernel boot and next system message and the next message something about the SAM being unavailable. I suspect it may be something with LASS or SAM but I don't know where to look. I thought about upgrading 2012R2 to 2019 but no idea how to it on AWS LightSail. The thing about Lightsail is that the images come with a built in license to use, so it's easy without having to figure out how to upgrade and transfer licenses. If you know how to upgrade that also would be completely worth trying.

I have one more insight, I created a new 2019 server, added the 50k new account but then just left the machine without rebooting. I logged out of the RDP session, wait for a few hours and tried to RDP back in and now it's hung on "Please wait for Local Session Manager" and it just times out, the console shows the CPU usage spikes to 100%. So it seems like something with Local Session Manager is causing the issue. I did the same with a 2022 server also and the same results (so it's not specific to 2019) Any ideas?

@psydii Thank you for the suggestion. I tried that but didn't make any difference. I started a new machine, made the changes to the registry, then ran the script to add the local users, everything works fine until I reboot and it's stuck at the spinning dots again. I also tried using a 8GB with 4 processor setup but it didn't make any difference either. Any other thoughts or suggestions. Is there to export all the server settings so I can compare the difference between the 2012R2 and 2019 server settings?

I've been using Windows server 2012R2 Standard for almost a decade now. I have apps running on the server which have created about 50,000 local users. These are local users created using WMIC scripts and eq. API's. The purpose is to allow for easy integration with IIS authentication. The app creates a local user account and IIS uses the built in Windows authentication to authenticate the user.It's been running great on 2021R2 but I recently decided to upgrade to a 2019 Datacenter edition so I created a new 2019 server and used the same scripts to recreate the 50,000 users on the 2019 server.However when I reboot the 2019 server it just hangs for hours at the spinning circle and takes up 100% CPU. It never get the login screen. I've never had this issues with 2012R2, to test it I spun up a new 2012R2 server and recreated the same 50,000 local users with the scripts and it took a few minutes to reboot but that was it. I start with a clean 2019 image, run the scripts to create the users and it's working great, the moment I reboot it just keeps spinning on the circle with 100% CPU utilization. When I do the exact same process with 2012R2 it works fine.Is there something different about 2019 server datacenter vs 2012r2 standard which causes the 2019 server to hang on boot when there are 50,000 local users? Is it trying to process all the users before presenting the login (I've set it to login automatically to an admin account but it never seems to get there). I'm at my wits end now and I'm open to suggestions on how to debug this further or what settings should I look at? I've checked out the group security and logon policies and they're both the same for the 2012 and 2019 servers. What am I overlooking?They're both running on AWS with 1GB RAM. Thanks in advance and please don't hesitate to throw out whatever ideas you may have. I cannot change the architecture at this time and the easiest way to integrate IIS with authentication is to create a local user account. They don't need to have local login privileges only network login (for IIS), so these accounts are added to a custom Group on the windows server and removed from the standard Users group.