50,000 users on 2019 hangs but works with 2012R2, any solution?

[coderr]

I've been using Windows server 2012R2 Standard for almost a decade now. I have apps running on the server which have created about 50,000 local users. These are local users created using WMIC scripts and eq. API's. The purpose is to allow for easy integration with IIS authentication. The app creates a local user account and IIS uses the built in Windows authentication to authenticate the user.It's been running great on 2021R2 but I recently decided to upgrade to a 2019 Datacenter edition so I created a new 2019 server and used the same scripts to recreate the 50,000 users on the 2019 server.However when I reboot the 2019 server it just hangs for hours at the spinning circle and takes up 100% CPU. It never get the login screen. I've never had this issues with 2012R2, to test it I spun up a new 2012R2 server and recreated the same 50,000 local users with the scripts and it took a few minutes to reboot but that was it.

 

I start with a clean 2019 image, run the scripts to create the users and it's working great, the moment I reboot it just keeps spinning on the circle with 100% CPU utilization. When I do the exact same process with 2012R2 it works fine.Is there something different about 2019 server datacenter vs 2012r2 standard which causes the 2019 server to hang on boot when there are 50,000 local users? Is it trying to process all the users before presenting the login (I've set it to login automatically to an admin account but it never seems to get there). I'm at my wits end now and I'm open to suggestions on how to debug this further or what settings should I look at? I've checked out the group security and logon policies and they're both the same for the 2012 and 2019 servers. What am I overlooking?They're both running on AWS with 1GB RAM. Thanks in advance and please don't hesitate to throw out whatever ideas you may have. I cannot change the architecture at this time and the easiest way to integrate IIS with authentication is to create a local user account. They don't need to have local login privileges only network login (for IIS), so these accounts are added to a custom Group on the windows server and removed from the standard Users group.

[psydii]

Take a look at whether your getting per user firewall rules created.

 

https://community.spiceworks.com/topic/2285411-server2019-rds-hundreds-of-firewall-rules-per-user-per-session

[coderr]

@psydii Thank you for the suggestion. I tried that but didn't make any difference. I started a new machine, made the changes to the registry, then ran the script to add the local users, everything works fine until I reboot and it's stuck at the spinning dots again. I also tried using a 8GB with 4 processor setup but it didn't make any difference either. Any other thoughts or suggestions. Is there to export all the server settings so I can compare the difference between the 2012R2 and 2019 server settings?

[coderr]

I have one more insight, I created a new 2019 server, added the 50k new account but then just left the machine without rebooting. I logged out of the RDP session, wait for a few hours and tried to RDP back in and now it's hung on "Please wait for Local Session Manager" and it just times out, the console shows the CPU usage spikes to 100%. So it seems like something with Local Session Manager is causing the issue. I did the same with a 2022 server also and the same results (so it's not specific to 2019) Any ideas? Edited March 27, 2023 by coderr

[Koldov]

Sorry, I can absolutely NOT help you with your issue, or offer any insights into what is going on but...

 

From your OP these are cloud based VMs and you have no issue spinning up new ones and flattening old ones...?

 

Just a thought out of left-field... could you create your 2012R2 instance and all the users etc. Then in-place upgrade to 2019 (got to admit I'm not sure how easy that is on a cloud server).

 

Also is there any way you could use a modified script to create a much smaller amount of users to test? If it starts with a good amount of users you can at least then export some of the settings/logs/stats/IIS settings/check users firewall/resources/collect memory dumps etc.

[chaplic]

So Im guessing something enumerating something and let’s say it’s 10 seconds per user then it’s gonna be forever. I’d try going hardcore disabling services especially things like WMI to see if that helps. Id love to get a procmon trace (and you can configure it to do boot time logging) but the question is, how do you get it off the machine

 

Having just said disable all services, can you enable powershell removing and get into it via powershell from another machine when it’s spinning away to see what its doing?

[coderr]

Yes, I just kill the old VM and spin up a new one to try the next iteration.

 

I tried limiting the number of users, at 100 users it's loading fine. At 3000 users it takes about 15 minutes, at 50K users it's taking 12 hours to complete the boot login. When it did eventually complete the login after 12 hours I looked the system logs and there was a 3 hour gap between the kernel boot and next system message and the next message something about the SAM being unavailable. I suspect it may be something with LASS or SAM but I don't know where to look.

 

I thought about upgrading 2012R2 to 2019 but no idea how to it on AWS LightSail. The thing about Lightsail is that the images come with a built in license to use, so it's easy without having to figure out how to upgrade and transfer licenses. If you know how to upgrade that also would be completely worth trying.

[chaplic]

For giggles Id try turning it into a domain controller; 50k users isn’t going to be a big deal and there’s lots of sizing info out there (you might find your app just works!)

if it’s still slow then it’s not directly related to user accounts

[coderr]

Dont' mind trying it out but can you share your thought on how that might change things? Would that change how the users are stored or processed? After installing the DC component do I need to do something different to migrate the users from local to DC? DC is new to me. My apps just uses the built in Windows authentication for IIS, assuming that won't change with DC. Thanks

[chaplic]

Youve got an issue because, I suspect, no-one else in the world has 50k accounts on an individual computer. AD having 50k accounts wouldn't be unusual.

 

Install DNS on the server, point the DNS settings on the nic at itself, then configure DNS forwarding for your ISPs DNS, run dcpromo. The local user accounts will be upgraded to domain user accounts. There's a reasonable chance that everything might still work. On the flip side if something is still slow there's something well odd going on.

[localzuk]

How many cores do your VMs have? 1GB RAM is too low if you're using the desktop experience - you need minimum 2GB.

 

But, as mentioned by @chaplic, 50k local accounts is getting into the world of extremely niche use cases for Windows Server. I'd say you'd be better relooking at how your login/auth processes are working - either move to a service like Active Directory, or move your users into an SQL database and handle authentication differently.

 

You really shouldn't be using Windows logins for website auth.

[Koldov]

OP says they have also tried using a 8GB with 4 processor setup.

 

I think it's going to be one of those situations where it might not be the best way to do it, or it might not be the best practise and there may be a whole new better way to do it, if you forget everything you know and completely change everything about the entire process, that's got to be tough.

 

I think it we've all had those times where something that's worked just fine for 10 years suddenly doesn't work anymore for no 'apparent' reason thanks to some kind of 'progress' and it's mind blowing... So the only way to get over it is to try and understand why...

 

EDIT: Probably not what you want to hear given the above, but if you are just trying to get away from Server 2012R2 due to its EOL, did you try 2016 (although I think anything based off a newer OS might give you the same issues as it's obviously doing something very differently from the OS 2012 is based on)?

 

Edited March 29, 2023 by Koldov

[psydii]

How many cores do your VMs have? 1GB RAM is too low if you're using the desktop experience - you need minimum 2GB.

 

But, as mentioned by @chaplic, 50k local accounts is getting into the world of extremely niche use cases for Windows Server. I'd say you'd be better relooking at how your login/auth processes are working - either move to a service like Active Directory, or move your users into an SQL database and handle authentication differently.

 

You really shouldn't be using Windows logins for website auth.

 

Quite, but it might be worth opening a ticket with Microsoft Support.

[coderr]

Thank you all for your suggestions, I do want to try to get the bottom of this. Like it was mentioned it's crazy that it's working on 2012R2 but not a newer version. Infact I tried 2016, 2019 and 2022 servers on AWS just for kicks and with each newer version of the server OS it got slower and slower. It's insane!

 

Okay coming back, so the 2019 server with 50K users takes about 12 hours to complete the boot up and login process.

 

 

I'm almost convinced it's a SAM issue. When I look at the server system event logs (without any users created, clean install 2019), the Directory-Services-SAM events, there are 3 SAM event logs that complete within about 1 second. On the server with the 50k users, those same 3 events are 12 hours apart, the first one right after Wininit and the the next one 12 hours later (when login takes place)

 

So the first one after Wininit:

> Remote calls to the SAM database are being restricted using the default security descriptor: O:SYG:SYD:(A;;RC;;;BA).

For more information please see http://go.microsoft.com/fwlink/?LinkId=787651.

 

The second one after 12 hours:

> The domain is configured with the following minimum password length-related settings.

MinimumPasswordLength: 0

MinimumPasswordLengthAudit: -1

For more information see https://go.microsoft.com/fwlink/?LinkId=2097191.

 

And the third one immediately thereafter:

> The security account manager is now logging periodic summary events for remote clients that call legacy password change or set RPC methods.

For more information please see https://go.microsoft.com/fwlink/?linkid=2150956.

 

Any thoughts on how to get SAM to speed up or not load all profiles at boot up?

[psydii]

I'm hesitant to recommend lowering the security posture of a server, but this might point you in the right direction:

https://learn.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/network-access-restrict-clients-allowed-to-make-remote-sam-calls

 

Basically Since 2016 there is a 'new' security limitation on accessing the SAM. It might be that it is this check that is slowing you down. I don't think there is a documented way to disable the check.

 

A check that takes 100ms per account (perhaps there is a timeout? I wonder what account/subsystem is calling the sam at this time?) would account for the hour delay during boot.

 

It might be time to make friends with this: https://learn.microsoft.com/en-us/windows-hardware/test/wpt/windows-performance-analyzer

 

It can record *everything* that is going on at boot time. Try comparing a clean 2019 boot trace to one captured from your problem server.

[coderr]

I found the culprit, it's LSASS (using ProcMon and boot logs thanks for the tip @chaplic), it's iterating through every user account in the registry HKLM\SAM\Domains\Account\Users and for every user it's making 4 calls to the registry

2:59:15.2752917 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 8,897, Name: XXXX

2:59:15.2753102 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\XXXX SUCCESS Desired Access: Read

2:59:15.2753356 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\XXXX\(Default) SUCCESS Type:

2:59:15.2753501 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names\XXXX SUCCESS

 

So the question is how do I stop LSASS from doing this? For some reason 2012R2 isn't doing it but 2019 and 2022 servers are doing this and it's taking them 12 hours to complete it before logging in. Is there a registry setting to control this?

[chaplic]

Before it starts doing that, you can see what other keys LSASS is looking up (including ones that dont exist) and you might find one that is named 'DontDoThisSlowStuff' or similar. Bit of a crap shoot but cannot see microsoft offering up a support response.

[dmj]

So the question is how do I stop LSASS from doing this? For some reason 2012R2 isn't doing it but 2019 and 2022 servers are doing this and it's taking them 12 hours to complete it before logging in. Is there a registry setting to control this?

Don't. There are much better ways to solve this as other have pointed out. Putting 50k users as local logins for a web application isn't the answer to this problem, annoying as it might be.

[Koldov]

So the question is how do I stop LSASS from doing this?

 

Isn't this possibly something LSASS has to do? I'm talking out of my a$$ here obviously as I'm no expert, but from the little I know isn't LSASS a security process?

 

I can only imagine in more recent iterations of Windows Server, this has got more and more 'hardened' with much more emphasis being put on the security of user accounts because so many threats try to attack targeting credentials.

 

On the other hand if it's only doing this at boot... does it run fine after that? How many times is it going to need to reboot?

[psydii]

*Only* LSASS should be doing that.

 

However, "why is it doing that specifically" it not an unreasonable question.

 

What happens on the 2012R2 box? Is something else querying for the users (and it is LSASs's job to respond), or does the post 2016 LSASS query all accounts in the SAM each boot time as part of the changes made (linked in my previous post)?

 

As per @chaplic's suggestion, I wonder how much work it would be to re-tool it for having the accounts in AD stood up for just this task...? Moving from local auth to domain auth in windows applications might be the least painful solution*. This is however upgrading from a 1994-style solution to a 1999-style solution and might be a false economy.

 

 

 

*and for the love of god, don't have the Domain doing anything other than serving this application, if it gets compromised it is toast.

[coderr]

LASS also appears to be doing the same/similar thing in 2012R2 (according to the procmon logs) but it's just doing it a *hit ton faster, it's completes what's it's doing for the 50K users in under 3 minutes where as 2019 and 2022 are both taking between 12-15 hours to do it at boot time. After boot is completed it appears to be okay but when running on AWS you don't get 12 hours of 100% CPU, you'll run out of CPU credits, you only get about 150 minutes of 100% CPU (less than 2 hours) so you can't just wait for it. MS really did quite a number on their newer servers to take such a performance hit and it appears to be registry related.

 

This is an old application and I'm not sure it can rewritten at this time. I will look into AD but I'm; hoping there's a tweak to speed up LASS/SAM.

@psydii can you recommend a easy process to setup AD to serve this application? I'm assuming there are similar API/batch commands to automate creation/deletion of users.

Edited March 30, 2023 by coderr

[coderr]

So I'm looking the registry of 2021R2 vs 2019 and I see that in 2012R2 there is no entries for any of the users under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users, however in 2019 there's an entry for each user under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users and Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users\Names

 

In both 2012R2 and 2019 I see all the users are saved under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users

 

The one difference that I'm seeing in the procmon boot logs is that in 2012R2 there are other activities happening along side with the lsass SAM enumeration. For example I see wmiprvse.exe working alongside lsass.exe throughout the process where as the 2019 boot logs are dominated only by lsass.exe for 9 hours straight.

 

One other difference I see is in how LSASS.exe is reading SAM is very different in 2012R2 vs 2019. In 2019 it's a straight out reading the database incrementing the index number one at a time where as in 2012R2 it's using a different pattern to read the SAM database from the registry (but alongside other things)

 

But I think the biggest difference I'm seeing is that there are other activities going on in parallel with LSASS.exe (including logonui, msmpeng etc), so basically in 2012R2 things are moving in parallel with LSASS and once it starts the login process LSASS just seems to stop enumerating the SAM users, where as in 2019 it's a serial process, unless LSASS completes, nothing else happens.

[coderr]

Looking at the procmon logs for the 2019 server more carefully I noticed that it's not that LSASS is taking a long time to parse the 50K users from SAM, it appears to be stuck in a loop!! LSASS keep reading all the SAM entries over and over again in a loop for 12 hours! Here's an excerpt from the logs showing when it ends one loop and then starts another loop. It's the exact same loop everytime. I can't see the end of the loop because procmon timed out after a few hours of logging but for those few hours it ends the last user, then reads the services NTDS and then restart the SAM user loop again. Anyone may any sense of this?

 

 

2:59:42.4202794 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 39,993, Name: X

2:59:42.4202969 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS Desired Access: Read

2:59:42.4203111 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\X\(Default) SUCCESS Type:

2:59:42.4203232 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS

2:59:42.4203319 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names NO MORE ENTRIES Index: 39,994, Length: 84

2:59:42.4203506 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS

2:59:42.4203699 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users SUCCESS Index: 9, Name: XX

2:59:42.4203963 PM lsass.exe 680 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0

2:59:42.4204086 PM lsass.exe 680 RegOpenKey HKLM\SYSTEM\CurrentControlSet\Services\NTDS REPARSE Desired Access: Query Value

2:59:42.4204269 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS Desired Access: Query Value

2:59:42.4204447 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Services\NTDS\DirectoryServiceExtPt NAME NOT FOUND Length: 144

2:59:42.4204594 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS

2:59:42.4204755 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\DOMAINS\Account\Users\Names SUCCESS Desired Access: Read

2:59:42.4205331 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 0, Name: XXX

2:59:42.4205462 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\XXX SUCCESS Desired Access: Read

2:59:42.4205604 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\XXX\(Default) SUCCESS Type:

[coderr]

Before it starts doing that, you can see what other keys LSASS is looking up (including ones that dont exist) and you might find one that is named 'DontDoThisSlowStuff' or similar. Bit of a crap shoot but cannot see microsoft offering up a support response.

Here's a list of keys that LSASS reads before starting to enumerate SAM (and before it ends up on the reading loop). This from 2019, the only difference between 2012R2 and 2019 are the SampPasswordLegacyModeEnabled and PasswordLegacyMode keys which aren't there in the 2012R2 logs. The rest are identical. Anything here strikes a note with you?

2:59:12.0661138 PM lsass.exe 680 RegOpenKey HKLM\SECURITY\Policy\LastPassCompleted SUCCESS Desired Access: Read

2:59:12.0661454 PM lsass.exe 680 RegQueryValue HKLM\SECURITY\Policy\LastPassCompleted\(Default) BUFFER OVERFLOW Length: 12

2:59:12.0661616 PM lsass.exe 680 RegCloseKey HKLM\SECURITY\Policy\LastPassCompleted SUCCESS

2:59:12.0661779 PM lsass.exe 680 RegOpenKey HKLM\SECURITY\Policy\LastPassCompleted SUCCESS Desired Access: Read

2:59:12.0661823 PM lsass.exe 680 RegCreateKey HKLM\System\CurrentControlSet\Control\Lsa\Audit\PerUserAuditing\System REPARSE Desired Access: Query Value, Set Value

2:59:12.0662621 PM lsass.exe 680 RegCreateKey HKLM\System\CurrentControlSet\Control\Lsa\Audit\PerUserAuditing\System SUCCESS Desired Access: Query Value, Set Value, Disposition: REG_OPENED_EXISTING_KEY

2:59:12.0662635 PM lsass.exe 680 RegQueryValue HKLM\SECURITY\Policy\LastPassCompleted\(Default) SUCCESS Type: REG_NONE, Length: 4, Data: 02 00 00 00

2:59:12.0662892 PM lsass.exe 680 RegCloseKey HKLM\SECURITY\Policy\LastPassCompleted SUCCESS

2:59:12.0663144 PM lsass.exe 680 RegEnumValue HKLM\System\CurrentControlSet\Control\Lsa\Audit\PerUserAuditing\System NO MORE ENTRIES Index: 0, Length: 686

2:59:12.0663565 PM lsass.exe 680 RegCreateKey HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy REPARSE Desired Access: Query Value, Set Value

2:59:12.0663709 PM lsass.exe 680 RegCreateKey HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy SUCCESS Desired Access: Query Value, Set Value, Disposition: REG_OPENED_EXISTING_KEY

2:59:12.0663958 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy\AuditPolicySD BUFFER OVERFLOW Length: 12

2:59:12.0664096 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy\AuditPolicySD BUFFER OVERFLOW Length: 144

2:59:12.0664220 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\Audit\AuditPolicy\AuditPolicySD SUCCESS Type: REG_BINARY, Length: 324, Data: 01 00 00 00 D0 8C 9D DF 01 15 D1 11 8C 7A 00 C0

2:59:12.0665151 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers REPARSE Desired Access: Read

2:59:12.0665332 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS Desired Access: Read

2:59:12.0665594 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS

2:59:12.0665896 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration REPARSE Desired Access: Read

2:59:12.0666161 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS Desired Access: Read

2:59:12.0666512 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS

2:59:12.0667472 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers REPARSE Desired Access: Read

2:59:12.0667756 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS Desired Access: Read

2:59:12.0668072 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS

2:59:12.0668302 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration REPARSE Desired Access: Read

2:59:12.0668534 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS Desired Access: Read

2:59:12.0668739 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS

2:59:12.0669259 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers REPARSE Desired Access: Read

2:59:12.0669440 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS Desired Access: Read

2:59:12.0669749 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Providers SUCCESS

2:59:12.0670113 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration REPARSE Desired Access: Read

2:59:12.0670454 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS Desired Access: Read

2:59:12.0670852 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Cryptography\Configuration SUCCESS

2:59:12.0672185 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Hvsi REPARSE Desired Access: Read

2:59:12.0672340 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Hvsi SUCCESS Desired Access: Read

2:59:12.0672602 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Hvsi\IsHvsiContainer NAME NOT FOUND Length: 16

2:59:12.0672764 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Hvsi SUCCESS

2:59:12.0683589 PM lsass.exe 680 RegOpenKey HKLM\SYSTEM\CurrentControlSet\Control\Lsa REPARSE Desired Access: Query Value

2:59:12.0683973 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\Lsa SUCCESS Desired Access: Query Value

2:59:12.0684279 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LookupLogLevel NAME NOT FOUND Length: 16

2:59:12.0684636 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupReturnSidTypeDeleted NAME NOT FOUND Length: 16

2:59:12.0684966 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupRestrictIsolatedNameLevel NAME NOT FOUND Length: 16

2:59:12.0685116 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsarpcServerAllowRemotedSecretOperations NAME NOT FOUND Length: 16

2:59:12.0685250 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupCacheRefreshTime NAME NOT FOUND Length: 16

2:59:12.0685373 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupCacheExpireTime NAME NOT FOUND Length: 16

2:59:12.0685492 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaLookupCacheMaxSize NAME NOT FOUND Length: 16

2:59:12.0685618 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LsaAllowReturningUnencryptedSecrets NAME NOT FOUND Length: 16

2:59:12.0685783 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Lsa SUCCESS

2:59:12.0867158 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\NoLmHash SUCCESS Type: REG_DWORD, Length: 4, Data: 1

2:59:12.0867387 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamReplicatePasswordsUrgently NAME NOT FOUND Length: 16

2:59:12.0867539 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\ForceGuest SUCCESS Type: REG_DWORD, Length: 4, Data: 0

2:59:12.0867682 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\LimitBlankPasswordUse SUCCESS Type: REG_DWORD, Length: 4, Data: 1

2:59:12.0867821 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamAccountLockoutTestMode NAME NOT FOUND Length: 16

2:59:12.0867951 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableListenOnTCP NAME NOT FOUND Length: 16

2:59:12.0868081 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SampPasswordLegacyModeEnabled NAME NOT FOUND Length: 16

2:59:12.0868228 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\IgnoreGCFailures NAME NOT FOUND Length: 16

2:59:12.0868397 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamNoGcLogonEnforceKerberosIpCheck NAME NOT FOUND Length: 16

2:59:12.0868540 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamNoGcLogonEnforceNTLMCheck NAME NOT FOUND Length: 16

2:59:12.0868671 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableSingleObjectRepl NAME NOT FOUND Length: 16

2:59:12.0868799 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableRSOOnPDCForward NAME NOT FOUND Length: 16

2:59:12.0868935 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableResetBadPwdCountForward NAME NOT FOUND Length: 16

2:59:12.0869071 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamConnectedAccountsExist NAME NOT FOUND Length: 16

2:59:12.0869200 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamDisableOutboundRSO NAME NOT FOUND Length: 16

2:59:12.0869910 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictAnonymous SUCCESS Type: REG_DWORD, Length: 4, Data: 0

2:59:12.0870048 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictAnonymousSam SUCCESS Type: REG_DWORD, Length: 4, Data: 1

2:59:12.0870192 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictRemoteSamEventThrottlingWindow NAME NOT FOUND Length: 16

2:59:12.0870324 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictRemoteSamAuditOnlyMode NAME NOT FOUND Length: 16

2:59:12.0870448 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictRemoteSam BUFFER OVERFLOW Length: 12

2:59:12.0870624 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\RestrictRemoteSam SUCCESS Type: REG_SZ, Length: 70, Data: O:BAG:BAD:(A;;RC;;;BA)(A;;RC;;;WD)

2:59:12.0871097 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\ExtendedSidEmulationMode NAME NOT FOUND Length: 16

2:59:12.0871233 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamLogSize NAME NOT FOUND Length: 16

2:59:12.0871360 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamLogLevel NAME NOT FOUND Length: 16

2:59:12.0871488 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamRestrictOwfPasswordChange NAME NOT FOUND Length: 16

2:59:12.0871614 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\MaxSamConnections NAME NOT FOUND Length: 16

2:59:12.0871739 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\dsrmAdminLogonBehavior NAME NOT FOUND Length: 16

2:59:12.0871865 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\SamMaxQueueLengthForPDCForward NAME NOT FOUND Length: 16

2:59:12.0871987 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\EnableClaimsTransformationEcho NAME NOT FOUND Length: 16

2:59:12.0872110 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\EnumerationCachePurgeInterval NAME NOT FOUND Length: 16

2:59:12.0872236 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\EnumerationCacheEntryLifetime NAME NOT FOUND Length: 16

2:59:12.0872359 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\Lsa\PasswordLegacyMode NAME NOT FOUND Length: 16

2:59:12.0872517 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\Lsa SUCCESS

2:59:12.0872765 PM lsass.exe 680 ReadFile C:\Windows\System32\samsrv.dll SUCCESS Offset: 849,408, Length: 8,192, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal

2:59:12.0878818 PM lsass.exe 680 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0

2:59:12.0879005 PM lsass.exe 680 RegOpenKey HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\SAM NAME NOT FOUND Desired Access: Read

2:59:12.0879238 PM lsass.exe 680 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0

2:59:12.0879354 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\SAM REPARSE Desired Access: Read

2:59:12.0879493 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Control\SAM SUCCESS Desired Access: Read

2:59:12.0879705 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Control\SAM\SamDuplicateAccountNamePolicy NAME NOT FOUND Length: 16

2:59:12.0879856 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Control\SAM SUCCESS

2:59:12.0880707 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Services\SAMSS REPARSE Desired Access: All Access

2:59:12.0880839 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Services\SAMSS SUCCESS Desired Access: All Access

2:59:12.0881091 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Services\SamSs\PostPromoteBoot NAME NOT FOUND Length: 0

2:59:12.0881228 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Services\SamSs SUCCESS

 

 

Edited March 31, 2023 by coderr

[psydii]

I haven’t done an in-place-upgraded of a production SAM to ADDS for about 22 years. I seem to recall one just installs the AD role on the server with the accounts and it just happens.

 

(So realising what I just said, don’t worry about standing up a second server… just add the role to the existing server)

 

The SAM still exists for disaster recovery purposes, so it is possible that lsass will still parse it on boot even when AD is stood up.

 

I think it must be worth opening a ticket with Microsoft about this, probably with the Windows Server team.

Edited March 31, 2023 by psydii