EduTech Posted June 21, 2011 Posted June 21, 2011 (edited) Afternoon All, Well, Whilst I'm waiting on Microsoft calling me back to have a look into this issue I thought I would throw it up here in case anyone has ever come accross this before. History: - No One Knows Previous Administrator Passwords - No One Knows Directory Restore Password - No Documentation has ever been made - No other Admin Account can login === So that gave me a mad morning, but now I am getting the following error message also. Directory Services could not start because of the following error: The system cannot find the file specified. Error status: 0xc000000f. Please click Ok to shutdown this system and reboot into Directory Services Restore Mode, check the event log for more detail information. Now that is all well and good, but i don't know the Directory Restore Login? I have been on site here for 4 days now and have built up an aweful lot of documentation but still have a long way to go. So before anyone starts screaming none of this (documentation does not exist) is down to me any one got any ideas? or am i best just waiting for MSFT to hopefully break into it (if they can). Oh! and backups were failing for the past 4 months, I managed to get a sucesful backup last week before this happened. so fingers crossed that backup is all well and good. also, without going into detail the previous IT guy was here for the first day but aint seen him since.. I was asked to lock him out of the system yesterday (change password etc.) but that is all that was done, and the new administrator password worked fine after changing it and rebooting the servers and logging in again worked fine.. but over night something weird happened. James. Edited June 21, 2011 by EduTech
iMash Posted June 21, 2011 Posted June 21, 2011 What version of Windows Server is it running? If it is Windows 2000 then as per the KB article Here then you could try using the backup software to restore Ntds.dit to %SystemRoot%\NTDS folder (default location of Ntds.dit..... the location can be changed at promotion time but usually isnt). As for breaking into the password I would be inclined to run something along the lines of ophcrack on a client and see if the admin password on the clients is the same as the directory services restore mode or admin on the server (Good reason to have your clients administrator account disabled, and the password completely different to any domain passwords). Are you only running a single server with no redundancy? That would be where I would start at least.........
EduTech Posted June 21, 2011 Author Posted June 21, 2011 What version of Windows Server is it running? If it is Windows 2000 then as per the KB article Here then you could try using the backup software to restore Ntds.dit to %SystemRoot%\NTDS folder (default location of Ntds.dit..... the location can be changed at promotion time but usually isnt). As for breaking into the password I would be inclined to run something along the lines of ophcrack on a client and see if the admin password on the clients is the same as the directory services restore mode or admin on the server (Good reason to have your clients administrator account disabled, and the password completely different to any domain passwords). Are you only running a single server with no redundancy? That would be where I would start at least......... Yeah I have already come accross that KB Article for Server 2000, This server is running Small Business Server 2003 Standard. There is no Failover etc. I plan to get rid of SBS in the very near future and do it properly, but being here 4 days you can understand it's not something i could of sorted in that amount of time. I have tried all of the passwords which i have found from doing a scan, none of which work. I even phoned the support company that installed this a few years ago and they gave me a few to try but they did not work either. James.
iMash Posted June 21, 2011 Posted June 21, 2011 (edited) I have found a few things from a website that is for the exchange of experts information (am unsure of the rules of links to 3rd party sites on here can PM you the link if needed). They were for Server 2003. But i can imagine that SBS will be very very similar (Never used SBS myself). That suggested using ERD Commander to reset the domain admin account. I was always under the impression that as a DC does not use a SAM you would not be able to reset it via ERD Commander but others on the site believed you could. So may be worth a try? The other suggestion was........... I just ran into this issue yesterday. I spent hours struggling to find an answer scouring the internet. Luckily, I found out that since I was a MS certified partner, they (Microsoft) would help me for free! Instead of trying to find the original password, the answer is actually rather simple. Do this: Install a parallel copy of the OS (making note of the new password that you create during install). After install, activate product, and simply copy over the new SAM, SOFTWARE, DEFAULT, SYSTEM, and SECURITY from "%systemroot%\system32\config" to your old installation with the same directory. Reboot into the old Operatiing system (Having to activate again). Then, you can restore from a backup a previous copy of the files listed above (I used the previous night's backup system state and restored the whole stinking thing) to bring your Active Directory and settings back. Your server will (fingers crossed) be back to normal and your system will boot up normally. Mine was as if it never happened and lost no data, settings, etc. during the process. The procedure took me approx. 4 hours on a 2003 server including OS install. Edited June 21, 2011 by iMash
EduTech Posted June 21, 2011 Author Posted June 21, 2011 I have found a few things from a website that is for the exchange of experts information (am unsure of the rules of links to 3rd party sites on here can PM you the link if needed). They were for Server 2003. But i can imagine that SBS will be very very similar (Never used SBS myself). That suggested using ERD Commander to reset the domain admin account. I was always under the impression that as a DC does not use a SAM you would not be able to reset it via ERD Commander but others on the site believed you could. So may be worth a try? The other suggestion was........... I was under the impression ERD Commander would not work to be honest as that had crossed my mind also. I'll keep the lata in mind as it might get to me having to do that... joys! it is going to be a long night indeed! James.
Michael Posted June 21, 2011 Posted June 21, 2011 A system restore is definitely needed here, as clearly it appears AD and possibly other services are corrupted. Is there no RAID setup on this system at all? I presume you've tried just a blank password to restore the directory?
EduTech Posted June 21, 2011 Author Posted June 21, 2011 A system restore is definitely needed here, as clearly it appears AD and possibly other services are corrupted. Is there no RAID setup on this system at all? I presume you've tried just a blank password to restore the directory? Yup, Tried Blank Password, and the usual common passwords. It is RAID, type I will have to get back to you as I have not looked yet. James.
RobBaxter Posted June 21, 2011 Posted June 21, 2011 @EduTech that is a total bollox point. Hiren's BootCD 14.0 - All in one Bootable CD » www.hiren.info may help in this case. This is my little O Crap CD. I can't remember, but there is a tool for replacing the System Restore Password. With any luck you can just boot the server, replace it and use that to fix the root of the issue. Security 101 Keep server in locked secure location, cause if your at the terminal and have enought time, no password is strong enough. Unless its BitLocked ofc Hope it helps buddy and good luck!
IanT Posted June 21, 2011 Posted June 21, 2011 Hey James, I bet your having fun, sounds like it! Some suggestions as I've played around with SBS 2003: Offline Windows Password & Registry Editor If you manage to get back into the server: How To Reset the Directory Services Restore Mode Administrator Account Password in Windows Server 2003 Reset Directory Service Restore Mode Administrator Password for windows 2008 via GPO
KK20 Posted June 21, 2011 Posted June 21, 2011 (edited) new administrator password worked fine after changing it and rebooting the servers and logging in again worked fine.. but over night something weird happened. Did you suspend VPNs? I suspect foul play or a severely corrupted AD database. If you have successfully logged in with an administrator account then either AD has turned very corrupt in which case it would be restore from backups anyway or someone has killed the account password. Obviously if you restore from backup you wont have an administrator account yet again (although the previous IT guy will have his account active again). Is anyones PERSONAL account a member of domain admins? Ntds.dit is the usual culprit of directory services errors. Since you havent reinstalled then the certificates on the server are still valid so if a backup exists then try copying that back over. TBH though you can either waste time cracking the admin account or start from scratch. I dont think you can reset the password in the same way as a normal windows machine as there isnt a local admin password on a DC. Domain controllers store their password in the AD directory - only after a DCPROMO demotion will you get a local admin password (having said that I have never used SBS therefore SBS could behave differently to a regular DC server) Not nice but you arent a miracle worker! Edited June 21, 2011 by KK20
EduTech Posted June 21, 2011 Author Posted June 21, 2011 Security is something I am getting on top off, they had 5900, 3389 and various other ports wide open and did not know the password the to WatchGuard Firewall so I had to rebuild that from scratch on thursday. I have not been able to lock down 3389 as of yet because I have to deploy a new solution quickly and I don't want to have to do it twice when the least line goes live in a couple of days so i'm just keeping my fingers crossed for the time being. (that will get locked down, and people will only be able to login using an SSL VPN via the Watchguard Firewall). No User Accounts can login, Only the Administrator, Myself and another account has Domain Admin / Enterprise Admin Rights and neither of them work. I am not sure if the previous guy has done something malicious over the weekend that has only caused an issue since me rebooting because last night even though i could auhenticate fine, it did feel a bit ropy and then around 10pm the server went offline.. and i came in this morning and it was powered on but had crashed so i had to reboot it and that is when i could not login at all (and i got quite worried) and then after trying a few things, safe mode, restore mode etc. this error now comes up with regards to the ntds.dit file etc. I just hope to god that this backup is in a good state because it has not been tested or anything. __ One thing that made me think this morning is that an SBS Server normally sits at the pre logon screen for a while doing it's business because of how much it has to deal with, but yet it got to the logon box in less than 30 seconds which is odd.. as if all the services was not started so i just assumed that maybe the logon/ad services were not starting which is why i could then not authenticate would make sense, but it leaves me stuck. MD wants to know how it has happened, as he is convinced the previous guy has done it and I can't say nowt as i don't know and won't be saying anything UNTIL i have concrete facts. and even then I want to stay out of it as much as possible. (they really upset him, and he has threatened with causing problems in the past like this). anyway, no call from MSFT as of yet even then the SLA is less than 1hr and that was 2hrs ago! I'm serously going to have to start doing some of the above soon. Thanks for your help chaps, James.
Michael Posted June 21, 2011 Posted June 21, 2011 It is RAID, type I will have to get back to you as I have not looked yet. Well I hope it's RAID1 or RAID5 and not RAID0. That would be a little silly... I would check the status of the RAID array in the RAID BIOS. This should give you a better indication what's going on. Also, the fact the server is POSTing OK, hopefully the issue isn't hardware related. The link IanT posted looks promising and would definitely be something I would explore. If the backup doesn't work, then it doesn't work. You can't be held responsible for this! It sounds to me you're doing a fab job given the circumstances.
ozydave Posted June 21, 2011 Posted June 21, 2011 Hello I have a SAM password cracker (for emergencies). Create a boot cd from the iso boot the server. It's a command line tool. Have used it here when I first took up post. I done the job for me. PM me if your interested All the best
EduTech Posted June 21, 2011 Author Posted June 21, 2011 Right, There are 2 arrays (RAID 1 and RAID 5) for reference. Tried a few more things with MSFT this afternoon, but they now basically want me to restore the system state from a backup (LTO3 Media) to another server (which i have not got) so they can grab somes files, and then using Server 2008/Windows 7 and a USB Drive with the files on they want me to copy the files to the Server and then I am guessing they are hoping it will then work. I am going to struggle to get that data of an LTO3 Catridge because I don't know anyone with one at the moment, I am going to get a copy of the server using Symantec System Recovery or something so i can at least get access to the data using a Virtual Machine if needs be. And then I shall fight with Microsoft with sorting that server. I can't re-build it, no chance with no documentation etc. just a complete nightmare! Don't help that I have the MD jumping up and down because he wants to know if he can take legal action against someone as he thinks that certain person did it maliciously, but to be honest that is the least of my worries right now i need to damn thing up and running. Joy oh Joy! I cannot believe how much of a mess this entire network is, nothing right about it at all and they are an international company! eek! i have not stopped dripping with sweat all day.. :/ scary stuff! I don't think it is hardware related. I'll update everyone in the morning, for now i need to find some drivers and get another box and a LTO3 Backup Drive to restore this media. James.
EduTech Posted June 21, 2011 Author Posted June 21, 2011 The link IanT posted looks promising and would definitely be something I would explore. If the backup doesn't work, then it doesn't work. You can't be held responsible for this! It sounds to me you're doing a fab job given the circumstances. Thanks Mate, I will do that but at the moment you can't even login to the server anymore because of the error it shows with regards to directory services. Joyful! James.
featured_spectre Posted June 21, 2011 Posted June 21, 2011 James, when you log in, find out if there is any form of software like logmein or something to that effect, or if the VPNs have had recent log ins that weren't you. If yes to either then get as many details as possible and disable both. From there you should be able to determine if any malicious action took place. As for the password....ERD Commander works on SBS no problem so you should be able to change the password. Also whilst there, change domain passwords and any admin passwords. Letter number symbol combinations. Tell the boss its a security measure until you find out whats happened.
sted Posted June 21, 2011 Posted June 21, 2011 and if/when you get it back up a a dc do a search for any account with domain/enterprise admin level and change passwords/disable its possible they had an account hidden somewhere
EduTech Posted June 21, 2011 Author Posted June 21, 2011 James, when you log in, find out if there is any form of software like logmein or something to that effect, or if the VPNs have had recent log ins that weren't you. If yes to either then get as many details as possible and disable both. From there you should be able to determine if any malicious action took place. As for the password....ERD Commander works on SBS no problem so you should be able to change the password. Also whilst there, change domain passwords and any admin passwords. Letter number symbol combinations. Tell the boss its a security measure until you find out whats happened. Yeah I will be looking into how it happened once i get it up and running, I know what the password is Admin wise because i set it up the reason it is not working is because Logon Services / Directory Services is shafted! so it's like disabling the local admin password on a laptop and that dropping of the domain sometimes you do get a bit stuck. So at the moment the passwords are not an issue, the fact is Directory Services is really really broken and in order to fix it i need the files from my backup that i have but it is on an LTO3 tape and I don't have a spare drive on another Server and the one they have is internal. SO IF ANYONE IN BIRMINGHAM/WEST MIDLANDS HAS AN LTO3 DRIVE I WOULD LIKE TO COME AND USE IT PLEASE YOU WILL BE DOING ME A HUGE FAVOUR! IF NOT I SHALL ORDER ONE NEXT DAY. Passwords that I set had Numbers, Symbols, Upper and Lower Case Letters so it's not like he could of guessed it afterwards. I was going to be resetting all USER accounts this afternoon to be changed, but that kind of never happened because it is broke. James.
EduTech Posted June 21, 2011 Author Posted June 21, 2011 and if/when you get it back up a a dc do a search for any account with domain/enterprise admin level and change passwords/disable its possible they had an account hidden somewhere I already checked Domain Admin / Enterprise Admins and only myself, admin account and some SQL Account was in there and I changed all passwords to very secure passwords James.
featured_spectre Posted June 21, 2011 Posted June 21, 2011 Active directory is easy to fix...get the SBS install disk, Uninstalled ADUC, reinstall it and tell it to pick up the old domain tree files. It will pick them up and kick it back into shape. Only thing you may have to do is re do some of the GPO stuff as I found doing that sometimes loses GPO settings, so in that instance export your GPO settings and reimport them after. Disclaimer - this method has been used twice by me in the past on server 2008r2. Whether it works for 2003 SBS remains to be seen and I take no responsibility for loss of server functionality if it fails.
PiqueABoo Posted June 21, 2011 Posted June 21, 2011 the reason it is not working is because Logon Services / Directory Services is shafted! Seconded and although AD usually survives them, I think the most likely explanation for this is an abrupt reset e.g. power outage or h/w glitch, as opposed to something malicious.
EduTech Posted June 22, 2011 Author Posted June 22, 2011 I have managed to hunt down someone with a spare backup server which i have put my Storageworks Ultrium 920 (LTO3) drive into and I have managed to get the system state from the backup i created (so that is a good start). I have taken an image of the server as it is, just incase somthing crazy happens during this process I still will be able to get access to any data using a Virtual Machine and attaching the VHD. (just a precaution). That has nearly finished, I will then boot into Windows Server 2008 R2 and copy the Files required on to the broken box replacing the broken/corrupt files and then touch wood it should boot. The .dic files is restored, and it looks like that is the main culprit that needs replacing. I shall let you know more soon as I have then copy done. James.
EduTech Posted June 22, 2011 Author Posted June 22, 2011 Quick Update whilst i intake some coffee.... Server is now back online, Microsoft are going to look at my event logs BUT! here is where it gets scary/annoying/making me want to cry! - Many IMPORTANT Folders have been deleted from server, including Main Database - Exchange has been removed - Backup Exec has been removed - Anti-Virus has been removed - amongst a few other things that DO NOT just disappear! - Event Logs were clean until the morning of Tuesday where events were being logged - Passwords to WatchGuard Firewall no longer work - CCTV has been turned off for quite a bit of time (I was not even aware of it until i mentioned can we see) - the NTDS.DIT File was actually MISSING as oppose to being there and corrupt So, one thing i think i can gather is this was not done by accident also i believe you cannot reset a WatchGuard Firewall without being on-site to put it into safe mode so it leads me to think it was actually done on site. (need to think in a bit more detail) I have so much to do, my hair is falling out! I have a backup which is missing data because someone unselected some important things (i.e. the database) and obviously i were not to know this as whom would of though someone would do such a thing. Complete Nightmare! another late night for me... loads of stuff i need to fix and get back online... and i need to work out how the bloody hell this has happened ready for the police tomorrow! Not Fun, and definitely not something i expected to be dealing after being here for 4 days!
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now