Corrupt AD-is reinstall my only option?

[cheeseslice]

Hi,

 

I have been asked to look at a small network. 1 2008r2 server and about 60 workstations.

 

Initially there was a problem that required them to demote the DC, I don't know what it was. When they then used dcpromo to promote the server again it worked but the dns server stopped working, giving errors about not being able to access the AD database.

 

This is where I was called in. I had to use dcpromo with the force removal switch. I deleted all existing entries in the dns(it started working again) I then attempted to clean up the metadata data, though this failed too.

 

There is no backup to restore to, I have scoured the forums and I can't find anyone with a similar problem so I'm kind of stuck now as I have never seen this before. The problem is obviously AD but I just can't seem to clean it up.

 

Any advice would be appreciated.

[sukh]

What exactly is the issue?

 

How many Dc, setup up, sites, dns integrated?

[cheeseslice]

Hi,

 

They had a single DC, running active directory with integrated DNS and DHCP.

 

The issue is that after they demoted and then promoted this server it broke everything. The DNS server will not run and they are constantly getting error messages from the file server role.

 

When I demoted the server again the dns started to work.

 

Logs show an error message 4000 mentioning the fate that the AD directory is corrupt/inaccessible.

 

I need to get this machine working the way it has been before but the only way I can see to achieve this is by wiping and starting again.

[sukh]

a single dc, single forest, single domain, then when you dcpormo and remove there isnt anything left, running dcpromo again, did you create a new forest/domain with the same name?

[m25man]

Only an authorative restore from a backup that included the AD is going to recover from this.

The database was wiped and a new Domain SID created when it was demoted and promoted back again.

 

All of the clients are now orphans as are all of the users, you my friend are in a **** storm that nobody wants.

Unless your being paid by the hour I would get on the first bus outta there.

[sukh]

be careful you dont get the blame for this, but as you have no backups, it aint looking good.

 

Whoever removed AD would have got a warning mentioning the consequences, they must have acknowleded that.

 

Going to take some time to set it all up, with 60 pc's and a file server to permission may take a few days, but its all the other stuff that ad may have contained, from 3rd party apps, or maybe even exch. Groups/dl/certificates and what ever else may have been there.

[cheeseslice]

Thanks for all the replies.

 

So am I right in thinking I should just format the os partition and start again? I have tried recreating the forest and domain using the same information as the original and as a completely new domain, no matter what I do the end result is errors about a corrupt AD and nothing working.

 

I was actually considering recommending that they either install 2008r2 with hyper v or esxi and then run 2 virtual dc's to try and prevent this from happening again. They have the licences but there is no chance of an additional server being bought.

 

Someone mentioned charging by the hour, I was actually going to ask about this in another post. Do you charge by the hour or by the job?

 

I got this job by recommendation from another job I just completed. I had to wipe a windows 7 machine, reinstall the software, fix a corrupt vista profile and rewrite some group policies. This doesn't seem like a lot of work but I was On site for 3 hours doing the windows 7 machine(I was asked to try and repair the corrupt hard drive before wiping) the another 4 hours fixing the corrupt profile, rejoining the machine to the domain and setting up and testing the policies remotely.

 

I don't know what the going rate is but at £25 p/h that's £150. Is that excessive?

[sukh]

Yea, virtualise, least you have 2 dc although of the hardware fails you stuffed again unless there is some tolerance.

 

I would charge by the hour for this job.

[cheeseslice]

It was too early this morning. 7x£25 is obviously £175 not £150. Which makes it worse as I was thinking £150 sounded too expensive!

[m25man]

Even if you got a new copy of the OS installed and actually got AD up and running you have 60 user accounts to create 60 machines to rejoin to the domain 60 user profiles to fix and untold ACL issues to address as the security on all of your existing folders and files will be screwed.

An MSP will be charging £750 a day and that could be a weeks work to clean up!

 

Only then will they see the value of a good backup.

 

If I were them I'd be contacting a solicitor to see if I could sue somebody, but alas you said that you used the dcpromo /force remove option which is rather like putting your head in the lions mouth to see if his teeth are ok!

 

Its now only your word to say that the other guy screwed up.... Be very careful what you say to the client.

 

Hindsight is of no use to you here, if your going to have to fix this you need a plan and a good one or it could take weeks to mop this mess up.

[cheeseslice]

Hi,

 

They have no issue with who is responsible, they know that everything had already been messed up well before I got involved.

 

I suppose I really wanted advice on wether I'm going down the right path just wiping the os and starting again from scratch?

 

The other point about charging for this work. I haven't actually billed the first customer who recommended me and my intention was to be charging him for 7 hours work which may seem excessive to him.

 

I find that people with small companies only really appreciate work that they can physically see. Tell them it took 4 hours to fix a corrupted profile, set up roaming profiles for the admin department and create some group policies and they think I'm sticking my arm in!

[m25man]

What you charge is relative to your overheads, your free to negotiate from minimum wage to £150 ph +

I don't expect they would bat an eyelid at a Solicitors bill for £500 or an Accountant...

 

They deserve to pay a high price for their stupidity, that type of business disruption can cause businesses to fail don't undervalue yourself or others that may have to follow in your footsteps.

[Jamo]

It seems you will have to completely start again, I would get the original network up and running before considering virtualization though. Too many changes at once can be an absolute nightmare!!!

 

If the network is already gone you might as well start up another server leaving the busted one as is and use it to at least migrate data across. You will have to join each and every pc back to the new domain and recreate all the users. All shares and folder permissions on shares will have to be reassigned (bet there are no docs on that!!) and then all GPO's recreated etc. Its a mammoth task. Its the same as going in to a place with no AD, but now you have data to take care of as well.

[mrwobbly]

With regard to the £25 ph.

 

What do electricians and plumbers charge them per hour? Its £35 to £45 ph around here to schools, I’m sure it’s probably more to businesses. They also generate funds from the mark up on supplying parts. Is your skill, knowledge and time spent learning your craft worth less than those trades?

 

Working as self employed you have to add cost to allow for all those extra expenses you incur.

[Steve21]

In terms of the £25 per hour, I don't think it's unreasonable, "HOWEVER" I think it should have be agreed before the work was started. No use turning around after and saying a figure if they didn't actually agree to it as such.

 

End of day I'm assuming something must have been agreed beforehand, unless it was more of a favour? What's been signed so far etc?

 

Steve

[cheeseslice]

I have done work for this person before and when he is stuck with something I can fix quickly by remote I don't normally charge him anything but I was originally asked to look at a corrupt hard drive and ended up fixing a multitude of problems. Then i was informed the new admin department were on the network but that some of the drive mappings weren't working, profiles corrupt etc. This is why a price wasn't really discussed.

 

So a 2008r2/windows 7 network working pefectly overnight turned into a mash up of xp/vista/apple/android/iOS devices. with them wondering why group policies wouldn't apply to xp machines or why new staff couldn't access specific folders.

 

Fast forward to this new job. I was in the middle of repairing the corrupt desktop when I get introduced to someone, a 5 minute walk and I'm being shown the mess we have been talking about.

 

In hindsight I should have given him a quote or estimate but it happened quite quickly.

 

Normally my jobs are by word of mouth and I have a discussion with them before hand to talk about their requirements and budget.

[Steve21]

I don't normally charge him anything

 

 

 

Aye, Don't get me wrong, just saying that if you do sometimes do favours etc, before getting yourself into days if not weeks of work, probably best to sit down and discuss this, else they may assume it's free, you spend ages on it, and get nothing. Then everyone gets into a fight over it etc etc End of day, best protect yourself first. IF they don't want to pay, that's their choice. But if you turn around after doing it all, and say pay me, legally there was no agreements etc etc.

 

Always protect number one (yourself)

 

Steve

[cheeseslice]

I have just been working on the server. I thought I would be smart, remove every role, install hyper-v then install a new virtual version of 2008r2 to get me up and running.

 

Looks like even this is doomed, keep getting error messages about invalid credentials when i try to connect to the new virtual machine, think tomorrow I'll start wiping!

[sukh]

AD is gone, so I would format and start fresh. If any data is on the server Files etc...back them up before formatting.

[NikChillin]

I think you should step back a bit here, get all the information and go and discuss the problem in detail, timescale and costs to repair. If you took the car to the garage you would expect to be notified prior to work being done.

They might think is is a simple 'restore to earlier date' thing....

[JJonas]

AD is gone, so I would format and start fresh. If any data is on the server Files etc...back them up more than once before formatting.

 

Belt and braces. If possible check you can restore you really dont want to make the problem any worse.

 

Maybe you need to level with them about how long this is going to take and how much it is going to cost?

 

Could you quote them for giving them a server they can log onto from some machines?

 

Then a quote to add an additional machine

 

and separate quotes for setting up additional things

 

might ease the shock a bit.

[plexer]

I'd hope in 6 months they are back up and running again?

 

Checks the posting dates people

 

Ben

[Michael]

I'm still finding it hard to believe that someone demoted a single DC then expected everything to be restored from what exactly? Definitely a head scratching moment...

[chazzy2501]

before you blow the os away backup the event logs, this will show that a user prior to you (date and time) demoted the only dc! it may even have the warning in the log.

[edutech4schools]

I charge £40ph. I would stop what your doing and explain the situation to your client and explain how long it could take and how much you charge. Than if they agree you know where you stand.