Open Directory Problem

[SC92]

Hi all,

 

Bit of a head scratcher here. A few days before half term we had to shut down all servers due to air con failing. For reasons i'm unaware of our mac server was not shutdown before the power switch was hit. The next morning our macs would not let anybody log in, not even admins, so after a few restarts of the server this seemed to rectify itself. All was working right up until easter, so back I came after a 2 week holiday to find that nobody can log into our macs yet again, only this time admin users are able to log in, but not students/teaching staff etc. The strange thing is our ODadmin account seems to have just been wiped off the face of the earth, the home directory for it still exists, but it doesn't now show up in Sys Prefs > Users. Can't log in to it either, which means I also cannot log into WGM and authenticate to LDAP.. So I can't do much at the moment! I have attached the error WGM throws at me when trying to authenticate with the correct credentials to LDAP:

 

 

This is also thrown at me when trying to go into Server Prefs:

 

 

Any ideas?!

 

EDIT: We're using 10.7.3 clients and 10.6.8 server.

Edited April 17, 2012 by SC92
info

[JR-PCS]

Looks like your Open Directory DB has corrupted. You need to unload slapd, run db_recover and then load slapd. If your unsure how to do this, or a bit worried of making it worse it might be worth getting an apple solutions expert or consultant, like me ;-) to fix it for you.

[SC92]

Hi JR-PCS, thanks for the reply. Are there any guides out there on doing this? I'm not really all that worried about making it worse, as it currently can't get that much worse as nothing is working at the moment!

[JR-PCS]

Run these commands from terminal.

 

sudo launchctl unload /System/Library/LaunchDaemons/org.openldap.slapd.plist

 

sudo /usr/bin/db_recover -h /var/db/openldap/openldap-data

 

sudo launchctl load /System/Library/LaunchDaemons/org.openldap.slapd.plist

[SC92]

I get an error from terminal after trying to run the first one; 'launchctl: Error unloading: org.openldap.slapd'.

 

Sam

[JR-PCS]

try the other two commands, as it might be that slapd is not loaded.

[SC92]

I'm able to run the last command, which successfully loads org.openldap.slapd it seems, but then running the second command after that gives me:

 

db_recover: checkpoint LSN record [785] [6635693] not found

db_recover: PANIC: DB_NOTFOUND: No matching key/data pair found

db_recover: DB_ENV->open: DB_RUNRECOVERY: Fatal error, run database recovery

 

Looks as if it wants me to run a recovery, but not sure what/where to find it!

[JR-PCS]

you are running the recovery tool (db_recover). You want slapd to be unloaded when you run the db_recover, it sounded like you loaded slapd first. However it seems like its not finding your database.

 

Could you try running the three commands in the correct order again and take a screenshot of terminal when running those commands. Also go to Server Admin > Open Directory > Logs and give me a screenshot of the LDAP log.

[SC92]

Here are the logs, hope they're of some help!

 

EDIT: Having just read the logs, looks like they need to be restored from backup?!

Edited April 18, 2012 by SC92

[JR-PCS]

Yes, it doesn't seem to be able to recover the db. You could try this, but it probably won't work either. (notice it has a -c in the recover command this time)

 

sudo launchctl unload /System/Library/LaunchDaemons/org.openldap.slapd.plist

 

sudo /usr/bin/db_recover -c -h /var/db/openldap/openldap-data

 

sudo launchctl load /System/Library/LaunchDaemons/org.openldap.slapd.plist

 

Have you ever taken an archive of Open Directory? If you haven't and don't have a server backup to use as a restore you may have no choice but to change your OD server to standalone, restart and then promote to OD master. This will give you a blank OD, which you will need to setup again.

[SC92]

It seemed like I was able to run the above 3 commands without fail, as in, no errors were returned when running them and all looked successful. How will I know if this had sorted the issue?!

[JR-PCS]

Is everything ok in Workgroup manager now, or does that still error?

[SC92]

Is everything ok in Workgroup manager now, or does that still error?

 

Can't even begin to thank you enough, all is now working! Was dreading the thought of rebuilding OD from scratch, thanks so much!

[JR-PCS]

Can't even begin to thank you enough, all is now working! Was dreading the thought of rebuilding OD from scratch, thanks so much!

 

Glad I could help. But your school might want to consider a backup plan, in case this ever happens again!

[Ashm]

Glad I could help. But your school might want to consider a backup plan, in case this ever happens again!

 

What do you recommend for backup? I currently backup our xserve via rsync and take regular archives of the Open Directory DB. Planning to use Carbon Copy Cloner & script Open Directory DB backup before CCC backup.

 

Thanks

[JR-PCS]

What you're planning to do is what I would recommend. You could have two drives that you alternate.

 

What do you recommend for backup? I currently backup our xserve via rsync and take regular archives of the Open Directory DB. Planning to use Carbon Copy Cloner & script Open Directory DB backup before CCC backup.

 

Thanks