I'm going to assume a few things here. First that you're trying to authenticate to the mac server so that you can mount a file share. My primary question would be, have you given restricted access rights to AFP or SMB services?
Next, in terms of the SSO go to the command line and type kinit see if you can check out a tgt. If you cannot check out a tgt then there's something wrong with your AD server and I can't help you. To check out a tgt you do not need the mac to be kerberized. If you can successfully check the tgt out then see if you can login into the mac server as an ad account. Do this from the login screen and not remotely, upon successful login check the klist com and again to see if you got a tgt assigned.
Next, repeat this on a mac client that is bound to both AD and OD. Login as an AD user and see if you get a tgt. if you do, the try to authenticate to the AFP server and check afterwards. You should see some extra tickets.