Server services are finicky, and Mavericks server is no different. For a long time now, my OS X Open Directory server has refused to accept certificates in the Server app.
The certificates were a bit of a mess: there were outdated certificates from Apple, the root certificate of a Windows domain that I deleted six months ago, that kind of thing. Worse, there were different certificates with the same name. The chances of screwing things up in a couple of years time was not insignificant. I decided, in the end, it’d be better to have a clean start, to replace all the certificates I’d been using on the server. I run my own PKI, using OpenBSD, so this wasn’t so difficult.
When I first replaced the certificates, the old Open Directory problem was still there. It wouldn’t accept the new certificate, even though all the other services would (which resolved another problem, the reason I’d gone through the exercise).
Then I realised I’d forgotten to update my certificate revocation server. Since I’d created a new CA for this renewal, that was necessary. So I fixed my omission.
Once I’d done that, Open Directory accepted my new certificate. I cured the problem by accident.
Anyone else who’s having the same problem might find it worthwhile to check their certification authority properly maintains a Certificate Revocation List. That’s not the only reason why the Open Directory server might sulk, but it seems to be one of them.