Had a weird issue this week- in preparation for an overseas trip I am trying to make all of the infrastructure a bit more robust. In doing that I created a new Open Directory Master server and attempted to set up at least one Replica running under ESXi. But no matter what I did, it kept failing. Now I’ve been doing this for a long time and though I had it fairly well polished by now, but obviously something very odd is happening.

So I went back and created the server on a real Mac with no host abstraction- and it worked straight away!

Interestingly I took logs from 2 attempts- one with the ESXi VM Master and one with the Physical Master- no other changes made on the VM Replica- nothing. Let’s take a look.

Here’s the successful replica build-

2016-03-06 02:13:30 +0000 slapconfig -createreplica

2016-03-06 02:13:31 +0000 1 Creating computer record for replica

2016-03-06 02:13:39 +0000 command: /usr/sbin/slapconfig -delkeychain /LDAPv3/127.0.0.1 od-replica.servicemax.private$

2016-03-06 02:13:39 +0000 slapconfig -delkeychain

2016-03-06 02:13:39 +0000 Added computer password to keychain

2016-03-06 02:13:39 +0000 Adding ldap and host service principals

2016-03-06 02:13:40 +0000 2 Creating ldap replicator user

2016-03-06 02:13:41 +0000 ServerID for this replica 2

2016-03-06 02:13:42 +0000 command: /usr/bin/sntp -s time.asia.apple.com.

2016-03-06 02:13:46 +0000 3 Updating local replica configuration

2016-03-06 02:13:46 +0000 4 Gathering replication data from the master

2016-03-06 02:13:46 +0000 5 Copying master database to new replica

2016-03-06 02:13:46 +0000 Removed directory at path /var/db/openldap/openldap-data.

2016-03-06 02:13:46 +0000 Retrieved DB_CONFIG for dc=server,dc=servicemax,dc=private from master

2016-03-06 02:13:46 +0000 Retrieved DB_CONFIG for cn=authdata from master

2016-03-06 02:13:46 +0000 Starting LDAP server (slapd)

2016-03-06 02:13:46 +0000 slapd started

2016-03-06 02:13:46 +0000 Stopping LDAP server (slapd)

2016-03-06 02:13:49 +0000 command: /usr/sbin/slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d

2016-03-06 02:13:49 +0000 command: /usr/sbin/slapadd -c -w -l /var/db/openldap/openldap-data/backup.ldif

2016-03-06 02:13:49 +0000 command: /usr/sbin/slapadd -c -w -b cn=authdata -l /var/db/openldap/authdata/authdata.ldif

2016-03-06 02:13:49 +0000 

2016-03-06 02:13:49 +0000 56db925d slapd is running in import mode – only use if importing large data 

56db925d bdb_monitor_db_open: monitoring disabled; configure monitor database to enable

2016-03-06 02:13:49 +0000 6 Starting new replica

2016-03-06 02:13:49 +0000 Starting LDAP server (slapd)

2016-03-06 02:13:49 +0000 slapd started

2016-03-06 02:13:49 +0000 set AccountPolicy Override account

2016-03-06 02:13:49 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-06 02:13:49 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-06 02:13:49 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID

2016-03-06 02:13:49 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-06 02:13:49 +0000 Starting password server

2016-03-06 02:13:55 +0000 7 Enabling local Kerberos server

2016-03-06 02:13:55 +0000 Configuring Kerberos server, realm is SERVER.SERVICEMAX.PRIVATE

2016-03-06 02:13:55 +0000 command: /usr/sbin/sso_util configure -x -k -r SERVER.SERVICEMAX.PRIVATE -f /LDAPv3/ldapi://%2Fvar%2Frun%2Fldapi -a diradmin -p **** -v 1 all

2016-03-06 02:13:58 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-06 02:13:58 +0000 Stopping LDAP server (slapd)

2016-03-06 02:13:59 +0000 Starting LDAP server (slapd)

2016-03-06 02:13:59 +0000 slapd started

2016-03-06 02:13:59 +0000 8 Enabling syncprov overlay on the replica

2016-03-06 02:13:59 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config objectClass=olcSyncProvConfig dn

2016-03-06 02:13:59 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-06 02:13:59 +0000 adding new entry “olcOverlay=syncprov,olcDatabase={1}bdb,cn=config”

2016-03-06 02:13:59 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-06 02:13:59 +0000 adding new entry “olcOverlay=syncprov,olcDatabase={2}bdb,cn=config”

2016-03-06 02:13:59 +0000 9 Adding replica to master

2016-03-06 02:13:59 +0000 Remote server (server.servicemax.private) ID: 1

2016-03-06 02:13:59 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b dc=server,dc=servicemax,dc=private uid=_ldap_replicator dn

2016-03-06 02:14:00 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID

2016-03-06 02:14:00 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config objectClass=olcSyncProvConfig dn

2016-03-06 02:14:00 +0000 default realm: SERVER.SERVICEMAX.PRIVATE

2016-03-06 02:14:00 +0000 Configuring multimaster

2016-03-06 02:14:00 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID

2016-03-06 02:14:00 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-06 02:14:00 +0000 modifying entry “cn=config”

modifying entry “olcDatabase={1}bdb,cn=config”

modifying entry “olcDatabase={1}bdb,cn=config”

modifying entry “olcDatabase={2}bdb,cn=config”

modifying entry “olcDatabase={2}bdb,cn=config”

2016-03-06 02:14:00 +0000 Updating ldapreplicas on server.servicemax.private as diradmin

2016-03-06 02:14:00 +0000 Updating ldapreplicas record

2016-03-06 02:14:00 +0000 Updating ldapreplicas plist.

2016-03-06 02:14:00 +0000 Binding to 127.0.0.1

2016-03-06 02:14:01 +0000 Configuring multimaster for (od-replica.servicemax.private) with ServerID (2)

2016-03-06 02:14:01 +0000 command: /usr/bin/ldapadd -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-06 02:14:01 +0000 Could not find root CA certificate in system keychain

2016-03-06 02:14:01 +0000 Could not find root CA certificate in system keychain

2016-03-06 02:14:01 +0000 10 Enabling intermediate CA

2016-03-06 02:14:01 +0000 Could not find matching identity in system keychain

2016-03-06 02:14:05 +0000 Deleting certificate od-replica.servicemax.private

2016-03-06 02:14:05 +0000 searching 002D009941FEA1BBF41BFFC31E2FACB244D23EB9

2016-03-06 02:14:05 +0000 command: /bin/launchctl load -w /System/Library/LaunchDaemons/com.apple.xscertd.plist

2016-03-06 02:14:05 +0000 command: /bin/launchctl load -w /System/Library/LaunchDaemons/com.apple.xscertd-helper.plist

2016-03-06 02:14:05 +0000 command: /bin/launchctl load -w /System/Library/LaunchDaemons/com.apple.xscertadmin.plist

2016-03-06 02:14:05 +0000 Removed directory at path /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/.

2016-03-06 02:14:05 +0000 Found identity name: od-replica.servicemax.private

2016-03-06 02:14:05 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-06 02:14:05 +0000 Replica Creation successfully completed

 

And here’s the previous one that didn’t work- Differences in Red

2016-03-05 06:14:07 +0000 NSMutableDictionary *_getRootDSE(const char *): rootDSE not found

2016-03-05 06:14:07 +0000 Error: Unable to determine the master’s software version.

2016-03-05 06:15:07 +0000 NSMutableDictionary *_getRootDSE(const char *): rootDSE not found

2016-03-05 06:15:07 +0000 Error: Unable to determine the master’s software version.

2016-03-05 06:26:17 +0000 slapconfig -createreplica

2016-03-05 06:26:18 +0000 1 Creating computer record for replica

2016-03-05 06:26:24 +0000 command: /usr/sbin/slapconfig -delkeychain /LDAPv3/127.0.0.1 od-replica.servicemax.private$

2016-03-05 06:26:24 +0000 slapconfig -delkeychain

2016-03-05 06:26:24 +0000 Added computer password to keychain

2016-03-05 06:26:24 +0000 Adding ldap and host service principals

2016-03-05 06:26:24 +0000 2 Creating ldap replicator user

2016-03-05 06:26:25 +0000 GetLastServerID: Error creating DSLDAPContainer: 77014 Can’t contact LDAP server (-1)

2016-03-05 06:26:25 +0000 ServerID for this replica 1

2016-03-05 06:26:25 +0000 SetLastServerID: Unable to create DSLDAPContainer: 77014 Can’t contact LDAP server (-1)

2016-03-05 06:26:25 +0000 Error setting last server id

2016-03-05 06:26:27 +0000 command: /usr/bin/sntp -s time.asia.apple.com.

2016-03-05 06:26:28 +0000 3 Updating local replica configuration

2016-03-05 06:26:28 +0000 4 Gathering replication data from the master

2016-03-05 06:26:28 +0000 5 Copying master database to new replica

2016-03-05 06:26:28 +0000 Removed directory at path /var/db/openldap/openldap-data.

2016-03-05 06:26:28 +0000 Retrieved DB_CONFIG for dc=server,dc=servicemax,dc=com,dc=au from master

2016-03-05 06:26:28 +0000 Retrieved DB_CONFIG for cn=authdata from master

2016-03-05 06:26:28 +0000 Starting LDAP server (slapd)

2016-03-05 06:26:28 +0000 slapd started

2016-03-05 06:26:28 +0000 Stopping LDAP server (slapd)

2016-03-05 06:26:31 +0000 command: /usr/sbin/slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d

2016-03-05 06:26:31 +0000 command: /usr/sbin/slapadd -c -w -l /var/db/openldap/openldap-data/backup.ldif

2016-03-05 06:26:31 +0000 command: /usr/sbin/slapadd -c -w -b cn=authdata -l /var/db/openldap/authdata/authdata.ldif

2016-03-05 06:26:31 +0000 

2016-03-05 06:26:31 +0000 56da7c17 slapd is running in import mode – only use if importing large data 

56da7c17 bdb_monitor_db_open: monitoring disabled; configure monitor database to enable

2016-03-05 06:26:31 +0000 6 Starting new replica

2016-03-05 06:26:31 +0000 Starting LDAP server (slapd)

2016-03-05 06:26:31 +0000 slapd started

2016-03-05 06:26:31 +0000 set AccountPolicy Override account

2016-03-05 06:26:31 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-05 06:26:31 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-05 06:26:31 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID

2016-03-05 06:26:31 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi

2016-03-05 06:26:31 +0000 Starting password server

2016-03-05 06:26:37 +0000 CopyHostGUID: Could not get query results

2016-03-05 06:26:37 +0000 FATAL : Could not retrieve HOST GUID for parent (server.servicemax.private)

2016-03-05 06:26:37 +0000 FATAL : Could not retrieve HOST GUID for parent (server.servicemax.private) (error = 78)

2016-03-05 06:26:37 +0000 Deleting Cert Authority related data

2016-03-05 06:26:37 +0000 OPENDIRECTORY_ROOT_CA_IDENTITY not found, unable to determine rootCA name from OPENDIRECTORY_ROOT_CA_CERTIFICATE, defaulting to configured value of (null)

2016-03-05 06:26:37 +0000 No intCAIdentity, not removing int CA from keychain

2016-03-05 06:26:37 +0000 command: /bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xscertd.plist

2016-03-05 06:26:37 +0000 command: /bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xscertd-helper.plist

2016-03-05 06:26:37 +0000 command: /bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xscertadmin.plist

2016-03-05 06:26:38 +0000 Stopping LDAP server (slapd)

2016-03-05 06:26:39 +0000 Stopping password server

2016-03-05 06:26:42 +0000 Removed all service principals from keytab for realm SERVER.SERVICEMAX.COM.AU

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.001.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.002.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.003.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.004.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.005.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.006.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/altSecurityIdentities.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-config-realname.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-generateduid.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-group-memberguid.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-group-nestedgroup.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-group-realname.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/cn.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/DB_CONFIG.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/dn2id.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/entryCSN.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/entryUUID.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/gidNumber.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/givenName.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/id2entry.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/ipHostNumber.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/log.0000000001.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/macAddress.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/memberUid.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/objectClass.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/ou.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/sn.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/uid.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/uidNumber.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.001.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.002.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.003.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.004.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.005.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.006.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/alock.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/authdata.ldif.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/authGUID.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/DB_CONFIG.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/dn2id.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/draft-krbPrincipalAliases.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/draft-krbPrincipalName.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/entryCSN.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/entryUUID.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/id2entry.bdb.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/log.0000000001.

2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/objectClass.bdb.

2016-03-05 06:26:42 +0000 Removed directory at path /var/db/openldap/authdata.

2016-03-05 06:26:42 +0000 Removed file at path /etc/openldap/slapd_macosxserver.conf.

2016-03-05 06:26:42 +0000 Removed file at path /etc/openldap/slapd.conf.

2016-03-05 06:26:42 +0000 Removed directory at path /etc/openldap/slapd.d/cn=config.

2016-03-05 06:26:42 +0000 Removed file at path /etc/openldap/slapd.d/cn=config.ldif.

2016-03-05 06:26:42 +0000 Removed directory at path /etc/openldap/slapd.d.

2016-03-05 06:26:42 +0000 Removed directory at path /etc/openldap/slapd.d.backup/cn=config.

2016-03-05 06:26:42 +0000 Removed file at path /etc/openldap/slapd.d.backup/cn=config.ldif.

2016-03-05 06:26:42 +0000 Removed directory at path /etc/openldap/slapd.d.backup.

2016-03-05 06:26:45 +0000 Stopping password server

2016-03-05 06:26:45 +0000 Removed file at path /etc/ntp_opendirectory.conf.

2016-03-05 06:26:45 +0000 Removed file at path /Library/Preferences/com.apple.openldap.plist.

 

Now, unfortunately I don’t know enough about the operating system to pick an exact fault, my current theory is this- I’ve been building these servers from an El Capitan installer that has an expired certificate. I ran the installer (in VMware Fusion) then quit before creating an account, keeping this file as the starting point for my VMs. If ( big if) the operating system is not generating that GUID before an account is created, that could mean that the OS is choking because of the expired certificate. Or perhaps there is a GUID clash because the root OS is the same?

As soon as possible I’ll build the server again from a new installer- that should help figure it out. I hope.