Had a weird issue this week- in preparation for an overseas trip I am trying to make all of the infrastructure a bit more robust. In doing that I created a new Open Directory Master server and attempted to set up at least one Replica running under ESXi. But no matter what I did, it kept failing. Now I’ve been doing this for a long time and though I had it fairly well polished by now, but obviously something very odd is happening.
So I went back and created the server on a real Mac with no host abstraction- and it worked straight away!
Interestingly I took logs from 2 attempts- one with the ESXi VM Master and one with the Physical Master- no other changes made on the VM Replica- nothing. Let’s take a look.
Here’s the successful replica build-
2016-03-06 02:13:30 +0000 slapconfig -createreplica
2016-03-06 02:13:31 +0000 1 Creating computer record for replica
2016-03-06 02:13:39 +0000 command: /usr/sbin/slapconfig -delkeychain /LDAPv3/127.0.0.1 od-replica.servicemax.private$
2016-03-06 02:13:39 +0000 slapconfig -delkeychain
2016-03-06 02:13:39 +0000 Added computer password to keychain
2016-03-06 02:13:39 +0000 Adding ldap and host service principals
2016-03-06 02:13:40 +0000 2 Creating ldap replicator user
2016-03-06 02:13:41 +0000 ServerID for this replica 2
2016-03-06 02:13:42 +0000 command: /usr/bin/sntp -s time.asia.apple.com.
2016-03-06 02:13:46 +0000 3 Updating local replica configuration
2016-03-06 02:13:46 +0000 4 Gathering replication data from the master
2016-03-06 02:13:46 +0000 5 Copying master database to new replica
2016-03-06 02:13:46 +0000 Removed directory at path /var/db/openldap/openldap-data.
2016-03-06 02:13:46 +0000 Retrieved DB_CONFIG for dc=server,dc=servicemax,dc=private from master
2016-03-06 02:13:46 +0000 Retrieved DB_CONFIG for cn=authdata from master
2016-03-06 02:13:46 +0000 Starting LDAP server (slapd)
2016-03-06 02:13:46 +0000 slapd started
2016-03-06 02:13:46 +0000 Stopping LDAP server (slapd)
2016-03-06 02:13:49 +0000 command: /usr/sbin/slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d
2016-03-06 02:13:49 +0000 command: /usr/sbin/slapadd -c -w -l /var/db/openldap/openldap-data/backup.ldif
2016-03-06 02:13:49 +0000 command: /usr/sbin/slapadd -c -w -b cn=authdata -l /var/db/openldap/authdata/authdata.ldif
2016-03-06 02:13:49 +0000
2016-03-06 02:13:49 +0000 56db925d slapd is running in import mode – only use if importing large data
56db925d bdb_monitor_db_open: monitoring disabled; configure monitor database to enable
2016-03-06 02:13:49 +0000 6 Starting new replica
2016-03-06 02:13:49 +0000 Starting LDAP server (slapd)
2016-03-06 02:13:49 +0000 slapd started
2016-03-06 02:13:49 +0000 set AccountPolicy Override account
2016-03-06 02:13:49 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-06 02:13:49 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-06 02:13:49 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID
2016-03-06 02:13:49 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-06 02:13:49 +0000 Starting password server
2016-03-06 02:13:55 +0000 7 Enabling local Kerberos server
2016-03-06 02:13:55 +0000 Configuring Kerberos server, realm is SERVER.SERVICEMAX.PRIVATE
2016-03-06 02:13:55 +0000 command: /usr/sbin/sso_util configure -x -k -r SERVER.SERVICEMAX.PRIVATE -f /LDAPv3/ldapi://%2Fvar%2Frun%2Fldapi -a diradmin -p **** -v 1 all
2016-03-06 02:13:58 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-06 02:13:58 +0000 Stopping LDAP server (slapd)
2016-03-06 02:13:59 +0000 Starting LDAP server (slapd)
2016-03-06 02:13:59 +0000 slapd started
2016-03-06 02:13:59 +0000 8 Enabling syncprov overlay on the replica
2016-03-06 02:13:59 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config objectClass=olcSyncProvConfig dn
2016-03-06 02:13:59 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-06 02:13:59 +0000 adding new entry “olcOverlay=syncprov,olcDatabase={1}bdb,cn=config”
2016-03-06 02:13:59 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-06 02:13:59 +0000 adding new entry “olcOverlay=syncprov,olcDatabase={2}bdb,cn=config”
2016-03-06 02:13:59 +0000 9 Adding replica to master
2016-03-06 02:13:59 +0000 Remote server (server.servicemax.private) ID: 1
2016-03-06 02:13:59 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b dc=server,dc=servicemax,dc=private uid=_ldap_replicator dn
2016-03-06 02:14:00 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID
2016-03-06 02:14:00 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config objectClass=olcSyncProvConfig dn
2016-03-06 02:14:00 +0000 default realm: SERVER.SERVICEMAX.PRIVATE
2016-03-06 02:14:00 +0000 Configuring multimaster
2016-03-06 02:14:00 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID
2016-03-06 02:14:00 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-06 02:14:00 +0000 modifying entry “cn=config”
modifying entry “olcDatabase={1}bdb,cn=config”
modifying entry “olcDatabase={1}bdb,cn=config”
modifying entry “olcDatabase={2}bdb,cn=config”
modifying entry “olcDatabase={2}bdb,cn=config”
2016-03-06 02:14:00 +0000 Updating ldapreplicas on server.servicemax.private as diradmin
2016-03-06 02:14:00 +0000 Updating ldapreplicas record
2016-03-06 02:14:00 +0000 Updating ldapreplicas plist.
2016-03-06 02:14:00 +0000 Binding to 127.0.0.1
2016-03-06 02:14:01 +0000 Configuring multimaster for (od-replica.servicemax.private) with ServerID (2)
2016-03-06 02:14:01 +0000 command: /usr/bin/ldapadd -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-06 02:14:01 +0000 Could not find root CA certificate in system keychain
2016-03-06 02:14:01 +0000 Could not find root CA certificate in system keychain
2016-03-06 02:14:01 +0000 10 Enabling intermediate CA
2016-03-06 02:14:01 +0000 Could not find matching identity in system keychain
2016-03-06 02:14:05 +0000 Deleting certificate od-replica.servicemax.private
2016-03-06 02:14:05 +0000 searching 002D009941FEA1BBF41BFFC31E2FACB244D23EB9
2016-03-06 02:14:05 +0000 command: /bin/launchctl load -w /System/Library/LaunchDaemons/com.apple.xscertd.plist
2016-03-06 02:14:05 +0000 command: /bin/launchctl load -w /System/Library/LaunchDaemons/com.apple.xscertd-helper.plist
2016-03-06 02:14:05 +0000 command: /bin/launchctl load -w /System/Library/LaunchDaemons/com.apple.xscertadmin.plist
2016-03-06 02:14:05 +0000 Removed directory at path /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/.
2016-03-06 02:14:05 +0000 Found identity name: od-replica.servicemax.private
2016-03-06 02:14:05 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-06 02:14:05 +0000 Replica Creation successfully completed
And here’s the previous one that didn’t work- Differences in Red
2016-03-05 06:14:07 +0000 NSMutableDictionary *_getRootDSE(const char *): rootDSE not found
2016-03-05 06:14:07 +0000 Error: Unable to determine the master’s software version.
2016-03-05 06:15:07 +0000 NSMutableDictionary *_getRootDSE(const char *): rootDSE not found
2016-03-05 06:15:07 +0000 Error: Unable to determine the master’s software version.
2016-03-05 06:26:17 +0000 slapconfig -createreplica
2016-03-05 06:26:18 +0000 1 Creating computer record for replica
2016-03-05 06:26:24 +0000 command: /usr/sbin/slapconfig -delkeychain /LDAPv3/127.0.0.1 od-replica.servicemax.private$
2016-03-05 06:26:24 +0000 slapconfig -delkeychain
2016-03-05 06:26:24 +0000 Added computer password to keychain
2016-03-05 06:26:24 +0000 Adding ldap and host service principals
2016-03-05 06:26:24 +0000 2 Creating ldap replicator user
2016-03-05 06:26:25 +0000 GetLastServerID: Error creating DSLDAPContainer: 77014 Can’t contact LDAP server (-1)
2016-03-05 06:26:25 +0000 ServerID for this replica 1
2016-03-05 06:26:25 +0000 SetLastServerID: Unable to create DSLDAPContainer: 77014 Can’t contact LDAP server (-1)
2016-03-05 06:26:25 +0000 Error setting last server id
2016-03-05 06:26:27 +0000 command: /usr/bin/sntp -s time.asia.apple.com.
2016-03-05 06:26:28 +0000 3 Updating local replica configuration
2016-03-05 06:26:28 +0000 4 Gathering replication data from the master
2016-03-05 06:26:28 +0000 5 Copying master database to new replica
2016-03-05 06:26:28 +0000 Removed directory at path /var/db/openldap/openldap-data.
2016-03-05 06:26:28 +0000 Retrieved DB_CONFIG for dc=server,dc=servicemax,dc=com,dc=au from master
2016-03-05 06:26:28 +0000 Retrieved DB_CONFIG for cn=authdata from master
2016-03-05 06:26:28 +0000 Starting LDAP server (slapd)
2016-03-05 06:26:28 +0000 slapd started
2016-03-05 06:26:28 +0000 Stopping LDAP server (slapd)
2016-03-05 06:26:31 +0000 command: /usr/sbin/slaptest -f /etc/openldap/slapd.conf -F /etc/openldap/slapd.d
2016-03-05 06:26:31 +0000 command: /usr/sbin/slapadd -c -w -l /var/db/openldap/openldap-data/backup.ldif
2016-03-05 06:26:31 +0000 command: /usr/sbin/slapadd -c -w -b cn=authdata -l /var/db/openldap/authdata/authdata.ldif
2016-03-05 06:26:31 +0000
2016-03-05 06:26:31 +0000 56da7c17 slapd is running in import mode – only use if importing large data
56da7c17 bdb_monitor_db_open: monitoring disabled; configure monitor database to enable
2016-03-05 06:26:31 +0000 6 Starting new replica
2016-03-05 06:26:31 +0000 Starting LDAP server (slapd)
2016-03-05 06:26:31 +0000 slapd started
2016-03-05 06:26:31 +0000 set AccountPolicy Override account
2016-03-05 06:26:31 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-05 06:26:31 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-05 06:26:31 +0000 command: /usr/bin/ldapsearch -x -LLL -H ldapi://%2Fvar%2Frun%2Fldapi -b cn=config -s base olcServerID
2016-03-05 06:26:31 +0000 command: /usr/bin/ldapmodify -c -x -H ldapi://%2Fvar%2Frun%2Fldapi
2016-03-05 06:26:31 +0000 Starting password server
2016-03-05 06:26:37 +0000 CopyHostGUID: Could not get query results
2016-03-05 06:26:37 +0000 FATAL : Could not retrieve HOST GUID for parent (server.servicemax.private)
2016-03-05 06:26:37 +0000 FATAL : Could not retrieve HOST GUID for parent (server.servicemax.private) (error = 78)
2016-03-05 06:26:37 +0000 Deleting Cert Authority related data
2016-03-05 06:26:37 +0000 OPENDIRECTORY_ROOT_CA_IDENTITY not found, unable to determine rootCA name from OPENDIRECTORY_ROOT_CA_CERTIFICATE, defaulting to configured value of (null)
2016-03-05 06:26:37 +0000 No intCAIdentity, not removing int CA from keychain
2016-03-05 06:26:37 +0000 command: /bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xscertd.plist
2016-03-05 06:26:37 +0000 command: /bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xscertd-helper.plist
2016-03-05 06:26:37 +0000 command: /bin/launchctl unload -w /System/Library/LaunchDaemons/com.apple.xscertadmin.plist
2016-03-05 06:26:38 +0000 Stopping LDAP server (slapd)
2016-03-05 06:26:39 +0000 Stopping password server
2016-03-05 06:26:42 +0000 Removed all service principals from keytab for realm SERVER.SERVICEMAX.COM.AU
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.001.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.002.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.003.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.004.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.005.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/__db.006.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/altSecurityIdentities.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-config-realname.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-generateduid.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-group-memberguid.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-group-nestedgroup.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/apple-group-realname.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/cn.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/DB_CONFIG.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/dn2id.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/entryCSN.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/entryUUID.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/gidNumber.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/givenName.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/id2entry.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/ipHostNumber.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/log.0000000001.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/macAddress.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/memberUid.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/objectClass.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/ou.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/sn.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/uid.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/openldap-data/uidNumber.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.001.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.002.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.003.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.004.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.005.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/__db.006.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/alock.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/authdata.ldif.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/authGUID.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/DB_CONFIG.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/dn2id.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/draft-krbPrincipalAliases.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/draft-krbPrincipalName.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/entryCSN.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/entryUUID.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/id2entry.bdb.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/log.0000000001.
2016-03-05 06:26:42 +0000 Removed file at path /var/db/openldap/authdata/objectClass.bdb.
2016-03-05 06:26:42 +0000 Removed directory at path /var/db/openldap/authdata.
2016-03-05 06:26:42 +0000 Removed file at path /etc/openldap/slapd_macosxserver.conf.
2016-03-05 06:26:42 +0000 Removed file at path /etc/openldap/slapd.conf.
2016-03-05 06:26:42 +0000 Removed directory at path /etc/openldap/slapd.d/cn=config.
2016-03-05 06:26:42 +0000 Removed file at path /etc/openldap/slapd.d/cn=config.ldif.
2016-03-05 06:26:42 +0000 Removed directory at path /etc/openldap/slapd.d.
2016-03-05 06:26:42 +0000 Removed directory at path /etc/openldap/slapd.d.backup/cn=config.
2016-03-05 06:26:42 +0000 Removed file at path /etc/openldap/slapd.d.backup/cn=config.ldif.
2016-03-05 06:26:42 +0000 Removed directory at path /etc/openldap/slapd.d.backup.
2016-03-05 06:26:45 +0000 Stopping password server
2016-03-05 06:26:45 +0000 Removed file at path /etc/ntp_opendirectory.conf.
2016-03-05 06:26:45 +0000 Removed file at path /Library/Preferences/com.apple.openldap.plist.
Now, unfortunately I don’t know enough about the operating system to pick an exact fault, my current theory is this- I’ve been building these servers from an El Capitan installer that has an expired certificate. I ran the installer (in VMware Fusion) then quit before creating an account, keeping this file as the starting point for my VMs. If ( big if) the operating system is not generating that GUID before an account is created, that could mean that the OS is choking because of the expired certificate. Or perhaps there is a GUID clash because the root OS is the same?
As soon as possible I’ll build the server again from a new installer- that should help figure it out. I hope.