“DSA is attempting to start after a long outage, perform a recovery procedure before starting”
Challenge: The IMPD (Identity Manager Provisioning Directory) Data DSAs have been offline for a while, e.g. 7 days+ (> 1 week), and the Symantec/CA Directory solution will, to protect the data, refuse to allow the DATA DSAs to start unless there is manual intervention to prevent the possibility of production data (Live DATA DSAs) being synced with older data (Offline DATA DSAs).
If we were concern, we would follow best practices and remove the offline DATA DSAs’ *.db & *.dp files, and replace the *.db with current copies of the Live DATA DSAs’ *.db files; generate temporary time files of *.dx and allow the time files of *.dp to rebuild themselves upon startup of the offline DATA DSAs.
However, if we are NOT concern, or the environment is non-production we can avoid the multiple shells, multiple commands to resync by using a combinations of bash shell commands. The proposal below outlines using the Symantec/CA Identity Suite virtual appliance, where both the IMPD and IMPS (Identity Manager Provisioning Server) components reside on the same servers.
Proposal: Use a single Linux host to send remote commands as a single user ID; sudo to the ‘dsa’ and ‘imps’ service IDs, and issue commands to address the restart process.
Pre-Work: For the Identity Suite vApp, recommend that .ssh keys be used to avoid using a password for the ‘config’ user IDs on all vApp nodes.
If using .SSH keys, do not forget to use this shortcut to cache the local session: eval `ssh-agent` && ssh-add
Steps: Issue the following bash commands with the correct IPs or hostnames.
If possible, wrap the remote commands in a for-loop. The below example uses the local ‘config’ user ID, to ssh to remote servers, then issues a local su to the ‘dsa’ service ID. The ‘dsa’ commands may need to be wrapped as shown below to allow multiple commands to be executed together. We have a quick hostname check, stop all IMPD DATA DSAs, find the time-stamp file that is preventing the startup of the IMPD DATA DSAs and remove it, restart all IMPD DATA DSA, and then move on to the next server with the for-loop. The ‘imps’ commands are similar with a quick hostname check, status check, stop and start process, another status check, then move on to the next server in the for-loop.
for i in {136..141}; do ssh -t config@192.168.242.$i "su - dsa -c \"hostname;dxserver stop all;pwd;find ./data/ -type f \( -name '*.dp' \) -delete ;dxserver start all \" "; done
for i in {136..141}; do ssh -t config@192.168.242.$i "su - imps -c \"hostname;imps status;imps stop;imps start;imps status \" "; done
View of for-loop commands output:
Additional: Process to assist with decision to sync or not sync.
Check if the number of total entries in each individual IMPD DATA DSA match with their peers (Multi-Write groups). Goal: Avoid any deltas > 1% between peers. The IMPD “main”, “co”, “inc” DATA DSA should be 100% in sync. We may see some minor flux in the “notify” DATA DSA, as this is temporary data used by the IMPS server to store data to be sent to the IME via the IME Call Back Process.
If there are any deltas, then we may export the IMPD DATA DSAs to LDIF files and then use the Symantec/CA Directory ldifdelta process to isolate and triage the deltas.
su - dsa OR [ sudo -iu dsa ]
export HISTIGNORE=' *' {USE THIS LINE TO FORCE HISTORY TO IGNORE ANY COMMANDS WITH A LEADING SPACE CHARACTER}
echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd {USE SPACE CHARACTER IN FRONT TO AVOID HISTORY USAGE}
# NOTIFY BRANCH (TCP 20404)
for i in {135..140}; do echo "########## 192.168.242.$i IMPD NOTIFY DATA DSA ##########";LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://192.168.242.$i:20404 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=notify,dc=etadb' '(objectClass=*)' dxTotalEntryCount | perl -p00e 's/\r?\n //g' ; done
# INC BRANCH (TCP 20398)
for i in {135..140}; do echo "########## 192.168.242.$i IMPD INC DATA DSA ##########";LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://192.168.242.$i:20398 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTInclusionContainerName=Inclusions,eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount | perl -p00e 's/\r?\n //g' ; done
# CO BRANCH (TCP 20396)
for i in {135..140}; do echo "########## 192.168.242.$i IMPD CO DATA DSA ##########";LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://192.168.242.$i:20396 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount | perl -p00e 's/\r?\n //g' ; done
# MAIN BRANCH (TCP 20394)
for i in {135..140}; do echo "########## 192.168.242.$i IMPD MAIN DATA DSA ##########";LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://192.168.242.$i:20394 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount | perl -p00e 's/\r?\n //g' ; done
NOTIFY DSA is temporary data and will have deltas. This DSA is used for the IME CALL BACK process.
Ensure the hostname entry is a FQDN or alias. It can not be an IP address if MS Exchange is to be managed through this connector, due to conflict with Kerberos authentication and IP addresses. If the object was created with an IP address, it may be changed via Jxplorer for two (2) attributes: eTADSprimaryServer and eTADSServerName.
2. General Information on the ADS Endpoint Logging Tab and where this information is stored. Only two (2) the Destination have value with current deployment, e.g. Text File & System Log (MS Windows Event viewer) for Active Directory (ADS). The “Text File” will output data to two (2) files: jcs\logs\ADS\<endpoint-name>.log and ccs\logs\ADS\<endpoint-name>.log
4. Additionally, the User ID may be in one of three (3) formats: UPN (serviceid@exchange.lab), NT ( domain\serviceid ), LDAP DN ( cn=serviceid,ou=people,dc=exchange,dc=lab). We recommend UPN or NT format to allow the embedded API features for MS Exchange powershell management to correctly function. If the ID is to be changed, a password update must be done as well, since the User ID is part of the seed for the encrypted password for the service ID to be stored in CA Directory on the ADS endpoint object.
Note: SASL is encrypted traffic. If wireshark is used to intercept the traffic, the service ID may be seen during initial authentication, but NOT the password nor the payload data.
Notes on SASL validation for Active Directory. {Pro: No need to worry about TLS certificates rotation on client connections – all TLS is managed by the server}
:: Search ADS / LDAP store what is offered for SASL (use -x for simple connection) ldapsearch -x -h dc2016.exchange.lab -p 389 -b “” -LLL -s base supportedSASLMechanisms
SASL appears to connect on TCP 636 briefly, then use TCP 389 extensively. Other ports are 80 (Service), 135 (lsass.exe for home folders), 6405 (lsass.exe). If Kerberos authentication is defined for the service ID, then other ports will be used, e.g. 3268/3269. TCP 4104/4105 are for the legacy CAM/CAFT agents (typically not used any more).
Recommendation: Add these TCP Ports to any Firewall between the IM JCS/CCS Server and the Active Directory Domain Controllers to improve performance and avoid time-out delays.
One of the challenges that IAM/IAG solutions may have is using single thread processing for select endpoints. For the CA/Symantec Identity Management solution, before IM r14.3cp2, we lived with a single-threaded connector to managed MS Active Directory endpoints.
To address this challenge, we deployed multiple connector servers. We allowed the IM Provisioning Server (IMPS) to use a built-in round-robin approach of load-balancing separate transactions to different connector servers, which would service the same Active Directory endpoints.
The IME may be running as fast as it can with its clustered deployment, but as soon as a task has MS Active Directory, and there is a bottleneck with the CCS Service. We begin to see the IME JMS queue reporting that it is stuck and the IME View Submitted Task reporting “In Progress” for all tasks. If the CCS service is restarted, all IME tasks are then reported as “Failed.”
This is/was the bottleneck for the solution for sites that have MS Active Directory for Birthright/DayOne Access.
We can now avoid this bottleneck. [*** (5/24/2021) – There is an enhancement to CP2 to address im_ccs.exe crashes during peak loads discovered using this testing process. ]
We now have full parallel provisioning to MS Active Directory from a single connector server (JCS/CCS).
The new attribute that regulates this behavior is eTADSMaxConnectionsInPool. This attribute will be applied on every existing ADS endpoint that is currently being managed by the IM Provisioning Server after CP2 is deployed. Note: The default value is 10, but we recommend after much testing, to match the value of the IMPS-> JCS and JCS->CCS to equal 200.
During testing within the IME using Bulk Tasks or the IM BLC, we can see that the CCS-> ADS traffic will reach20-30connections if allowed. You may set this attribute to a value of 200 via Jxplorer and/or an ldapmodify/dxmodify script.
To confirm the number of open connections is greater than one (1), we can issue a Bulk IM Task or use a performance tool like CA Directory dxsoak.
In this example, we will show case using CA Directory dxsoak to execute 100 parallel threads to create 100 ADS Accounts with MS Exchange Mailboxes. We will also enclose this script for download for others to review and use.
Performance Lab:
Pre-Steps:
Leverage CA Directory samples’ dxsoak binary (performance testing). You may wish to use CA Directory on an existing IM Provisioning Server (Linux OS) or you may deploy CA Directory (MS Windows version) to the JCS/CCS connector. Examples are provided for both OSes.
Create LDIF files for IM Provisioning Server and/or IM Connector Tier. This file is needed to ‘push’ the solution to-failure. The use of the IME Bulk Task and/or etautil scripts to the IM Provisioning Tier, will not provide the transaction speed we need to break the CCS service if possible.
Within the IM Provisioning Manager enable the ADS Endpoint TXT Logs on the Logging TAB, for all checkboxes.
Monitor the IMPS etatrans* logs, monitor the JCS ADS logs, monitor the CCS ADS logs, monitor the number of CCS-> ADS (LDAP/S – TCP 389/636) threads. [Suggest using MS Sysinternals Process Explorer and select im_ccs.exe & then TCP/IP TAB]
Monitor the MS ADS Domain via MS ADUC (AD Users & Computers UI) and MS Exchange Mailbox (Mailbox UI via Browser)
Execution:
6. Perform a UNIT TEST with dxmodify/ldapmodify to confirm the LDIF file input is correct with the correct suffix.
8. IMPS etatrans*.log – Count the number of operations per second. Note any RACE and/or data collisions, e.g. ADS accounts deleted prior to add via 100 threads or ADS account created multiple times attempted in different threads.
9. IM CCS ADS <endpoint>.log – Will only have useful data if the ADS Endpoint Logging TAB has been checked for TXT logs.
10. Finally, validate directly in MS Active Domain with the ADUC or similar tool & MS Exchange mailboxes being created/deleted.
11. Count the number of threads from im_ccs.exe to ADS – Suggest using MS Sysinternals Process Explorer tool and/or Powershell to count the number of connections.
MS Powershell Script to count the number of LDAP (TCP 389) connection from im_ccs.exe. [Note: TCP 389 is used more if the ADS Endpoint is setup to use SASL authentication. TCP 636 is used more if the ADS Endpoint is using the older TLS authentication]
$i=1
Do {
cls
(Get-NetTCPConnection -State Established -OwningProcess (Get-Process -name im_ccs).id -RemotePort 389).count
Start-Sleep -s 1
$i++
}
while ($i -le 5)
Direct Performance Testing to JCS/CCS Service
While this testing has limited value, it can offer satisfaction and assistance to troubleshoot any challenges. We can use the prior LDIF files with a slightly different suffix, dc=etasa (instead of dc=eta), to use dxsoak to push the connector tier to failure. This step helped provide memory dumps back to CA/Symantec Engineering teams to help isolate challenges within the parallel processing. CCS Service is only exposed via localhost. If you wish to test the CCS Service remotely, then update the MS Registry key for the CCS service to use the external IP address of the JCS/CCS Server. Rate observed = 25 K ids/hr
Script to generate 100 ADS Accounts with MS Exchange Mailbox Creation
You may wish to review this script and adjust it for your ADS / MS Exchange domains for testing. You can also create a simple LDIF file with password resets or ADS group membership adds. Just remember that the IMPS Service (TCP 20389/20390) uses the suffix dc=eta, and the IM JCS/CCS Services (TCP 20410/20411) & (TCP 20402/20403) use the suffix dc=etasa. Additionally, if using CA Directory dxsoak, only use the non-TLS ports, as this binary is not equipped for using TLS certs.
#!/bin/bash
#######################################################################################################################
# Name: Generate ADS Feed Files for IM Solution Provisioning/Connector Tiers
#
# Goal: Validate the new parallel processes from the IM Connector Tier to Active Directory with MS Exchange
#
#
# Generate ADS User LDIF file(s) for use with unit (dxmodify) and performance testing (dxsoak) to:
# - {Note: dxsoak will only work with non-TLS ports}
#
# IM JCS (20410) "dc=etasa" {Ensure MS Windows Firewall allows this port to be exposed}
# IM CCS (20402) "dc=etasa" {This port is localhost only, may open to network traffic via registry update}
# IMPS (20389) "dc=eta"
#
#
# Monitor:
#
# The IMPS etatrans*.log {exclude searches}
# The JCS daily log
# The JCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
# The CCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
#
# Execute per the examples provided during run of this file
#
#
# ANA 05/2021
#######################################################################################################################
# Unique Variables for an ADS Domain
NAMESPACE=exchange2016
ADSDOMAIN=exchange.lab
DCDOMAIN="DC=exchange,DC=lab"
OU=People
#######################################################################################################################
MAX=100
start=00001
counter=$start
echo "###############################################################"
echo "###############################################################"
START=`/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`
echo `/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`" = Current OS UTC time stamp"
echo "###############################################################"
FILE1=ads_user_with_exchange_dc_etasa.ldif
FILE2=ads_user_with_exchange_dc_eta.ldif
echo "" > $FILE1
while [ $counter -le $MAX ]
do
n=$((10000+counter)); n=${n#1}
tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
echo "Counter with leading zeros = $n at time: $tz"
cat << EOF >> $FILE1
dn: eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: add
objectClass: eTADSAccount
eTADSobjectClass: user
eTADSAccountName: firstname$n aaalastname$n
eTADSgivenName: firstname$n
eTADSsn: aaalastname$n
eTADSdisplayName: firstname$n aaalastname$n
eTADSuserPrincipalName: aaatestuser$n@$ADSDOMAIN
eTADSsAMAccountName: aaatestuser$n
eTPassword: Password01
eTADSpwdLastSet: -1
eTSuspended: 0
eTADSuserAccountControl: 0000000512
eTADSDescription: description $tz
eTADSphysicalDeliveryOfficeName: office
eTADStelephoneNumber: 111-222-3333
eTADSmail: aaatestuser$n@$ADSDOMAIN
eTADSwwwHomePage: web.page.lab
eTADSotherTelephone: 111-222-3333
eTADSurl: other.web.page.lab
eTADSstreetAddress: street address line01
eTADSpostOfficeBox: pobox 111
eTADSl: city
eTADSst: state
eTADSpostalCode: 11111
eTADSco: UNITED STATES
eTADSc: US
eTADScountryCode: 840
eTADSscriptPath: loginscript.cmd
eTADSprofilePath: \profile\path\here
eTADShomePhone: 111-222-3333
eTADSpager: 111-222-3333
eTADSmobile: 111-222-3333
eTADSfacsimileTelephoneNumber: 111-222-3333
eTADSipPhone: 111-222-3333
eTADSinfo: Notes Here
eTADSotherHomePhone: 111-222-3333
eTADSotherPager: 111-222-3333
eTADSotherMobile: 111-222-3333
eTADSotherFacsimileTelephoneNumber: 111-222-3333
eTADSotherIpPhone: 111-222-3333
eTADStitle: title
eTADSdepartment: department
eTADScompany: company
eTADSmanager: CN=manager_fn manager_ln,OU=$OU,$DCDOMAIN
eTADSmemberOf: CN=Backup Operators,CN=Builtin,$DCDOMAIN
eTADSlyncSIPAddressOption: 0000000000
eTADSdisplayNamePrintable: aaatestuser$n
eTADSmailNickname: aaatestuser$n
eTADShomeMDB: (Automatic Mailbox Distribution)
eTADShomeMTA: CN=DC001,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,$DCDOMAIN
eTAccountStatus: A
eTADSmsExchRecipientTypeDetails: 0000000001
eTADSmDBUseDefaults: TRUE
eTADSinitials: A
eTADSaccountExpires: 9223372036854775807
EOF
counter=$(( $counter + 00001 ))
done
# Create the delete ADS Process
start=00001
counter=$start
while [ $counter -le $MAX ]
do
n=$((10000+counter)); n=${n#1}
tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
echo "Counter with leading zeros = $n at time: $tz"
cat << EOF >> $FILE1
dn: eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: delete
EOF
counter=$(( $counter + 00001 ))
done
echo ""
echo "################################### ADS USER OBJECT STATS ################################################################"
echo "Number of add objects: `grep "changetype: add" $FILE1 | wc -l`"
echo "Number of delete objects: `grep "changetype: delete" $FILE1 | wc -l`"
rm -rf $FILE2
cp -r -p $FILE1 $FILE2
sed -i 's|,dc=im,dc=etasa|,dc=im,dc=eta|g' $FILE2
ls -lart $FILE1
ls -lart $FILE2
echo ""
echo "################################### SET ADS MAX CONNECTIONS IN POOL SIZE ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
LDAPTLS_REQCERT=never dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD" << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'
echo ""
echo "################################### CCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20402
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the CCS Service to test single thread with dxmodify or ldapmodify"
echo "dxmodify -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the CCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo ""
echo "################################### JCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20410
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the JCS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the JCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo ""
echo "################################### IMPS UNIT & PERF TEST ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
echo "Execute this command to the IMPS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "
echo "Execute this command to the IMPS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $IMPS_HOST:$IMPS_PORT -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "
Address the new bottleneck of MS Exchange / O365 Provisioning.
After parallel provisioning has been introduced with the new im_ccs.exe service, you may noticed that the number of transactions is still being throttled during performance testing.
Out-of-the-box MS Active Directory Global Throttling Policy has the parameter of PowerShellMaxConcurrency set to a default of 18 connection. Any provisioning that uses MS Powershell for MS Exchange and/or MS O365 will be impacted by this default parameter.
To address this bottleneck, we can create a new Throttling Policy and only assign the service ID that will be managing identities, to avoid a global change.
After this change has been made, restart the IM JCS/CCS Services, and retest again with your performance tools. Review the CCS ADS log for # of creations in 60 seconds, and you will be pleasantly surprise at the rate. The logs are the strong confirmation we are looking for.
Performance test (947 ADS accounts w/Exchange mailboxes in 60 seconds, 08:59:54 to 09:00:53) =>Rate of 15 ids/second (or 54 K ids/hr) with updated MaxPowershell = 100 thottlingpolicy.
The last bottleneck appears to be CPU availability to MS Exchange Supporting Services, w3wp.exe, the MS IIS Service. Which appears to be managing MS Powershell connections per its startup string of
The Symantec (CA/Broadcom) Directory solution provides a mechanism for routing LDAPv3 traffic to other solutions. This routing mechanism allows Symantec Directory to act as a virtual directory service for other directories, e.g., MS Active Directory, SunOne, Novell eDirectory, etc.
The Symantec Identity Suite solution uses the LDAP protocol for its mid-tier and connector-tier components. The Provisioning Server is exposed on TCP 20389/20390, the JCS (Java Connector Server) is exposed on TCP 20410/20411, and the CCS (C++ Connector Server) is exposed on TCP 20402/20403.
We wished to isolate provisioning data challenges within the Symantec Identity Management solution that was not fully viewable using the existing debugging logs & features of the provisioning tier & connector tiers. Using Symantec Directory, we can leverage the routing mechanism to build a MITM (man-in-the-middle) methodology to track all LDAP traffic through the Symantec Identity Manager connector tier.
We focused on the final leg of provisioning and created a process to track the JCS -> CCS LDAP traffic. We wanted to understand what and how the data was being sent from the JCS to the CCS to isolate issues to the CCS service and MS Active Directory. Using the trace level of Symantec Directory, we can capture all LDAP traffic, including binds/queries/add/modify actions.
The below steps showcase how to use Symantec Directory as an approved MITM process for troubleshooting exercises. We found this process more valuable than deploying Wireshark on the JCS/CCS Server and decoding the encrypted traffic for LDAP.
Background:
Symantec Directory documentation on routing. Please note the concept / feature of “set transparent-routing = true;” to avoid schema challenges when routing to other directory/ldap solutions.
The Symantec Identity Management connector tier may be deployed on MS Windows or Linux OS. If the CCS service is being used, then MS Windows OS is required for this MS Visual C++ component/service. As we are focused on the CCS service, we will introduce the Symantec Directory solution on the same MS Windows OS.
NOTE: We will keep the MITM process contained on a single host, and will not redirect the network traffic beyond the host.
Step 1: Deploy the latest Symantec Directory solution on MS Windows OS. This deployment is a blank slate for the next steps to follow.
Step 2: Copy the folders of schema, limits, and ssld from an existing Symantec Directory deployment of the Symantec Identity Manager solution. Using the existing schema files, references, and certificates will allow us to avoid any challenges during startup of the Router DSA due to the pre-defined provisioning/connector tier configurations. Please note when copying from a Linux OS version of Symantec Directory, we will need to update the path from Linux format to MS Windows format in the SSLD impd.dxc file for “cert-dir” and “ca-file” parameters.
Step 3: Create a new Router DSA DXI configuration file. This is the primary configuration file for Symantec Directory DSA. It will referenced the schema, knowledge, limits, and certificates for the DSA. Note the parameters for “transparent-routing” to avoid schema challenges with other solutions. Note the trace level used to trace the LDAP traffic in the Symantec Directory Router DSA trace log.
# DXserver/config/servers/admin_router_ccs_30402.dxi
# logging and tracing
close summary-log;
close trace-log;
source "../logging/default.dxc";
# schema
clear schema;
source "../schema/impd.dxg";
# access controls
clear access;
# source "../access/";
# ssld
source "../ssld/impd.dxc";
# knowledge
clear dsas;
source "../knowledge/admin_router_ccs_group.dxg";
# operational settings
source "../settings/default.dxc";
# service limits
source "../limits/impd.dxc";
# database - none - transparent router
set transparent-routing=TRUE;
# tunnel through eAdmin server error code and messages
set route-non-compliant-ldap-error-codes = true;
set trace=ldap,time,stats;
#set trace=dsa,time;
Step 4: Create the three (3) knowledge files. The “group” knowledge file will be used to redirect to the other two (2) knowledge files of the router DSA and the re-direct DSA to the CCS service.
# DXserver/config/knowledge/admin_router_ccs_group.dxg
# The admin_router_ccs_30402.dxc PORT 30402
# will be used for the IAMCS (JCS) CCS port override configuration file
# server_ccs.properties via proxyConnectionConfig.proxyServerPort=30402
source "admin_router_ccs_30402.dxc";
source "admin_ccs_server_01.dxc";
# DXserver/config/knowledge/admin_ccs_server_01.dxc
# This file is sourced by admin_router_ccs_group.dxg.
set dsa admin_ccs_server_01 =
{
prefix = <dc etasa>
dsa-name = <dc etasa><cn admin_ccs_server_01>
dsa-password = "secret"
address = ipv4 localhost port 20402
auth-levels = clear-password
dsp-idle-time = 100000
dsa-flags = load-share
trust-flags = allow-check-password, no-server-credentials, trust-conveyed-originator
link-flags = dsp-ldap
#link-flags = dsp-ldap, ssl-encryption
# Note: ssl will require update to /etc/hosts with: <IP_Address> eta_server
};
Step 5: Update the JCS configuration file that contains the TCP port that we will be redirecting to. In this example, we will declare TCP 30402 to be the new port.
Overview of all files updated and their relationship to each other.
Validation
Start up the solution in the following order. Ensure that the new Symantec Directory Router DSA is starting with no issue. If there are any syntax issues, isolate them with the command: dxserver -d start DSA_NAME.
Start the Router DSA first, then restart the im_jcs (JCS) service. The im_ccs (CCS) service will be auto-started by the JCS service. Wait one (1) minute, then check that both TCP Ports 20402 (CCS) and 30402 (Router DSA) are both in the LISTEN state. If we do not see these both ports, please stop and restart these services.
May use MS Sysinternals ProcessExplorer to monitor both services and using the TCP/IP tab, to view which ports are being used.
A view of the im_ccs.exe and dxserver.exe services and which TCP ports they are listening on.
Use a 3rd party LDAP client tool, such as Jxplorer to authenticate to both the CCS and the Router DSA ports, with the embedded service ID of “cn=root,dc=etasa”. We should see exactly the SAME data.
Use the IME or IMPS to perform a query to MS Active Directory (or any other endpoint that uses the CCS connector tier). We should now see the “cache” on the CCS service be populated with the endpoint information, and the base DN structure. We can now track all LDAP traffic through the Router DSA MITM process.
View of trace logs
We can monitor when the JCS first binds to the CCS service.
We can monitor when the IMPS via the JCS queries if the CCS is aware of the ADS endpoint.
Finally, we can view when the IMPS service decrypt its stored information on the Active Directory endpoint, and push this information to the CCS cache, to allow communication to MS Active Directory. Using Notepad++ we can tail the trace log.
Please note, this is a secure LDAP/S tunnel from the IMPS -> JCS -> CCS -> MS ADS.
We can now view how this data is pushed via this secure tunnel with the MITM process.
We can now monitor all traffic and assist with troubleshooting any CCS/MS-ADS challenges.
This same MITM methodology/process may also be used for the IMPS (TCP 20389/2039) and the JCS (TCP 20410/20411) services. We have used this process to capture the IME (JIAM) LDAP traffic to the IMPS Service, to isolate multiple queries for Child Provisioning Roles. Which has been used by the product team to enhance the solution to lower startup durations of the IME in the latest releases.
Binds/queries/add/modification all work with this approach, but we do see an issue with OID for IMPS ADS endpoint “explore process” on ADS OU object. We are reviewing how to address this last challenge that states “critical extension is unavailable” for a LDAP control property of the OU object. The OIDs captured appear to be related to SunOne/Iplanet.
The Symantec (CA) Identity Suite includes the Symantec (CA) Directory. This component is installed under the ‘dsa’ service ID. On the virtual appliance, this ‘dsa’ service ID does not have a password defined, and therefore no login is allowed.
As an enhancement, we would like to add in a SSH private key to allow authentication to the ‘dsa’ service ID from other virtual appliances and desktop usage with various tools, e.g. Putty, MobaXterm, WinSCP, etc. This enhancement will allow for a streamlined process to address out-of-sync Directory DATA DSAs with scp/Rsync copies without intermediate file shares or use of other service IDs.
Challenge:
The virtual appliance of Symantec (CA) Identity Suite r14.3 is built on CentOS 6.4. The OpenSSH services on this OS apparently do not use a private key format that can be used by desktop tools or the PuttyGen (keygen conversion tool). However, the private key may be used between vApp servers if using the FQDN (full qualified domain name). We noted that during testing, that localhost is not allowed due to localhost not defined in the SSHD “AllowedUsers” property file.
On newer virtual appliances vApp r14.4 with CentOS 8 Stream, this challenge does not exist, and we can use the OpenSSH private key, id_rsa, with the desktop tools as-is.
To assist with challenge and streamlining this process we have the following three (2) options:
Option 1: On newer OS, use OpenSSH process
After creating the private key, ./ssh/id_rsa, cat this file out to notepad, and save for use with the desktop tools
Generate this OpenSSH private/public key. The final command will help to validate this private key may be used for server to server communication.
On any OS (new/old) just use Putty-Gen tool to generate the private key. Update key comment/passphrase. After the private key is created, copy the TEXT “Public Key for pasting into OpenSSH authorized_keys file”. Just like it says, and then you may use the associated private key, id_rsa.ppk, with the desktop tools for the ‘dsa’ service ID.
Option 3: Combination of processes/tools
Important: .ssh/authorized_keys is updated and not overwritten.
A very common challenge we see is the modification of the CA/Symantec Connector Server Service(s) startup order for the embedded C++ (CCS) Connector. This CCS connector service on MS Windows OS is marked default as “Manual” startup.
Since the solution documentation is not clear on why this is configured as manual, we will see site’ administrators that will either change this service from “Manual” to “Automatic” or will start the CCS service manually themselves upon a restart.
However, either of these processes will impact the ability of the JCS Service from managing the CCS Services cache upon startup. The JCS will NOT be able to manage the CCS service for a number of minutes until it can resolve this challenge. Unfortunately, when this occurs, the traffic to any CCS managed endpoints will be placed in a long time out within the JCS Service. The IMPS (Provisioning Service) will think that it successfully handed off the task to the JCS/CCS tier, but the task will stay in a holding pattern until either the memory of the JCS is overwhelmed or the CCS Service restarts/crashes or a timeout of the task.
TL;DR – Please do not start the CCS Service manually. Only stop/start the JCS Service, wait a full minute and you should see the CCS Service start up. If the CCS Service does NOT start, investigate why.
JCS Service’s management of the CCS Service:
To understand how the JCS Service manages the CCS Service (via localhost TCP 20402), we can review two (2) files and use MS Sysinternals Process Explorer to view the JCS Service starting the CCS Service via the command “net start im_ccs”. The JCS Service will now have access to update the CCS service’s cache with information for a managed endpoint, e.g. Active Directory.
The two (2) JCS Service configuration files for CCS Service are:
C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\server_osgi_ccs.xml [File contains startup properties of how the JCS will manage timeouts to the CCS Service & connections pools]
C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\override\server_ccs.properties [File contains the bind credentials and the service port to communicate to on localhost:20402. The password hash will be PBES or AES format depending if FIPS is enabled.]
And finally a view of the startup of the CCS Service via JCS Service using MS Sysinternals Process Explorer https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer We can see that a child process is started from the JCS Service that will call the MS Windows “net.exe” command and execute “net start im_ccs”
Keeping the JCS Service and CCS Service as-is for startup processes will help avoid confusion for the provisioning tier of the CA/Symantec solution. Please only stop/start the JCS Service. If the CCS Service does not stop after 2 minutes, kill it. But never start the CCS by itself.
A view of the data path from IMPS (IM Provisioning Server) to Active Directory (manage endpoint) via the Connector tier.
Performance Improvements
While we may not adjust the startup from manual to automatic, we can enhance the default configurations for performance and timeout improvements. The JCS Service starts up with a default of 1 GB RAM. The JCS Service is 64 bit based on using 64 bit JAVA and the memory can be increased accordingly. After testing with large data sets, we recommend increasing the JCS JVM max memory from 1 GB to 4 GB. We can confirm after startup of the JCS will use over 1 GB of RAM with MS Sysinternals Process Explorer.
Other improvement include updating the JAVA that is supporting the JCS Service. CA/Symantec now recommends using AdoptOpenJDK. The documentation now explains how this may be updated in-place. Or as we prefer to reinstall and allow the installer to update the path statements for AdoptOpenJDK.
The below image below shows in the MS Windows Registry for the JCS Service (Procrun 2.0/im_jcs) the key value pairs that are updated for AdoptOpenJDK. https://adoptopenjdk.net/ If managing Active Directory, please review your OS environmental variables to control the behavior from the CCS Service to Active Directory.
After you restart the JCS Service, open the JCS Administration Console via http://localhost:20080/main or https://localhost:20443/main right click on the “Local Connector Server” ICON and it should display that AdoptOpenJDK is in use now. Only major release 8 is supported, avoid trying later releases (11,15) until support is confirmed.
Stability Improvements
The default JCS Service configuration file has knowledge of the connection pool and timeouts, but appears to be missing the “maxWait” token defined. If we are willing to wait 5-10 minutes for the JCS Service to reset its knowledge of the CCS service, we can leave the default. However for a large environment, we have found that lowering the wait times will greatly avoid the delays in transactions when there is stoppage. We have identified two (2) configuration parameters that will assist with the long term stability of the solution. Adding the “maxWait” of 60 seconds (60000 milliseconds) to the JCS configuration file for CCS service and updating the default IM Provisioning Server domain configuration parameter of “Connections/Refresh Time” to 90 seconds.
Troubleshooting and Logging
To assist with RCA efforts, we have the following recommendations. Enable verbose logging for both the JCS Service and the managed endpoint to isolate issues. You may also need to increase logging for the API Gateway or docker logs.
Below is the example to enable verbose logging.
To monitor the JCS logs, there are several tools that will assist, but we find that the latest releases of Notepad++ allow for “tailing” the active JCS logs.
Example of verbose logs for Active Directory via the CCS’s ADS and JCS logs.
Important Logging Note: Enable the new IM r14.3cp2 feature to auto rotate your CCS ADS log. Avoid stop/start of the CCS Service yourself, that may interrupt the JCS behavior to the CCS Service (error communicating to localhost:20402 will display in JCS logs). New file(s): Connector Server\ccs\data\ADS\<Endpoint_Name>.logconfig
Example of setting MS Windows OS Environmental variables with “setx” and description of the value of each variable for Active Directory/MS Exchange
1
[High Value. Will force AGENTLESS connection to Exchange 2010 & UP]
setx ADS_AGENTLESS_MODE 1 /m
2
[High Value. Default value = 2, Kerberos authentication for Exchange Powershell API]
setx ADS_AGENTLESS_AUTHMETHOD 2 /m
3
[High Value. Default value = 3. Increase to 100 and ALSO have Exchange Admin create a new quota for the service account used to create mailboxes. Default Exchange Powershell Quota is 18. New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 AND Set-Mailbox ServiceAccountID -ThrottlingPolicy MaxPowershell ]
setx ADS_AGENTLESS_MAXCONN 100 /m
4
[Monitor. Default value = 1. Error level ONLY, increase to level 3 for debugging powershell logging to MS Exchange]
setx ADS_AGENTLESS_LOGLEVEL 1 /m
5
[Medium Value. CCS service will wait 10 minutes for single account. Exchange Powershell Mailbox Quota of 18 and BLC with 100’s of users.]
setx ADS_CONFIRM_MAILBOX 600 /m
6
[Low Value. Mask the AD Failover List in the IM Prov Manager UI]
setx ADS_DISABLE_DCSTATUS 1 /m
7
[Low Value. Mask the viewing the default AD Primary Group in the IM Prov Manager UI]
setx ADS_DISABLE_PRIMARYGROUPNAME 1 /m
8
[High Value. Send the DC hostname to the Exchange server to query first instead of Exchange relying on its current pool]
setx ADS_E2K_SEND_DC 1 /m
9
[High Value. Requires service account can view all alternatives DC. May limit failover DC via properties file.]
setx ADS_FAILOVER 1 /m
10
[Medium Value. Performance if Terminal Services attribute are NOT being managed, e.g. changed in Account Templates or PX rules.]
setx ADS_WTS_TIMEOUT -1 /m
11
[Set “ADS_OPERATION_TIMEOUT” to -1 to disable the client side timeout functionality in the Environment Variable, otherwise 60]
setx ADS_OPERATION_TIMEOUT 60 /m
12
[The failover retry interval is the time that the Active Directory connector waits before checking the stopped server. The default retry interval is 15 minutes]
setx ADS_RETRY 15 /m
13
[To allow groups in unmanaged domains to be a part of synchronization, Defines whether the synchronization operation searches the global catalog. The value of x can be 0 or 1: 0: (Default) The synchronization operation queries the local catalog only. It does not consider universal groups in unmanaged domains. When x is set to 0, the y value has no effect. 1: Synchronization queries the global catalog to allow it to consider groups in unmanaged domains. y Defines which domains the synchronization operation considers. 0: Synchronization considers groups in both managed and unmanaged domains. 1: Synchronization considers groups in managed domains only.]
setx ADS_MANAGE_GROUPS 01 /m
14
[Monitor. Seems only valuable for debugging. Has performance hit but may assist for CCS debugging to ActiveDirectory.]
setx ADS_FORCELOG 1 /m
15
[Low Value. The IMPS service can page with lower limits. Impact if this value is > what AD default page limit size is. ]
setx ADS_SIZELIMIT 50000 /m
Reinstalling the JCS Service from the Virtual Appliance
If you are using the CA/Symantec Identity Suite virtual appliance, consider after patching the solutions on the virtual appliance, to re-installing the remote JCS Services. This will avoid any confusion on which patches are deployed on the remote JCS servers. Any patches on the virtual appliance will be incorporated into the new installer. We prefer to use the JCS only on the MS Windows OS, as it can service both JCS type managed endpoints & CCS type managed endpoints together. We also have full access to adjust the behavior of these service on MS Windows OS rather than the limited access provided by the virtual appliance for the JCS service.
Hopefully some of these notes will help you avoid any challenges with the connector tier and if you do, how to isolate the issues.
Advance Review: Review how CCS Service receives IMPS data via the JCS Tier.
The below example will load the DLL for the CCS Service (pass-through), then the information to bind to the ADS endpoint will be sent, then two (2) modify operations will be executed. This process emulates the IMPS behavior with the JCS and CCS. The bind information for the ADS endpoint that is stored in the CA Provisioning User Store, and queried/decrypted by the IMPS to send to the JCS as needed. Only after this information is stored in the CCS service, will the solution be able to explore or manage the ADS endpoint accounts.
If unable to re-install, please delete the CA install/registry tracking file under C:\Windows folder, C:\Windows\vpd.properties , then reboot before attempting a re-install of the JCS/CCS component.
ECS Services
These five (5) ECS Services are typically not active used & may be changed to manual for minor CPU relief. ECS features are retained for supporting libraries.
One of the challenges that IAM/IAG teams may have every few months is delivery or access for internal/external auditors to validate access within the IAM/IAG system and their managed endpoints.
Usually, auditors may directly access the 100’s system/endpoints/applications and randomly select a few or export the entire directory structure to review access. This effort takes time and possible 100’s of entitlements to grant temporary/expiry access to view. Auditors also prefer Excel or CSV files to review rather than fixed documents (PDF) to allow them to filter and isolate what interests them.
One process that may have value for your team is various tools with export functionality to CSV/XLS and the ability to query the 100’s-1000’s of systems from a single entry point.
A tool that we have found valuable over the years is SoftTerra LDAP Browser.
The multiple benefits from this tool for IAM/IAG are:
It is a read-only tool, so no mistakes can be made by granting too much access.
It has the ability to save queries that are popular and can be copied from other tools.
It has the ability to export the queries to CSV/XLS formats (plus others)
It can be used to pull reports from an IAM/IAG solution via their directory ports.
It can be used to pull reports from the managed applications (on-prem or SaaS) via the IAM provisioning directory ports.
The tool is free from SoftTerra, it is a limited version of their Administration tool
Example of the SoftTerra LDAP Browser tool used to query Active Directory, LDAP user stores, and Provisioning User Store & managed endpoints/applications.
A view to export Service Now (SNOW) accounts via the CA/Symantec Identity Manager Provisioning Server/Service (TCP 20390) via the LDAP/S protocol.
Why? The provisioning server may be viewed as a virtual directory/pass-through directory to the managed endpoints via its connector tier.
The below image shows SoftTerra LDAPBrowser used to connect to the Provisioning Server (TCP 20390). Then navigate to a Service Now (SNOW) managed endpoint, to query on all accounts and their respective profiles & entitlements. This same report/extract process may be done for mainframe/AS400 and client-server applications, e.g Active Directory, Unix, Databases, etc.
Enhance this process with defense-in-depth
We will not use the primary default administration account of the provisioning tier, “etaadmin”. Since this account has full access to change data.
Within the IAM/IAG solution, create an auditor account.
In the example below we create a new Global User, with the name “auditor”, a description, password, and a local “read-only admin profile” with an expiration date. This will allow the auditors to use the account as they wish (or you may grant this “read-only admin profile directly to their existing Global User ID). The account may still follow the same password reset expiration processes. If the account is marked as “restricted” in the CA/Symantec IM solution, then this account is limited how it may be changed to avoid any unexpected sync challenges to managed endpoints (if it was correlated to other accounts).
After the new Global User is created (or existing ID is added to the Admin Profile “ReadAdministrator”), update SoftTerra Credentials for the Provisioning Service. Below the new DN with “auditor” is shown in the credentials for login ID, e.g. “eTGlobalUserName=auditor,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta”
Now, the auditors may run as many reports as they would like, and export to spreadsheets or PDF files using a read-only account with a read-only tool.
Honorable mentions for other query tools.
Jxplorer is a useful & free java-based tool for reports, but this is a full edit tool & only exports out to LDIF format. http://jxplorer.org/
Apache Directory Studio is another very useful & free java-based tool for reports. This is a full edit tool. It does have the ability to export to many different formats. Since this tool does NOT need an MS Windows installer, and if the Desktop prevent installation, this is typically our 2nd choice to use. Extract and use the current java on the MS Windows OS or download AdoptOpenJDK and extract it to use with Apache Directory Studio. https://directory.apache.org/studio/ & https://adoptopenjdk.net/
SoftTerra LDAP Administrator is a paid and full edit tool. It has the same look-n-feel of the SoftTerra LDAP Browser tool. It is typically used by administrators of various LDAP solutions. We recommend this tool for your larger sites or if you would like a fast responsive tool on MS Windows OS. https://www.ldapadministrator.com/
If you have other recommendations, please leave a response.
Bonus Feature – SoftTerra AD Authentication
Both the SoftTerra tools allow binding using your existing authentication (on your desktop/laptop) into Active Directory. No need to create additional user ID for the auditors or yourself.
Perhaps the O365 or Outlook contacts process is not robust or too slow or perhaps you wish you had a more detail view of your internal active directory to view a manager’s direct reports. You can use this feature to view the the non-privacy attributes of your domain of all accounts with a read-only tool.
Step 01: Open a command-line prompt on your desktop/workstation after you have authenticated to your Active Directory domain & type set | findstr LOGONSERVER
Step 02: Install SoftTerra LDAP Browser Tool & Create a new profile
Step 03: Type the name of the Active Directory LOGONSERVER (aka Domain Controller) into the following fields & ensure “Use Secure Connection (SSL)” is selected (to avoid query issues).
Step 04: Click Next until you see “User Authentication Information” then select the radio button for “Currently logged on user (Active Directory)”, then click Finish button.
Step 05: After the profile is built, now click on the profile and watch it expand into a tree display of Active Directory. Select the branch that you believe has the list of users you would like to view, then select an individual user account, to see the values populated.
Step 06: If you wish to export this data to a spreadsheet (CSV/XLS), right click on the left object and select export option.
Step 07: You will have a series of options to export to & the file name it will write to.
Step 08: Advance search and export process. Select the branch that holds all the users you wish to view and export. Note: If the branch has 10,000 objects, this process may take minutes to complete depending on the query.
Step 09: The follow search windows will appear to help you create, save, and export your queries. Note that if you start to type in the field name, the list of the fields will start to appear.
Step 10: Ensure the FILTER is properly formed (use google to assist), and which attribute you wish to view or export is defined, then click search. If you are satisfied with your search, use the “Save Results” to export to a spreadsheet (CSV/XLS) or other format.
A useful feature with CA Directory for WAN latency challenges is the HUB model. This model allows sync of the data to occur to distant peer Multi-write DATA DSA, but does NOT impact the external application that is updating its own local Router/DATA DSAs.
To assist with understanding this HUB model, we have leverage the CA Directory samples of democorp & router to build out an architecture with six (6) DATA DSAs and two (2) router DSAs, to emulate two (2) data centers across the world. These samples are included with every CA Directory deployment under $DXHOME/samples/democorp & $DXHOME/samples/router.
This lab emulates two (2) of the three (3) data centers that are displayed within the CA documentation.
Below is a bash shell script used to create the lab environment that was created on a single host OS, with CA Directory samples of “democorp” and “router”. These samples were “copied” and updated via sed commands to ensure the DSA are unique for TCP Ports and naming convention. The below examples’ nomenclature will use democorpX for (A-F) and routerYY (AA or BB) .
These DATA DSA use the same suffix and are all referenced in the group knowledge file. The HUB model configuration will change the behavior for MW-DISP replication between data centers. MW-DISP replication will still be used for all local sync between DATA DSA in the same data center, and would ONLY be used between data centers for the DATA DSA that are designated as “HUB”s (aka multi-write-group-hub).
To test the value of HUB model with WAN latency, suggest that the same lab be executed on two (2) hosts, where one host has the VMWARE network latency enabled to 150/150 milli-seconds. Update the ip addresses within the $DXHOME/config/knowlege/*.dxc on both host OS, to reflect the correct hostnames for each data center in the DSAs.
The diagram below outlines the delta between various democorp DATA DSA to use the HUB model.
The changes below show the deltas within the *.dxc/*.dxg files within the knowledge folder for democorp MW HUB model.
The below image captures the only deltas in the *.dxi (startup files) for the democorp MW HUB model, located within the server folder. Note, if CA Directory management tool is deployed and used for democorp, all configurations will be in a single *.dxi file.
#!/bin/bash
##############################################
#
# Name: democorp_mw_hub_lab.sh
#
# Multi-Write HUB lab using CA Directory and the samples of
# democorp and router under DXHOME/samples
# A. Baugher, 04/2020 - ANA Technology Partner
#
# Assumptions:
# CA Directory is deployed & dxprofile is enabled for dsa user
# Execute script as dsa user
#
# Step 0. Clean-Up prior deployment
#
# Step 1. Auto deploy both democorp and router samples with: setup.sh -q
#
# Step 2. Make common changes in democorp prior to copying
#
# Step 3. Create six (6) copies of democorp and two (2) copies of router
#
# Step 4. Update the six (6) copies of democorp for:
# - name
# - ports
# - multi-write-group (HUB group)
# - DSA flags for MW & HUB-DSA
# - Group knowledge file reference
#
# Update the two (2) copies of router for:
# - name
# - ports
# - Group knowledge file reference
# - set write-precedence (for HUB-DSA)
#
# Step 5. Start all DSAs
#
# Step 6. Test with dxsearch query
#
# Step 7. Execute the dxsoak command with the service account & time command
#
# Step 8. Update democorpA to force a single delta between peer members of AA and BB
#
# Step 9. Create LDAP Export
#
# Step 10. Create LDAP Delta & Compare the various democorp DSA to validate sync operations
#
#
##############################################
#set -xv
echo ..
echo "#############################################################"
echo "Step 0. Clean up prior deployment of democorp and router"
echo "#############################################################"
dxserver stop all
sleep 5
kill -9 `ps -ef | grep dsa | grep democorp | grep -v grep | grep -v "democorp_mw_hub_lab" | awk '{print $2}'` > /dev/null 2>&1
kill -9 `ps -ef | grep dsa | grep router | grep -v grep | awk '{print $2}'` > /dev/null 2>&1
sleep 5
rm -rf $DXHOME/data/democorp*.*
rm -rf $DXHOME/config/knowledge/democorp*.*
rm -rf $DXHOME/config/knowledge/router*.*
rm -rf $DXHOME/config/servers/democorp*.*
rm -rf $DXHOME/config/servers/router*.*
rm -rf $DXHOME/logs/democorp*.*
rm -rf $DXHOME/logs/router*.*
rm -rf $DXHOME/backup/delta*.* > /dev/null 2>&1
rm -rf $DXHOME/backup/*.ldif > /dev/null 2>&1
echo ..
echo "#############################################################"
echo "Step 1a. Deploy clean version of democorp and router"
echo "#############################################################"
cd $DXHOME/samples/democorp
$DXHOME/samples/democorp/setup.sh -q > /dev/null 2>&1
cd $DXHOME/samples/router
$DXHOME/samples/router/setup.sh -q > /dev/null 2>&1
cd
echo ..
echo "#############################################################"
echo "Step 1b. Create service ID in democorp for later use"
echo "#############################################################"
cat << EOF > $DXHOME/diradmin.ldif
version: 1
dn: cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
cn: diradmin
sn: diradmin
givenName: diradmin
userPassword: Password01
EOF
dxmodify -a -c -h `hostname` -p 19389 -f $DXHOME/diradmin.ldif
echo ..
echo "#############################################################"
echo "Step 1c. Stop all running democorp & router DSAs"
echo "#############################################################"
dxserver stop all
sleep 10
echo ..
echo "#############################################################"
echo "Step 2a. Make common changes in pre-existing files before other modification"
echo "Update dsa-flags in democorp.dxc to allow Multi-Write with a HUB"
echo "#############################################################"
sed -i 's|ssl-auth|ssl-auth\n multi-write-group = hub_group_AA\n dsa-flags =|g' $DXHOME/config/knowledge/democorp.dxc
sed -i 's|dsa-flags =|dsa-flags = multi-write, no-service-while-recovering, load-share|g' $DXHOME/config/knowledge/democorp.dxc
echo ..
echo "#############################################################"
echo "Step 2b. Update MW recovery in democorp.dxi file"
echo "#############################################################"
sed -i 's|recovery = false;|recovery = true;|g' $DXHOME/config/servers/democorp.dxi
echo ..
echo "#############################################################"
echo "Step 3a. Create six (6) copies of democorp and two (2) routers"
echo "Copy democorp data folder contents"
echo "#############################################################"
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpA.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpA.tx > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpB.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpB.tx > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpC.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpC.tx > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpD.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpD.tx > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpE.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpE.tx > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpF.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpF.tx > /dev/null 2>&1
echo ..
echo "#############################################################"
echo "Step 3b. Copy autostart folder contents"
echo "#############################################################"
cp -r -p $DXHOME/config/autostart/democorp $DXHOME/config/autostart/democorpA
cp -r -p $DXHOME/config/autostart/democorp $DXHOME/config/autostart/democorpB
cp -r -p $DXHOME/config/autostart/democorp $DXHOME/config/autostart/democorpC
cp -r -p $DXHOME/config/autostart/democorp $DXHOME/config/autostart/democorpD
cp -r -p $DXHOME/config/autostart/democorp $DXHOME/config/autostart/democorpE
cp -r -p $DXHOME/config/autostart/democorp $DXHOME/config/autostart/democorpF
cp -r -p $DXHOME/config/autostart/router $DXHOME/config/autostart/routerAA
cp -r -p $DXHOME/config/autostart/router $DXHOME/config/autostart/routerBB
echo ..
echo "#############################################################"
echo "Step 3c. Copy knowledge folder contents"
echo "#############################################################"
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpA.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpB.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpC.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpD.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpE.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpF.dxc
cp -r -p $DXHOME/config/knowledge/router.dxc $DXHOME/config/knowledge/routerAA.dxc
cp -r -p $DXHOME/config/knowledge/router.dxc $DXHOME/config/knowledge/routerBB.dxc
cp -r -p $DXHOME/config/knowledge/sample.dxg $DXHOME/config/knowledge/groupAA.dxg
cp -r -p $DXHOME/config/knowledge/sample.dxg $DXHOME/config/knowledge/groupBB.dxg
echo ..
echo "#############################################################"
echo "Step 3d. Copy server folder contents"
echo "#############################################################"
cp -r -p $DXHOME/config/servers/democorp.dxi $DXHOME/config/servers/democorpA.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi $DXHOME/config/servers/democorpB.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi $DXHOME/config/servers/democorpC.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi $DXHOME/config/servers/democorpD.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi $DXHOME/config/servers/democorpE.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi $DXHOME/config/servers/democorpF.dxi
cp -r -p $DXHOME/config/servers/router.dxi $DXHOME/config/servers/routerAA.dxi
cp -r -p $DXHOME/config/servers/router.dxi $DXHOME/config/servers/routerBB.dxi
echo ..
echo "#############################################################"
echo "Step 4a. Update names & ports in democorp knowledge files"
echo "#############################################################"
sed -i 's|19389|29389|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|19390|29390|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPA =|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPA>|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|19389|29489|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|19390|29490|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPB =|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPB>|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|19389|29589|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|19390|29590|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPC =|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPC>|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|19389|29689|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|19390|29690|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPD =|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPD>|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|19389|29789|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|19390|29790|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPE =|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPE>|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|19389|29889|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|19390|29890|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPF =|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPF>|g' $DXHOME/config/knowledge/democorpF.dxc
echo ..
echo "#############################################################"
echo "Step 4b. Update knowledge files for router ports"
echo "#############################################################"
sed -i 's|19289|39289|g' $DXHOME/config/knowledge/routerAA.dxc
sed -i 's|19290|39290|g' $DXHOME/config/knowledge/routerAA.dxc
sed -i 's|dsa ROUTER =|dsa ROUTERAA =|g' $DXHOME/config/knowledge/routerAA.dxc
sed -i 's|19289|39389|g' $DXHOME/config/knowledge/routerBB.dxc
sed -i 's|19290|39390|g' $DXHOME/config/knowledge/routerBB.dxc
sed -i 's|dsa ROUTER =|dsa ROUTERBB =|g' $DXHOME/config/knowledge/routerBB.dxc
echo ..
echo "#############################################################"
echo "Step 4c. Update group knowledge file for three (3)MW Group HUB Peers "
echo "#############################################################"
sed -i 's|"router.dxc";|"routerAA.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
sed -i 's|"democorp.dxc";|"democorpA.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
sed -i 's|"democorpA.dxc";|"democorpA.dxc";\nsource "democorpB.dxc";\nsource "democorpC.dxc";\nsource "routerBB.dxc";\nsource "democorpD.dxc";\nsource "democorpE.dxc";\nsource "democorpF.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
sed -i 's|source "unspsc.dxc";|#source "unspsc.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
cp -r -p $DXHOME/config/knowledge/groupAA.dxg $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|"router.dxc";|"routerBB.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|"democorp.dxc";|"democorpD.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|"democorpD.dxc";|"democorpD.dxc";\nsource "democorpE.dxc";\nsource "democorpF.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|source "unspsc.dxc";|#source "unspsc.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
echo ..
echo "#############################################################"
echo "Step 4d. Update Server folder contents"
echo "#############################################################"
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/democorpA.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/democorpB.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/democorpC.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/democorpD.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/democorpE.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/democorpF.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/routerAA.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/routerBB.dxi
echo ..
echo "#############################################################"
echo "Step 4e. Update HUB Configurations in DSA knowledge and DSA routers"
echo "#############################################################"
sed -i 's|load-share|load-share, multi-write-group-hub|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|load-share|load-share, multi-write-group-hub|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|multi-write-group = hub_group_AA|multi-write-group = hub_group_BB|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|multi-write-group = hub_group_AA|multi-write-group = hub_group_BB|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|multi-write-group = hub_group_AA|multi-write-group = hub_group_BB|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|/knowledge/groupAA.dxg";|/knowledge/groupAA.dxg";\nset write-precedence = democorpA ,democorpB, democorpC;\n|g' $DXHOME/config/servers/routerAA.dxi
sed -i 's|/knowledge/groupBB.dxg";|/knowledge/groupBB.dxg";\nset write-precedence = democorpD ,democorpE, democorpF;\n|g' $DXHOME/config/servers/routerBB.dxi
echo ..
echo "#############################################################"
echo "Step 4f. Remove samples of router & democorp from starting "
echo "#############################################################"
rm -rf $DXHOME/config/servers/democorp.dxi
rm -rf $DXHOME/config/servers/router.dxi
rm -rf $DXHOME/config/autostart/democorp
rm -rf $DXHOME/config/autostart/router
echo ..
echo "#############################################################"
echo "Step 5. Start all DSAs"
echo "#############################################################"
dxcertgen certs > /dev/null 2>&1
dxserver start all
dxserver status
#exit
echo ..
echo "#############################################################"
echo "Step 6. Test all DSAs with dxsearch query"
echo "#############################################################"
# Comment out if too verbose
# Data DSAs
#dxsearch -h `hostname` -p 29389 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29489 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29589 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29689 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29789 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29889 -c -x -b o=DEMOCORP,c=AU
# Router DSAs
#dxsearch -h `hostname` -p 39289 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 39389 -c -x -b o=DEMOCORP,c=AU
# Data DSAs
#dxsearch -h `hostname` -p 29389 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01
#dxsearch -h `hostname` -p 29489 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01
#dxsearch -h `hostname` -p 29589 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01
#dxsearch -h `hostname` -p 29689 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01
#dxsearch -h `hostname` -p 29789 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01
#dxsearch -h `hostname` -p 29889 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01
# Router DSAs
#dxsearch -h `hostname` -p 39289 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01
#dxsearch -h `hostname` -p 39389 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01
echo ..
echo "#############################################################"
echo "Step 7. Execute the dxsoak command with the service account & time command"
echo "allow to run for over 5 sec to monitor changes for Multi-Write"
echo "may allow for longer times (1 hour) to get better performance metrics"
echo "#############################################################"
cd $DXHOME/samples/dxsoak
echo "Update democorpA (TCP 29389) to confirm MW to from democorpA (hub_group_AA) to democorpD (hub_group_BB)"
# Create a delete file first; then re-add entries
grep dn: democorp.eldf | grep ,ou=Services > democorp-del.eldf
sed -i 's|,c=AU|,c=AU\nchangetype: del\n|g' democorp-del.eldf
echo ..
echo "#############################################################"
echo "# Delete all DN entries with ou=Services: `wc -l democorp-del.eldf` on democorpA (TCP 29389)"
time ./dxsoak -c -t 2 -q 10 -l 5 -h `hostname`:29389 -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 -f democorp-del.eldf
echo ..
echo "#############################################################"
echo "# Re-Add all DN entries with ou=Services: `wc -l democorp.eldf` on democorpD (TCP 29689)"
time ./dxsoak -c -t 2 -q 10 -l 5 -h `hostname`:29689 -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 -f democorp.eldf
echo ..
echo "#############################################################"
echo "Step 8a. Update democorpA to force a single delta between peer members of AA and BB"
echo "#############################################################"
cd
cat << EOF > $DXHOME/diradmin_sn.ldif
dn: cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU
changetype: modify
replace: sn
sn: diradmin_AA_new_update
EOF
echo "#############################################################"
echo "# Query democorpA (TCP 29389) for sn value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29389 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 sn createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for sn value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 sn createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Make update on democorpA"
echo "#############################################################"
dxmodify -a -c -h `hostname` -p 29389 -f $DXHOME/diradmin_sn.ldif
echo "#############################################################"
echo "# Query democorpA (TCP 29389) for sn value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29389 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 sn createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for sn value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 sn createTimestamp modifyTimestamp
#exit
echo ..
echo "#############################################################"
echo "Step 8b. Update democorpF to force a reverse single delta between peer members of AA and BB"
echo "#############################################################"
cd
cat << EOF > $DXHOME/diradmin_givenName.ldif
dn: cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU
changetype: modify
replace: givenName
givenName: diradmin_BB_new_update
EOF
echo "#############################################################"
echo "# Query democorpC (TCP 29589) for givenName value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29589 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 givenName createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for givenName value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 givenName createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Update democorpF to show replication via democorpD (HUB) to democorpA (HUB) "
echo "#############################################################"
dxmodify -a -c -h `hostname` -p 29889 -f $DXHOME/diradmin_givenName.ldif
echo "#############################################################"
echo "# Query democorpC (TCP 29589) for givenName value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29589 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 givenName createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for givenName value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 givenName createTimestamp modifyTimestamp
echo ..
echo "###########################################################"
echo "Step 9b. Update CA Directory DSA to allow online backup ###"
echo "###########################################################"
echo " - Configure CA Directory to provide an data dump (zdb file) while DSA are online"
cp -r -p $DXHOME/config/settings/default.dxc.org $DXHOME/config/settings/default.dxc > /dev/null 2>&1
cp -r -p $DXHOME/config/settings/default.dxc $DXHOME/config/settings/default.dxc.org > /dev/null 2>&1
# Edit the DSA settings file to add in one line. dump dxgrid-db;
chmod 744 $DXHOME/config/settings/default.dxc
echo "dump dxgrid-db;" >> $DXHOME/config/settings/default.dxc
echo ..
echo "######################################################################################"
echo "Step 9c. Re-init all DSA to data dump the CA DSAs for democorp & router "
echo "######################################################################################"
echo " - This make take 5-30 seconds to complete "
dxserver init all > /dev/null 2>&1
# View for zdb or zd? (in-progress) files
sleep 10
echo ..
echo "#################################################################"
echo "Step 9d. Export DSA backup/offline zdb data files to LDIF file ###"
echo "#################################################################"
echo " - Export will happen after the backup/offline zdb files are fully created"
echo " - This make take 5-60 seconds to complete "
echo ..
echo "#################################################################"
echo "Step 9e. Set WHILE loop for DemocorpF DSA ###"
echo "#################################################################"
until [ -f $DXHOME/data/democorpF.zdb ]
do
echo " - Waiting till CA Directory has completed online data dump of DemocorpF DSA"
sleep 5
done
sleep 5
echo ..
echo "#################################################################"
echo "Step 9f. Execute dxdumbdb for Democorp DSA - FULL ###"
echo "#################################################################"
mkdir $DXHOME/backup > /dev/null 2>&1
cd $DXHOME/backup
dxdumpdb -z -f $DXHOME/backup/democorpA.ldif democorpA > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpB.ldif democorpB > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpC.ldif democorpC > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpD.ldif democorpD > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpE.ldif democorpE > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpF.ldif democorpF > /dev/null 2>&1
sleep 5
echo ..
echo "#################################################################"
echo "Step 10a. Perform LDIF DELTA compare between democorpA and democorpB within same HUB MW group"
echo "Look for any delta in the metrics > 0"
echo "#################################################################"
#ldifdelta -x -S DSANAME OLDFILE NEWFILE DELTAFILE
ldifdelta -x -S democorpA $DXHOME/backup/democorpA.ldif $DXHOME/backup/democorpB.ldif $DXHOME/backup/delta-between-A-and-B.ldif
echo "#################################################################"
echo "Step 10b. Perform LDIF DELTA compare between democorpD and democorpE within same HUB MW group"
echo "Look for any delta in the metrics > 0"
echo "#################################################################"
ldifdelta -x -S democorpC $DXHOME/backup/democorpD.ldif $DXHOME/backup/democorpE.ldif $DXHOME/backup/delta-between-D-and-E.ldif
echo "#################################################################"
echo "Step 10c. Perform LDIF DELTA compare between democorpC and democorpF across different HUB MW groups"
echo "Look for any delta in the metrics > 0"
echo "#################################################################"
ldifdelta -x -S democorpC $DXHOME/backup/democorpC.ldif $DXHOME/backup/democorpF.ldif $DXHOME/backup/delta-between-C-and-F.ldif
echo .
echo .
Ref: This HUB Model lab was built off a prior lab for MW Sync with air-gap requirements.
The prior releases of CA Identity Manager / Identity Suite have a bottleneck with the provisioning tier.
The top tier of the solution stack, Identity Manager Environment (IME/J2EE Application), may communicate to multiple Provisioning Servers (IMPS), but this configuration only has value for fail-over high availability.
This default deployment means we will have a “many-to-one” challenge, multiple IMEs experiencing a bottleneck with provisioning communication to a single IMPS server.
If this IMPS server is busy, then transactions for one or more IMEs are paused or may timeout. Unfortunately, the IME (J2EE) error messages or delays are not clear that this is a provisioning bottleneck challenge. Clients may attempt to resolve this challenge by increasing the number of IME and IMPS servers but will still be impacted by the provisioning bottleneck.
Two (2) prior methods used to overcome this bottleneck challenge were:
a) Pseudo hostname(s) entries, on the J2EE servers, for the Provisioning Tier, then rotate the order pseudo hostname(s) on the local J2EE host file to have their IP addresses access other IMPS. This methodology would give us a 1:1 configuration where one (1) IME is now locked to one (1) IMPS (by the pseudo hostname/IP address). This method is not perfect but ensures that all IMPS servers will be utilized if the number of IMPS servers equals IME (J2EE) servers. Noteworthy, this method is used by the CA identity Suite virtual appliance, where the pseudo hostname(s) are ca-prov-srv-01, ca-prov-srv-02, ca-prov-03, etc. (see image above)
b) A Router placed in-front of the IMPS (TCP 20389/20390), that contains “stickiness” to ensure that when round-robin model is used, that the same IMPS server is used for the IME that submitted a transaction, to avoid any concerns/challenges of possible”RACE” conditions, where a modify operations may occur before the create operation.
The “RACE” challenge is a concern of both of the methods above, but this risk is low, and can be managed with additional business rules that include pre-conditional checks, e.g., does the account exist before any modifications.
New CP2 Loading Balance Feature – No more bottleneck.
Identity Manager can now use round-robin load balancing support, without any restrictions on either type of provisioning operations or existing runtime limitations. This load balancing approach distributes client requests across a group of Provisioning servers.
This feature is managed in the IME tier, and will also address any RACE conditions/concerns.
No configuration changes are required on the IMPS tier. After updates of CP2, we can now use the IME Management console to export the directory.xml for the IMPS servers and update the XML tag for <Connection. This configuration may also be deployed to the Virtual Appliances.
Before applying this patch, we recommend collecting your metrics for feed operations that include multiple create operations, and modify operations to minimal of 1000 IDS, Monitor the IMPS etatrans logs as well; and the JCS/CCS logs. After the patch, run the same feed operations to determine the value of provisioning load-balance feature; and any provisioning delays that have been addressed. You may wish to increase the # of JCS/CCS servers (MS Windows) to speed up provisioning to Active Directory and other endpoints.
Restore processes may be done with snapshots-in-time for both databases and directories. We wished to provide clarity of the restoration steps after a snapshot-in-time is utilized for a directory. The methodology outlined below has the following goals: a) allow sites to prepare before they need the restoration steps, b) provide a training module to exercise samples included in a vendor solution.
In this scenario, we focused on the CA/Broadcom/Symantec Directory solution. The CA Directory provides several tools to automate online backup snapshots, but these processes stop at copies of the binary data files.
Additionally, we desired to walk-through the provided DAR (Disaster and Recovery) scenarios and determine what needed to be updated to reflect newer features; and how we may validate that we did accomplish a full restoration.
Finally, to assist with the decision tree model, where we need to triage and determine if a full restore is required, or may we select partial restoration via extracts and imports of selected data.
Cluster Out-of-Sync Scenario
Awareness
The first indicator that a userstore (CA Directory DATA DSA) is out-of-sync will be the CA Directory logs themselves, e.g. alarm or trace logs.
Another indication will be inconsistent query results for a user object that returns different results when using a front-end router to the DATA DSAs.
After awareness of the issue, the team will exercise a triage process to determine the extent of the out-of-sync data. For a quick check, one may execute LDAP queries direct to the TCP port of each DATA DSA on each host, and examine the results directory or even the total number of entries, e.g. dxTotalEntryCount.
The returned count value will help determine if the number of entries for each DATA DSA on the peer MW hosts are out-of-sync for ADD or DEL operations. The challenge/GAP with this method is it will not show any delta due to modify operations on the user objects themselves, e.g. address field changed.
Example of LDAP queries (dxsearch/ldapsearch) to CA Directory DATA DSA for the CA Identity Management solution (4 DATA DSA and 1 ROUTER DSA)
su - dsa OR [ sudo -iu dsa ]
echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd
# NOTIFY BRANCH (TCP 20404)
LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://`hostname`:20404 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=notify,dc=etadb' '(objectClass=*)' dxTotalEntryCount
dn: dc=notify,dc=etadb
# INC BRANCH (TCP 20398)
LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://`hostname`:20398 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTInclusionContainerName=Inclusions,eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount
# CO BRANCH (TCP 20396)
LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://`hostname`:20396 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount
# MAIN BRANCH (TCP 20394)
LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://`hostname`:20394 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount
# ALL BRANCHES - Router Port (TCP 20391)
LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://`hostname`:20391 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=etadb' '(objectClass=*)' dxTotalEntryCount
# Scroll to see entire line
A better process to identify the delta(s) will be automating the daily backup process, to build out LDIF files for each peer MW DATA DSA and then performing a delta process between the LDIF files. We will walk through this more involve step later in this blog entry.
Recovery Processes
The below link has examples from CA/Broadcom/Symantec with recovery notes of CA Directory DATA DSA that are out-of-sync due to extended downtime or outage window.
The below image pulled from the document (page 9.) shows CA Directory r12.x using the latest recovery processes of “multiwrite-DISP” (MW-DISP) mode.
This recovery process of MW-DISP is default for the CA Identity Management DATA DSAs via the install wizard tools, when they create the IMPD DATA DSAs.
The modified version we have started for CA Directory r14.x adds some clarity to the <dsaname>.dx files; and which steps may be adjusted to support the split data structure for the four (4) IMPD DATA DSAs.
The same time flow diagram was used. Extra notes were added for clarity, and if possible, examples of commands that will be used to assist with direct automation of each step (or maybe pasted in an SSH session window, as the dsa service ID).
Step 1, implicit in the identification/triage process, is to determine what userstore data is out-of-sync and how large a delta do we have. If the DSA service has been shut down (either deliberately or via a startup issue), if the shutdown delay is more than a few days, then the CA Directory process will check the date stamp in the <dsaname>.dp file and the transaction in the <dsaname>.tx file; if the dates are too large CA Directory will refuse to start the DATA DSA and issue a warning message.
Step 2, we will leverage the dxdisp <dsaname> command to generate a new time-stamp file <dsaname>.dx, that will be used to prevent unnecessary sync operations with any data older than the date stamp in this file.
This command should be issued for every DATA DSA on the same host—Especially true for split DATA DSAs, e.g. IMPD (CA Identity Manager’s Provisioning Directories). In our example below, to assist with this step, we use a combination of commands with a while-loop to issue the dxdisp command.
This command can be executed regardless if the DSA is running or shutdown. If an existing <dsaname>.dx file exists, any additional execution of dxdisp will add updated time-stamps to this file.
Note: The <dsaname>.dx file will be removed upon restart of the DATA DSA.
STEP 2: ISSUE DXDISP COMMAND [ Create time-stamp file for re-sync use ] ON ALL IMPD SERVERS.
su - dsa OR [ sudo -iu dsa ]
bash
dxserver status | grep -v router | awk '{print $1}' | while IFS='' read -r LINE || [ -n "$LINE" ] ; do dxdisp "$LINE" ;done ; echo ; find $DXHOME -name "*.dx" -exec ls -larth {} \;
# Scroll to see entire line
Step 3 will then ask for an updated online backup to be executed.
In earlier release of CA Directory, this required a telnet/ssh connection to the dxconsole of each DATA DSA. Or using the DSA configuration files to contain a dump dxgrid-db; command that would be executed with dxserver init all command.
In newer releases of CA Directory, we can leverage the dxserver onlinebackup <dsaname> process.
This step can be a challenge to dump all DATA DSAs at the same time, using manual procedures.
Fortunately, we can automate this with a single bash shell process; and as an enhancement, we can also generate the LDIF extracts of each DATA DSA for later delta compare operations.
Note: The DATA DSA must be running (started) for the onlinebackup process to function correctly. If unsure, issue a dxserver status or dxserver start all prior.
Retain the LDIF files from the “BAD” DATA DSA Servers for analysis.
STEP 3a-3c: ON ALL IMPD DATA DSA SERVERS - ISSUE ONLINE BACKUP PROCESS
su - dsa OR [ sudo -iu dsa ]
bash
dxserver status | grep started | grep -v router | awk '{print $1}' | while IFS='' read -r LINE || [ -n "$LINE" ] ; do dxserver onlinebackup "$LINE" ; sleep 10; dxdumpdb -w -z -f /tmp/`date '+%Y%m%d_%H%M%S_%s'`_$LINE.ldif $LINE ;done ; echo ; find $DXHOME -name "*.zdb" -exec ls -larth {} \; ; echo ; ls -larth --time-style=full-iso /tmp/*.ldif | grep `date '+%Y-%m-%d'`
# Scroll to see entire line
Step 4a Walks through the possible copy operations from “GOOD” to the “BAD” DATA DSA host, for the <dsaname>.zdb files. The IMPD DATA DSA will require that three (3) of four (4) zdb files are copied, to ensure no impact to referential integrity between the DATA DSA.
The preferred model to copy data from one remote host to another is via the compressed rsync process over SSH, as this is a rapid process for the CA Directory db / zdb files.
Below are the code blocks that demonstrate examples how to copy data from one DSA server to another DSA server.
# RSYNC METHOD
sudo -iu dsa
time rsync --progress -e 'ssh -ax' -avz --exclude "User*" --exclude "*.dp" --exclude "*.tx" dsa@192.168.242.135:./data/ $DXHOME/data
# Scroll to see entire line
# SCP METHOD
sudo -iu dsa
scp REMOTE_ID@$HOST:./data/<folder_impd_data_dsa_name>/*.zdb /tmp/dsa_data
/usr/bin/mv /tmp/dsa_data/<incorrect_dsaname>.zdb $DXHOME/data/<folder_impd_data_dsa_name>/<correct_dsaname>.db
# Scroll to see entire line
Step 4b Walk through the final steps before restarting the “BAD” DATA DSA.
The ONLY files that should be in the data folders are <dsaname>.db (binary data file) and <dsaname>.dx (ASCII time-stamp file). Ensure that the copied <prior-hostname-dsaname>.zdb file has been renamed to the correct hostname & extension for <dsaname>.db
Remove the prior <dsaname>.dp (ASCII time-stamp file) { the DATA DSA will auto replace this file with the *.dx file contents } and the <dsaname>.tx (binary data transaction file).
Step 5a Startup the DATA DSA with the command
dxserver start all
If there is any issue with a DATA or ROUTER DSA not starting, then issue the same command with the debug switch (-d)
dxserver -d start <dsaname>
Use the output from the above debug process to address any a) syntax challenges, or b) older PID/LCK files ($DXHOME/pid)
Step 5b Finally, use dxsearch/ldapsearch to query a unit-test of authentication with the primary service ID. Use other unit/use-case tests as needed to confirm data is now synced.
bash
echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd
LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://`hostname`:20394 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s base -b 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' '(objectClass=*)' | perl -p00e 's/\r?\n //g'
# Scroll to see entire line
LDIF Recovery Processes
The steps above are for recovery via a 100% replacement method, where the assumption is that the “bad” DSA server does NOT have any data worth keeping or wish to be reviewed.
We wish to clarify a process/methodology, where the “peer” Multi-write DSA may be out-of-sync. Still, we are not sure “which” is truly the “good DSA” to select, or perhaps we wished to merge data from multiple DSA before we declare one to be the “good DSA” (with regards to the completeness of data).
Using CA Directory commands, we can join them together to automate snapshots and exports to LDIF files. These LDIF files can then be compared against their peers MW DATA DSA exports or even to themselves at different snapshot export times. As long as we have the LDIF exports, we can recover from any DAR scenario.
Example of using CA Directory dxserver and dxdumpdb commands (STEP 3) with the ldifdelta and dxmodify commands.
The output from ldifdelta may be imported to any remote peer MW DATA DSA server to sync via dxmodify to that hostname, to force a sync for the few objects that may be out-of-sync, e.g. Password Hashes or other.
The below images demonstrate a delta that exists between two (2) time snapshots. The CA Directory tool, ldifdelta, can identify and extract the modified entry to the user object.
The following examples will show how to re-import this delta using dxmodify command to the DATA DSA with no other modifications required to the input LDIF file.
In the testing example below, before any update to an object, let’s capture a snapshot-in-time and the LDIF files for each DATA DSA.
Lets make an update to a user object using any tool we wish, or command line process like ldapmodify.
Next, lets capture a new snapshot-in-time after the update, so we will be able to utilize the ldifdelta tool.
We can use the ldifdelta tool to create the delta LDIF input file. After we review this file, and accept the changes, we can then submit this LDIF file to the remote peer MW DATA DSA that are out-of-sync.
Hope this has value to you and any challenges you may have with your environment.
ANA Technology Partner 