“DSA is attempting to start after a long outage, perform a recovery procedure before starting”
Challenge: The IMPD (Identity Manager Provisioning Directory) Data DSAs have been offline for a while, e.g. 7 days+ (> 1 week), and the Symantec/CA Directory solution will, to protect the data, refuse to allow the DATA DSAs to start unless there is manual intervention to prevent the possibility of production data (Live DATA DSAs) being synced with older data (Offline DATA DSAs).
If we were concern, we would follow best practices and remove the offline DATA DSAs’ *.db & *.dp files, and replace the *.db with current copies of the Live DATA DSAs’ *.db files; generate temporary time files of *.dx and allow the time files of *.dp to rebuild themselves upon startup of the offline DATA DSAs.
Example to recover from an outage: https://anapartner.com/2020/08/21/directory-backup-and-restore-dar-scenarios/
However, if we are NOT concern, or the environment is non-production we can avoid the multiple shells, multiple commands to resync by using a combinations of bash shell commands. The proposal below outlines using the Symantec/CA Identity Suite virtual appliance, where both the IMPD and IMPS (Identity Manager Provisioning Server) components reside on the same servers.
Proposal: Use a single Linux host to send remote commands as a single user ID; sudo to the ‘dsa’ and ‘imps’ service IDs, and issue commands to address the restart process.
Pre-Work: For the Identity Suite vApp, recommend that .ssh keys be used to avoid using a password for the ‘config’ user IDs on all vApp nodes.
Example to setup .SSH keys for ‘config’ user ID: https://anapartner.com/2020/05/01/avoid-locking-a-userid-in-a-virtual-appliance/
If using .SSH keys, do not forget to use this shortcut to cache the local session: eval `ssh-agent` && ssh-add
Steps: Issue the following bash commands with the correct IPs or hostnames.
If possible, wrap the remote commands in a for-loop. The below example uses the local ‘config’ user ID, to ssh to remote servers, then issues a local su to the ‘dsa’ service ID. The ‘dsa’ commands may need to be wrapped as shown below to allow multiple commands to be executed together. We have a quick hostname check, stop all IMPD DATA DSAs, find the time-stamp file that is preventing the startup of the IMPD DATA DSAs and remove it, restart all IMPD DATA DSA, and then move on to the next server with the for-loop. The ‘imps’ commands are similar with a quick hostname check, status check, stop and start process, another status check, then move on to the next server in the for-loop.
for i in {136..141}; do ssh -t config@192.168.242.$i "su - dsa -c \"hostname;dxserver stop all;pwd;find ./data/ -type f \( -name '*.dp' \) -delete ;dxserver start all \" "; done
for i in {136..141}; do ssh -t config@192.168.242.$i "su - imps -c \"hostname;imps status;imps stop;imps start;imps status \" "; done
View of for-loop commands output:
Additional: Process to assist with decision to sync or not sync.
Check if the number of total entries in each individual IMPD DATA DSA match with their peers (Multi-Write groups). Goal: Avoid any deltas > 1% between peers. The IMPD “main”, “co”, “inc” DATA DSA should be 100% in sync. We may see some minor flux in the “notify” DATA DSA, as this is temporary data used by the IMPS server to store data to be sent to the IME via the IME Call Back Process.
If there are any deltas, then we may export the IMPD DATA DSAs to LDIF files and then use the Symantec/CA Directory ldifdelta process to isolate and triage the deltas.
su - dsa OR [ sudo -iu dsa ]
export HISTIGNORE=' *' {USE THIS LINE TO FORCE HISTORY TO IGNORE ANY COMMANDS WITH A LEADING SPACE CHARACTER}
echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd {USE SPACE CHARACTER IN FRONT TO AVOID HISTORY USAGE}
# NOTIFY BRANCH (TCP 20404)
for i in {135..140}; do echo "########## 192.168.242.$i IMPD NOTIFY DATA DSA ##########";LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://192.168.242.$i:20404 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=notify,dc=etadb' '(objectClass=*)' dxTotalEntryCount | perl -p00e 's/\r?\n //g' ; done
# INC BRANCH (TCP 20398)
for i in {135..140}; do echo "########## 192.168.242.$i IMPD INC DATA DSA ##########";LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://192.168.242.$i:20398 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTInclusionContainerName=Inclusions,eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount | perl -p00e 's/\r?\n //g' ; done
# CO BRANCH (TCP 20396)
for i in {135..140}; do echo "########## 192.168.242.$i IMPD CO DATA DSA ##########";LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://192.168.242.$i:20396 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount | perl -p00e 's/\r?\n //g' ; done
# MAIN BRANCH (TCP 20394)
for i in {135..140}; do echo "########## 192.168.242.$i IMPD MAIN DATA DSA ##########";LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://192.168.242.$i:20394 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount | perl -p00e 's/\r?\n //g' ; done
