Month: August 2020

Multi-Write HUB Model with democorp

August 24, 2020August 24, 2020by Alan BaugherLeave a comment

A useful feature with CA Directory for WAN latency challenges is the HUB model. This model allows sync of the data to occur to distant peer Multi-write DATA DSA, but does NOT impact the external application that is updating its own local Router/DATA DSAs.

To assist with understanding this HUB model, we have leverage the CA Directory samples of democorp & router to build out an architecture with six (6) DATA DSAs and two (2) router DSAs, to emulate two (2) data centers across the world. These samples are included with every CA Directory deployment under $DXHOME/samples/democorp & $DXHOME/samples/router.

This lab emulates two (2) of the three (3) data centers that are displayed within the CA documentation.

Ref: https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-identity-and-access-management/directory/14-1/ca-directory-concepts/directory-replication/multiwrite-mw-groups-hubs/topology-sample-and-disaster-recovery.html

This lab may be replicated with near-real-world WAN latency within VMWARE Workstation feature.

https://www.vmware.com/products/workstation-pro.html

Below is a bash shell script used to create the lab environment that was created on a single host OS, with CA Directory samples of “democorp” and “router”. These samples were “copied” and updated via sed commands to ensure the DSA are unique for TCP Ports and naming convention. The below examples’ nomenclature will use democorpX for (A-F) and routerYY (AA or BB) .

These DATA DSA use the same suffix and are all referenced in the group knowledge file. The HUB model configuration will change the behavior for MW-DISP replication between data centers. MW-DISP replication will still be used for all local sync between DATA DSA in the same data center, and would ONLY be used between data centers for the DATA DSA that are designated as “HUB”s (aka multi-write-group-hub).

To test the value of HUB model with WAN latency, suggest that the same lab be executed on two (2) hosts, where one host has the VMWARE network latency enabled to 150/150 milli-seconds. Update the ip addresses within the $DXHOME/config/knowlege/*.dxc on both host OS, to reflect the correct hostnames for each data center in the DSAs.

The diagram below outlines the delta between various democorp DATA DSA to use the HUB model.

The changes below show the deltas within the *.dxc/*.dxg files within the knowledge folder for democorp MW HUB model.

The below image captures the only deltas in the *.dxi (startup files) for the democorp MW HUB model, located within the server folder. Note, if CA Directory management tool is deployed and used for democorp, all configurations will be in a single *.dxi file.

#!/bin/bash
##############################################
#
# Name: democorp_mw_hub_lab.sh
#
# Multi-Write HUB lab using CA Directory and the samples of
# democorp and router under DXHOME/samples
# A. Baugher, 04/2020 - ANA Technology Partner
#
# Assumptions:
#   CA Directory is deployed & dxprofile is enabled for dsa user
#   Execute script as dsa user
#
# Step 0.  Clean-Up prior deployment
#
# Step 1.  Auto deploy both democorp and router samples with: setup.sh -q
#
# Step 2.  Make common changes in democorp prior to copying
#
# Step 3.  Create six (6) copies of democorp and two (2) copies of router
#
# Step 4.  Update the six (6) copies of democorp for:
#     - name
#     - ports
#     - multi-write-group  (HUB group)
#     - DSA flags for MW & HUB-DSA
#     - Group knowledge file reference
#
#        Update the two (2) copies of router for:
#    - name
#    - ports
#    - Group knowledge file reference
#    - set write-precedence  (for HUB-DSA)
#
# Step 5. Start all DSAs
#
# Step 6. Test with dxsearch query
#
# Step 7. Execute the dxsoak command with the service account & time command
#
# Step 8. Update democorpA to force a single delta between peer members of AA and BB
#
# Step 9.  Create LDAP Export
#
# Step 10.  Create LDAP Delta & Compare the various democorp DSA to validate sync operations
#
#
##############################################
#set -xv
echo ..
echo "#############################################################"
echo "Step 0.  Clean up prior deployment of democorp and router"
echo "#############################################################"
dxserver stop all
sleep 5
kill -9 `ps -ef | grep dsa | grep democorp | grep -v grep | grep -v "democorp_mw_hub_lab" | awk '{print $2}'` >   /dev/null 2>&1
kill -9 `ps -ef | grep dsa | grep router   | grep -v grep | awk '{print $2}'` >   /dev/null 2>&1
sleep 5
rm -rf $DXHOME/data/democorp*.*
rm -rf $DXHOME/config/knowledge/democorp*.*
rm -rf $DXHOME/config/knowledge/router*.*
rm -rf $DXHOME/config/servers/democorp*.*
rm -rf $DXHOME/config/servers/router*.*
rm -rf $DXHOME/logs/democorp*.*
rm -rf $DXHOME/logs/router*.*
rm -rf $DXHOME/backup/delta*.*  > /dev/null 2>&1
rm -rf $DXHOME/backup/*.ldif > /dev/null 2>&1


echo ..
echo "#############################################################"
echo "Step 1a. Deploy clean version of democorp and router"
echo "#############################################################"
cd  $DXHOME/samples/democorp
$DXHOME/samples/democorp/setup.sh -q  > /dev/null 2>&1
cd $DXHOME/samples/router
$DXHOME/samples/router/setup.sh -q    > /dev/null 2>&1

cd
echo ..
echo "#############################################################"
echo "Step 1b. Create service ID in democorp for later use"
echo "#############################################################"
cat << EOF > $DXHOME/diradmin.ldif
version: 1
dn: cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
cn: diradmin
sn: diradmin
givenName: diradmin
userPassword: Password01
EOF

dxmodify -a -c -h `hostname` -p 19389 -f $DXHOME/diradmin.ldif

echo ..
echo "#############################################################"
echo "Step 1c.  Stop all running democorp & router DSAs"
echo "#############################################################"
dxserver stop all
sleep 10

echo ..
echo "#############################################################"
echo "Step 2a. Make common changes in pre-existing files before other modification"
echo "Update dsa-flags in democorp.dxc to allow Multi-Write with a HUB"
echo "#############################################################"
sed -i 's|ssl-auth|ssl-auth\n    multi-write-group = hub_group_AA\n     dsa-flags     =|g' $DXHOME/config/knowledge/democorp.dxc
sed -i 's|dsa-flags     =|dsa-flags     = multi-write, no-service-while-recovering, load-share|g' $DXHOME/config/knowledge/democorp.dxc

echo ..
echo "#############################################################"
echo "Step 2b. Update MW recovery in democorp.dxi file"
echo "#############################################################"
sed -i 's|recovery = false;|recovery = true;|g' $DXHOME/config/servers/democorp.dxi

echo ..
echo "#############################################################"
echo "Step 3a. Create six (6) copies of democorp and two (2) routers"
echo "Copy democorp data folder contents"
echo "#############################################################"
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpA.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpA.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpB.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpB.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpC.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpC.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpD.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpD.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpE.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpE.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpF.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpF.tx  > /dev/null 2>&1

echo ..
echo "#############################################################"
echo "Step 3b. Copy autostart folder contents"
echo "#############################################################"
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpA
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpB
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpC
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpD
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpE
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpF
cp -r -p $DXHOME/config/autostart/router    $DXHOME/config/autostart/routerAA
cp -r -p $DXHOME/config/autostart/router    $DXHOME/config/autostart/routerBB

echo ..
echo "#############################################################"
echo "Step 3c. Copy knowledge folder contents"
echo "#############################################################"
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpA.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpB.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpC.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpD.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpE.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpF.dxc
cp -r -p $DXHOME/config/knowledge/router.dxc   $DXHOME/config/knowledge/routerAA.dxc
cp -r -p $DXHOME/config/knowledge/router.dxc   $DXHOME/config/knowledge/routerBB.dxc
cp -r -p $DXHOME/config/knowledge/sample.dxg   $DXHOME/config/knowledge/groupAA.dxg
cp -r -p $DXHOME/config/knowledge/sample.dxg   $DXHOME/config/knowledge/groupBB.dxg

echo ..
echo "#############################################################"
echo "Step 3d. Copy server folder contents"
echo "#############################################################"
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpA.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpB.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpC.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpD.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpE.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpF.dxi
cp -r -p $DXHOME/config/servers/router.dxi     $DXHOME/config/servers/routerAA.dxi
cp -r -p $DXHOME/config/servers/router.dxi     $DXHOME/config/servers/routerBB.dxi

echo ..
echo "#############################################################"
echo "Step 4a.  Update names & ports in democorp knowledge files"
echo "#############################################################"
sed -i 's|19389|29389|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|19390|29390|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPA =|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPA>|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|19389|29489|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|19390|29490|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPB =|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPB>|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|19389|29589|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|19390|29590|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPC =|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPC>|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|19389|29689|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|19390|29690|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPD =|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPD>|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|19389|29789|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|19390|29790|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPE =|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPE>|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|19389|29889|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|19390|29890|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPF =|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPF>|g' $DXHOME/config/knowledge/democorpF.dxc

echo ..
echo "#############################################################"
echo "Step 4b. Update knowledge files for router ports"
echo "#############################################################"
sed -i 's|19289|39289|g' $DXHOME/config/knowledge/routerAA.dxc
sed -i 's|19290|39290|g' $DXHOME/config/knowledge/routerAA.dxc
sed -i 's|dsa ROUTER =|dsa ROUTERAA =|g' $DXHOME/config/knowledge/routerAA.dxc
sed -i 's|19289|39389|g' $DXHOME/config/knowledge/routerBB.dxc
sed -i 's|19290|39390|g' $DXHOME/config/knowledge/routerBB.dxc
sed -i 's|dsa ROUTER =|dsa ROUTERBB =|g' $DXHOME/config/knowledge/routerBB.dxc

echo ..
echo "#############################################################"
echo "Step 4c. Update group knowledge file for three (3)MW Group HUB Peers "
echo "#############################################################"
sed -i 's|"router.dxc";|"routerAA.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
sed -i 's|"democorp.dxc";|"democorpA.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
sed -i 's|"democorpA.dxc";|"democorpA.dxc";\nsource "democorpB.dxc";\nsource "democorpC.dxc";\nsource "routerBB.dxc";\nsource "democorpD.dxc";\nsource "democorpE.dxc";\nsource "democorpF.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
sed -i 's|source "unspsc.dxc";|#source "unspsc.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg

cp -r -p $DXHOME/config/knowledge/groupAA.dxg $DXHOME/config/knowledge/groupBB.dxg

#sed -i 's|"router.dxc";|"routerBB.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|"democorp.dxc";|"democorpD.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|"democorpD.dxc";|"democorpD.dxc";\nsource "democorpE.dxc";\nsource "democorpF.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|source "unspsc.dxc";|#source "unspsc.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg

echo ..
echo "#############################################################"
echo "Step 4d.  Update Server folder contents"
echo "#############################################################"
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/democorpA.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/democorpB.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/democorpC.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/democorpD.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/democorpE.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/democorpF.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/routerAA.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/routerBB.dxi


echo ..
echo "#############################################################"
echo "Step 4e.  Update HUB Configurations in DSA knowledge and DSA routers"
echo "#############################################################"
sed -i 's|load-share|load-share, multi-write-group-hub|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|load-share|load-share, multi-write-group-hub|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|multi-write-group = hub_group_AA|multi-write-group = hub_group_BB|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|multi-write-group = hub_group_AA|multi-write-group = hub_group_BB|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|multi-write-group = hub_group_AA|multi-write-group = hub_group_BB|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|/knowledge/groupAA.dxg";|/knowledge/groupAA.dxg";\nset  write-precedence = democorpA ,democorpB, democorpC;\n|g' $DXHOME/config/servers/routerAA.dxi
sed -i 's|/knowledge/groupBB.dxg";|/knowledge/groupBB.dxg";\nset  write-precedence = democorpD ,democorpE, democorpF;\n|g' $DXHOME/config/servers/routerBB.dxi

echo ..
echo "#############################################################"
echo "Step 4f.  Remove samples of router & democorp from starting "
echo "#############################################################"
rm -rf $DXHOME/config/servers/democorp.dxi
rm -rf $DXHOME/config/servers/router.dxi
rm -rf $DXHOME/config/autostart/democorp
rm -rf $DXHOME/config/autostart/router

echo ..
echo "#############################################################"
echo "Step 5. Start all DSAs"
echo "#############################################################"
dxcertgen certs > /dev/null 2>&1
dxserver start all

dxserver status

#exit

echo ..
echo "#############################################################"
echo "Step 6. Test all DSAs with dxsearch query"
echo "#############################################################"
# Comment out if too verbose
# Data DSAs
#dxsearch -h `hostname` -p 29389 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29489 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29589 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29689 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29789 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29889 -c -x -b o=DEMOCORP,c=AU
# Router DSAs
#dxsearch -h `hostname` -p 39289 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 39389 -c -x -b o=DEMOCORP,c=AU

# Data DSAs
#dxsearch -h `hostname` -p 29389 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29489 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29589 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29689 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29789 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29889 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
# Router DSAs
#dxsearch -h `hostname` -p 39289 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 39389 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01

echo ..
echo "#############################################################"
echo "Step 7. Execute the dxsoak command with the service account & time command"
echo "allow to run for over 5 sec to monitor changes for Multi-Write"
echo "may allow for longer times (1 hour) to get better performance metrics"
echo "#############################################################"
cd $DXHOME/samples/dxsoak
echo "Update democorpA (TCP 29389) to confirm MW to from democorpA (hub_group_AA) to democorpD (hub_group_BB)"
# Create a delete file first; then re-add entries
grep dn: democorp.eldf | grep ,ou=Services > democorp-del.eldf
sed -i 's|,c=AU|,c=AU\nchangetype: del\n|g' democorp-del.eldf

echo ..
echo "#############################################################"
echo "# Delete all DN entries with ou=Services: `wc -l democorp-del.eldf` on democorpA (TCP 29389)"
time ./dxsoak -c -t 2 -q 10 -l 5 -h `hostname`:29389 -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 -f democorp-del.eldf

echo ..
echo "#############################################################"
echo "# Re-Add all DN entries with ou=Services: `wc -l democorp.eldf` on democorpD (TCP 29689)"
time ./dxsoak -c -t 2 -q 10 -l 5 -h `hostname`:29689 -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 -f democorp.eldf


echo ..
echo "#############################################################"
echo "Step 8a. Update democorpA to force a single delta between peer members of AA and BB"
echo "#############################################################"
cd
cat << EOF > $DXHOME/diradmin_sn.ldif
dn: cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU
changetype: modify
replace: sn
sn: diradmin_AA_new_update
EOF

echo "#############################################################"
echo "# Query democorpA (TCP 29389) for sn value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29389 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 sn createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for sn value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 sn createTimestamp modifyTimestamp

echo "#############################################################"
echo "# Make update on democorpA"
echo "#############################################################"
dxmodify -a -c -h `hostname` -p 29389 -f $DXHOME/diradmin_sn.ldif

echo "#############################################################"
echo "# Query democorpA (TCP 29389) for sn value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29389 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 sn createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for sn value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 sn createTimestamp modifyTimestamp

#exit
echo ..
echo "#############################################################"
echo "Step 8b. Update democorpF to force a reverse single delta between peer members of AA and BB"
echo "#############################################################"
cd
cat << EOF > $DXHOME/diradmin_givenName.ldif
dn: cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU
changetype: modify
replace: givenName
givenName: diradmin_BB_new_update
EOF


echo "#############################################################"
echo "# Query democorpC (TCP 29589) for givenName value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29589 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 givenName createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for givenName value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 givenName createTimestamp modifyTimestamp

echo "#############################################################"
echo "# Update democorpF to show replication via democorpD (HUB) to democorpA (HUB) "
echo "#############################################################"
dxmodify -a -c -h `hostname` -p 29889 -f $DXHOME/diradmin_givenName.ldif

echo "#############################################################"
echo "# Query democorpC (TCP 29589) for givenName value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29589 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 givenName createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for givenName value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 givenName createTimestamp modifyTimestamp




echo ..
echo "###########################################################"
echo "Step 9b. Update CA Directory DSA to allow online backup ###"
echo "###########################################################"
echo " - Configure CA Directory to provide an data dump (zdb file) while DSA are online"
cp -r -p $DXHOME/config/settings/default.dxc.org $DXHOME/config/settings/default.dxc  > /dev/null 2>&1
cp -r -p $DXHOME/config/settings/default.dxc $DXHOME/config/settings/default.dxc.org  > /dev/null 2>&1
# Edit the DSA settings file to add in one line.  dump dxgrid-db;
chmod 744 $DXHOME/config/settings/default.dxc
echo "dump dxgrid-db;" >> $DXHOME/config/settings/default.dxc



echo ..
echo "######################################################################################"
echo "Step 9c. Re-init all DSA to data dump the CA DSAs for democorp & router "
echo "######################################################################################"
echo " - This make take 5-30 seconds to complete "
dxserver init all    > /dev/null 2>&1
# View for zdb or zd? (in-progress) files
sleep 10



echo ..
echo "#################################################################"
echo "Step 9d. Export DSA backup/offline zdb data files to LDIF file ###"
echo "#################################################################"
echo " - Export will happen after the backup/offline zdb files are fully created"
echo " - This make take 5-60 seconds  to complete "
echo ..
echo "#################################################################"
echo "Step 9e. Set WHILE loop for DemocorpF DSA ###"
echo "#################################################################"
until [ -f $DXHOME/data/democorpF.zdb ]
do
     echo " - Waiting till CA Directory has completed online data dump of DemocorpF DSA"
     sleep 5
done
sleep 5
echo ..
echo "#################################################################"
echo "Step 9f. Execute dxdumbdb for Democorp DSA - FULL ###"
echo "#################################################################"
mkdir $DXHOME/backup  > /dev/null 2>&1
cd $DXHOME/backup
dxdumpdb -z -f $DXHOME/backup/democorpA.ldif democorpA   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpB.ldif democorpB   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpC.ldif democorpC   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpD.ldif democorpD   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpE.ldif democorpE   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpF.ldif democorpF   > /dev/null 2>&1
sleep 5

echo ..
echo "#################################################################"
echo "Step 10a. Perform LDIF DELTA compare between democorpA and democorpB within same HUB MW group"
echo "Look for any delta in the metrics > 0"
echo "#################################################################"
#ldifdelta -x -S DSANAME  OLDFILE NEWFILE DELTAFILE
ldifdelta -x -S democorpA $DXHOME/backup/democorpA.ldif  $DXHOME/backup/democorpB.ldif $DXHOME/backup/delta-between-A-and-B.ldif
echo "#################################################################"
echo "Step 10b. Perform LDIF DELTA compare between democorpD and democorpE within same HUB MW group"
echo "Look for any delta in the metrics > 0"
echo "#################################################################"
ldifdelta -x -S democorpC $DXHOME/backup/democorpD.ldif  $DXHOME/backup/democorpE.ldif $DXHOME/backup/delta-between-D-and-E.ldif
echo "#################################################################"
echo "Step 10c. Perform LDIF DELTA compare between democorpC and democorpF across different HUB MW groups"
echo "Look for any delta in the metrics > 0"
echo "#################################################################"
ldifdelta -x -S democorpC $DXHOME/backup/democorpC.ldif  $DXHOME/backup/democorpF.ldif $DXHOME/backup/delta-between-C-and-F.ldif

echo .
echo .

Ref: This HUB Model lab was built off a prior lab for MW Sync with air-gap requirements.

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?MessageKey=62ccc41d-7c37-4728-ad1e-c82e7a8acc38&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

Load Balancing Provisioning Tier

August 24, 2020August 24, 2020by Alan BaugherLeave a comment

The prior releases of CA Identity Manager / Identity Suite have a bottleneck with the provisioning tier.

The top tier of the solution stack, Identity Manager Environment (IME/J2EE Application), may communicate to multiple Provisioning Servers (IMPS), but this configuration only has value for fail-over high availability.

This default deployment means we will have a “many-to-one” challenge, multiple IMEs experiencing a bottleneck with provisioning communication to a single IMPS server.

If this IMPS server is busy, then transactions for one or more IMEs are paused or may timeout. Unfortunately, the IME (J2EE) error messages or delays are not clear that this is a provisioning bottleneck challenge. Clients may attempt to resolve this challenge by increasing the number of IME and IMPS servers but will still be impacted by the provisioning bottleneck.

Two (2) prior methods used to overcome this bottleneck challenge were:

a) Pseudo hostname(s) entries, on the J2EE servers, for the Provisioning Tier, then rotate the order pseudo hostname(s) on the local J2EE host file to have their IP addresses access other IMPS. This methodology would give us a 1:1 configuration where one (1) IME is now locked to one (1) IMPS (by the pseudo hostname/IP address). This method is not perfect but ensures that all IMPS servers will be utilized if the number of IMPS servers equals IME (J2EE) servers. Noteworthy, this method is used by the CA identity Suite virtual appliance, where the pseudo hostname(s) are ca-prov-srv-01, ca-prov-srv-02, ca-prov-03, etc. (see image above)

<Connection
  host="ca-prov-srv-primary" port="20390"
  failover="ca-prov-srv-01:20390,ca-prov-srv-02:20390,ca-prov-srv-03:20390,ca-prov-srv-04:20390“
/>

b) A Router placed in-front of the IMPS (TCP 20389/20390), that contains “stickiness” to ensure that when round-robin model is used, that the same IMPS server is used for the IME that submitted a transaction, to avoid any concerns/challenges of possible”RACE” conditions, where a modify operations may occur before the create operation.

The “RACE” challenge is a concern of both of the methods above, but this risk is low, and can be managed with additional business rules that include pre-conditional checks, e.g., does the account exist before any modifications.

Ref: RACE https://en.wikipedia.org/wiki/Race_condition

Example of one type of RACE condition that may be seen.

Ref: PX Rule Engine: https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-identity-and-access-management/identity-manager/14-3/Release-Notes/Cumulative-Patches/Latest-Cumulative-Patch-14_3-CP2.html

New CP2 Loading Balance Feature – No more bottleneck.

Identity Manager can now use round-robin load balancing support, without any restrictions on either type of provisioning operations or existing runtime limitations. This load balancing approach distributes client requests across a group of Provisioning servers.

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-identity-and-access-management/identity-manager/14-3/Release-Notes/release-features-and-enhancement/Identity-Manager-14_3-CP2.html#concept.dita_b51ab03e-6e77-49be-8235-e50ee477247a_LoadBalancing

This feature is managed in the IME tier, and will also address any RACE conditions/concerns.

No configuration changes are required on the IMPS tier. After updates of CP2, we can now use the IME Management console to export the directory.xml for the IMPS servers and update the XML tag for <Connection. This configuration may also be deployed to the Virtual Appliances.

<Connection   
  host="ca-prov-srv-primary" port="20390”   
  loadbalance="ca-prov-srv-02:20390,ca-prov-srv-03:20390,ca-prov-srv-04:20390“   
  failover="ca-prov-srv-01:20390,ca-prov-srv-02:20390,ca-prov-srv-03:20390,ca-prov-srv-04:20390“ 
/>

View of CP2 to download.

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-identity-and-access-management/identity-manager/14-3/Release-Notes/Cumulative-Patches/Latest-Cumulative-Patch-14_3-CP2.html

Before applying this patch, we recommend collecting your metrics for feed operations that include multiple create operations, and modify operations to minimal of 1000 IDS, Monitor the IMPS etatrans logs as well; and the JCS/CCS logs. After the patch, run the same feed operations to determine the value of provisioning load-balance feature; and any provisioning delays that have been addressed. You may wish to increase the # of JCS/CCS servers (MS Windows) to speed up provisioning to Active Directory and other endpoints.

Disaster Recovery Scenarios for Directories

August 21, 2020October 9, 2020by Alan Baugher1 Comment

Restore processes may be done with snapshots-in-time for both databases and directories. We wished to provide clarity of the restoration steps after a snapshot-in-time is utilized for a directory. The methodology outlined below has the following goals: a) allow sites to prepare before they need the restoration steps, b) provide a training module to exercise samples included in a vendor solution.

In this scenario, we focused on the CA/Broadcom/Symantec Directory solution. The CA Directory provides several tools to automate online backup snapshots, but these processes stop at copies of the binary data files.

Additionally, we desired to walk-through the provided DAR (Disaster and Recovery) scenarios and determine what needed to be updated to reflect newer features; and how we may validate that we did accomplish a full restoration.

Finally, to assist with the decision tree model, where we need to triage and determine if a full restore is required, or may we select partial restoration via extracts and imports of selected data.

Cluster Out-of-Sync Scenario

Awareness

The first indicator that a userstore (CA Directory DATA DSA) is out-of-sync will be the CA Directory logs themselves, e.g. alarm or trace logs.

Another indication will be inconsistent query results for a user object that returns different results when using a front-end router to the DATA DSAs.

After awareness of the issue, the team will exercise a triage process to determine the extent of the out-of-sync data. For a quick check, one may execute LDAP queries direct to the TCP port of each DATA DSA on each host, and examine the results directory or even the total number of entries, e.g. dxTotalEntryCount.

The returned count value will help determine if the number of entries for each DATA DSA on the peer MW hosts are out-of-sync for ADD or DEL operations. The challenge/GAP with this method is it will not show any delta due to modify operations on the user objects themselves, e.g. address field changed.

Example of LDAP queries (dxsearch/ldapsearch) to CA Directory DATA DSA for the CA Identity Management solution (4 DATA DSA and 1 ROUTER DSA)

su - dsa    OR [ sudo -iu dsa ]
echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd

# NOTIFY BRANCH (TCP 20404) 
LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://`hostname`:20404 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=notify,dc=etadb' '(objectClass=*)' dxTotalEntryCount
dn: dc=notify,dc=etadb

# INC BRANCH (TCP 20398)
LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://`hostname`:20398 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTInclusionContainerName=Inclusions,eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount

# CO BRANCH (TCP 20396)
LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://`hostname`:20396 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount

# MAIN BRANCH (TCP 20394)
LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://`hostname`:20394 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount

# ALL BRANCHES - Router Port (TCP 20391)
LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://`hostname`:20391 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=etadb' '(objectClass=*)' dxTotalEntryCount

# Scroll to see entire line

A better process to identify the delta(s) will be automating the daily backup process, to build out LDIF files for each peer MW DATA DSA and then performing a delta process between the LDIF files. We will walk through this more involve step later in this blog entry.

Recovery Processes

The below link has examples from CA/Broadcom/Symantec with recovery notes of CA Directory DATA DSA that are out-of-sync due to extended downtime or outage window.

The below image pulled from the document (page 9.) shows CA Directory r12.x using the latest recovery processes of “multiwrite-DISP” (MW-DISP) mode.

This recovery process of MW-DISP is default for the CA Identity Management DATA DSAs via the install wizard tools, when they create the IMPD DATA DSAs.

https://knowledge.broadcom.com/external/article?articleId=54088

The above document is dated, and still mentions additional file structures that have been retired, e.g. oc/zoc, at,zat.

An enhancement request has been submitted for both of these requests:

https://community.broadcom.com/participate/ideation-home/viewidea?IdeationKey=c71a304b-a689-4894-ac1c-786c9a2b2d0d

The modified version we have started for CA Directory r14.x adds some clarity to the <dsaname>.dx files; and which steps may be adjusted to support the split data structure for the four (4) IMPD DATA DSAs.

The same time flow diagram was used. Extra notes were added for clarity, and if possible, examples of commands that will be used to assist with direct automation of each step (or maybe pasted in an SSH session window, as the dsa service ID).

Step 1, implicit in the identification/triage process, is to determine what userstore data is out-of-sync and how large a delta do we have. If the DSA service has been shut down (either deliberately or via a startup issue), if the shutdown delay is more than a few days, then the CA Directory process will check the date stamp in the <dsaname>.dp file and the transaction in the <dsaname>.tx file; if the dates are too large CA Directory will refuse to start the DATA DSA and issue a warning message.

Step 2, we will leverage the dxdisp <dsaname> command to generate a new time-stamp file <dsaname>.dx, that will be used to prevent unnecessary sync operations with any data older than the date stamp in this file.

This command should be issued for every DATA DSA on the same host—Especially true for split DATA DSAs, e.g. IMPD (CA Identity Manager’s Provisioning Directories). In our example below, to assist with this step, we use a combination of commands with a while-loop to issue the dxdisp command.

This command can be executed regardless if the DSA is running or shutdown. If an existing <dsaname>.dx file exists, any additional execution of dxdisp will add updated time-stamps to this file.

Note: The <dsaname>.dx file will be removed upon restart of the DATA DSA.

STEP 2: ISSUE DXDISP COMMAND [ Create time-stamp file for re-sync use ] ON ALL IMPD SERVERS.

su - dsa OR [ sudo -iu dsa ]
bash
dxserver status | grep -v router | awk '{print $1}' | while IFS='' read -r LINE || [ -n "$LINE" ] ; do dxdisp "$LINE" ;done ; echo ; find $DXHOME -name "*.dx" -exec ls -larth {} \;

# Scroll to see entire line

Step 3 will then ask for an updated online backup to be executed.

In earlier release of CA Directory, this required a telnet/ssh connection to the dxconsole of each DATA DSA. Or using the DSA configuration files to contain a dump dxgrid-db; command that would be executed with dxserver init all command.

In newer releases of CA Directory, we can leverage the dxserver onlinebackup <dsaname> process.

This step can be a challenge to dump all DATA DSAs at the same time, using manual procedures.

Fortunately, we can automate this with a single bash shell process; and as an enhancement, we can also generate the LDIF extracts of each DATA DSA for later delta compare operations.

Note: The DATA DSA must be running (started) for the onlinebackup process to function correctly. If unsure, issue a dxserver status or dxserver start all prior.

Retain the LDIF files from the “BAD” DATA DSA Servers for analysis.

STEP 3a-3c: ON ALL IMPD DATA DSA SERVERS - ISSUE ONLINE BACKUP PROCESS
su - dsa OR [ sudo -iu dsa ]
bash

dxserver status | grep started | grep -v router | awk '{print $1}' | while IFS='' read -r LINE || [ -n "$LINE" ] ; do dxserver onlinebackup "$LINE" ; sleep 10; dxdumpdb -w -z -f /tmp/`date '+%Y%m%d_%H%M%S_%s'`_$LINE.ldif $LINE ;done ; echo ; find $DXHOME -name "*.zdb" -exec ls -larth {} \; ; echo ; ls -larth --time-style=full-iso /tmp/*.ldif | grep  `date '+%Y-%m-%d'`

# Scroll to see entire line

Step 4a Walks through the possible copy operations from “GOOD” to the “BAD” DATA DSA host, for the <dsaname>.zdb files. The IMPD DATA DSA will require that three (3) of four (4) zdb files are copied, to ensure no impact to referential integrity between the DATA DSA.

The preferred model to copy data from one remote host to another is via the compressed rsync process over SSH, as this is a rapid process for the CA Directory db / zdb files.

https://anapartner.com/2020/05/03/wan-latency-rsync-versus-scp/

Below are the code blocks that demonstrate examples how to copy data from one DSA server to another DSA server.

# RSYNC METHOD
sudo -iu dsa

time rsync --progress -e 'ssh -ax' -avz --exclude "User*" --exclude "*.dp" --exclude "*.tx" dsa@192.168.242.135:./data/ $DXHOME/data

# Scroll to see entire line

# SCP METHOD   
sudo -iu dsa

scp   REMOTE_ID@$HOST:./data/<folder_impd_data_dsa_name>/*.zdb   /tmp/dsa_data
/usr/bin/mv  /tmp/dsa_data/<incorrect_dsaname>.zdb   $DXHOME/data/<folder_impd_data_dsa_name>/<correct_dsaname>.db

# Scroll to see entire line

Step 4b Walk through the final steps before restarting the “BAD” DATA DSA.

The ONLY files that should be in the data folders are <dsaname>.db (binary data file) and <dsaname>.dx (ASCII time-stamp file). Ensure that the copied <prior-hostname-dsaname>.zdb file has been renamed to the correct hostname & extension for <dsaname>.db

Remove the prior ~~<dsaname>.dp (ASCII time-stamp file)~~ { the DATA DSA will auto replace this file with the *.dx file contents } and the <dsaname>.tx (binary data transaction file).

Step 5a Startup the DATA DSA with the command

dxserver start all

If there is any issue with a DATA or ROUTER DSA not starting, then issue the same command with the debug switch (-d)

dxserver -d start <dsaname>

Use the output from the above debug process to address any a) syntax challenges, or b) older PID/LCK files ($DXHOME/pid)

Step 5b Finally, use dxsearch/ldapsearch to query a unit-test of authentication with the primary service ID. Use other unit/use-case tests as needed to confirm data is now synced.

bash
echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd

LDAPTLS_REQCERT=never dxsearch -LLL -H ldaps://`hostname`:20394 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s base -b 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' '(objectClass=*)' | perl -p00e 's/\r?\n //g'

# Scroll to see entire line

LDIF Recovery Processes

The steps above are for recovery via a 100% replacement method, where the assumption is that the “bad” DSA server does NOT have any data worth keeping or wish to be reviewed.

We wish to clarify a process/methodology, where the “peer” Multi-write DSA may be out-of-sync. Still, we are not sure “which” is truly the “good DSA” to select, or perhaps we wished to merge data from multiple DSA before we declare one to be the “good DSA” (with regards to the completeness of data).

Using CA Directory commands, we can join them together to automate snapshots and exports to LDIF files. These LDIF files can then be compared against their peers MW DATA DSA exports or even to themselves at different snapshot export times. As long as we have the LDIF exports, we can recover from any DAR scenario.

Example of using CA Directory dxserver and dxdumpdb commands (STEP 3) with the ldifdelta and dxmodify commands.

The output from ldifdelta may be imported to any remote peer MW DATA DSA server to sync via dxmodify to that hostname, to force a sync for the few objects that may be out-of-sync, e.g. Password Hashes or other.

dxserver status | grep started | grep -v router | awk '{print $1}' | while IFS='' read -r LINE || [ -n "$LINE" ] ; do dxserver onlinebackup "$LINE" ; sleep 10; dxdumpdb -z -f /tmp/`date '+%Y%m%d_%H%M%S_%s'`_$LINE.ldif $LINE ;done ; echo ; find $DXHOME -name "*.zdb" -exec ls -larth {} \; ; echo ; ls -larth --time-style=full-iso /tmp/*.ldif | grep  `date '+%Y-%m-%d'`

ldifdelta -x -S ca-prov-srv-01-impd-co  /tmp/20200819_122820_1597858100_ca-prov-srv-01-impd-co.ldif   /tmp/20200819_123108_1597858268_ca-prov-srv-01-impd-co.ldif  |  perl -p00e 's/\r?\n //g'  >   /tmp/delta_file_ca-prov-srv-01-impd-co.ldif   ; cat /tmp/delta_file_ca-prov-srv-01-impd-co.ldif

echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd
dxmodify -v -c -h`hostname` -p 20391  -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -f /tmp/delta_file_ca-prov-srv-01-impd-co.ldif

# Scroll to see entire line

The below images demonstrate a delta that exists between two (2) time snapshots. The CA Directory tool, ldifdelta, can identify and extract the modified entry to the user object.

The following examples will show how to re-import this delta using dxmodify command to the DATA DSA with no other modifications required to the input LDIF file.

In the testing example below, before any update to an object, let’s capture a snapshot-in-time and the LDIF files for each DATA DSA.