Month: June 2021

The hidden cost of Entropy to your business

Posted on by Leave a comment

On Linux OS, there are two (2) device drivers that provide entropy “noise” for components that require encryption, e.g. the /dev/random and the /dev/urandom device drivers. The /dev/random is a “blocking” device driver. When the “noise” is low, any component that relies on this driver will be “stalled” until enough entropy is returned. We can measure the entropy from a range of 0-4096. Where a value over 1000 is excellent. Any value in the double or single digits will impact the performance of the OS and solutions with delays. The root cause of these delays is not evident during troubleshooting, and typically there are no warning nor error messages related to entropy.

watch -n 1 cat /proc/sys/kernel/random/entropy_avail

The Symantec Identity Suite solution, when deployed on Linux OS is typically deployed with the JVM switch -Djava.security.egd=file:/dev/./urandom for any component that uses Java (Oracle or AdoptOpenJDK), e.g. Wildfly (IM/IG/IP) and IAMCS (JCS). This JVM variable is sufficient for most use-cases to manage the encryption/hash needs of the solution.

However, for any component that does not provide a mechanism to use the alternative of /dev/urandom driver, the Linux OS vendors offer tools such as the “rng-tools” package. We can review what OS RNGD service is available using package tools, e.g. 

dnf list installed | grep -i rng

If the Symantec Identity Suite or other solutions are deployed as standalone components, then we may adjust the Linux OS as we need with no restrictions to add the RNGD daemon as we wish. One favorite is the HAVEGED daemon over the default OS RNGD.

See prior notes on value and testing for Entropy on Linux OS (standalone deployments):

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=2197&MID=720771&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

https://community.broadcom.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=7747b411-2e1e-4bc2-8284-9b8856790ef9

Challenge for vApp

The challenge for Virtual Appliances is that we are limited to what functionality the Symantec Product Team provides for us to leverage. The RNGD service was available on the vApp r14.3, but was disabled for OS challenges with 100% utilization with CentOS 6.4. The service is still installed, but the actual binary is non-executable.

https://knowledge.broadcom.com/external/article/97774/ca-identity-suite-low-entropy-on-virtual.html
https://knowledge.broadcom.com/external/article/139759/ca-identity-suite-142-vapp-rngd-proces.html
https://broadcom-stage.adobecqms.net/us/en/symantec-security-software/identity-security/identity-suite/14-3/virtual-appliance/administering-virtual-appliance/using-the-login-shell.html

A new Virtual Appliance patch would be required to re-enable this RNGD on vApp r14.3cp2. We have access via sudo, to /sbin/chkconfig, /sbin/service to re-enable this service, but as the binary is not executable, we cannot progress any further. We can see the alias in the documentation still exist, but the OS alias was removed in the cp2 update.

However, since vApp r14.4 was release, we can focus on this Virtual Appliance which is running Centos 8 stream. The RNGD service here is disabled (masked) but can be re-enabled for our use with the sudo command. There is no current documented method for RNGD on vApp r14.4 at this time, but the steps below will show an approved way using the ‘config’ userID and sudo commands.

Confirm that the “rng-tools” package is installed and that the RNGD binary is executable. We can also see that the RNGD service is “masked”. Masked services are prevented from starting manually or automatically as an extra safety measure when we wish for tighter control over our systems.

If we test OS entropy for this vApp r14.4 server without RNGD, we can monitor how a simple BASH shell script that emulates a password being generated will impact the “entropy” of /dev/random. The below script will reduce the entropy to low numbers. This process will now impact the OS itself and any components that reference /dev/random. We can observe with “lsof /dev/random” that the java programs will still reference /dev/random; even though most activity is going to /dev/urandom.

Using the time command in the BASH shell script, we can see that the response is rapid for the first 20+ iterations, but as soon as the entropy is depleted, each execution is delayed by 10-30x times.

counter=1;MAX=100;while [ $counter -le $MAX ]; do echo "##########  $counter ##########" ; time dd if=/dev/random bs=8 count=1 2> /dev/null | base64; counter=$(( $counter + 1 )); done;

Enable RNGD on vApp r14.4 & Testing

Now let’s see what RNGD service will do for us when it is enabled. Let’s follow the steps below to unmask, enable, and start the RNGD service as the ‘config’ userID. We have access to sudo to the Centos 8 Stream command of /sbin/systemctl.

sudo /usr/bin/systemctl status rngd.service
ls -lart /etc/systemd/system/rngd.service
sudo /usr/bin/systemctl unmask rngd.service
sudo /usr/bin/systemctl enable rngd.service
cat /usr/lib/systemd/system/rngd.service
sudo /usr/bin/systemctl start rngd.service
sudo /usr/bin/systemctl status rngd.service
ps -ef | grep rngd | grep -v grep

After the RNGD service is enabled, test again with the same prior BASH shell script but bump the loops to 1000 or higher. Note using the time command we can see that each loop finishes within a fraction of a second. 

counter=1;MAX=1000;while [ $counter -le $MAX ]; do echo "##########  $counter ##########" ; time dd if=/dev/random bs=8 count=1 2> /dev/null | base64; counter=$(( $counter + 1 )); done;

Summary

Aim to keep the solution footprint small and the right-sized to solve the business’ needs. Do not accept the default performance; avoid over-purchasing to scale to your expected growth.

Use the JVM switch wherever there is a java process, e.g. BLC or home-grown ETL (extract-transform-load) processes.

-Djava.security.egd=file:/dev/./urandom

If you suspect a dependence may impact the OS or other processes on /dev/random, then enable the OS RNGD and perform your testing. Monitor with the top command to ensure RNGD service is providing value and not impacting the solution.

Rollback for Active Directory and Mainframe (TSS/ACF2/RACF) Entitlements

Posted on by Leave a comment

One business risk to manage when new business logic is being promoted to production environments is how to plan for a rollback process, where prior state data is restored, especially for an application/endpoint that is critical for a business; and as important to users as their login credentials and access.

In this entry, we showcase how to use CA Directory to snapshot an endpoint on a scheduled basis (daily/hourly) and have the process prepare a rollback delta file for user’s entitlements.

Understanding how queries may be direct to an endpoint/application or via the CA Identity Manager provisioning tier, we can speed up this process rapidly for sites that have millions of identities in an endpoint.

#!/bin/bash
##############################################################################
#
#  POC to demostrate process to snapshot endpoint data on a daily basis
#  and to allow a format for roll back
#
#  1.  Review ADS with dxsearch/dxmodify
#  2.  Create ADS representative Router DSA with CA Directory
#  3.  Create ldif delta of snapshot data
#  4.  Convert 'replace' to 'add' to ensure Roll back process is a 'merge'
#      and NOT an 'overwrite' of entitlements
#
#
#
#  A. Baugher, ANA, 11/2019
#
##############################################################################

##########  Secure password for script ########
FILE=/tmp/.ads.hash.pwd
#rm -rf $FILE $FILE.salt

[[ -f $FILE ]]
echo "Check if $FILE exists:  $?"
[[ -s $FILE ]]
echo "Check if $FILE is populated: $?"

if [[ ! -s $FILE && ! -s $FILE.salt ]]
then
 # File did not have any data
 #  Run script once with pwd then replace with junk data in script
  SALT=$RANDOM$RANDOM$RANDOM
  PWD=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  ENCPWD=$(echo $PWD      | openssl enc -aes-256-cbc -a    -salt -pass pass:$SALT)
  echo  $ENCPWD > $FILE
  echo  $SALT > $FILE.salt
  chmod 600 $FILE $FILE.salt
fi


if [[ -s $FILE && -s $FILE.salt ]]
then
  ENCPWD=`cat  $FILE`
  SALT=`cat $FILE.salt`
  echo  "$PWD and $SALT for $ENCPWD"
  MYPWD=$(echo  "$ENCPWD" | openssl enc -aes-256-cbc -a -d -salt -pass pass:$SALT)
  echo "$PWD and $SALT for $MYPWD"
else
  echo "Missing password encrypted data and salt"
  exit 1
fi

#exit

echo ""
echo "##############################################################################"
echo "Step 0 # Remove prior ads schema files"
echo "##############################################################################"
ADS_SCHEMA=ads_schema
ADS_SUFFIX="dc=exchange,dc=lab"
RANDOM_PORT=50389
rm -rf $DXHOME/config/knowledge/$ADS_SCHEMA.dxc
rm -rf $DXHOME/config/servers/$ADS_SCHEMA.dxi
rm -rf $DXHOME/config/schema/$ADS_SCHEMA.dxc


echo ""
echo "##############################################################################"
echo "Step 1 # Create new router DSA"
echo "##############################################################################"
echo "dxnewdsa -t router $ADS_SCHEMA $RANDOM_PORT $ADS_SUFFIX"
dxnewdsa -t router $ADS_SCHEMA $RANDOM_PORT $ADS_SUFFIX


echo""
echo "##############################################################################"
echo "Step 2 # Create temporary LDIF file of ADS schema"
echo "##############################################################################"
cd $DXHOME/config/schema
ADS_BIND_DN="CN=Administrator,CN=Users,DC=exchange,DC=lab"
ADS_BIND_PWD=$MYPWD
ADS_PASSFILE=/tmp/.ads.pwd
echo -n $MYPWD > $ADS_PASSFILE
chmod 600 $ADS_PASSFILE
ADS_SERVER=dc2016.exchange.lab
ADS_PORT=389
echo "dxschemaldif -v -D $ADS_BIND_DN -w ADS_BIND_PASSWORD_HERE $ADS_SERVER:$ADS_PORT > $ADS_SCHEMA.ldif"
dxschemaldif -v -D $ADS_BIND_DN -w $ADS_BIND_PWD $ADS_SERVER:$ADS_PORT > $ADS_SCHEMA.ldif

echo ""
echo "##############################################################################"
echo "Step 3 # Replace unknown SYNTAX with closely related SYNTAX known by CA Directory r12.6.5"
echo "##############################################################################"
echo  "sed -i 's|1.2.840.113556.1.4.1221|1.3.6.1.4.1.1466.115.121.1.26|g' $ADS_SCHEMA.ldif"
sed -i 's|1.2.840.113556.1.4.1221|1.3.6.1.4.1.1466.115.121.1.26|g' $ADS_SCHEMA.ldif


echo ""
echo "##############################################################################"
echo "Step 4 - # Create CA Directory Schema DXC File from LDIF Schema File"
echo "##############################################################################"
echo "ldif2dxc -f $ADS_SCHEMA.ldif -b bad.ldif -x default.dxg -v $ADS_SCHEMA.dxc"
ldif2dxc -f $ADS_SCHEMA.ldif -b bad.ldif -x default.dxg -v $ADS_SCHEMA.dxc


echo ""
echo "##############################################################################"
echo "Step 5 - # Update router DSA schema reference"
echo "##############################################################################"
echo "sed -i \"s|source \"../schema/default.dxg\";|source \"../schema/default.dxg\";\nsource \"../schema/$ADS_SCHEMA.dxc\"; |g\"  $DXHOME/config                                         /servers/$ADS_SCHEMA.dxi"
sed -i "s|source \"../schema/default.dxg\";|source \"../schema/default.dxg\";\nsource \"../schema/$ADS_SCHEMA.dxc\"; |g"  $DXHOME/config/servers                                         /$ADS_SCHEMA.dxi


echo ""
echo "##############################################################################"
echo "Step 6 - # Query ADS endpoint for snapshot 1 "
echo "##############################################################################"
echo "dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX '(objectClass=User)'  memberOf  > snapshot_1_                                         $ADS_SCHEMA.ldif "
echo "ldifsort snapshot_1_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif "
dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX "(objectClass=User)" memberOf |  perl -p00e 's/\r?\                                         n //g' > snapshot_1_$ADS_SCHEMA.ldif
ldifsort snapshot_1_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif


echo ""
echo "##############################################################################"
echo "Step 7 - # Query ADS endpoint for snapshot 2"
echo "##############################################################################"
echo "dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX '(objectClass=User)'  memberOf  > snapshot_2_                                         $ADS_SCHEMA.ldif "
echo "ldifsort snapshot_2_$ADS_SCHEMA.ldif  snapshot_2_sorted_$ADS_SCHEMA.ldif "
dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX "(objectClass=User)"  memberOf | perl -p00e 's/\r?\                                         n //g'  > snapshot_2_$ADS_SCHEMA.ldif
ldifsort snapshot_2_$ADS_SCHEMA.ldif  snapshot_2_sorted_$ADS_SCHEMA.ldif


echo ""
echo "##############################################################################"
echo "Step 8 - # Find the delta for any removed objects"
echo "##############################################################################"
echo "ldifdelta -x -S $ADS_SCHEMA snapshot_2_sorted_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif"
ldifdelta -x -S $ADS_SCHEMA snapshot_2_sorted_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif

echo ""
echo "##############################################################################"
echo "Step 9a:  Convert from User ldapmodify syntax of 'overwrite' of 'replace' "
echo "##############################################################################"
ldifdelta -S $ADS_SCHEMA snapshot_2_sorted_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif  user_mod_syntax_input.ldif >/dev/null 2>&1
cat user_mod_syntax_input.ldif | perl -p00e 's/\r?\n //g'  > user_mod_syntax.ldif
cat user_mod_syntax.ldif
echo "##############################################################################"
echo "Step 9b: Convert to ADS Group ldapmodify syntax with a 'merge' of 'add' for the group objects"
echo "##############################################################################"
perl /opt/CA/Directory/dxserver/samples/dxsoak/convert.pl user_mod_syntax.ldif > group_mod_syntax_input.ldif
cat group_mod_syntax_input.ldif  | perl -p00e 's/\r?\n //g' > group_mod_syntax.ldif
cat group_mod_syntax.ldif
echo "##############################################################################"

Example of output from above script:

[dsa@vapp0001]$ ./active_directory_user_delta_via_ca_dir_tools-lab.sh
Check if /tmp/.ads.hash.pwd exists:  0
Check if /tmp/.ads.hash.pwd is populated: 0
/opt/CA/Directory/dxserver/samples/dxsoak and 31936904511291 for U2FsdGVkX195Ti6A8GdFTG6Kmrf6xDcOhrd2aPWVezc=
/opt/CA/Directory/dxserver/samples/dxsoak and 31936904511291 for CAdemo123

20200427150345,505.0Z = Current OS UTC time stamp
##############################################################################
Step 0 # Remove prior ads schema files
##############################################################################

20200427150345,509.0Z = Current OS UTC time stamp
##############################################################################
Step 1 # Create new router DSA
##############################################################################
dxnewdsa -t router ads_schema 50389 dc=exchange,dc=lab
Writing the knowledge file...
knowledge file written
Writing the initialization file...
Initialization file written
Starting the DSA 'ads_schema'...
ads_schema starting

ads_schema started

20200427150345,513.0Z = Current OS UTC time stamp
##############################################################################
Step 2 # Create temporary LDIF file of ADS schema
##############################################################################
dxschemaldif -v -D CN=Administrator,CN=Users,DC=exchange,DC=lab -w ADS_BIND_PASSWORD_HERE dc2016.exchange.lab:389 > ads_schema.ldif
>> Issuing LDAP v3 synchronous bind to 'dc2016.exchange.lab:389'...
>> Fetching root DSE 'subschemaSubentry' attribute...
>> Downloading schema from 'CN=Aggregate,CN=Schema,CN=Configuration,DC=exchange,DC=lab'...
>> Received (4527) values
>> Done.

20200427150345,539.0Z = Current OS UTC time stamp
##############################################################################
Step 3 # Replace unknown SYNTAX with closely related SYNTAX known by CA Directory r12.6.5
##############################################################################
sed -i 's|1.2.840.113556.1.4.1221|1.3.6.1.4.1.1466.115.121.1.26|g' ads_schema.ldif

20200427150345,560.0Z = Current OS UTC time stamp
##############################################################################
Step 4 - # Create CA Directory Schema DXC File from LDIF Schema File
##############################################################################
ldif2dxc -f ads_schema.ldif -b bad.ldif -x default.dxg -v ads_schema.dxc
>> Opening input file 'ads_schema.ldif' ...
>> Opening existing dxserver schema file '/opt/CA/Directory/dxserver/config/schema/default.dxg' ...
>> Opening bad file 'bad.ldif' ...
>> Opening output file '/opt/CA/Directory/dxserver/config/schema/ads_schema.dxc' ...
>> Processing dxserver schema group file '/opt/CA/Directory/dxserver/config/schema/default.dxg'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/x500.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/cosine.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/umich.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/inetop.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/dxserver.dxc'...
>> Loaded (248) existing dxserver schema entries
>> Loading LDIF records...
>> Loading LDIF record number (1)...
>> Skipping attr: 'objectClass'
>> Skipping attr: 'objectClass'
>> Processing loaded LDIF records...
>> Moving objectClasses to end of list...
>> Sorting attrs/objectClasses so parents precede their children...
>> Processing attributeTypes...
>> Defaulting 'directoryString' syntax without any (required) matching rules to 'caseIgnoreString'...

[Remove repeating lines x 1000]

>> Processing objectClasses...
>> Skipping existing schema entry 'top' with oid '2.5.6.0'...
>> Skipping existing schema entry 'locality' with oid '2.5.6.3'...
>> Skipping existing schema entry 'device' with oid '2.5.6.14'...
>> Skipping existing schema entry 'certificationAuthority' with oid '2.5.6.16'...
>> Skipping existing schema entry 'groupOfNames' with oid '2.5.6.9'...
>> Skipping existing schema entry 'organizationalRole' with oid '2.5.6.8'...
>> Skipping existing schema entry 'organizationalUnit' with oid '2.5.6.5'...
>> Skipping existing schema entry 'domain' with oid '1.2.840.113556.1.5.66'...
>> Skipping existing schema entry 'rFC822LocalPart' with oid '0.9.2342.19200300.100.4.14'...
>> Skipping existing schema entry 'applicationProcess' with oid '2.5.6.11'...
>> Skipping existing schema entry 'document' with oid '0.9.2342.19200300.100.4.6'...
>> Skipping existing schema entry 'room' with oid '0.9.2342.19200300.100.4.7'...
>> Skipping existing schema entry 'domainRelatedObject' with oid '0.9.2342.19200300.100.4.17'...
>> Skipping existing schema entry 'country' with oid '2.5.6.2'...
>> Skipping existing schema entry 'friendlyCountry' with oid '0.9.2342.19200300.100.4.18'...
>> Skipping existing schema entry 'groupOfUniqueNames' with oid '2.5.6.17'...
>> Skipping existing schema entry 'organization' with oid '2.5.6.4'...
>> Skipping existing schema entry 'simpleSecurityObject' with oid '0.9.2342.19200300.100.4.19'...
>> Skipping existing schema entry 'person' with oid '2.5.6.6'...
>> Skipping existing schema entry 'organizationalPerson' with oid '2.5.6.7'...
>> Skipping existing schema entry 'inetOrgPerson' with oid '2.16.840.1.113730.3.2.2'...
>> Skipping existing schema entry 'residentialPerson' with oid '2.5.6.10'...
>> Skipping existing schema entry 'applicationEntity' with oid '2.5.6.12'...
>> Skipping existing schema entry 'dSA' with oid '2.5.6.13'...
>> Skipping existing schema entry 'cRLDistributionPoint' with oid '2.5.6.19'...
>> Skipping existing schema entry 'documentSeries' with oid '0.9.2342.19200300.100.4.9'...
>> Skipping existing schema entry 'account' with oid '0.9.2342.19200300.100.4.5'...
>> Converting LDIF records to DXserver schema format...
>> Converted (4398) of (4525) schema records

20200427150345,894.0Z = Current OS UTC time stamp
##############################################################################
Step 5 - # Update router DSA schema reference
##############################################################################
sed -i "s|source "../schema/default.dxg";|source "../schema/default.dxg";\nsource "../schema/ads_schema.dxc"; |g"  /opt/CA/Directory/dxserver/config/servers/ads_schema.dxi

20200427150345,897.0Z = Current OS UTC time stamp
##############################################################################
step 6 - # Update an ADS account with memberOf for testing with initial conditions
##############################################################################
dxmodify -c -H ldap://dc2016.exchange.lab:389 -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd << EOF >/dev/null 2>&1
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab

modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
ldap_modify: Already exists (68)
        additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0


modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
ldap_modify: Already exists (68)
        additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0


adding new entry CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab

modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab


20200427150345,909.0Z = Current OS UTC time stamp
##############################################################################
Step 7 - # Query ADS endpoint for snapshot 1
##############################################################################
dxsearch -LLL -h dc2016.exchange.lab -p 389 -x -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd -b dc=exchange,dc=lab '(&(objectClass=User)(memberOf=*))' memberOf |  perl -p00e 's/\r?\n //g' > snapshot_1_ads_schema.ldif
ldifsort snapshot_1_ads_schema.ldif  snapshot_1_sorted_ads_schema.ldif

creating buckets
creating sort cluster 1 of size 200
sorting 0 records
creating sort cluster 2 of size 200
sorting 200 records
creating sort cluster 3 of size 200
sorting 400 records
3 buckets created

sorting 588 records
588 records sorted, 0 bad records

20200427150345,940.0Z = Current OS UTC time stamp
##############################################################################
Step 8 - # Update an ADS account with memberOf for testing after snapshot
##############################################################################
dxmodify -c -H ldap://dc2016.exchange.lab:389 -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd << EOF
Ignore the error msg:  DSID-031A1254, problem 5003 (WILL_NOT_PERFORM)
This error will occur if a non-existant value is removed from the group's member attribute
##############################################################################

ldap_initialize( ldap://dc2016.exchange.lab:389 )
delete member:
        CN=Test User 001,CN=Users,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modify complete

delete member:
        CN=eeeee,CN=Users,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modify complete

delete member:
        CN=Test User 001,CN=Users,DC=exchange,DC=lab
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
ldap_modify: Server is unwilling to perform (53)
        additional info: 00000561: SvcErr: DSID-031A1254, problem 5003 (WILL_NOT_PERFORM), data 0


delete member:
        CN=alantest,CN=Users,DC=exchange,DC=lab
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modify complete

delete member:
        CN=eeeee,CN=Users,DC=exchange,DC=lab
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
ldap_modify: Server is unwilling to perform (53)
        additional info: 00000561: SvcErr: DSID-031A1254, problem 5003 (WILL_NOT_PERFORM), data 0


add member:
        CN=alantest,CN=Users,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modify complete

deleting entry "CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab"
delete complete

20200427150345,954.0Z = Current OS UTC time stamp
##############################################################################
Step 9 - # Query ADS endpoint for snapshot 2
##############################################################################
dxsearch -LLL -h dc2016.exchange.lab -p 389 -x -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd -b dc=exchange,dc=lab '(&(objectClass=User)(memberOf=*))'  memberOf | perl -p00e 's/\r?\n //g'  > snapshot_2_ads_schema.ldif
ldifsort snapshot_2_ads_schema.ldif  snapshot_2_sorted_ads_schema.ldif

creating buckets
creating sort cluster 1 of size 200
sorting 0 records
creating sort cluster 2 of size 200
sorting 200 records
creating sort cluster 3 of size 200
sorting 400 records
3 buckets created

sorting 587 records
587 records sorted, 0 bad records

20200427150345,985.0Z = Current OS UTC time stamp
##############################################################################
Step 10 - # Find the delta for any removed objects
##############################################################################
ldifdelta -x -S ads_schema snapshot_2_sorted_ads_schema.ldif  snapshot_1_sorted_ads_schema.ldif
dn: CN=eeeee,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-

dn: CN=alantest,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=la
 b
-

dn: CN=Test User 001,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-

dn: CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab
changetype: add
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab


ldifdelta summary:
         587 entries in old file
         588 entries in new file
Produced:
           1 add entry records
           0 delete entry records
           3 modify entry records

20200427150346,070.0Z = Current OS UTC time stamp
##############################################################################
Step 11a:  Convert from User ldapmodify syntax of 'overwrite' of 'replace'
##############################################################################
dn: CN=eeeee,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-

dn: CN=alantest,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
-

dn: CN=Test User 001,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-

dn: CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab
changetype: add
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab


20200427150346,163.0Z = Current OS UTC time stamp
##############################################################################
Step 11b: Convert to ADS Group ldapmodify syntax with a 'merge' of 'add' for the group objects
##############################################################################
dn: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=eeeee,CN=Users,DC=exchange,DC=lab
member: CN=Test User 001,CN=Users,DC=exchange,DC=lab

dn: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=alantest,CN=Users,DC=exchange,DC=lab

# Ignoring Users: [CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab <-> CN=Account Operators,CN=Builtin,DC=exchange,DC=lab] Reason: User NOT present in the latest Snapshot! Cannot add to group.

20200427150346,172.0Z = Current OS UTC time stamp
##############################################################################
Step 11c: Query ADS Group member(s) before Roll back process
##############################################################################
dn: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab

20200427150346,185.0Z = Current OS UTC time stamp
##############################################################################
Step 12: Roll back change to ADS User membershipOf to ADS
##############################################################################
Ignore the false positive warning message of: (ENTRY_EXISTS) - This is the 'merge' process
##############################################################################
dxmodify -c -H ldap://dc2016.exchange.lab:389 -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd -f group_mod_syntax.ldif

modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab

modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab

modifying entry CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
ldap_modify: Already exists (68)
        additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0


modifying entry CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
ldap_modify: Already exists (68)
        additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0




20200427150346,194.0Z = Current OS UTC time stamp
##############################################################################
Step 13: Query ADS Group member after Roll back process
##############################################################################
dn: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=eeeee,CN=Users,DC=exchange,DC=lab
member: CN=Test User 001,CN=Users,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab

Avoid the noise – IMPS etatrans alias/function “tap”

Posted on by Leave a comment

Monitoring use-cases within solutions that use various logs can be onerous when there is “noise” or low-value events in the logs. For provisioning use-cases, we prefer to focus on the “CrUD” use-cases and actions.

The CA/Symantec Identity Manager solution has a mid-tier component, IM Provisioning Server, that captures quite a bit of information useful for monitoring for success/failure. The default Log Level of the primary log file, etatrans*.log, is log level = 7. This log level will capture all possible searches and information of activity within the Provisioning Server’s service and transactions to its connector tier.

We can reduce some of the “noise” of searches/information and focus on the “CRuD” actions of “add/mod/del” by reducing the log level to level = 3.

This help as well to reduce the impact to the disk spaces and roll-over of the etatrans*.log file during bulk tasks or feed tasks.

Challenges:

However, even with log level = 3, we still have some “noise” in the etatrans*.log.

Additional “pain points”, the etatrans*.log file is renamed upon every restart of the IMPS service and during rollover at a size of 1 MB.

Resolution:

To assist with “finding” the current file, and to remove the noise, we have created the following “function/alias” for the IMPS user ID.

  1. Log into the IMPS service ID: sudo su – imps {Ensure you use the “dash” character to ensure the .profile is sourced when you switch IDs}
  2. Edit the .profile file: vi .profile
  3. The current file will only have one line, that sources the primary IMPS environmental information: . /etc/.profile_imps

4. Add the following body after the IMPS environmental profile line

function tap () {
cd $ETAHOME/logs
a=$(ls -rt $ETAHOME/logs | grep etatrans | tail -1)
pwd
echo "Tail current log file with exclusions: "$a
tail -F $a | grep -v -e ":LDAP" -e ":Config" -e "AUDITCONFIG" -e ":EtaServer" -e ":Bind " -e ":Unbind " -e ":Search "
}
export -f tap

This new “function/alias” will cd to the correct folder of logs, then tail the correct etatrans*.log file, and exclude the noise of non-CrUD activity. Using the new alias of “tap” on all provisioning servers, will allow us to isolate any challenges during use-case validation.

5. Exit out of the IMPS user ID account; then re-sudo back into this account, and test the “tap” alias.

6. While using the “tap” alias, exercise use-cases within the IM Provisioning Manager (GUI) and the IM User Console (browser); monitor the “Add/Mod/Deletes”. You will also be able to see the “Child” updates to endpoints and updates to the IMPS notification queue (IME Callback).

Using X11 on Virtual Appliances

Posted on by Leave a comment

In this blog example, we will explore expanding the ability of Virtual Appliances to use X11 programs where possible instead of requiring a 2nd server to host the solutions’ client tools.

We will review how to enable the following client tools: CX (Symantec IM Connector Xpress), WF Designer (Symantec Workflow Designer), Wildfly/JBOSS Management UI, Jxplorer (LDAP Management UI), and Apache Directory Studio (LDAP Management UI). Note there is no java version for the Symantec Identity Manager GUI (32bit VC++ client)

Challenge:

The Symantec Identity Suite Virtual Appliance is locked down from updating libraries as the ‘root’ user and the default login service ID of ‘config’ does not have access to the package installer, e.g. yum.

The Symantec Identity Suite Virtual Appliance like other tools, have an enhancement request process to add new functionality. While we wait for this to be delivered, we can address these gaps ourselves with knowledge of what X11 functionality is required to be enabled with the limited resources we have available to ourselves.

https://community.broadcom.com/participate/ideation-home/viewidea?IdeationKey=34adb887-a1c0-4d59-a977-4b65f4274425

To first identify what drivers may be needed, we can use the Linux OS “strace” command to capture which files are “open” or “openat” and used by the programs.

Example for tracing the files/drivers for Java (Oracle or AdoptOpenJDK) jconsole program:

STRACE

Using “strace -e trace=open,openat /opt/CA/java/bin/policytool” we can see the files that were found ” = 3″ versus those not found “= -1 ENOENT (No such file or directory)”. Some files are required for Centos 8 Stream and others for Centos 6.4

Using this iterative process above will help to identify either the primary file used or a supporting file required to start and use the UI version of the program.

We can use both Java jconsole or policytool to help identify the drivers required. There may be a different of drivers version or additional ones require for the OS of the virtual appliances.

Trace Example with Java Policytool program and compare between Centos 8 Stream (openat) and Centos 6.4 (open)

Now that we have a process to help identify the drivers required, we can walk through the challenges and the value statement.

First challenge for X11 access, is the DISPLAY environment variable must be defined. Otherwise we will see this error message: “No X11 DISPLAY variable“

To address this challenge, we could manually set this value every time, but why not use our bash shell to manage this.

DISPLAY OS Variable

Add the below two (2) lines to config’s .bash_profile

DISPLAY=$(who -m | gawk -F'(' '{ print $2}' | gawk -F')' '{print $1}'):0.0;export DISPLAY
env | grep DISPLAY

Our DISPLAY variable will now be updated every time we log into the vApp with the config userID.

We should now see this:

When using the “strace” method, we may see that we have a file on the OS, but it is reporting it as not found due to an internal dependency.

Example for policytool, we can see that the file libXext.so.6 (or link) is not referenced correctly.

If we search the OS of the vApp, we can find this file (and its link) with no issue.

The file libXext.so.6 (or libXext.so.6.4.0) requires a supporting file of libX11.so.6 (libX11.so.6.3.0). As we search for these files, we can now start collecting them from nonVapp OS servers (that do have access to package updates), and make them available to the ‘config’ userID via scp/rsync.

On another server, that has these file (same OS release), find and copy these files.

After we have identified all the required files that are missing from the vApp r14.3 (Centos 6.4) or r14.4 (Centos 8 Stream), we can package them up for the ‘config’ userID and scp/rsync them to the vApp.

Soft Links

Before we use these files, we may need to validate that the soft-links are properly defined. If you have any issues, use strace to help identify the missing soft-link to the real file.

LD_LIBRARY_PATH

LD_LIBRARY_PATH is the OS variable we will use to redirect the libraries files (x86/x64) for the ‘config’ userID. Again edit the config’s .bash_profile and ensure the following lines exist:

#### ANA - Add X11 process & supporting libraries to vApp r14.3 config service ID ####
DISPLAY=$(who -m | gawk -F'(' '{ print $2}' | gawk -F')' '{print $1}'):0.0;export DISPLAY
env | grep -i DISPLAY

LD_LIBRARY_PATH=/tmp/x11_libraries_for_centos6_vapp143/usr/lib64:/tmp/x11_libraries_for_centos6_vapp143/usr/lib;export LD_LIBRARY_PATH
env | grep -i LD_LIBRARY_PATH
#### ANA - Add X11 process & supporting libraries to vApp config service ID ####

OR

#### ANA - Add X11 process & supporting libraries to vApp r14.4 config service ID ####
DISPLAY=$(who -m | gawk -F'(' '{ print $2}' | gawk -F')' '{print $1}'):0.0;export DISPLAY
env | grep -i DISPLAY

LD_LIBRARY_PATH=/tmp/x11_for_centos8_for_vapp144/usr/lib64:/tmp/x11_for_centos8_for_vapp144/usr/lib;export LD_LIBRARY_PATH
env | grep -i LD_LIBRARY_PATH
#### ANA - Add X11 process & supporting libraries to vApp config service ID ####

We should now see the following upon login:

Summary Page of X11 Functionality for vApp r14.3cp2 (Centos 6.4)

Centos 6.4 OS Files required for X11 functionality 

/tmp/x11_libraries_for_centos6_vapp143/usr/lib64:
-rwxr-xr-x 1 config config   62176 Jun 18 15:04 libXi.so.6.1.0
-rwxr-xr-x 1 config config   38272 Jun 18 15:04 libXrender.so.1.3.0
-rwxr-xr-x 1 config config   21952 Jun 18 15:04 libXtst.so.6.1.0
-rwxrwxr-x 1 config config   74336 Jun 18 15:04 libXext.so.6.4.0
-rwxr-xr-x 1 config config 1297928 Jun 18 15:04 libX11.so.6.3.0
lrwxrwxrwx 1 config config      14 Jun 18 15:05 libXi.so.6 -> libXi.so.6.1.0
lrwxrwxrwx 1 config config      16 Jun 18 15:03 libXtst.so.6 -> libXtst.so.6.1.0
lrwxrwxrwx 1 config config      19 Jun 18 15:01 libXrender.so.1 -> libXrender.so.1.3.0
lrwxrwxrwx 1 config config      15 Jun 18 14:52 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 14:14 libXext.so.6 -> libXext.so.6.4.0

/tmp/x11_libraries_for_centos6_vapp143/usr/lib:
-rwxr-xr-x 1 config config   59180 Jun 18 15:04 libXi.so.6.1.0
-rwxr-xr-x 1 config config   20044 Jun 18 15:04 libXtst.so.6.1.0
-rwxrwxr-x 1 config config   68588 Jun 18 15:04 libXext.so.6.4.0
-rwxr-xr-x 1 config config 1279168 Jun 18 15:04 libX11.so.6.3.0
lrwxrwxrwx 1 config config      14 Jun 18 15:05 libXi.so.6 -> libXi.so.6.1.0
lrwxrwxrwx 1 config config      16 Jun 18 15:03 libXtst.so.6 -> libXtst.so.6.1.0
lrwxrwxrwx 1 config config      15 Jun 18 14:52 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 14:14 libXext.so.6 -> libXext.so.6.4.0

Summary Page of X11 Functionality for vApp r14.4 (Centos 8 Stream)

Centos 8 Stream’s OS Files required for X11 functionality 

/tmp/x11_for_centos8_for_vapp144/usr/lib64:
-rwxrwxr-x 1 config config  170208 Jun 18 17:34 libxcb.so.1.1.0
-rwxr-xr-x 1 config config   49256 Jun 18 17:34 libXrender.so.1.3.0
-rwxr-xr-x 1 config config   29104 Jun 18 17:34 libXtst.so.6.1.0
-rwxr-xr-x 1 config config   80728 Jun 18 17:34 libXext.so.6.4.0
-rwxr-xr-x 1 config config   70720 Jun 18 17:34 libXi.so.6.1.0
-rwxr-xr-x 1 config config 1343952 Jun 18 17:34 libX11.so.6.3.0
-rwxr-xr-x 1 config config   16352 Jun 18 17:34 libXau.so.6.0.0
lrwxrwxrwx 1 config config      15 Jun 18 17:35 libXau.so.6 -> libXau.so.6.0.0
lrwxrwxrwx 1 config config      15 Jun 18 17:33 libxcb.so.1 -> libxcb.so.1.1.0
lrwxrwxrwx 1 config config      14 Jun 18 17:32 libXi.so.6 -> libXi.so.6.1.0
lrwxrwxrwx 1 config config      16 Jun 18 17:31 libXtst.so.6 -> libXtst.so.6.1.0
lrwxrwxrwx 1 config config      19 Jun 18 17:30 libXrender.so.1 -> libXrender.so.1.3.0
lrwxrwxrwx 1 config config      15 Jun 18 17:28 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 17:27 libXext.so.6 -> libXext.so.6.4.0

/tmp/x11_for_centos8_for_vapp144/usr/lib:
-rwxrwxr-x 1 config config  181952 Jun 18 17:34 libxcb.so.1.1.0
-rwxr-xr-x 1 config config   78200 Jun 18 17:34 libXi.so.6.1.0
-rwxr-xr-x 1 config config   87788 Jun 18 17:34 libXext.so.6.4.0
-rwxr-xr-x 1 config config   15700 Jun 18 17:34 libXau.so.6.0.0
-rwxr-xr-x 1 config config 1411660 Jun 18 17:34 libX11.so.6.3.0
lrwxrwxrwx 1 config config      15 Jun 18 17:28 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 17:27 libXext.so.6 -> libXext.so.6.4.0

Final Value Statement – X11 UI on vApp

Jxplorer

Script to add jxplorer on the vApp (if you have internet access to the vApp). This script will maintain the configuration file “connections.txt” where hostname/ports/userDN are stored for Jxplorer.

#!/bin/bash
##############################################
#  Name: add_jxplorer.sh
#  Goal: Add Jxplorer (jar) to vApp r14.4 (Centos 8 Stream) with X11 enabled
#  Ref: http://jxplorer.org/downloads/users.html
# ANA 2021
##############################################
cd
mkdir -p jxplorer;cd jxplorer
find . -type f -not -name 'connections.txt' -delete
curl -OL https://netactuate.dl.sourceforge.net/project/jxplorer/jxplorer/version%203.3.1.2/jxplorer-3.3.1.2-linux-installer.run
chmod 555 jxplorer-3.3.1.2-linux-installer.run; ./jxplorer-3.3.1.2-linux-installer.run --unattendedmodeui minimal --mode unattended
pwd
./jxplorer.sh >/dev/null &
echo "Done"

Wildfly / JBoss CLI X11 UI

Use for managing the standalone-full-ha.xml file via jboss-cli.sh scripts and to update values.

Ensure you have created a management user credential to access the running Wildfly/JBoss release.

config@vapp14401 VAPP-14.4.0 (192.168.2.210):~ > sudo /opt/CA/wildfly-idm/bin/add-user.sh -m -u jboss-admin -p Password01!
Added user 'jboss-admin' to file '/opt/CA/wildfly-idm/standalone/configuration/mgmt-users.properties'
Added user 'jboss-admin' to file '/opt/CA/wildfly-idm/domain/configuration/mgmt-users.properties'
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~ >
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~ > /opt/CA/wildfly-idm/bin/jboss-cli.sh   --connect  --user=jboss-admin  --password=Password01!  --gui

Next Steps

We can use the X11 functionality for the IM Workpoint Designer tool, the Connector Xpress (CX) UI tool, and any other tools, e.g. Symantec Layer7 Management UI (manager.jar)

Side Note:

The IM Workpoint Designer tool and other tools have been removed from the vApp r14.4 IAMSuite samples.

Installed IAMSuite tools only under config service ID, to determine if there is any value. Do not see any X11 client applications under this installed component.

Workpoint Designer

Extract the workpoint designer from the standalone deployment tools to a media folder.

Update the shell script files to be executable, replace the localhost entry for another host alias that will resolve to an IP address that the IM solution with Workpoint is actively listening to. Then run the designer from the virtual appliance.

config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media > unzip CA-IG_WorkpointDesigner.zip  > /dev/null
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media > cd CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin/
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > chmod 555 *.sh
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > cp -r -p ../conf/workpoint-client.properties ../conf/workpoint-client.properties.org
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > sed -i 's|localhost|caim-srv|g' ../conf/workpoint-client.properties
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > ./Designer.sh

Ref: https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-manager/14-4/administrating/workflow/how-to-use-the-workpoint-method/configure-workpoint-administrative-tools.html

Connector Xpress

Connector Xpress only has a MS Windows installer, but we can still use this component on Linux OS. Install the CX UI on MS Windows, then zip up the installed folder with all sub-folders. Copy this compress file over to a media folder for the ‘config’ userID and extract the file.

Review the startup file of “ConnectorXpress.bat” and we will create a version for Linux OS. Copy the last line with the conxp.jar file to a new bash script file. Update the file path from MS Windows format, to Linux OS format.

We can now use CX UI from the vApp.

Restart remote IMPD DATA DSAs after long outage

Posted on by Leave a comment

“DSA is attempting to start after a long outage, perform a recovery procedure before starting”

Challenge:   The IMPD (Identity Manager Provisioning Directory) Data DSAs have been offline for a while, e.g. 7 days+ (> 1 week), and the Symantec/CA Directory solution will, to protect the data, refuse to allow the DATA DSAs to start unless there is manual intervention to prevent the possibility of production data (Live DATA DSAs) being synced with older data (Offline DATA DSAs).

If we were concern, we would follow best practices and remove the offline DATA DSAs’ *.db & *.dp files, and replace the *.db with current copies of the Live DATA DSAs’ *.db files; generate temporary time files of *.dx and allow the time files of *.dp to rebuild themselves upon startup of the offline DATA DSAs.

Example to recover from an outage: https://anapartner.com/2020/08/21/directory-backup-and-restore-dar-scenarios/

However, if we are NOT concern, or the environment is non-production we can avoid the multiple shells, multiple commands to resync by using a combinations of bash shell commands. The proposal below outlines using the Symantec/CA Identity Suite virtual appliance, where both the IMPD and IMPS (Identity Manager Provisioning Server) components reside on the same servers.

Proposal:   Use a single Linux host to send remote commands as a single user ID; sudo to the ‘dsa’ and ‘imps’ service IDs, and issue commands to address the restart process.

Pre-Work:   For the Identity Suite vApp, recommend that .ssh keys be used to avoid using a password for the ‘config’ user IDs on all vApp nodes.

Example to setup .SSH keys for ‘config’ user ID: https://anapartner.com/2020/05/01/avoid-locking-a-userid-in-a-virtual-appliance/

If using .SSH keys, do not forget to use this shortcut to cache the local session: eval `ssh-agent` && ssh-add

Steps:   Issue the following bash commands with the correct IPs or hostnames.  

If possible, wrap the remote commands in a for-loop. The below example uses the local ‘config’ user ID, to ssh to remote servers, then issues a local su to the ‘dsa’ service ID. The ‘dsa’ commands may need to be wrapped as shown below to allow multiple commands to be executed together. We have a quick hostname check, stop all IMPD DATA DSAs, find the time-stamp file that is preventing the startup of the IMPD DATA DSAs and remove it, restart all IMPD DATA DSA, and then move on to the next server with the for-loop. The ‘imps’ commands are similar with a quick hostname check, status check, stop and start process, another status check, then move on to the next server in the for-loop.

for i in {136..141}; do ssh  -t config@192.168.242.$i "su - dsa -c \"hostname;dxserver stop all;pwd;find ./data/ -type f \( -name '*.dp' \) -delete  ;dxserver start all \" "; done

for i in {136..141}; do ssh  -t config@192.168.242.$i "su - imps -c \"hostname;imps status;imps stop;imps start;imps status \" "; done

View of for-loop commands output:

Additional: Process to assist with decision to sync or not sync.

Check if the number of total entries in each individual IMPD DATA DSA match with their peers (Multi-Write groups). Goal: Avoid any deltas > 1% between peers. The IMPD “main”, “co”, “inc” DATA DSA should be 100% in sync. We may see some minor flux in the “notify” DATA DSA, as this is temporary data used by the IMPS server to store data to be sent to the IME via the IME Call Back Process.

If there are any deltas, then we may export the IMPD DATA DSAs to LDIF files and then use the Symantec/CA Directory ldifdelta process to isolate and triage the deltas.

su - dsa    OR [ sudo -iu dsa ]
export HISTIGNORE=' *'             {USE THIS LINE TO FORCE HISTORY TO IGNORE ANY COMMANDS WITH A LEADING SPACE CHARACTER}
 echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd     {USE SPACE CHARACTER IN FRONT TO AVOID HISTORY USAGE}


# NOTIFY BRANCH (TCP 20404) 

for i in {135..140}; do echo "##########  192.168.242.$i IMPD NOTIFY DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20404 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=notify,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done

# INC BRANCH (TCP 20398)

for i in {135..140}; do echo "##########  192.168.242.$i IMPD INC DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20398 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTInclusionContainerName=Inclusions,eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done

# CO BRANCH (TCP 20396)

for i in {135..140}; do echo "##########  192.168.242.$i IMPD CO DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20396 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done

# MAIN BRANCH (TCP 20394)

for i in {135..140}; do echo "##########  192.168.242.$i IMPD MAIN DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20394 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done
NOTIFY DSA is temporary data and will have deltas. This DSA is used for the IME CALL BACK process.