vApp IMPS PAM AD Authentication “Jump Server”

The virtual appliance and standalone deployment of Symantec (CA) Identity Suite allow for redirecting authentication for the J2EE tier application through Symantec SSO or directly to an Active Directory domain, instead of the existing userstore for the solution.

Challenge:

The standalone deployment of Symantec (CA) Identity Suite on MS Windows OS allowed for the mid-tier component to utilize PAM modules to redirect to AD authentication for the Global User.

However this PAM feature does not exist for Provisioning Servers on the virtual appliance.

To be clear, there are no expectations this feature will be introduced in the future roadmap for the solution, as the primary UI will be the web browser.

Review:

Symantec (CA) Identity Suite architecture for virtual appliance versus standalone deployment architecture.

  • The standalone deployment architecture has both MS Windows and Linux components of all tiers.
  • The vApp deployment architecture has primary Linux components and few MS Windows components.
    • The vApp MS Windows components do not include the IMPS (Provisioning Server)

Proposal:

To address this requirement of enabling AD authentication to the vApp Provisioning Server, we will introduce the concept of a “jump server”.

The “jump server” will utilize the standalone deployment of Symantec Identity Provisioning Server on an MS Windows OS. This “jump server” will be deployed as an “alternative server” integrated into the existing vApp Provisioning Directory deployment.

We will select deployment configuration ONLY of the Provisioning Server itself. We do not require the embedded CCS Service.

We will integrate this “jump server” deployment with the existing Symantec Identity solution.

Ensure the imps_datakey encryption seed file is in sync between all components vApp and standalone.

To avoid impacting the existing vApp deployment, we will NOT integrate the “jump server” deployment to the IME. The IME’s Directory XML for the Provisioning Directory will not be updated.

Important Note: The Symantec/CA Directory solution is required as a pre-step.

Summary of deployment steps:

  • Select a MS Window OS workstation (clean or with JCS/CCS Services) that may be part of the MS AD Domain
    • Option 1: [RECOMMENDED & PREFERRED] If using a clean OS, install MS .NetFramework 3.5.1 for the provisioning component.
      • Open cmd as administrator to deploy:  DISM /Online /Enable-Feature /All /FeatureName:NetFx3
    • Option 2: [MED-HIGH RISK] If using “side-deployment” on an existing JCS/CCS server (MS Win OS), we will need to make modifications to this server.
      • Will need to rename the file  C:\Windows\vpd.properties   to avoid conflict with the JCS/CCS component naming convention in this “registry” file. (see below screen shot)
      • Will require a post-install execution of the IMPS pwdmgr tool to address an MS Registry path conflict between the CCS and IMPS components.
  • Ensure all CA Directory hostnames are in DNS or in the MS Windows local host file (C:\Windows\System32\drivers\etc\hosts ) otherwise this “jump server” deployment will fail when it tries to validate all possible directory nodes’ hostnames and build the respective Directory knowledge files.
  • Create a reference file for the new IMPS router dxc file on at least one of the existing vApp Identity Suite Directory Server otherwise this “jump server” deployment will fail due to trust issue when testing connections to other directory nodes’ hostnames.
  • Deploy Symantec/CA Directory (if not already done) – default configurations. Otherwise, you will see this error message
  • Deploy IMPS MS Windows – Only IMPS (no CCS) with Alternative Server Selection Configuration & update to latest CP patches. Note: For “side-deployment” only: If the vpd.properties file was not renamed, then a name collision will occur due to this registry file, if using the JCS/CCS server to side-deploy. It is low risk to change this file, as it is used to prevent deployments of lower release version of components over the prior installed higher release versions of the same component. If there is a concern, all components can be reinstalled as needed. Do not forget to install the latest CP patches to ensure this “jump server” is the same binary level as the vApp solution.
  • Review of additional notes during deployment of “jump server”. Note: For “side-deployment” only: On the page that ask for the Identity Suite Directory connection information, you will see the solution attempt to load env variables that do not exist. Override these value and enter the Directory hostname, Port 20394, and the default bind DN credentials for a Directory userID: eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=im,dc=etadb
  • Deploy IMPM Manager GUI if needed.
  • Post-Deployment – Update IMPM Manager GUI preference to ONLY connect to the new IMPS server on MS Windows. Use the “Enable Failover” checkbox and place the IP/hostname first in the list. Hint: Remove the other IMPS servers from this list or you may add an extra digit to the IMPS servers entries you wish to save, but prevent auto-connectivity to them. Confirm able to authenticate directly through the solution using prior credentials for your service ID: etaadmin or imadmin. This will validate connectivity to the existing vApp Identity Suite solution.
  • On the “jump server” under the Provisioning Server\pam\ADS folder copy the etapam.dll to the IMPS \bin folder. Then copy the etapam_id.conf configuration file to the \pam parent folder. Update the parameters in this file. Set the enable= parameter to yes. Set the domain= to either the MS AD Domain or use the FQDN hostname of the ADS Domain Controller (DC). If we use the FQDN hostname of the DC the “jump server” does NOT have to be made a member of the MS AD Domain. Save the file and restart the “CA Identity Manager – Provisioning Server”
  • Validate PAM functionality in the IMPS etatrans log is enabled. We will see two (2) entries: PAM: Initialization started (same for all use-cases) and PAM: Not enabled or No PAM managed endpoint. We want “PAM: No PAM managed endpoint” – That is an extra feature we could enable, but do not require for the “jump server” scenario.
  • Validate PAM functionality with MS Sysinternals. Ensure that we copied the etapam.dll to the bin folder and that the configuration file is being read.
  • Test authentication using IMPM Manager login as IMPS Manager Global User that has same userID format as AD sAMAccountName. Negative Use-Case testing: Create a new AD domain user that does NOT exist as a Global User and attempt to authentication. Test with etaadmin or other Global User that does NOT have a match AD sAMAcccount name entry. Review the IMPS etatrans logs on the “jump server”

Update the IMPS encrypted seed file imps_datakey as needed.

Note: The MS Win version of IMPS encrypted seed file may be different than the vApp seed.

If this step is skipped, there will be no obvious error message with the exception that a bind has failed for communication to the JCS/CCS services.

After this file is updated, we will need to re-install IMPS service to ensure that all prior encryption passwords are replaced with new passwords using the new seed file. Basically, we need to install the MS Win version of IMPS Server twice, e.g. standard install, change the seed file value, re-install with update all components and updated passwords.

CCS Service conflict with “side-loading” IMPS Service {“side-loading” methodology}

The “side loading “process of deploying the “jump server” IMPS Provisioning Server on the JCS/CCS Server will impact starting of the CCS service. The installation will update the MS Registry with extra branches and updated shared attribute values between the CCS service and IMPS service, e.g. ETAHOME.

This challenge is a strong reason why we may choose the “clean” installation methodology, to avoid this conflict and possible support challenge.

To address this concern, update the new registry values that store the embedded reversible encrypted password for the CCS Service. Use the password reset tool “pwdmgr” and reset the “Connector Server” for both “eta” & “im” domain to the prior stored password. If the imps_datakey file is not in sync between all provisioning servers (& ccs service), then we will see failed bind connections error messages in the logs.

We will now be able to stop/start the JCS service, and see the embedded CCS service stop and start as well.

Example of challenge and error messages if imps_datakey is not updated and in sync.

Use the following command, csfconfig.exe, under the newly deployed IMPS bin folder to view the JCS connectors defined to the solution stack.

C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin show
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:


C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>echo Password01 > c:\imps.pwd


C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin add name=pamjcs host=192.168.242.143 pass=c:\imps.pwd br-add=@ debug=yes port=20411
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:
Created CS object with name = pamjcs

C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin remove name=pamjcs 
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:

We will see both error status when the imps_datakey file is out-of-sync with others. Please ensure the Linux & MS Win versions are in sync.

You may view the file imps_datakey being referenced with the pwdmgr tool:

su - imps
strace -e trace=open,openat  pwdmrg

open(“/opt/CA/IdentityManager/ProvisioningServer/data/tls/keymgmt/imps_datakey”, O_RDONLY) = 5

You wish to monitor what accounts (embedded) are updated with the IMPS pwdmgr tool: su – imps and execute the two commands in a different SSH shell to monitor the pwdmgr.log that was enabled.

eta-env action=set name=eta_pwdmgr_log type=string value=true
tail -f $ETAHOME/bin/pwdmgr.log

Enablement of extra functionality (bypass the no-sync option on Global User password update)

You may wish to keep the Global User and AD password in sync. If they are not, then you will have two passwords that will work for the Global User account. The newer PAM AD authentication credentials, and the older Global User password. The etapam.dll module data path appears to check for PAM AD first, and if it fails, then it will check the Global User eTPassword field as well.

Enable the AD endpoint in the etapam_id.conf file. The type and domain will be as shown, e.g. Active Directory and im (for the vApp). The endpoint-name will be free-form and whatever you may have named your AD endpoint in the IMPS GUI.

Monitor the startup of the PAM module within the IMPS etatrans*.log

Perform a use-case test with changing a Global User account without correlation to an AD endpoint; and then retest with a Global User that is correlated to an AD endpoint. Do both test with NO SYNC operation

If the Global User is already correlated to an AD endpoint account, then we will see a “Child Modify” operation to the correlated AD endpoint account’s Password within the IMPS etatrans*.log.

One “gotcha”. There appears to be a check against the AD password policy. If the new password does not fit the AD password policy, the following error message will appear, “ETA_E_0007 <MGU>, Global user XXXXXXX modification failed: PAM account password updated failed: Account password must match global user password.

The hidden cost of Entropy to your business

On Linux OS, there are two (2) device drivers that provide entropy “noise” for components that require encryption, e.g. the /dev/random and the /dev/urandom device drivers. The /dev/random is a “blocking” device driver. When the “noise” is low, any component that relies on this driver will be “stalled” until enough entropy is returned. We can measure the entropy from a range of 0-4096. Where a value over 1000 is excellent. Any value in the double or single digits will impact the performance of the OS and solutions with delays. The root cause of these delays is not evident during troubleshooting, and typically there are no warning nor error messages related to entropy.

watch -n 1 cat /proc/sys/kernel/random/entropy_avail

The Symantec Identity Suite solution, when deployed on Linux OS is typically deployed with the JVM switch -Djava.security.egd=file:/dev/./urandom for any component that uses Java (Oracle or AdoptOpenJDK), e.g. Wildfly (IM/IG/IP) and IAMCS (JCS). This JVM variable is sufficient for most use-cases to manage the encryption/hash needs of the solution.

However, for any component that does not provide a mechanism to use the alternative of /dev/urandom driver, the Linux OS vendors offer tools such as the “rng-tools” package. We can review what OS RNGD service is available using package tools, e.g.

dnf list installed | grep -i rng

If the Symantec Identity Suite or other solutions are deployed as standalone components, then we may adjust the Linux OS as we need with no restrictions to add the RNGD daemon as we wish. One favorite is the HAVEGED daemon over the default OS RNGD.

See prior notes on value and testing for Entropy on Linux OS (standalone deployments):

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=2197&MID=720771&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

https://community.broadcom.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=7747b411-2e1e-4bc2-8284-9b8856790ef9

Challenge for vApp

The challenge for Virtual Appliances is that we are limited to what functionality the Symantec Product Team provides for us to leverage. The RNGD service was available on the vApp r14.3, but was disabled for OS challenges with 100% utilization with CentOS 6.4. The service is still installed, but the actual binary is non-executable.

https://knowledge.broadcom.com/external/article/97774/ca-identity-suite-low-entropy-on-virtual.html
https://knowledge.broadcom.com/external/article/139759/ca-identity-suite-142-vapp-rngd-proces.html
https://broadcom-stage.adobecqms.net/us/en/symantec-security-software/identity-security/identity-suite/14-3/virtual-appliance/administering-virtual-appliance/using-the-login-shell.html

A new Virtual Appliance patch would be required to re-enable this RNGD on vApp r14.3cp2. We have access via sudo, to /sbin/chkconfig, /sbin/service to re-enable this service, but as the binary is not executable, we cannot progress any further. We can see the alias in the documentation still exist, but the OS alias was removed in the cp2 update.

However, since vApp r14.4 was release, we can focus on this Virtual Appliance which is running Centos 8 stream. The RNGD service here is disabled (masked) but can be re-enabled for our use with the sudo command. There is no current documented method for RNGD on vApp r14.4 at this time, but the steps below will show an approved way using the ‘config’ userID and sudo commands.

Confirm that the “rng-tools” package is installed and that the RNGD binary is executable. We can also see that the RNGD service is “masked”. Masked services are prevented from starting manually or automatically as an extra safety measure when we wish for tighter control over our systems.

If we test OS entropy for this vApp r14.4 server without RNGD, we can monitor how a simple BASH shell script that emulates a password being generated will impact the “entropy” of /dev/random. The below script will reduce the entropy to low numbers. This process will now impact the OS itself and any components that reference /dev/random. We can observe with “lsof /dev/random” that the java programs will still reference /dev/random; even though most activity is going to /dev/urandom.

Using the time command in the BASH shell script, we can see that the response is rapid for the first 20+ iterations, but as soon as the entropy is depleted, each execution is delayed by 10-30x times.

counter=1;MAX=100;while [ $counter -le $MAX ]; do echo "##########  $counter ##########" ; time dd if=/dev/random bs=8 count=1 2> /dev/null | base64; counter=$(( $counter + 1 )); done;

Enable RNGD on vApp r14.4 & Testing

Now let’s see what RNGD service will do for us when it is enabled. Let’s follow the steps below to unmask, enable, and start the RNGD service as the ‘config’ userID. We have access to sudo to the Centos 8 Stream command of /sbin/systemctl.

sudo /usr/bin/systemctl status rngd.service
ls -lart /etc/systemd/system/rngd.service
sudo /usr/bin/systemctl unmask rngd.service
sudo /usr/bin/systemctl enable rngd.service
cat /usr/lib/systemd/system/rngd.service
sudo /usr/bin/systemctl start rngd.service
sudo /usr/bin/systemctl status rngd.service
ps -ef | grep rngd | grep -v grep

After the RNGD service is enabled, test again with the same prior BASH shell script but bump the loops to 1000 or higher. Note using the time command we can see that each loop finishes within a fraction of a second.

counter=1;MAX=1000;while [ $counter -le $MAX ]; do echo "##########  $counter ##########" ; time dd if=/dev/random bs=8 count=1 2> /dev/null | base64; counter=$(( $counter + 1 )); done;

Summary

Aim to keep the solution footprint small and the right-sized to solve the business’ needs. Do not accept the default performance; avoid over-purchasing to scale to your expected growth.

Use the JVM switch wherever there is a java process, e.g. BLC or home-grown ETL (extract-transform-load) processes.

-Djava.security.egd=file:/dev/./urandom

If you suspect a dependence may impact the OS or other processes on /dev/random, then enable the OS RNGD and perform your testing. Monitor with the top command to ensure RNGD service is providing value and not impacting the solution.

Rollback for Active Directory and Mainframe (TSS/ACF2/RACF) Entitlements

One business risk to manage when new business logic is being promoted to production environments is how to plan for a rollback process, where prior state data is restored, especially for an application/endpoint that is critical for a business; and as important to users as their login credentials and access.

In this entry, we showcase how to use CA Directory to snapshot an endpoint on a scheduled basis (daily/hourly) and have the process prepare a rollback delta file for user’s entitlements.

Understanding how queries may be direct to an endpoint/application or via the CA Identity Manager provisioning tier, we can speed up this process rapidly for sites that have millions of identities in an endpoint.

#!/bin/bash
##############################################################################
#
#  POC to demostrate process to snapshot endpoint data on a daily basis
#  and to allow a format for roll back
#
#  1.  Review ADS with dxsearch/dxmodify
#  2.  Create ADS representative Router DSA with CA Directory
#  3.  Create ldif delta of snapshot data
#  4.  Convert 'replace' to 'add' to ensure Roll back process is a 'merge'
#      and NOT an 'overwrite' of entitlements
#
#
#
#  A. Baugher, ANA, 11/2019
#
##############################################################################

##########  Secure password for script ########
FILE=/tmp/.ads.hash.pwd
#rm -rf $FILE $FILE.salt

[[ -f $FILE ]]
echo "Check if $FILE exists:  $?"
[[ -s $FILE ]]
echo "Check if $FILE is populated: $?"

if [[ ! -s $FILE && ! -s $FILE.salt ]]
then
 # File did not have any data
 #  Run script once with pwd then replace with junk data in script
  SALT=$RANDOM$RANDOM$RANDOM
  PWD=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  ENCPWD=$(echo $PWD      | openssl enc -aes-256-cbc -a    -salt -pass pass:$SALT)
  echo  $ENCPWD > $FILE
  echo  $SALT > $FILE.salt
  chmod 600 $FILE $FILE.salt
fi


if [[ -s $FILE && -s $FILE.salt ]]
then
  ENCPWD=`cat  $FILE`
  SALT=`cat $FILE.salt`
  echo  "$PWD and $SALT for $ENCPWD"
  MYPWD=$(echo  "$ENCPWD" | openssl enc -aes-256-cbc -a -d -salt -pass pass:$SALT)
  echo "$PWD and $SALT for $MYPWD"
else
  echo "Missing password encrypted data and salt"
  exit 1
fi

#exit

echo ""
echo "##############################################################################"
echo "Step 0 # Remove prior ads schema files"
echo "##############################################################################"
ADS_SCHEMA=ads_schema
ADS_SUFFIX="dc=exchange,dc=lab"
RANDOM_PORT=50389
rm -rf $DXHOME/config/knowledge/$ADS_SCHEMA.dxc
rm -rf $DXHOME/config/servers/$ADS_SCHEMA.dxi
rm -rf $DXHOME/config/schema/$ADS_SCHEMA.dxc


echo ""
echo "##############################################################################"
echo "Step 1 # Create new router DSA"
echo "##############################################################################"
echo "dxnewdsa -t router $ADS_SCHEMA $RANDOM_PORT $ADS_SUFFIX"
dxnewdsa -t router $ADS_SCHEMA $RANDOM_PORT $ADS_SUFFIX


echo""
echo "##############################################################################"
echo "Step 2 # Create temporary LDIF file of ADS schema"
echo "##############################################################################"
cd $DXHOME/config/schema
ADS_BIND_DN="CN=Administrator,CN=Users,DC=exchange,DC=lab"
ADS_BIND_PWD=$MYPWD
ADS_PASSFILE=/tmp/.ads.pwd
echo -n $MYPWD > $ADS_PASSFILE
chmod 600 $ADS_PASSFILE
ADS_SERVER=dc2016.exchange.lab
ADS_PORT=389
echo "dxschemaldif -v -D $ADS_BIND_DN -w ADS_BIND_PASSWORD_HERE $ADS_SERVER:$ADS_PORT > $ADS_SCHEMA.ldif"
dxschemaldif -v -D $ADS_BIND_DN -w $ADS_BIND_PWD $ADS_SERVER:$ADS_PORT > $ADS_SCHEMA.ldif

echo ""
echo "##############################################################################"
echo "Step 3 # Replace unknown SYNTAX with closely related SYNTAX known by CA Directory r12.6.5"
echo "##############################################################################"
echo  "sed -i 's|1.2.840.113556.1.4.1221|1.3.6.1.4.1.1466.115.121.1.26|g' $ADS_SCHEMA.ldif"
sed -i 's|1.2.840.113556.1.4.1221|1.3.6.1.4.1.1466.115.121.1.26|g' $ADS_SCHEMA.ldif


echo ""
echo "##############################################################################"
echo "Step 4 - # Create CA Directory Schema DXC File from LDIF Schema File"
echo "##############################################################################"
echo "ldif2dxc -f $ADS_SCHEMA.ldif -b bad.ldif -x default.dxg -v $ADS_SCHEMA.dxc"
ldif2dxc -f $ADS_SCHEMA.ldif -b bad.ldif -x default.dxg -v $ADS_SCHEMA.dxc


echo ""
echo "##############################################################################"
echo "Step 5 - # Update router DSA schema reference"
echo "##############################################################################"
echo "sed -i \"s|source \"../schema/default.dxg\";|source \"../schema/default.dxg\";\nsource \"../schema/$ADS_SCHEMA.dxc\"; |g\"  $DXHOME/config                                         /servers/$ADS_SCHEMA.dxi"
sed -i "s|source \"../schema/default.dxg\";|source \"../schema/default.dxg\";\nsource \"../schema/$ADS_SCHEMA.dxc\"; |g"  $DXHOME/config/servers                                         /$ADS_SCHEMA.dxi


echo ""
echo "##############################################################################"
echo "Step 6 - # Query ADS endpoint for snapshot 1 "
echo "##############################################################################"
echo "dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX '(objectClass=User)'  memberOf  > snapshot_1_                                         $ADS_SCHEMA.ldif "
echo "ldifsort snapshot_1_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif "
dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX "(objectClass=User)" memberOf |  perl -p00e 's/\r?\                                         n //g' > snapshot_1_$ADS_SCHEMA.ldif
ldifsort snapshot_1_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif


echo ""
echo "##############################################################################"
echo "Step 7 - # Query ADS endpoint for snapshot 2"
echo "##############################################################################"
echo "dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX '(objectClass=User)'  memberOf  > snapshot_2_                                         $ADS_SCHEMA.ldif "
echo "ldifsort snapshot_2_$ADS_SCHEMA.ldif  snapshot_2_sorted_$ADS_SCHEMA.ldif "
dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX "(objectClass=User)"  memberOf | perl -p00e 's/\r?\                                         n //g'  > snapshot_2_$ADS_SCHEMA.ldif
ldifsort snapshot_2_$ADS_SCHEMA.ldif  snapshot_2_sorted_$ADS_SCHEMA.ldif


echo ""
echo "##############################################################################"
echo "Step 8 - # Find the delta for any removed objects"
echo "##############################################################################"
echo "ldifdelta -x -S $ADS_SCHEMA snapshot_2_sorted_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif"
ldifdelta -x -S $ADS_SCHEMA snapshot_2_sorted_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif

echo ""
echo "##############################################################################"
echo "Step 9a:  Convert from User ldapmodify syntax of 'overwrite' of 'replace' "
echo "##############################################################################"
ldifdelta -S $ADS_SCHEMA snapshot_2_sorted_$ADS_SCHEMA.ldif  snapshot_1_sorted_$ADS_SCHEMA.ldif  user_mod_syntax_input.ldif >/dev/null 2>&1
cat user_mod_syntax_input.ldif | perl -p00e 's/\r?\n //g'  > user_mod_syntax.ldif
cat user_mod_syntax.ldif
echo "##############################################################################"
echo "Step 9b: Convert to ADS Group ldapmodify syntax with a 'merge' of 'add' for the group objects"
echo "##############################################################################"
perl /opt/CA/Directory/dxserver/samples/dxsoak/convert.pl user_mod_syntax.ldif > group_mod_syntax_input.ldif
cat group_mod_syntax_input.ldif  | perl -p00e 's/\r?\n //g' > group_mod_syntax.ldif
cat group_mod_syntax.ldif
echo "##############################################################################"


Example of output from above script:

[dsa@vapp0001]$ ./active_directory_user_delta_via_ca_dir_tools-lab.sh
Check if /tmp/.ads.hash.pwd exists:  0
Check if /tmp/.ads.hash.pwd is populated: 0
/opt/CA/Directory/dxserver/samples/dxsoak and 31936904511291 for U2FsdGVkX195Ti6A8GdFTG6Kmrf6xDcOhrd2aPWVezc=
/opt/CA/Directory/dxserver/samples/dxsoak and 31936904511291 for CAdemo123

20200427150345,505.0Z = Current OS UTC time stamp
##############################################################################
Step 0 # Remove prior ads schema files
##############################################################################

20200427150345,509.0Z = Current OS UTC time stamp
##############################################################################
Step 1 # Create new router DSA
##############################################################################
dxnewdsa -t router ads_schema 50389 dc=exchange,dc=lab
Writing the knowledge file...
knowledge file written
Writing the initialization file...
Initialization file written
Starting the DSA 'ads_schema'...
ads_schema starting

ads_schema started

20200427150345,513.0Z = Current OS UTC time stamp
##############################################################################
Step 2 # Create temporary LDIF file of ADS schema
##############################################################################
dxschemaldif -v -D CN=Administrator,CN=Users,DC=exchange,DC=lab -w ADS_BIND_PASSWORD_HERE dc2016.exchange.lab:389 > ads_schema.ldif
>> Issuing LDAP v3 synchronous bind to 'dc2016.exchange.lab:389'...
>> Fetching root DSE 'subschemaSubentry' attribute...
>> Downloading schema from 'CN=Aggregate,CN=Schema,CN=Configuration,DC=exchange,DC=lab'...
>> Received (4527) values
>> Done.

20200427150345,539.0Z = Current OS UTC time stamp
##############################################################################
Step 3 # Replace unknown SYNTAX with closely related SYNTAX known by CA Directory r12.6.5
##############################################################################
sed -i 's|1.2.840.113556.1.4.1221|1.3.6.1.4.1.1466.115.121.1.26|g' ads_schema.ldif

20200427150345,560.0Z = Current OS UTC time stamp
##############################################################################
Step 4 - # Create CA Directory Schema DXC File from LDIF Schema File
##############################################################################
ldif2dxc -f ads_schema.ldif -b bad.ldif -x default.dxg -v ads_schema.dxc
>> Opening input file 'ads_schema.ldif' ...
>> Opening existing dxserver schema file '/opt/CA/Directory/dxserver/config/schema/default.dxg' ...
>> Opening bad file 'bad.ldif' ...
>> Opening output file '/opt/CA/Directory/dxserver/config/schema/ads_schema.dxc' ...
>> Processing dxserver schema group file '/opt/CA/Directory/dxserver/config/schema/default.dxg'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/x500.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/cosine.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/umich.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/inetop.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/dxserver.dxc'...
>> Loaded (248) existing dxserver schema entries
>> Loading LDIF records...
>> Loading LDIF record number (1)...
>> Skipping attr: 'objectClass'
>> Skipping attr: 'objectClass'
>> Processing loaded LDIF records...
>> Moving objectClasses to end of list...
>> Sorting attrs/objectClasses so parents precede their children...
>> Processing attributeTypes...
>> Defaulting 'directoryString' syntax without any (required) matching rules to 'caseIgnoreString'...

[Remove repeating lines x 1000]

>> Processing objectClasses...
>> Skipping existing schema entry 'top' with oid '2.5.6.0'...
>> Skipping existing schema entry 'locality' with oid '2.5.6.3'...
>> Skipping existing schema entry 'device' with oid '2.5.6.14'...
>> Skipping existing schema entry 'certificationAuthority' with oid '2.5.6.16'...
>> Skipping existing schema entry 'groupOfNames' with oid '2.5.6.9'...
>> Skipping existing schema entry 'organizationalRole' with oid '2.5.6.8'...
>> Skipping existing schema entry 'organizationalUnit' with oid '2.5.6.5'...
>> Skipping existing schema entry 'domain' with oid '1.2.840.113556.1.5.66'...
>> Skipping existing schema entry 'rFC822LocalPart' with oid '0.9.2342.19200300.100.4.14'...
>> Skipping existing schema entry 'applicationProcess' with oid '2.5.6.11'...
>> Skipping existing schema entry 'document' with oid '0.9.2342.19200300.100.4.6'...
>> Skipping existing schema entry 'room' with oid '0.9.2342.19200300.100.4.7'...
>> Skipping existing schema entry 'domainRelatedObject' with oid '0.9.2342.19200300.100.4.17'...
>> Skipping existing schema entry 'country' with oid '2.5.6.2'...
>> Skipping existing schema entry 'friendlyCountry' with oid '0.9.2342.19200300.100.4.18'...
>> Skipping existing schema entry 'groupOfUniqueNames' with oid '2.5.6.17'...
>> Skipping existing schema entry 'organization' with oid '2.5.6.4'...
>> Skipping existing schema entry 'simpleSecurityObject' with oid '0.9.2342.19200300.100.4.19'...
>> Skipping existing schema entry 'person' with oid '2.5.6.6'...
>> Skipping existing schema entry 'organizationalPerson' with oid '2.5.6.7'...
>> Skipping existing schema entry 'inetOrgPerson' with oid '2.16.840.1.113730.3.2.2'...
>> Skipping existing schema entry 'residentialPerson' with oid '2.5.6.10'...
>> Skipping existing schema entry 'applicationEntity' with oid '2.5.6.12'...
>> Skipping existing schema entry 'dSA' with oid '2.5.6.13'...
>> Skipping existing schema entry 'cRLDistributionPoint' with oid '2.5.6.19'...
>> Skipping existing schema entry 'documentSeries' with oid '0.9.2342.19200300.100.4.9'...
>> Skipping existing schema entry 'account' with oid '0.9.2342.19200300.100.4.5'...
>> Converting LDIF records to DXserver schema format...
>> Converted (4398) of (4525) schema records

20200427150345,894.0Z = Current OS UTC time stamp
##############################################################################
Step 5 - # Update router DSA schema reference
##############################################################################
sed -i "s|source "../schema/default.dxg";|source "../schema/default.dxg";\nsource "../schema/ads_schema.dxc"; |g"  /opt/CA/Directory/dxserver/config/servers/ads_schema.dxi

20200427150345,897.0Z = Current OS UTC time stamp
##############################################################################
step 6 - # Update an ADS account with memberOf for testing with initial conditions
##############################################################################
dxmodify -c -H ldap://dc2016.exchange.lab:389 -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd << EOF >/dev/null 2>&1
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab

modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
ldap_modify: Already exists (68)
        additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0


modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
ldap_modify: Already exists (68)
        additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0


adding new entry CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab

modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab


20200427150345,909.0Z = Current OS UTC time stamp
##############################################################################
Step 7 - # Query ADS endpoint for snapshot 1
##############################################################################
dxsearch -LLL -h dc2016.exchange.lab -p 389 -x -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd -b dc=exchange,dc=lab '(&(objectClass=User)(memberOf=*))' memberOf |  perl -p00e 's/\r?\n //g' > snapshot_1_ads_schema.ldif
ldifsort snapshot_1_ads_schema.ldif  snapshot_1_sorted_ads_schema.ldif

creating buckets
creating sort cluster 1 of size 200
sorting 0 records
creating sort cluster 2 of size 200
sorting 200 records
creating sort cluster 3 of size 200
sorting 400 records
3 buckets created

sorting 588 records
588 records sorted, 0 bad records

20200427150345,940.0Z = Current OS UTC time stamp
##############################################################################
Step 8 - # Update an ADS account with memberOf for testing after snapshot
##############################################################################
dxmodify -c -H ldap://dc2016.exchange.lab:389 -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd << EOF
Ignore the error msg:  DSID-031A1254, problem 5003 (WILL_NOT_PERFORM)
This error will occur if a non-existant value is removed from the group's member attribute
##############################################################################

ldap_initialize( ldap://dc2016.exchange.lab:389 )
delete member:
        CN=Test User 001,CN=Users,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modify complete

delete member:
        CN=eeeee,CN=Users,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modify complete

delete member:
        CN=Test User 001,CN=Users,DC=exchange,DC=lab
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
ldap_modify: Server is unwilling to perform (53)
        additional info: 00000561: SvcErr: DSID-031A1254, problem 5003 (WILL_NOT_PERFORM), data 0


delete member:
        CN=alantest,CN=Users,DC=exchange,DC=lab
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modify complete

delete member:
        CN=eeeee,CN=Users,DC=exchange,DC=lab
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
ldap_modify: Server is unwilling to perform (53)
        additional info: 00000561: SvcErr: DSID-031A1254, problem 5003 (WILL_NOT_PERFORM), data 0


add member:
        CN=alantest,CN=Users,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modify complete

deleting entry "CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab"
delete complete

20200427150345,954.0Z = Current OS UTC time stamp
##############################################################################
Step 9 - # Query ADS endpoint for snapshot 2
##############################################################################
dxsearch -LLL -h dc2016.exchange.lab -p 389 -x -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd -b dc=exchange,dc=lab '(&(objectClass=User)(memberOf=*))'  memberOf | perl -p00e 's/\r?\n //g'  > snapshot_2_ads_schema.ldif
ldifsort snapshot_2_ads_schema.ldif  snapshot_2_sorted_ads_schema.ldif

creating buckets
creating sort cluster 1 of size 200
sorting 0 records
creating sort cluster 2 of size 200
sorting 200 records
creating sort cluster 3 of size 200
sorting 400 records
3 buckets created

sorting 587 records
587 records sorted, 0 bad records

20200427150345,985.0Z = Current OS UTC time stamp
##############################################################################
Step 10 - # Find the delta for any removed objects
##############################################################################
ldifdelta -x -S ads_schema snapshot_2_sorted_ads_schema.ldif  snapshot_1_sorted_ads_schema.ldif
dn: CN=eeeee,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-

dn: CN=alantest,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=la
 b
-

dn: CN=Test User 001,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-

dn: CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab
changetype: add
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab


ldifdelta summary:
         587 entries in old file
         588 entries in new file
Produced:
           1 add entry records
           0 delete entry records
           3 modify entry records

20200427150346,070.0Z = Current OS UTC time stamp
##############################################################################
Step 11a:  Convert from User ldapmodify syntax of 'overwrite' of 'replace'
##############################################################################
dn: CN=eeeee,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-

dn: CN=alantest,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
-

dn: CN=Test User 001,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-

dn: CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab
changetype: add
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab


20200427150346,163.0Z = Current OS UTC time stamp
##############################################################################
Step 11b: Convert to ADS Group ldapmodify syntax with a 'merge' of 'add' for the group objects
##############################################################################
dn: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=eeeee,CN=Users,DC=exchange,DC=lab
member: CN=Test User 001,CN=Users,DC=exchange,DC=lab

dn: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=alantest,CN=Users,DC=exchange,DC=lab

# Ignoring Users: [CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab <-> CN=Account Operators,CN=Builtin,DC=exchange,DC=lab] Reason: User NOT present in the latest Snapshot! Cannot add to group.

20200427150346,172.0Z = Current OS UTC time stamp
##############################################################################
Step 11c: Query ADS Group member(s) before Roll back process
##############################################################################
dn: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab

20200427150346,185.0Z = Current OS UTC time stamp
##############################################################################
Step 12: Roll back change to ADS User membershipOf to ADS
##############################################################################
Ignore the false positive warning message of: (ENTRY_EXISTS) - This is the 'merge' process
##############################################################################
dxmodify -c -H ldap://dc2016.exchange.lab:389 -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd -f group_mod_syntax.ldif

modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab

modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab

modifying entry CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
ldap_modify: Already exists (68)
        additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0


modifying entry CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
ldap_modify: Already exists (68)
        additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0




20200427150346,194.0Z = Current OS UTC time stamp
##############################################################################
Step 13: Query ADS Group member after Roll back process
##############################################################################
dn: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=eeeee,CN=Users,DC=exchange,DC=lab
member: CN=Test User 001,CN=Users,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab

dn: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab


Avoid the noise – IMPS etatrans alias/function “tap”

Monitoring use-cases within solutions that use various logs can be onerous when there is “noise” or low-value events in the logs. For provisioning use-cases, we prefer to focus on the “CrUD” use-cases and actions.

The CA/Symantec Identity Manager solution has a mid-tier component, IM Provisioning Server, that captures quite a bit of information useful for monitoring for success/failure. The default Log Level of the primary log file, etatrans*.log, is log level = 7. This log level will capture all possible searches and information of activity within the Provisioning Server’s service and transactions to its connector tier.

We can reduce some of the “noise” of searches/information and focus on the “CRuD” actions of “add/mod/del” by reducing the log level to level = 3.

This help as well to reduce the impact to the disk spaces and roll-over of the etatrans*.log file during bulk tasks or feed tasks.

Challenges:

However, even with log level = 3, we still have some “noise” in the etatrans*.log.

Additional “pain points”, the etatrans*.log file is renamed upon every restart of the IMPS service and during rollover at a size of 1 MB.

Resolution:

To assist with “finding” the current file, and to remove the noise, we have created the following “function/alias” for the IMPS user ID.

  1. Log into the IMPS service ID: sudo su – imps {Ensure you use the “dash” character to ensure the .profile is sourced when you switch IDs}
  2. Edit the .profile file: vi .profile
  3. The current file will only have one line, that sources the primary IMPS environmental information: . /etc/.profile_imps

4. Add the following body after the IMPS environmental profile line

function tap () {
cd $ETAHOME/logs
a=$(ls -rt $ETAHOME/logs | grep etatrans | tail -1)
pwd
echo "Tail current log file with exclusions: "$a
tail -F $a | grep -v -e ":LDAP" -e ":Config" -e "AUDITCONFIG" -e ":EtaServer" -e ":Bind " -e ":Unbind " -e ":Search "
}
export -f tap

This new “function/alias” will cd to the correct folder of logs, then tail the correct etatrans*.log file, and exclude the noise of non-CrUD activity. Using the new alias of “tap” on all provisioning servers, will allow us to isolate any challenges during use-case validation.

5. Exit out of the IMPS user ID account; then re-sudo back into this account, and test the “tap” alias.

6. While using the “tap” alias, exercise use-cases within the IM Provisioning Manager (GUI) and the IM User Console (browser); monitor the “Add/Mod/Deletes”. You will also be able to see the “Child” updates to endpoints and updates to the IMPS notification queue (IME Callback).

Using X11 on Virtual Appliances

In this blog example, we will explore expanding the ability of Virtual Appliances to use X11 programs where possible instead of requiring a 2nd server to host the solutions’ client tools.

We will review how to enable the following client tools: CX (Symantec IM Connector Xpress), WF Designer (Symantec Workflow Designer), Wildfly/JBOSS Management UI, Jxplorer (LDAP Management UI), and Apache Directory Studio (LDAP Management UI). Note there is no java version for the Symantec Identity Manager GUI (32bit VC++ client)

Challenge:

The Symantec Identity Suite Virtual Appliance is locked down from updating libraries as the ‘root’ user and the default login service ID of ‘config’ does not have access to the package installer, e.g. yum.

The Symantec Identity Suite Virtual Appliance like other tools, have an enhancement request process to add new functionality. While we wait for this to be delivered, we can address these gaps ourselves with knowledge of what X11 functionality is required to be enabled with the limited resources we have available to ourselves.

https://community.broadcom.com/participate/ideation-home/viewidea?IdeationKey=34adb887-a1c0-4d59-a977-4b65f4274425

To first identify what drivers may be needed, we can use the Linux OS “strace” command to capture which files are “open” or “openat” and used by the programs.

Example for tracing the files/drivers for Java (Oracle or AdoptOpenJDK) jconsole program:

STRACE

Using “strace -e trace=open,openat /opt/CA/java/bin/policytool” we can see the files that were found ” = 3″ versus those not found “= -1 ENOENT (No such file or directory)”. Some files are required for Centos 8 Stream and others for Centos 6.4

Using this iterative process above will help to identify either the primary file used or a supporting file required to start and use the UI version of the program.

We can use both Java jconsole or policytool to help identify the drivers required. There may be a different of drivers version or additional ones require for the OS of the virtual appliances.

Trace Example with Java Policytool program and compare between Centos 8 Stream (openat) and Centos 6.4 (open)

Now that we have a process to help identify the drivers required, we can walk through the challenges and the value statement.

First challenge for X11 access, is the DISPLAY environment variable must be defined. Otherwise we will see this error message: “No X11 DISPLAY variable

To address this challenge, we could manually set this value every time, but why not use our bash shell to manage this.

DISPLAY OS Variable

Add the below two (2) lines to config’s .bash_profile

DISPLAY=$(who -m | gawk -F'(' '{ print $2}' | gawk -F')' '{print $1}'):0.0;export DISPLAY
env | grep DISPLAY

Our DISPLAY variable will now be updated every time we log into the vApp with the config userID.

We should now see this:

When using the “strace” method, we may see that we have a file on the OS, but it is reporting it as not found due to an internal dependency.

Example for policytool, we can see that the file libXext.so.6 (or link) is not referenced correctly.

If we search the OS of the vApp, we can find this file (and its link) with no issue.

The file libXext.so.6 (or libXext.so.6.4.0) requires a supporting file of libX11.so.6 (libX11.so.6.3.0). As we search for these files, we can now start collecting them from nonVapp OS servers (that do have access to package updates), and make them available to the ‘config’ userID via scp/rsync.

On another server, that has these file (same OS release), find and copy these files.

After we have identified all the required files that are missing from the vApp r14.3 (Centos 6.4) or r14.4 (Centos 8 Stream), we can package them up for the ‘config’ userID and scp/rsync them to the vApp.

Soft Links

Before we use these files, we may need to validate that the soft-links are properly defined. If you have any issues, use strace to help identify the missing soft-link to the real file.

LD_LIBRARY_PATH

LD_LIBRARY_PATH is the OS variable we will use to redirect the libraries files (x86/x64) for the ‘config’ userID. Again edit the config’s .bash_profile and ensure the following lines exist:

#### ANA - Add X11 process & supporting libraries to vApp r14.3 config service ID ####
DISPLAY=$(who -m | gawk -F'(' '{ print $2}' | gawk -F')' '{print $1}'):0.0;export DISPLAY
env | grep -i DISPLAY

LD_LIBRARY_PATH=/tmp/x11_libraries_for_centos6_vapp143/usr/lib64:/tmp/x11_libraries_for_centos6_vapp143/usr/lib;export LD_LIBRARY_PATH
env | grep -i LD_LIBRARY_PATH
#### ANA - Add X11 process & supporting libraries to vApp config service ID ####

OR

#### ANA - Add X11 process & supporting libraries to vApp r14.4 config service ID ####
DISPLAY=$(who -m | gawk -F'(' '{ print $2}' | gawk -F')' '{print $1}'):0.0;export DISPLAY
env | grep -i DISPLAY

LD_LIBRARY_PATH=/tmp/x11_for_centos8_for_vapp144/usr/lib64:/tmp/x11_for_centos8_for_vapp144/usr/lib;export LD_LIBRARY_PATH
env | grep -i LD_LIBRARY_PATH
#### ANA - Add X11 process & supporting libraries to vApp config service ID ####

We should now see the following upon login:

Summary Page of X11 Functionality for vApp r14.3cp2 (Centos 6.4)

Centos 6.4 OS Files required for X11 functionality

/tmp/x11_libraries_for_centos6_vapp143/usr/lib64:
-rwxr-xr-x 1 config config   62176 Jun 18 15:04 libXi.so.6.1.0
-rwxr-xr-x 1 config config   38272 Jun 18 15:04 libXrender.so.1.3.0
-rwxr-xr-x 1 config config   21952 Jun 18 15:04 libXtst.so.6.1.0
-rwxrwxr-x 1 config config   74336 Jun 18 15:04 libXext.so.6.4.0
-rwxr-xr-x 1 config config 1297928 Jun 18 15:04 libX11.so.6.3.0
lrwxrwxrwx 1 config config      14 Jun 18 15:05 libXi.so.6 -> libXi.so.6.1.0
lrwxrwxrwx 1 config config      16 Jun 18 15:03 libXtst.so.6 -> libXtst.so.6.1.0
lrwxrwxrwx 1 config config      19 Jun 18 15:01 libXrender.so.1 -> libXrender.so.1.3.0
lrwxrwxrwx 1 config config      15 Jun 18 14:52 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 14:14 libXext.so.6 -> libXext.so.6.4.0

/tmp/x11_libraries_for_centos6_vapp143/usr/lib:
-rwxr-xr-x 1 config config   59180 Jun 18 15:04 libXi.so.6.1.0
-rwxr-xr-x 1 config config   20044 Jun 18 15:04 libXtst.so.6.1.0
-rwxrwxr-x 1 config config   68588 Jun 18 15:04 libXext.so.6.4.0
-rwxr-xr-x 1 config config 1279168 Jun 18 15:04 libX11.so.6.3.0
lrwxrwxrwx 1 config config      14 Jun 18 15:05 libXi.so.6 -> libXi.so.6.1.0
lrwxrwxrwx 1 config config      16 Jun 18 15:03 libXtst.so.6 -> libXtst.so.6.1.0
lrwxrwxrwx 1 config config      15 Jun 18 14:52 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 14:14 libXext.so.6 -> libXext.so.6.4.0

Summary Page of X11 Functionality for vApp r14.4 (Centos 8 Stream)

Centos 8 Stream’s OS Files required for X11 functionality

/tmp/x11_for_centos8_for_vapp144/usr/lib64:
-rwxrwxr-x 1 config config  170208 Jun 18 17:34 libxcb.so.1.1.0
-rwxr-xr-x 1 config config   49256 Jun 18 17:34 libXrender.so.1.3.0
-rwxr-xr-x 1 config config   29104 Jun 18 17:34 libXtst.so.6.1.0
-rwxr-xr-x 1 config config   80728 Jun 18 17:34 libXext.so.6.4.0
-rwxr-xr-x 1 config config   70720 Jun 18 17:34 libXi.so.6.1.0
-rwxr-xr-x 1 config config 1343952 Jun 18 17:34 libX11.so.6.3.0
-rwxr-xr-x 1 config config   16352 Jun 18 17:34 libXau.so.6.0.0
lrwxrwxrwx 1 config config      15 Jun 18 17:35 libXau.so.6 -> libXau.so.6.0.0
lrwxrwxrwx 1 config config      15 Jun 18 17:33 libxcb.so.1 -> libxcb.so.1.1.0
lrwxrwxrwx 1 config config      14 Jun 18 17:32 libXi.so.6 -> libXi.so.6.1.0
lrwxrwxrwx 1 config config      16 Jun 18 17:31 libXtst.so.6 -> libXtst.so.6.1.0
lrwxrwxrwx 1 config config      19 Jun 18 17:30 libXrender.so.1 -> libXrender.so.1.3.0
lrwxrwxrwx 1 config config      15 Jun 18 17:28 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 17:27 libXext.so.6 -> libXext.so.6.4.0

/tmp/x11_for_centos8_for_vapp144/usr/lib:
-rwxrwxr-x 1 config config  181952 Jun 18 17:34 libxcb.so.1.1.0
-rwxr-xr-x 1 config config   78200 Jun 18 17:34 libXi.so.6.1.0
-rwxr-xr-x 1 config config   87788 Jun 18 17:34 libXext.so.6.4.0
-rwxr-xr-x 1 config config   15700 Jun 18 17:34 libXau.so.6.0.0
-rwxr-xr-x 1 config config 1411660 Jun 18 17:34 libX11.so.6.3.0
lrwxrwxrwx 1 config config      15 Jun 18 17:28 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 17:27 libXext.so.6 -> libXext.so.6.4.0

Final Value Statement – X11 UI on vApp

Jxplorer

Script to add jxplorer on the vApp (if you have internet access to the vApp). This script will maintain the configuration file “connections.txt” where hostname/ports/userDN are stored for Jxplorer.

#!/bin/bash
##############################################
#  Name: add_jxplorer.sh
#  Goal: Add Jxplorer (jar) to vApp r14.4 (Centos 8 Stream) with X11 enabled
#  Ref: http://jxplorer.org/downloads/users.html
# ANA 2021
##############################################
cd
mkdir -p jxplorer;cd jxplorer
find . -type f -not -name 'connections.txt' -delete
curl -OL https://netactuate.dl.sourceforge.net/project/jxplorer/jxplorer/version%203.3.1.2/jxplorer-3.3.1.2-linux-installer.run
chmod 555 jxplorer-3.3.1.2-linux-installer.run; ./jxplorer-3.3.1.2-linux-installer.run --unattendedmodeui minimal --mode unattended
pwd
./jxplorer.sh >/dev/null &
echo "Done"

Wildfly / JBoss CLI X11 UI

Use for managing the standalone-full-ha.xml file via jboss-cli.sh scripts and to update values.

Ensure you have created a management user credential to access the running Wildfly/JBoss release.

config@vapp14401 VAPP-14.4.0 (192.168.2.210):~ > sudo /opt/CA/wildfly-idm/bin/add-user.sh -m -u jboss-admin -p Password01!
Added user 'jboss-admin' to file '/opt/CA/wildfly-idm/standalone/configuration/mgmt-users.properties'
Added user 'jboss-admin' to file '/opt/CA/wildfly-idm/domain/configuration/mgmt-users.properties'
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~ >
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~ > /opt/CA/wildfly-idm/bin/jboss-cli.sh   --connect  --user=jboss-admin  --password=Password01!  --gui

Next Steps

We can use the X11 functionality for the IM Workpoint Designer tool, the Connector Xpress (CX) UI tool, and any other tools, e.g. Symantec Layer7 Management UI (manager.jar)

Side Note:

The IM Workpoint Designer tool and other tools have been removed from the vApp r14.4 IAMSuite samples.

Installed IAMSuite tools only under config service ID, to determine if there is any value. Do not see any X11 client applications under this installed component.

Workpoint Designer

Extract the workpoint designer from the standalone deployment tools to a media folder.

Update the shell script files to be executable, replace the localhost entry for another host alias that will resolve to an IP address that the IM solution with Workpoint is actively listening to. Then run the designer from the virtual appliance.

config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media > unzip CA-IG_WorkpointDesigner.zip  > /dev/null
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media > cd CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin/
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > chmod 555 *.sh
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > cp -r -p ../conf/workpoint-client.properties ../conf/workpoint-client.properties.org
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > sed -i 's|localhost|caim-srv|g' ../conf/workpoint-client.properties
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > ./Designer.sh

Ref: https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-manager/14-4/administrating/workflow/how-to-use-the-workpoint-method/configure-workpoint-administrative-tools.html

Connector Xpress

Connector Xpress only has a MS Windows installer, but we can still use this component on Linux OS. Install the CX UI on MS Windows, then zip up the installed folder with all sub-folders. Copy this compress file over to a media folder for the ‘config’ userID and extract the file.

Review the startup file of “ConnectorXpress.bat” and we will create a version for Linux OS. Copy the last line with the conxp.jar file to a new bash script file. Update the file path from MS Windows format, to Linux OS format.

We can now use CX UI from the vApp.

Restart remote IMPD DATA DSAs after long outage

“DSA is attempting to start after a long outage, perform a recovery procedure before starting”

Challenge:   The IMPD (Identity Manager Provisioning Directory) Data DSAs have been offline for a while, e.g. 7 days+ (> 1 week), and the Symantec/CA Directory solution will, to protect the data, refuse to allow the DATA DSAs to start unless there is manual intervention to prevent the possibility of production data (Live DATA DSAs) being synced with older data (Offline DATA DSAs).

If we were concern, we would follow best practices and remove the offline DATA DSAs’ *.db & *.dp files, and replace the *.db with current copies of the Live DATA DSAs’ *.db files; generate temporary time files of *.dx and allow the time files of *.dp to rebuild themselves upon startup of the offline DATA DSAs.

Example to recover from an outage: https://anapartner.com/2020/08/21/directory-backup-and-restore-dar-scenarios/

However, if we are NOT concern, or the environment is non-production we can avoid the multiple shells, multiple commands to resync by using a combinations of bash shell commands. The proposal below outlines using the Symantec/CA Identity Suite virtual appliance, where both the IMPD and IMPS (Identity Manager Provisioning Server) components reside on the same servers.

Proposal:   Use a single Linux host to send remote commands as a single user ID; sudo to the ‘dsa’ and ‘imps’ service IDs, and issue commands to address the restart process.

Pre-Work:   For the Identity Suite vApp, recommend that .ssh keys be used to avoid using a password for the ‘config’ user IDs on all vApp nodes.

Example to setup .SSH keys for ‘config’ user ID: https://anapartner.com/2020/05/01/avoid-locking-a-userid-in-a-virtual-appliance/

If using .SSH keys, do not forget to use this shortcut to cache the local session: eval `ssh-agent` && ssh-add

Steps:   Issue the following bash commands with the correct IPs or hostnames.  

If possible, wrap the remote commands in a for-loop. The below example uses the local ‘config’ user ID, to ssh to remote servers, then issues a local su to the ‘dsa’ service ID. The ‘dsa’ commands may need to be wrapped as shown below to allow multiple commands to be executed together. We have a quick hostname check, stop all IMPD DATA DSAs, find the time-stamp file that is preventing the startup of the IMPD DATA DSAs and remove it, restart all IMPD DATA DSA, and then move on to the next server with the for-loop. The ‘imps’ commands are similar with a quick hostname check, status check, stop and start process, another status check, then move on to the next server in the for-loop.

for i in {136..141}; do ssh  -t config@192.168.242.$i "su - dsa -c \"hostname;dxserver stop all;pwd;find ./data/ -type f \( -name '*.dp' \) -delete  ;dxserver start all \" "; done

for i in {136..141}; do ssh  -t config@192.168.242.$i "su - imps -c \"hostname;imps status;imps stop;imps start;imps status \" "; done

View of for-loop commands output:

Additional: Process to assist with decision to sync or not sync.

Check if the number of total entries in each individual IMPD DATA DSA match with their peers (Multi-Write groups). Goal: Avoid any deltas > 1% between peers. The IMPD “main”, “co”, “inc” DATA DSA should be 100% in sync. We may see some minor flux in the “notify” DATA DSA, as this is temporary data used by the IMPS server to store data to be sent to the IME via the IME Call Back Process.

If there are any deltas, then we may export the IMPD DATA DSAs to LDIF files and then use the Symantec/CA Directory ldifdelta process to isolate and triage the deltas.

su - dsa    OR [ sudo -iu dsa ]
export HISTIGNORE=' *'             {USE THIS LINE TO FORCE HISTORY TO IGNORE ANY COMMANDS WITH A LEADING SPACE CHARACTER}
 echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd     {USE SPACE CHARACTER IN FRONT TO AVOID HISTORY USAGE}


# NOTIFY BRANCH (TCP 20404) 

for i in {135..140}; do echo "##########  192.168.242.$i IMPD NOTIFY DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20404 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=notify,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done

# INC BRANCH (TCP 20398)

for i in {135..140}; do echo "##########  192.168.242.$i IMPD INC DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20398 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTInclusionContainerName=Inclusions,eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done

# CO BRANCH (TCP 20396)

for i in {135..140}; do echo "##########  192.168.242.$i IMPD CO DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20396 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done

# MAIN BRANCH (TCP 20394)

for i in {135..140}; do echo "##########  192.168.242.$i IMPD MAIN DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20394 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done


NOTIFY DSA is temporary data and will have deltas. This DSA is used for the IME CALL BACK process.

ADS Endpoint Configuration Challenges and Hints

  1. Ensure the hostname entry is a FQDN or alias. It can not be an IP address if MS Exchange is to be managed through this connector, due to conflict with Kerberos authentication and IP addresses. If the object was created with an IP address, it may be changed via Jxplorer for two (2) attributes: eTADSprimaryServer and eTADSServerName.

2. General Information on the ADS Endpoint Logging Tab and where this information is stored. Only two (2) the Destination have value with current deployment, e.g. Text File & System Log (MS Windows Event viewer) for Active Directory (ADS). The “Text File” will output data to two (2) files: jcs\logs\ADS\<endpoint-name>.log and ccs\logs\ADS\<endpoint-name>.log

3. Use the MS Event Viewer on the ADS Domain Controller, or use the MS Event Viewer to remotely view the transactions on the remote ADS DC. Select the event codes of 627,628,4723,4724,4738 to start with. Other codes may be added that are useful. Ref: https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/appendix-l–events-to-monitor

4. Additionally, the User ID may be in one of three (3) formats: UPN (serviceid@exchange.lab), NT ( domain\serviceid ), LDAP DN ( cn=serviceid,ou=people,dc=exchange,dc=lab). We recommend UPN or NT format to allow the embedded API features for MS Exchange powershell management to correctly function. If the ID is to be changed, a password update must be done as well, since the User ID is part of the seed for the encrypted password for the service ID to be stored in CA Directory on the ADS endpoint object.

5. SASL versus TLS authentication checkboxes. We can tested the ADS authentication availability using ldapsearch binary. Ports used by Active Directory for authentication by client tools, https://docs.microsoft.com/en-us/troubleshoot/windows-server/identity/config-firewall-for-ad-domains-and-trusts

Note: SASL is encrypted traffic. If wireshark is used to intercept the traffic, the service ID may be seen during initial authentication, but NOT the password nor the payload data.

Notes on SASL validation for Active Directory. {Pro: No need to worry about TLS certificates rotation on client connections – all TLS is managed by the server}

:: Search ADS / LDAP store what is offered for SASL (use -x for simple connection)
ldapsearch -x -h dc2016.exchange.lab -p 389 -b “” -LLL -s base supportedSASLMechanisms

EXAMPLE OUTPUT

[root@oracle ~]# ldapsearch -x -h dc2016.exchange.lab -p 389 -b “” -LLL -s base supportedSASLMechanisms
dn:
supportedSASLMechanisms: GSSAPI
supportedSASLMechanisms: GSS-SPNEGO
supportedSASLMechanisms: EXTERNAL
supportedSASLMechanisms: DIGEST-MD5

:: On Linux OS, execute rpm -qa to search for SASL installed modules/libraries.
rpm -qa | grep cyrus

EXAMPLE OUTPUT

[root@oracle ~]# rpm -qa | grep cyrus
cyrus-sasl-gssapi-2.1.26-23.el7.x86_64
cyrus-sasl-lib-2.1.26-23.el7.x86_64
cyrus-sasl-md5-2.1.26-23.el7.x86_64

:: On Linux OS, install missing SASL libraries & ldapsearch (ldap-client)
yum -y install cyrus-sasl-md5 cyrus-sasl-gssapi openldap-clients

TESTING DIFFERING AUTHENTICATION MECHANISMS #### (may remove -d9 debug switch to view cleaner results)

TLS

LDAPTLS_REQCERT=never ldapsearch -d9 -LLL -H ldaps://dc2016.exchange.lab:636 -w CAdemo123 -D “CN=Administrator,CN=Users,DC=exchange,DC=lab” -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

Start TLS

LDAPTLS_REQCERT=never ldapsearch -d9 -Z -LLL -H ldap://dc2016.exchange.lab:389 -w CAdemo123 -D “CN=Administrator,CN=Users,DC=exchange,DC=lab” -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

Digest-MD5

ldapsearch -d9 -LLL -H ldap://dc2016.exchange.lab -w CAdemo123 -Y DIGEST-MD5 -U Administrator -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

Kerberos (GSS)

ldapsearch -d9 -LLL -H ldap://dc2016.exchange.lab -w CAdemo123 -Y GSSAPI -U Administrator -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

6. TCP/UDP Ports required for Active Directory Endpoint management per CA Documentation https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-manager/14-4/reference/default-ports-for-ca-identity-manager-and-associated-components.html

SASL appears to connect on TCP 636 briefly, then use TCP 389 extensively. Other ports are 80 (Service), 135 (lsass.exe for home folders), 6405 (lsass.exe). If Kerberos authentication is defined for the service ID, then other ports will be used, e.g. 3268/3269. TCP 4104/4105 are for the legacy CAM/CAFT agents (typically not used any more).

Recommendation: Add these TCP Ports to any Firewall between the IM JCS/CCS Server and the Active Directory Domain Controllers to improve performance and avoid time-out delays.

MS Active Directory References on SASL.

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/989e0748-0953-455d-9d37-d08dfbf3998b

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/a98c1f56-8246-4212-8c4e-d92da1a9563b

Parallel provisioning for Active Directory and MS Exchange mailboxes – Improve Birthright/DayOne Access

One of the challenges that IAM/IAG solutions may have is using single thread processing for select endpoints. For the CA/Symantec Identity Management solution, before IM r14.3cp2, we lived with a single-threaded connector to managed MS Active Directory endpoints.

To address this challenge, we deployed multiple connector servers. We allowed the IM Provisioning Server (IMPS) to use a built-in round-robin approach of load-balancing separate transactions to different connector servers, which would service the same Active Directory endpoints.

The IME may be running as fast as it can with its clustered deployment, but as soon as a task has MS Active Directory, and there is a bottleneck with the CCS Service. We begin to see the IME JMS queue reporting that it is stuck and the IME View Submitted Task reporting “In Progress” for all tasks. If the CCS service is restarted, all IME tasks are then reported as “Failed.”

This is/was the bottleneck for the solution for sites that have MS Active Directory for Birthright/DayOne Access.

We can now avoid this bottleneck. [*** (5/24/2021) – There is an enhancement to CP2 to address im_ccs.exe crashes during peak loads discovered using this testing process. ]

Via the newly delivered enhancement https://community.broadcom.com/participate/ideation-home/viewidea?IdeationKey=7154e15b-085d-469e-bff0-ac588ff6bd5b .

We now have full parallel provisioning to MS Active Directory from a single connector server (JCS/CCS).

The new attribute that regulates this behavior is eTADSMaxConnectionsInPool. This attribute will be applied on every existing ADS endpoint that is currently being managed by the IM Provisioning Server after CP2 is deployed. Note: The default value is 10, but we recommend after much testing, to match the value of the IMPS-> JCS and JCS->CCS to equal 200.

During testing within the IME using Bulk Tasks or the IM BLC, we can see that the CCS-> ADS traffic will reach 20-30 connections if allowed. You may set this attribute to a value of 200 via Jxplorer and/or an ldapmodify/dxmodify script.

echo "############### SET ADS MAX CONNECTIONS IN POOL SIZE ##################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
NAMESPACE=exchange2016
LDAPTLS_REQCERT=never dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD" << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'

To confirm the number of open connections is greater than one (1), we can issue a Bulk IM Task or use a performance tool like CA Directory dxsoak.

In this example, we will show case using CA Directory dxsoak to execute 100 parallel threads to create 100 ADS Accounts with MS Exchange Mailboxes. We will also enclose this script for download for others to review and use.

Performance Lab:

Pre-Steps:

  1. Leverage CA Directory samples’ dxsoak binary (performance testing). You may wish to use CA Directory on an existing IM Provisioning Server (Linux OS) or you may deploy CA Directory (MS Windows version) to the JCS/CCS connector. Examples are provided for both OSes.
  2. Create LDIF files for IM Provisioning Server and/or IM Connector Tier. This file is needed to ‘push’ the solution to-failure. The use of the IME Bulk Task and/or etautil scripts to the IM Provisioning Tier, will not provide the transaction speed we need to break the CCS service if possible.
  3. Within the IM Provisioning Manager enable the ADS Endpoint TXT Logs on the Logging TAB, for all checkboxes.
  4. Monitor the IMPS etatrans* logs, monitor the JCS ADS logs, monitor the CCS ADS logs, monitor the number of CCS-> ADS (LDAP/S – TCP 389/636) threads. [Suggest using MS Sysinternals Process Explorer and select im_ccs.exe & then TCP/IP TAB]
  5. Monitor the MS ADS Domain via MS ADUC (AD Users & Computers UI) and MS Exchange Mailbox (Mailbox UI via Browser)

Execution:

6. Perform a UNIT TEST with dxmodify/ldapmodify to confirm the LDIF file input is correct with the correct suffix.

time dxmodify -H ldap://192.168.242.135:20389 -c -x -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldif

7. Perform the PERFORMANCE TEST with dxsoak binary with the same LDIF file & correct suffix. Rate observed = 23 K ids/hr

./dxsoak -c -l 60 -t 100 -h 192.168.242.135:20389 -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldif

Observations:

8. IMPS etatrans*.log – Count the number of operations per second. Note any RACE and/or data collisions, e.g. ADS accounts deleted prior to add via 100 threads or ADS account created multiple times attempted in different threads.

9. IM CCS ADS <endpoint>.log – Will only have useful data if the ADS Endpoint Logging TAB has been checked for TXT logs.

10. Finally, validate directly in MS Active Domain with the ADUC or similar tool & MS Exchange mailboxes being created/deleted.

11. Count the number of threads from im_ccs.exe to ADS – Suggest using MS Sysinternals Process Explorer tool and/or Powershell to count the number of connections.

MS Powershell Script to count the number of LDAP (TCP 389) connection from im_ccs.exe. [Note: TCP 389 is used more if the ADS Endpoint is setup to use SASL authentication. TCP 636 is used more if the ADS Endpoint is using the older TLS authentication]

$i=1
Do {
cls
(Get-NetTCPConnection -State Established -OwningProcess (Get-Process -name im_ccs).id -RemotePort 389).count
Start-Sleep -s 1
$i++
}
while ($i -le 5)

Direct Performance Testing to JCS/CCS Service

While this testing has limited value, it can offer satisfaction and assistance to troubleshoot any challenges. We can use the prior LDIF files with a slightly different suffix, dc=etasa (instead of dc=eta), to use dxsoak to push the connector tier to failure. This step helped provide memory dumps back to CA/Symantec Engineering teams to help isolate challenges within the parallel processing. CCS Service is only exposed via localhost. If you wish to test the CCS Service remotely, then update the MS Registry key for the CCS service to use the external IP address of the JCS/CCS Server. Rate observed = 25 K ids/hr

Script to generate 100 ADS Accounts with MS Exchange Mailbox Creation

You may wish to review this script and adjust it for your ADS / MS Exchange domains for testing. You can also create a simple LDIF file with password resets or ADS group membership adds. Just remember that the IMPS Service (TCP 20389/20390) uses the suffix dc=eta, and the IM JCS/CCS Services (TCP 20410/20411) & (TCP 20402/20403) use the suffix dc=etasa. Additionally, if using CA Directory dxsoak, only use the non-TLS ports, as this binary is not equipped for using TLS certs.

#!/bin/bash
#######################################################################################################################
# Name:  Generate ADS Feed Files for IM Solution Provisioning/Connector Tiers
#
# Goal:  Validate the new parallel processes from the IM Connector Tier to Active Directory with MS Exchange
#
#
# Generate ADS User LDIF file(s) for use with unit (dxmodify) and performance testing (dxsoak) to:
#  - {Note: dxsoak will only work with non-TLS ports}
#
# IM JCS (20410)  "dc=etasa"    {Ensure MS Windows Firewall allows this port to be exposed}
# IM CCS (20402)  "dc=etasa"    {This port is localhost only, may open to network traffic via registry update}
# IMPS (20389)    "dc=eta"
#
#
# Monitor:  
#
# The IMPS etatrans*.log  {exclude searches}
# The JCS daily log
# The JCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
# The CCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
#
# Execute per the examples provided during run of this file
#
#
# ANA 05/2021
#######################################################################################################################

# Unique Variables for an ADS Domain
NAMESPACE=exchange2016
ADSDOMAIN=exchange.lab
DCDOMAIN="DC=exchange,DC=lab"
OU=People

#######################################################################################################################


MAX=100
start=00001
counter=$start
echo "###############################################################"
echo "###############################################################"
START=`/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`
echo `/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`" = Current OS UTC time stamp"
echo "###############################################################"
FILE1=ads_user_with_exchange_dc_etasa.ldif
FILE2=ads_user_with_exchange_dc_eta.ldif
echo "" > $FILE1
while [ $counter -le $MAX ]
do
    n=$((10000+counter)); n=${n#1}
    tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
   echo "Counter with leading zeros = $n   at time:  $tz"


cat << EOF >> $FILE1
dn:  eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: add
objectClass:  eTADSAccount
eTADSobjectClass:  user
eTADSAccountName:  firstname$n aaalastname$n
eTADSgivenName:  firstname$n
eTADSsn:  aaalastname$n
eTADSdisplayName:  firstname$n aaalastname$n
eTADSuserPrincipalName:  aaatestuser$n@$ADSDOMAIN
eTADSsAMAccountName:  aaatestuser$n
eTPassword:  Password01
eTADSpwdLastSet:  -1
eTSuspended:  0
eTADSuserAccountControl:  0000000512
eTADSDescription:  description $tz
eTADSphysicalDeliveryOfficeName:  office
eTADStelephoneNumber:  111-222-3333
eTADSmail:  aaatestuser$n@$ADSDOMAIN
eTADSwwwHomePage:  web.page.lab
eTADSotherTelephone:  111-222-3333
eTADSurl:  other.web.page.lab
eTADSstreetAddress:  street address line01
eTADSpostOfficeBox:  pobox 111
eTADSl:  city
eTADSst:  state
eTADSpostalCode:  11111
eTADSco:  UNITED STATES
eTADSc:  US
eTADScountryCode:  840
eTADSscriptPath:  loginscript.cmd
eTADSprofilePath:  \profile\path\here
eTADShomePhone:  111-222-3333
eTADSpager:  111-222-3333
eTADSmobile:  111-222-3333
eTADSfacsimileTelephoneNumber:  111-222-3333
eTADSipPhone:  111-222-3333
eTADSinfo:  Notes Here
eTADSotherHomePhone:  111-222-3333
eTADSotherPager:  111-222-3333
eTADSotherMobile:  111-222-3333
eTADSotherFacsimileTelephoneNumber:  111-222-3333
eTADSotherIpPhone:  111-222-3333
eTADStitle:  title
eTADSdepartment:  department
eTADScompany:  company
eTADSmanager:  CN=manager_fn manager_ln,OU=$OU,$DCDOMAIN
eTADSmemberOf:  CN=Backup Operators,CN=Builtin,$DCDOMAIN
eTADSlyncSIPAddressOption: 0000000000
eTADSdisplayNamePrintable: aaatestuser$n
eTADSmailNickname: aaatestuser$n
eTADShomeMDB: (Automatic Mailbox Distribution)
eTADShomeMTA: CN=DC001,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,$DCDOMAIN
eTAccountStatus: A
eTADSmsExchRecipientTypeDetails: 0000000001
eTADSmDBUseDefaults: TRUE
eTADSinitials: A
eTADSaccountExpires: 9223372036854775807

EOF
 counter=$(( $counter + 00001 ))
done


#  Create the delete ADS Process
start=00001
counter=$start
while [ $counter -le $MAX ]
do
    n=$((10000+counter)); n=${n#1}
    tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
   echo "Counter with leading zeros = $n   at time:  $tz"


cat << EOF >> $FILE1
dn:  eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: delete

EOF
 counter=$(( $counter + 00001 ))
done

echo ""
echo "################################### ADS USER OBJECT STATS ################################################################"
echo "Number of add objects: `grep "changetype: add" $FILE1 | wc -l`"
echo "Number of delete objects: `grep "changetype: delete" $FILE1 | wc -l`"
rm -rf $FILE2
cp -r -p $FILE1 $FILE2
sed -i 's|,dc=im,dc=etasa|,dc=im,dc=eta|g' $FILE2
ls -lart $FILE1
ls -lart $FILE2

echo ""
echo "################################### SET ADS MAX CONNECTIONS IN POOL SIZE ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
LDAPTLS_REQCERT=never dxmodify  -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD"  << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL  -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'

echo ""
echo "################################### CCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20402
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the CCS Service to test single thread with dxmodify or ldapmodify"
echo "dxmodify  -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the CCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "

echo ""
echo "################################### JCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20410
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the JCS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify  -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the JCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "


echo ""
echo "################################### IMPS UNIT & PERF TEST ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
echo "Execute this command to the IMPS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify  -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "
echo "Execute this command to the IMPS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $IMPS_HOST:$IMPS_PORT -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "




Address the new bottleneck of MS Exchange / O365 Provisioning.

After parallel provisioning has been introduced with the new im_ccs.exe service, you may noticed that the number of transactions is still being throttled during performance testing.

Out-of-the-box MS Active Directory Global Throttling Policy has the parameter of PowerShellMaxConcurrency set to a default of 18 connection. Any provisioning that uses MS Powershell for MS Exchange and/or MS O365 will be impacted by this default parameter.

To address this bottleneck, we can create a new Throttling Policy and only assign the service ID that will be managing identities, to avoid a global change.

Example: New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 & Set-Mailbox “User Name” -ThrottlingPolicy MaxPowershell

After this change has been made, restart the IM JCS/CCS Services, and retest again with your performance tools. Review the CCS ADS log for # of creations in 60 seconds, and you will be pleasantly surprise at the rate. The logs are the strong confirmation we are looking for.

Performance test (947 ADS accounts w/Exchange mailboxes in 60 seconds, 08:59:54 to  09:00:53) => Rate of 15 ids/second   (or 54 K ids/hr) with updated MaxPowershell = 100 thottlingpolicy.

The last bottleneck appears to be CPU availability to MS Exchange Supporting Services, w3wp.exe, the MS IIS Service. Which appears to be managing MS Powershell connections per its startup string of

" c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangePowerShellAppPool" -v "v4.0" -c "C:\Program Files\Microsoft\Exchange Server\V15\bin\GenericAppPoolConfigWithGCServerEnabledFalse.config" -a \.\pipe\iisipme304c50e-6b42-4b26-83a4-229ee037be5d -h "C:\inetpub\temp\apppools\MSExchangePowerShellAppPool\MSExchangePowerShellAppPool.config" -w "" -m 0"

LDAP MITM Methodology to isolate data challenge

The Symantec (CA/Broadcom) Directory solution provides a mechanism for routing LDAPv3 traffic to other solutions. This routing mechanism allows Symantec Directory to act as a virtual directory service for other directories, e.g., MS Active Directory, SunOne, Novell eDirectory, etc.


The Symantec Identity Suite solution uses the LDAP protocol for its mid-tier and connector-tier components. The Provisioning Server is exposed on TCP 20389/20390, the JCS (Java Connector Server) is exposed on TCP 20410/20411, and the CCS (C++ Connector Server) is exposed on TCP 20402/20403.


We wished to isolate provisioning data challenges within the Symantec Identity Management solution that was not fully viewable using the existing debugging logs & features of the provisioning tier & connector tiers. Using Symantec Directory, we can leverage the routing mechanism to build a MITM (man-in-the-middle) methodology to track all LDAP traffic through the Symantec Identity Manager connector tier.


We focused on the final leg of provisioning and created a process to track the JCS -> CCS LDAP traffic. We wanted to understand what and how the data was being sent from the JCS to the CCS to isolate issues to the CCS service and MS Active Directory. Using the trace level of Symantec Directory, we can capture all LDAP traffic, including binds/queries/add/modify actions.

The below steps showcase how to use Symantec Directory as an approved MITM process for troubleshooting exercises. We found this process more valuable than deploying Wireshark on the JCS/CCS Server and decoding the encrypted traffic for LDAP.

Background:

Symantec Directory documentation on routing. Please note the concept / feature of “set transparent-routing = true;” to avoid schema challenges when routing to other directory/ldap solutions.

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/directory/14-1/ca-directory-concepts/directory-distribution-and-routing.html

MITM Methodology for JCS->CCS Service:

The Symantec Identity Management connector tier may be deployed on MS Windows or Linux OS. If the CCS service is being used, then MS Windows OS is required for this MS Visual C++ component/service. As we are focused on the CCS service, we will introduce the Symantec Directory solution on the same MS Windows OS.

NOTE: We will keep the MITM process contained on a single host, and will not redirect the network traffic beyond the host.

Step 1: Deploy the latest Symantec Directory solution on MS Windows OS. This deployment is a blank slate for the next steps to follow.

Step 2: Copy the folders of schema, limits, and ssld from an existing Symantec Directory deployment of the Symantec Identity Manager solution. Using the existing schema files, references, and certificates will allow us to avoid any challenges during startup of the Router DSA due to the pre-defined provisioning/connector tier configurations. Please note when copying from a Linux OS version of Symantec Directory, we will need to update the path from Linux format to MS Windows format in the SSLD impd.dxc file for “cert-dir” and “ca-file” parameters.

# DXserver/config/ssld/impd.dxc

set ssl = {
cert-dir = "C:\Program Files\CA\Directory\dxserver\config\ssld\personalities"
ca-file = "C:\Program Files\CA\Directory\dxserver\config\ssld\impd_trusted.pem"
cipher = "HIGH:!SSLv2:!EXP:!aNULL:!eNULL"
#protocol = tlsv12
fips = false
};

Step 3: Create a new Router DSA DXI configuration file. This is the primary configuration file for Symantec Directory DSA. It will referenced the schema, knowledge, limits, and certificates for the DSA. Note the parameters for “transparent-routing” to avoid schema challenges with other solutions. Note the trace level used to trace the LDAP traffic in the Symantec Directory Router DSA trace log.

# DXserver/config/servers/admin_router_ccs_30402.dxi

# logging and tracing 
close summary-log; 
close trace-log; 
source "../logging/default.dxc"; 
 
# schema 
clear schema; 
source "../schema/impd.dxg";
 
# access controls 
clear access; 
# source "../access/"; 
 
# ssld
source "../ssld/impd.dxc";

# knowledge 
clear dsas; 
source "../knowledge/admin_router_ccs_group.dxg"; 
 
# operational settings 
source "../settings/default.dxc"; 
 
# service limits 
source "../limits/impd.dxc"; 

# database  - none - transparent router
set transparent-routing=TRUE;

# tunnel through eAdmin server error code and  messages
set route-non-compliant-ldap-error-codes = true;

set trace=ldap,time,stats;
#set trace=dsa,time;

Step 4: Create the three (3) knowledge files. The “group” knowledge file will be used to redirect to the other two (2) knowledge files of the router DSA and the re-direct DSA to the CCS service.

# DXserver/config/knowledge/admin_router_ccs_group.dxg 
# The admin_router_ccs_30402.dxc PORT 30402 
# will be used for the IAMCS (JCS) CCS port override configuration file
# server_ccs.properties via proxyConnectionConfig.proxyServerPort=30402

source "admin_router_ccs_30402.dxc";
source "admin_ccs_server_01.dxc";
# DXserver/config/knowledge/admin_router_ccs_30402.dxc 
# This file is sourced by admin_router_ccs_group.dxg.
 
set dsa admin_router_ccs_30402 =  
{ 
    prefix        = <> 
    dsa-name      = <dc etasa><cn admin_router_ccs_30402> 
    dsa-password  = "secret"
    address       = ipv4 localhost port 30402
    snmp-port     = 22500
    console-port  = 22501
    auth-levels   = clear-password
    dsp-idle-time = 100000 
    trust-flags = allow-check-password, trust-conveyed-originator
    link-flags    = ssl-encryption-remote
};
# DXserver/config/knowledge/admin_ccs_server_01.dxc
# This file is sourced by admin_router_ccs_group.dxg.

set dsa admin_ccs_server_01 =  
{ 
     prefix        = <dc etasa> 
     dsa-name      = <dc etasa><cn admin_ccs_server_01> 
     dsa-password  = "secret"
     address       = ipv4 localhost port 20402
     auth-levels   = clear-password
     dsp-idle-time = 100000
     dsa-flags     = load-share
     trust-flags   = allow-check-password, no-server-credentials, trust-conveyed-originator
     link-flags    = dsp-ldap
     #link-flags    = dsp-ldap, ssl-encryption
     # Note:  ssl will require update to /etc/hosts with:  <IP_Address>  eta_server

};

Step 5: Update the JCS configuration file that contains the TCP port that we will be redirecting to. In this example, we will declare TCP 30402 to be the new port.

#C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\override\server_ccs.properties

ccsWindowsController.ccsScriptPath=C:\\Program Files (x86)\\CA\\Identity Manager\\Connector Server\\ccs\\bin
proxyCCSManager.enabled=true
proxyCCSManager.startupWait=30
proxyConnectionConfig.proxyServerHostname=localhost
#proxyConnectionConfig.proxyServerPort=20402
proxyConnectionConfig.proxyServerPort=30402
proxyConnectionConfig.proxyServerUser=cn=root,dc=etasa
proxyConnectionConfig.proxyServerPassword={AES}pbj27RvWGakDKCr+DhRH4Q==
proxyConnectionConfig.proxyServerUseSsl=false
proxyCCSManager.controller.ref=ccsWindowsController

Overview of all files updated and their relationship to each other.

Validation

Start up the solution in the following order. Ensure that the new Symantec Directory Router DSA is starting with no issue. If there are any syntax issues, isolate them with the command: dxserver -d start DSA_NAME.

Start the Router DSA first, then restart the im_jcs (JCS) service. The im_ccs (CCS) service will be auto-started by the JCS service. Wait one (1) minute, then check that both TCP Ports 20402 (CCS) and 30402 (Router DSA) are both in the LISTEN state. If we do not see these both ports, please stop and restart these services.

May use MS Sysinternals ProcessExplorer to monitor both services and using the TCP/IP tab, to view which ports are being used.

A view of the im_ccs.exe and dxserver.exe services and which TCP ports they are listening on.

Use a 3rd party LDAP client tool, such as Jxplorer to authenticate to both the CCS and the Router DSA ports, with the embedded service ID of “cn=root,dc=etasa”. We should see exactly the SAME data.

Use the IME or IMPS to perform a query to MS Active Directory (or any other endpoint that uses the CCS connector tier). We should now see the “cache” on the CCS service be populated with the endpoint information, and the base DN structure. We can now track all LDAP traffic through the Router DSA MITM process.

View of trace logs

We can monitor when the JCS first binds to the CCS service.

We can monitor when the IMPS via the JCS queries if the CCS is aware of the ADS endpoint.

Finally, we can view when the IMPS service decrypt its stored information on the Active Directory endpoint, and push this information to the CCS cache, to allow communication to MS Active Directory. Using Notepad++ we can tail the trace log.

Please note, this is a secure LDAP/S tunnel from the IMPS -> JCS -> CCS -> MS ADS.

We can now view how this data is pushed via this secure tunnel with the MITM process.

> [88] 
> [88] <-- #1 LDAP MESSAGE messageID 5
> [88] AddRequest
> [88]  entry: eTADSDirectoryName=ads2016,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
> [88]  attributes:
> [88]   type: eTADSobjectCategory
> [88]   value: CN=Domain-DNS,CN=Schema,CN=Configuration,DC=exchange,DC=lab
> [88]   type: eTADSdomainFunctionality
> [88]   value: 7
> [88]   type: eTADSUseSSL
> [88]   value: 3
> [88]   type: eTADSexchangeGroups
> [88]   value: CN=Mailbox Database 0840997559,CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=ExchangeLab,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=lab
> [88]   value: CN=im,CN=Databases,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=ExchangeLab,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=lab
> [88]   type: eTLogWindowsEventSeverity
> [88]   value: FE
> [88]   type: eTAccountResumable
> [88]   value: 1
> [88]   type: eTADSnetBIOS
> [88]   value: EXCHANGE
> [88]   type: eTLogStdoutSeverity
> [88]   value: FE
> [88]   type: eTLog
> [88]   value: 0
> [88]   type: eTLogUnicenterSeverity
> [88]   value: FE
> [88]   type: eTADSlockoutDuration
> [88]   value: -18000000000
> [88]   type: objectClass
> [88]   value: eTADSDirectory
> [88]   type: eTLogETSeverity
> [88]   value: FE
> [88]   type: eTADSmsExchSystemObjectsObjectVersion
> [88]   value: 13240
> [88]   type: eTADSsettings
> [88]   value: 3
> [88]   type: eTADSconfig
> [88]   value: ExpirePwd=0
> [88]   value: HomeDirInheritPermission=0
> [88]   type: eTLogDestination
> [88]   value: F
> [88]   type: eTADSUserContainer
> [88]   value: CN=BuiltIn;CN=Users
> [88]   type: eTADSbackupDirs
> [88]   value: 000;DEFAULT;192.168.242.156;0
> [88]   value: 001;DEFAULT;dc2016.exchange.lab;0
> [88]   value: 002;site1;server1.domain.com;0
> [88]   value: 003;site1;server2.domain.com;0
> [88]   value: 004;site2;server3.domain.com;0
> [88]   value: 005;site2;server4.domain.com;0
> [88]   type: eTADSuseFailover
> [88]   value: 1
> [88]   type: eTLogAuditSeverity
> [88]   value: FE
> [88]   type: eTADS-DefaultContext
> [88]   value: exchange.lab
> [88]   type: eTADSforestFunctionality
> [88]   value: 7
> [88]   type: eTADSAuthDN
> [88]   value: Administrator
> [88]   type: eTADSlyncMaxConnection
> [88]   value: 5
> [88]   type: eTADShomeMTA
> [88]   value: CN=Microsoft MTA,CN=EXCHANGE2016,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=ExchangeLab,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=lab
> [88]   type: eTADSAuthPWD
> [88]   value: CAdemo123
> [88]   type: eTADSexchangelegacyDN
> [88]   value: /o=ExchangeLab/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=EXCHANGE2016/cn=Microsoft Private MDB
> [88]   type: eTLogFileSeverity
> [88]   value: F
> [88]   type: eTADSprimaryServer
> [88]   value: dc2016.exchange.lab
> [88]   type: eTADScontainers
> [88]   value: CN=Builtin,DC=exchange,DC=lab
> [88]   value: CN=Computers,DC=exchange,DC=lab
> [88]   value: OU=Domain Controllers,DC=exchange,DC=lab
> [88]   value: OU=Explore,DC=exchange,DC=lab
> [88]   value: CN=ForeignSecurityPrincipals,DC=exchange,DC=lab
> [88]   value: CN=Keys,DC=exchange,DC=lab
> [88]   value: CN=Managed Service Accounts,DC=exchange,DC=lab
> [88]   value: OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
> [88]   value: OU=o365,DC=exchange,DC=lab
> [88]   value: OU=People,DC=exchange,DC=lab
> [88]   value: CN=Program Data,DC=exchange,DC=lab
> [88]   value: CN=Users,DC=exchange,DC=lab
> [88]   value: DC=ForestDnsZones,DC=exchange,DC=lab
> [88]   value: DC=DomainDnsZones,DC=exchange,DC=lab
> [88]   type: eTADSTimeBoundMembershipsEnabled
> [88]   value: 0
> [88]   type: eTADSexchange
> [88]   value: 1
> [88]   type: eTADSdomainControllerFunctionality
> [88]   value: 7
> [88]   type: eTADSexchangeStores
> [88]   value: CN=EXCHANGE2016,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=ExchangeLab,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=lab
> [88]   value: CN=Mailbox,CN=Transport Configuration,CN=EXCHANGE2016,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=ExchangeLab,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=lab
> [88]   value: CN=Frontend,CN=Transport Configuration,CN=EXCHANGE2016,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=ExchangeLab,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=lab
> [88]   type: eTADSKeepCamCaftFiles
> [88]   value: 0
> [88]   type: eTADSmsExchSchemaVersion
> [88]   value: 15333
> [88]   type: eTADSCamCaftTimeout
> [88]   value: 0000001800
> [88]   type: eTADSMaxConnectionsInPool
> [88]   value: 0000000101
> [88]   type: eTADSPortNum
> [88]   value: 389
> [88]   type: eTADSDCDomain
> [88]   value: DC=exchange,DC=lab
> [88]   type: eTADSServerName
> [88]   value: 192.168.242.156
> [88]   type: eTADSDirectoryName
> [88]   value: ads2016
> [88]   type: eTAccountDeletable
> [88]   value: 1
> [88] controls:
> [88]   controlType: 2.16.840.1.113730.3.4.2
> [88]   non-critical

We can now monitor all traffic and assist with troubleshooting any CCS/MS-ADS challenges.

This same MITM methodology/process may also be used for the IMPS (TCP 20389/2039) and the JCS (TCP 20410/20411) services. We have used this process to capture the IME (JIAM) LDAP traffic to the IMPS Service, to isolate multiple queries for Child Provisioning Roles. Which has been used by the product team to enhance the solution to lower startup durations of the IME in the latest releases.

Binds/queries/add/modification all work with this approach, but we do see an issue with OID for IMPS ADS endpoint “explore process” on ADS OU object. We are reviewing how to address this last challenge that states “critical extension is unavailable” for a LDAP control property of the OU object. The OIDs captured appear to be related to SunOne/Iplanet.

Authenticate to vApp ‘dsa’ user ID via ssh private key

The Symantec (CA) Identity Suite includes the Symantec (CA) Directory. This component is installed under the ‘dsa’ service ID. On the virtual appliance, this ‘dsa’ service ID does not have a password defined, and therefore no login is allowed.

As an enhancement, we would like to add in a SSH private key to allow authentication to the ‘dsa’ service ID from other virtual appliances and desktop usage with various tools, e.g. Putty, MobaXterm, WinSCP, etc. This enhancement will allow for a streamlined process to address out-of-sync Directory DATA DSAs with scp/Rsync copies without intermediate file shares or use of other service IDs.

Challenge:

The virtual appliance of Symantec (CA) Identity Suite r14.3 is built on CentOS 6.4. The OpenSSH services on this OS apparently do not use a private key format that can be used by desktop tools or the PuttyGen (keygen conversion tool). However, the private key may be used between vApp servers if using the FQDN (full qualified domain name). We noted that during testing, that localhost is not allowed due to localhost not defined in the SSHD “AllowedUsers” property file.

On newer virtual appliances vApp r14.4 with CentOS 8 Stream, this challenge does not exist, and we can use the OpenSSH private key, id_rsa, with the desktop tools as-is.

To assist with challenge and streamlining this process we have the following three (2) options:

Option 1: On newer OS, use OpenSSH process

After creating the private key, ./ssh/id_rsa, cat this file out to notepad, and save for use with the desktop tools

Generate this OpenSSH private/public key. The final command will help to validate this private key may be used for server to server communication.

echo y | ssh-keygen -t rsa -b 4096 -N Password02 -C "$USER@$HOSTNAME" -f .ssh/id_rsa ; ls -lart .ssh ; cat .ssh/id_rsa ; cat .ssh/id_rsa.pub >> .ssh/authorized_keys ; chmod 600 .ssh/authorized_keys ; ssh -v -i .ssh/id_rsa $USER@`hostname`

Option 2: Skip the OpenSSH process, use PuttyGen

On any OS (new/old) just use Putty-Gen tool to generate the private key. Update key comment/passphrase. After the private key is created, copy the TEXT “Public Key for pasting into OpenSSH authorized_keys file”. Just like it says, and then you may use the associated private key, id_rsa.ppk, with the desktop tools for the ‘dsa’ service ID.

Option 3: Combination of processes/tools

Important: .ssh/authorized_keys is updated and not overwritten.