vApp IMPS PAM AD Authentication “Jump Server”

The virtual appliance and standalone deployment of Symantec (CA) Identity Suite allow for redirecting authentication for the J2EE tier application through Symantec SSO or directly to an Active Directory domain, instead of the existing userstore for the solution.

Challenge:

The standalone deployment of Symantec (CA) Identity Suite on MS Windows OS allowed for the mid-tier component to utilize PAM modules to redirect to AD authentication for the Global User.

However this PAM feature does not exist for Provisioning Servers on the virtual appliance.

To be clear, there are no expectations this feature will be introduced in the future roadmap for the solution, as the primary UI will be the web browser.

Review:

Symantec (CA) Identity Suite architecture for virtual appliance versus standalone deployment architecture.

  • The standalone deployment architecture has both MS Windows and Linux components of all tiers.
  • The vApp deployment architecture has primary Linux components and few MS Windows components.
    • The vApp MS Windows components do not include the IMPS (Provisioning Server)

Proposal:

To address this requirement of enabling AD authentication to the vApp Provisioning Server, we will introduce the concept of a “jump server”.

The “jump server” will utilize the standalone deployment of Symantec Identity Provisioning Server on an MS Windows OS. This “jump server” will be deployed as an “alternative server” integrated into the existing vApp Provisioning Directory deployment.

We will select deployment configuration ONLY of the Provisioning Server itself. We do not require the embedded CCS Service.

We will integrate this “jump server” deployment with the existing Symantec Identity solution.

Ensure the imps_datakey encryption seed file is in sync between all components vApp and standalone.

To avoid impacting the existing vApp deployment, we will NOT integrate the “jump server” deployment to the IME. The IME’s Directory XML for the Provisioning Directory will not be updated.

Important Note: The Symantec/CA Directory solution is required as a pre-step.

Summary of deployment steps:

  • Select a MS Window OS workstation (clean or with JCS/CCS Services) that may be part of the MS AD Domain
    • Option 1: [RECOMMENDED & PREFERRED] If using a clean OS, install MS .NetFramework 3.5.1 for the provisioning component.
      • Open cmd as administrator to deploy:  DISM /Online /Enable-Feature /All /FeatureName:NetFx3
    • Option 2: [MED-HIGH RISK] If using “side-deployment” on an existing JCS/CCS server (MS Win OS), we will need to make modifications to this server.
      • Will need to rename the file  C:\Windows\vpd.properties   to avoid conflict with the JCS/CCS component naming convention in this “registry” file. (see below screen shot)
      • Will require a post-install execution of the IMPS pwdmgr tool to address an MS Registry path conflict between the CCS and IMPS components.
  • Ensure all CA Directory hostnames are in DNS or in the MS Windows local host file (C:\Windows\System32\drivers\etc\hosts ) otherwise this “jump server” deployment will fail when it tries to validate all possible directory nodes’ hostnames and build the respective Directory knowledge files.
  • Create a reference file for the new IMPS router dxc file on at least one of the existing vApp Identity Suite Directory Server otherwise this “jump server” deployment will fail due to trust issue when testing connections to other directory nodes’ hostnames.
  • Deploy Symantec/CA Directory (if not already done) – default configurations. Otherwise, you will see this error message
  • Deploy IMPS MS Windows – Only IMPS (no CCS) with Alternative Server Selection Configuration & update to latest CP patches. Note: For “side-deployment” only: If the vpd.properties file was not renamed, then a name collision will occur due to this registry file, if using the JCS/CCS server to side-deploy. It is low risk to change this file, as it is used to prevent deployments of lower release version of components over the prior installed higher release versions of the same component. If there is a concern, all components can be reinstalled as needed. Do not forget to install the latest CP patches to ensure this “jump server” is the same binary level as the vApp solution.
  • Review of additional notes during deployment of “jump server”. Note: For “side-deployment” only: On the page that ask for the Identity Suite Directory connection information, you will see the solution attempt to load env variables that do not exist. Override these value and enter the Directory hostname, Port 20394, and the default bind DN credentials for a Directory userID: eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=im,dc=etadb
  • Deploy IMPM Manager GUI if needed.
  • Post-Deployment – Update IMPM Manager GUI preference to ONLY connect to the new IMPS server on MS Windows. Use the “Enable Failover” checkbox and place the IP/hostname first in the list. Hint: Remove the other IMPS servers from this list or you may add an extra digit to the IMPS servers entries you wish to save, but prevent auto-connectivity to them. Confirm able to authenticate directly through the solution using prior credentials for your service ID: etaadmin or imadmin. This will validate connectivity to the existing vApp Identity Suite solution.
  • On the “jump server” under the Provisioning Server\pam\ADS folder copy the etapam.dll to the IMPS \bin folder. Then copy the etapam_id.conf configuration file to the \pam parent folder. Update the parameters in this file. Set the enable= parameter to yes. Set the domain= to either the MS AD Domain or use the FQDN hostname of the ADS Domain Controller (DC). If we use the FQDN hostname of the DC the “jump server” does NOT have to be made a member of the MS AD Domain. Save the file and restart the “CA Identity Manager – Provisioning Server”
  • Validate PAM functionality in the IMPS etatrans log is enabled. We will see two (2) entries: PAM: Initialization started (same for all use-cases) and PAM: Not enabled or No PAM managed endpoint. We want “PAM: No PAM managed endpoint” – That is an extra feature we could enable, but do not require for the “jump server” scenario.
  • Validate PAM functionality with MS Sysinternals. Ensure that we copied the etapam.dll to the bin folder and that the configuration file is being read.
  • Test authentication using IMPM Manager login as IMPS Manager Global User that has same userID format as AD sAMAccountName. Negative Use-Case testing: Create a new AD domain user that does NOT exist as a Global User and attempt to authentication. Test with etaadmin or other Global User that does NOT have a match AD sAMAcccount name entry. Review the IMPS etatrans logs on the “jump server”

Update the IMPS encrypted seed file imps_datakey as needed.

Note: The MS Win version of IMPS encrypted seed file may be different than the vApp seed.

If this step is skipped, there will be no obvious error message with the exception that a bind has failed for communication to the JCS/CCS services.

After this file is updated, we will need to re-install IMPS service to ensure that all prior encryption passwords are replaced with new passwords using the new seed file. Basically, we need to install the MS Win version of IMPS Server twice, e.g. standard install, change the seed file value, re-install with update all components and updated passwords.

CCS Service conflict with “side-loading” IMPS Service {“side-loading” methodology}

The “side loading “process of deploying the “jump server” IMPS Provisioning Server on the JCS/CCS Server will impact starting of the CCS service. The installation will update the MS Registry with extra branches and updated shared attribute values between the CCS service and IMPS service, e.g. ETAHOME.

This challenge is a strong reason why we may choose the “clean” installation methodology, to avoid this conflict and possible support challenge.

To address this concern, update the new registry values that store the embedded reversible encrypted password for the CCS Service. Use the password reset tool “pwdmgr” and reset the “Connector Server” for both “eta” & “im” domain to the prior stored password. If the imps_datakey file is not in sync between all provisioning servers (& ccs service), then we will see failed bind connections error messages in the logs.

We will now be able to stop/start the JCS service, and see the embedded CCS service stop and start as well.

Example of challenge and error messages if imps_datakey is not updated and in sync.

Use the following command, csfconfig.exe, under the newly deployed IMPS bin folder to view the JCS connectors defined to the solution stack.

C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin show
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:


C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>echo Password01 > c:\imps.pwd


C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin add name=pamjcs host=192.168.242.143 pass=c:\imps.pwd br-add=@ debug=yes port=20411
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:
Created CS object with name = pamjcs

C:\Program Files (x86)\CA\Identity Manager\Provisioning Server\bin>csfconfig.exe auth=etaadmin remove name=pamjcs 
EtaSSL.initialize: CRYPTO_library_init: 1
EtaSSL.initialize: SSL_library_init: 1
Enter your authentication password:

We will see both error status when the imps_datakey file is out-of-sync with others. Please ensure the Linux & MS Win versions are in sync.

You may view the file imps_datakey being referenced with the pwdmgr tool:

su - imps
strace -e trace=open,openat  pwdmrg

open(“/opt/CA/IdentityManager/ProvisioningServer/data/tls/keymgmt/imps_datakey”, O_RDONLY) = 5

You wish to monitor what accounts (embedded) are updated with the IMPS pwdmgr tool: su – imps and execute the two commands in a different SSH shell to monitor the pwdmgr.log that was enabled.

eta-env action=set name=eta_pwdmgr_log type=string value=true
tail -f $ETAHOME/bin/pwdmgr.log

Enablement of extra functionality (bypass the no-sync option on Global User password update)

You may wish to keep the Global User and AD password in sync. If they are not, then you will have two passwords that will work for the Global User account. The newer PAM AD authentication credentials, and the older Global User password. The etapam.dll module data path appears to check for PAM AD first, and if it fails, then it will check the Global User eTPassword field as well.

Enable the AD endpoint in the etapam_id.conf file. The type and domain will be as shown, e.g. Active Directory and im (for the vApp). The endpoint-name will be free-form and whatever you may have named your AD endpoint in the IMPS GUI.

Monitor the startup of the PAM module within the IMPS etatrans*.log

Perform a use-case test with changing a Global User account without correlation to an AD endpoint; and then retest with a Global User that is correlated to an AD endpoint. Do both test with NO SYNC operation

If the Global User is already correlated to an AD endpoint account, then we will see a “Child Modify” operation to the correlated AD endpoint account’s Password within the IMPS etatrans*.log.

One “gotcha”. There appears to be a check against the AD password policy. If the new password does not fit the AD password policy, the following error message will appear, “ETA_E_0007 <MGU>, Global user XXXXXXX modification failed: PAM account password updated failed: Account password must match global user password.

The hidden cost of Entropy to your business

On Linux OS, there are two (2) device drivers that provide entropy “noise” for components that require encryption, e.g. the /dev/random and the /dev/urandom device drivers. The /dev/random is a “blocking” device driver. When the “noise” is low, any component that relies on this driver will be “stalled” until enough entropy is returned. We can measure the entropy from a range of 0-4096. Where a value over 1000 is excellent. Any value in the double or single digits will impact the performance of the OS and solutions with delays. The root cause of these delays is not evident during troubleshooting, and typically there are no warning nor error messages related to entropy.

watch -n 1 cat /proc/sys/kernel/random/entropy_avail

The Symantec Identity Suite solution, when deployed on Linux OS is typically deployed with the JVM switch -Djava.security.egd=file:/dev/./urandom for any component that uses Java (Oracle or AdoptOpenJDK), e.g. Wildfly (IM/IG/IP) and IAMCS (JCS). This JVM variable is sufficient for most use-cases to manage the encryption/hash needs of the solution.

However, for any component that does not provide a mechanism to use the alternative of /dev/urandom driver, the Linux OS vendors offer tools such as the “rng-tools” package. We can review what OS RNGD service is available using package tools, e.g.

dnf list installed | grep -i rng

If the Symantec Identity Suite or other solutions are deployed as standalone components, then we may adjust the Linux OS as we need with no restrictions to add the RNGD daemon as we wish. One favorite is the HAVEGED daemon over the default OS RNGD.

See prior notes on value and testing for Entropy on Linux OS (standalone deployments):

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=2197&MID=720771&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

https://community.broadcom.com/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=7747b411-2e1e-4bc2-8284-9b8856790ef9

Challenge for vApp

The challenge for Virtual Appliances is that we are limited to what functionality the Symantec Product Team provides for us to leverage. The RNGD service was available on the vApp r14.3, but was disabled for OS challenges with 100% utilization with CentOS 6.4. The service is still installed, but the actual binary is non-executable.

https://knowledge.broadcom.com/external/article/97774/ca-identity-suite-low-entropy-on-virtual.html
https://knowledge.broadcom.com/external/article/139759/ca-identity-suite-142-vapp-rngd-proces.html
https://broadcom-stage.adobecqms.net/us/en/symantec-security-software/identity-security/identity-suite/14-3/virtual-appliance/administering-virtual-appliance/using-the-login-shell.html

A new Virtual Appliance patch would be required to re-enable this RNGD on vApp r14.3cp2. We have access via sudo, to /sbin/chkconfig, /sbin/service to re-enable this service, but as the binary is not executable, we cannot progress any further. We can see the alias in the documentation still exist, but the OS alias was removed in the cp2 update.

However, since vApp r14.4 was release, we can focus on this Virtual Appliance which is running Centos 8 stream. The RNGD service here is disabled (masked) but can be re-enabled for our use with the sudo command. There is no current documented method for RNGD on vApp r14.4 at this time, but the steps below will show an approved way using the ‘config’ userID and sudo commands.

Confirm that the “rng-tools” package is installed and that the RNGD binary is executable. We can also see that the RNGD service is “masked”. Masked services are prevented from starting manually or automatically as an extra safety measure when we wish for tighter control over our systems.

If we test OS entropy for this vApp r14.4 server without RNGD, we can monitor how a simple BASH shell script that emulates a password being generated will impact the “entropy” of /dev/random. The below script will reduce the entropy to low numbers. This process will now impact the OS itself and any components that reference /dev/random. We can observe with “lsof /dev/random” that the java programs will still reference /dev/random; even though most activity is going to /dev/urandom.

Using the time command in the BASH shell script, we can see that the response is rapid for the first 20+ iterations, but as soon as the entropy is depleted, each execution is delayed by 10-30x times.

counter=1;MAX=100;while [ $counter -le $MAX ]; do echo "##########  $counter ##########" ; time dd if=/dev/random bs=8 count=1 2> /dev/null | base64; counter=$(( $counter + 1 )); done;

Enable RNGD on vApp r14.4 & Testing

Now let’s see what RNGD service will do for us when it is enabled. Let’s follow the steps below to unmask, enable, and start the RNGD service as the ‘config’ userID. We have access to sudo to the Centos 8 Stream command of /sbin/systemctl.

sudo /usr/bin/systemctl status rngd.service
ls -lart /etc/systemd/system/rngd.service
sudo /usr/bin/systemctl unmask rngd.service
sudo /usr/bin/systemctl enable rngd.service
cat /usr/lib/systemd/system/rngd.service
sudo /usr/bin/systemctl start rngd.service
sudo /usr/bin/systemctl status rngd.service
ps -ef | grep rngd | grep -v grep

After the RNGD service is enabled, test again with the same prior BASH shell script but bump the loops to 1000 or higher. Note using the time command we can see that each loop finishes within a fraction of a second.

counter=1;MAX=1000;while [ $counter -le $MAX ]; do echo "##########  $counter ##########" ; time dd if=/dev/random bs=8 count=1 2> /dev/null | base64; counter=$(( $counter + 1 )); done;

Summary

Aim to keep the solution footprint small and the right-sized to solve the business’ needs. Do not accept the default performance; avoid over-purchasing to scale to your expected growth.

Use the JVM switch wherever there is a java process, e.g. BLC or home-grown ETL (extract-transform-load) processes.

-Djava.security.egd=file:/dev/./urandom

If you suspect a dependence may impact the OS or other processes on /dev/random, then enable the OS RNGD and perform your testing. Monitor with the top command to ensure RNGD service is providing value and not impacting the solution.

Avoid the noise – IMPS etatrans alias/function “tap”

Monitoring use-cases within solutions that use various logs can be onerous when there is “noise” or low-value events in the logs. For provisioning use-cases, we prefer to focus on the “CrUD” use-cases and actions.

The CA/Symantec Identity Manager solution has a mid-tier component, IM Provisioning Server, that captures quite a bit of information useful for monitoring for success/failure. The default Log Level of the primary log file, etatrans*.log, is log level = 7. This log level will capture all possible searches and information of activity within the Provisioning Server’s service and transactions to its connector tier.

We can reduce some of the “noise” of searches/information and focus on the “CRuD” actions of “add/mod/del” by reducing the log level to level = 3.

This help as well to reduce the impact to the disk spaces and roll-over of the etatrans*.log file during bulk tasks or feed tasks.

Challenges:

However, even with log level = 3, we still have some “noise” in the etatrans*.log.

Additional “pain points”, the etatrans*.log file is renamed upon every restart of the IMPS service and during rollover at a size of 1 MB.

Resolution:

To assist with “finding” the current file, and to remove the noise, we have created the following “function/alias” for the IMPS user ID.

  1. Log into the IMPS service ID: sudo su – imps {Ensure you use the “dash” character to ensure the .profile is sourced when you switch IDs}
  2. Edit the .profile file: vi .profile
  3. The current file will only have one line, that sources the primary IMPS environmental information: . /etc/.profile_imps

4. Add the following body after the IMPS environmental profile line

function tap () {
cd $ETAHOME/logs
a=$(ls -rt $ETAHOME/logs | grep etatrans | tail -1)
pwd
echo "Tail current log file with exclusions: "$a
tail -F $a | grep -v -e ":LDAP" -e ":Config" -e "AUDITCONFIG" -e ":EtaServer" -e ":Bind " -e ":Unbind " -e ":Search "
}
export -f tap

This new “function/alias” will cd to the correct folder of logs, then tail the correct etatrans*.log file, and exclude the noise of non-CrUD activity. Using the new alias of “tap” on all provisioning servers, will allow us to isolate any challenges during use-case validation.

5. Exit out of the IMPS user ID account; then re-sudo back into this account, and test the “tap” alias.

6. While using the “tap” alias, exercise use-cases within the IM Provisioning Manager (GUI) and the IM User Console (browser); monitor the “Add/Mod/Deletes”. You will also be able to see the “Child” updates to endpoints and updates to the IMPS notification queue (IME Callback).

Using X11 on Virtual Appliances

In this blog example, we will explore expanding the ability of Virtual Appliances to use X11 programs where possible instead of requiring a 2nd server to host the solutions’ client tools.

We will review how to enable the following client tools: CX (Symantec IM Connector Xpress), WF Designer (Symantec Workflow Designer), Wildfly/JBOSS Management UI, Jxplorer (LDAP Management UI), and Apache Directory Studio (LDAP Management UI). Note there is no java version for the Symantec Identity Manager GUI (32bit VC++ client)

Challenge:

The Symantec Identity Suite Virtual Appliance is locked down from updating libraries as the ‘root’ user and the default login service ID of ‘config’ does not have access to the package installer, e.g. yum.

The Symantec Identity Suite Virtual Appliance like other tools, have an enhancement request process to add new functionality. While we wait for this to be delivered, we can address these gaps ourselves with knowledge of what X11 functionality is required to be enabled with the limited resources we have available to ourselves.

https://community.broadcom.com/participate/ideation-home/viewidea?IdeationKey=34adb887-a1c0-4d59-a977-4b65f4274425

To first identify what drivers may be needed, we can use the Linux OS “strace” command to capture which files are “open” or “openat” and used by the programs.

Example for tracing the files/drivers for Java (Oracle or AdoptOpenJDK) jconsole program:

STRACE

Using “strace -e trace=open,openat /opt/CA/java/bin/policytool” we can see the files that were found ” = 3″ versus those not found “= -1 ENOENT (No such file or directory)”. Some files are required for Centos 8 Stream and others for Centos 6.4

Using this iterative process above will help to identify either the primary file used or a supporting file required to start and use the UI version of the program.

We can use both Java jconsole or policytool to help identify the drivers required. There may be a different of drivers version or additional ones require for the OS of the virtual appliances.

Trace Example with Java Policytool program and compare between Centos 8 Stream (openat) and Centos 6.4 (open)

Now that we have a process to help identify the drivers required, we can walk through the challenges and the value statement.

First challenge for X11 access, is the DISPLAY environment variable must be defined. Otherwise we will see this error message: “No X11 DISPLAY variable

To address this challenge, we could manually set this value every time, but why not use our bash shell to manage this.

DISPLAY OS Variable

Add the below two (2) lines to config’s .bash_profile

DISPLAY=$(who -m | gawk -F'(' '{ print $2}' | gawk -F')' '{print $1}'):0.0;export DISPLAY
env | grep DISPLAY

Our DISPLAY variable will now be updated every time we log into the vApp with the config userID.

We should now see this:

When using the “strace” method, we may see that we have a file on the OS, but it is reporting it as not found due to an internal dependency.

Example for policytool, we can see that the file libXext.so.6 (or link) is not referenced correctly.

If we search the OS of the vApp, we can find this file (and its link) with no issue.

The file libXext.so.6 (or libXext.so.6.4.0) requires a supporting file of libX11.so.6 (libX11.so.6.3.0). As we search for these files, we can now start collecting them from nonVapp OS servers (that do have access to package updates), and make them available to the ‘config’ userID via scp/rsync.

On another server, that has these file (same OS release), find and copy these files.

After we have identified all the required files that are missing from the vApp r14.3 (Centos 6.4) or r14.4 (Centos 8 Stream), we can package them up for the ‘config’ userID and scp/rsync them to the vApp.

Soft Links

Before we use these files, we may need to validate that the soft-links are properly defined. If you have any issues, use strace to help identify the missing soft-link to the real file.

LD_LIBRARY_PATH

LD_LIBRARY_PATH is the OS variable we will use to redirect the libraries files (x86/x64) for the ‘config’ userID. Again edit the config’s .bash_profile and ensure the following lines exist:

#### ANA - Add X11 process & supporting libraries to vApp r14.3 config service ID ####
DISPLAY=$(who -m | gawk -F'(' '{ print $2}' | gawk -F')' '{print $1}'):0.0;export DISPLAY
env | grep -i DISPLAY

LD_LIBRARY_PATH=/tmp/x11_libraries_for_centos6_vapp143/usr/lib64:/tmp/x11_libraries_for_centos6_vapp143/usr/lib;export LD_LIBRARY_PATH
env | grep -i LD_LIBRARY_PATH
#### ANA - Add X11 process & supporting libraries to vApp config service ID ####

OR

#### ANA - Add X11 process & supporting libraries to vApp r14.4 config service ID ####
DISPLAY=$(who -m | gawk -F'(' '{ print $2}' | gawk -F')' '{print $1}'):0.0;export DISPLAY
env | grep -i DISPLAY

LD_LIBRARY_PATH=/tmp/x11_for_centos8_for_vapp144/usr/lib64:/tmp/x11_for_centos8_for_vapp144/usr/lib;export LD_LIBRARY_PATH
env | grep -i LD_LIBRARY_PATH
#### ANA - Add X11 process & supporting libraries to vApp config service ID ####

We should now see the following upon login:

Summary Page of X11 Functionality for vApp r14.3cp2 (Centos 6.4)

Centos 6.4 OS Files required for X11 functionality

/tmp/x11_libraries_for_centos6_vapp143/usr/lib64:
-rwxr-xr-x 1 config config   62176 Jun 18 15:04 libXi.so.6.1.0
-rwxr-xr-x 1 config config   38272 Jun 18 15:04 libXrender.so.1.3.0
-rwxr-xr-x 1 config config   21952 Jun 18 15:04 libXtst.so.6.1.0
-rwxrwxr-x 1 config config   74336 Jun 18 15:04 libXext.so.6.4.0
-rwxr-xr-x 1 config config 1297928 Jun 18 15:04 libX11.so.6.3.0
lrwxrwxrwx 1 config config      14 Jun 18 15:05 libXi.so.6 -> libXi.so.6.1.0
lrwxrwxrwx 1 config config      16 Jun 18 15:03 libXtst.so.6 -> libXtst.so.6.1.0
lrwxrwxrwx 1 config config      19 Jun 18 15:01 libXrender.so.1 -> libXrender.so.1.3.0
lrwxrwxrwx 1 config config      15 Jun 18 14:52 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 14:14 libXext.so.6 -> libXext.so.6.4.0

/tmp/x11_libraries_for_centos6_vapp143/usr/lib:
-rwxr-xr-x 1 config config   59180 Jun 18 15:04 libXi.so.6.1.0
-rwxr-xr-x 1 config config   20044 Jun 18 15:04 libXtst.so.6.1.0
-rwxrwxr-x 1 config config   68588 Jun 18 15:04 libXext.so.6.4.0
-rwxr-xr-x 1 config config 1279168 Jun 18 15:04 libX11.so.6.3.0
lrwxrwxrwx 1 config config      14 Jun 18 15:05 libXi.so.6 -> libXi.so.6.1.0
lrwxrwxrwx 1 config config      16 Jun 18 15:03 libXtst.so.6 -> libXtst.so.6.1.0
lrwxrwxrwx 1 config config      15 Jun 18 14:52 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 14:14 libXext.so.6 -> libXext.so.6.4.0

Summary Page of X11 Functionality for vApp r14.4 (Centos 8 Stream)

Centos 8 Stream’s OS Files required for X11 functionality

/tmp/x11_for_centos8_for_vapp144/usr/lib64:
-rwxrwxr-x 1 config config  170208 Jun 18 17:34 libxcb.so.1.1.0
-rwxr-xr-x 1 config config   49256 Jun 18 17:34 libXrender.so.1.3.0
-rwxr-xr-x 1 config config   29104 Jun 18 17:34 libXtst.so.6.1.0
-rwxr-xr-x 1 config config   80728 Jun 18 17:34 libXext.so.6.4.0
-rwxr-xr-x 1 config config   70720 Jun 18 17:34 libXi.so.6.1.0
-rwxr-xr-x 1 config config 1343952 Jun 18 17:34 libX11.so.6.3.0
-rwxr-xr-x 1 config config   16352 Jun 18 17:34 libXau.so.6.0.0
lrwxrwxrwx 1 config config      15 Jun 18 17:35 libXau.so.6 -> libXau.so.6.0.0
lrwxrwxrwx 1 config config      15 Jun 18 17:33 libxcb.so.1 -> libxcb.so.1.1.0
lrwxrwxrwx 1 config config      14 Jun 18 17:32 libXi.so.6 -> libXi.so.6.1.0
lrwxrwxrwx 1 config config      16 Jun 18 17:31 libXtst.so.6 -> libXtst.so.6.1.0
lrwxrwxrwx 1 config config      19 Jun 18 17:30 libXrender.so.1 -> libXrender.so.1.3.0
lrwxrwxrwx 1 config config      15 Jun 18 17:28 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 17:27 libXext.so.6 -> libXext.so.6.4.0

/tmp/x11_for_centos8_for_vapp144/usr/lib:
-rwxrwxr-x 1 config config  181952 Jun 18 17:34 libxcb.so.1.1.0
-rwxr-xr-x 1 config config   78200 Jun 18 17:34 libXi.so.6.1.0
-rwxr-xr-x 1 config config   87788 Jun 18 17:34 libXext.so.6.4.0
-rwxr-xr-x 1 config config   15700 Jun 18 17:34 libXau.so.6.0.0
-rwxr-xr-x 1 config config 1411660 Jun 18 17:34 libX11.so.6.3.0
lrwxrwxrwx 1 config config      15 Jun 18 17:28 libX11.so.6 -> libX11.so.6.3.0
lrwxrwxrwx 1 config config      16 Jun 18 17:27 libXext.so.6 -> libXext.so.6.4.0

Final Value Statement – X11 UI on vApp

Jxplorer

Script to add jxplorer on the vApp (if you have internet access to the vApp). This script will maintain the configuration file “connections.txt” where hostname/ports/userDN are stored for Jxplorer.

#!/bin/bash
##############################################
#  Name: add_jxplorer.sh
#  Goal: Add Jxplorer (jar) to vApp r14.4 (Centos 8 Stream) with X11 enabled
#  Ref: http://jxplorer.org/downloads/users.html
# ANA 2021
##############################################
cd
mkdir -p jxplorer;cd jxplorer
find . -type f -not -name 'connections.txt' -delete
curl -OL https://netactuate.dl.sourceforge.net/project/jxplorer/jxplorer/version%203.3.1.2/jxplorer-3.3.1.2-linux-installer.run
chmod 555 jxplorer-3.3.1.2-linux-installer.run; ./jxplorer-3.3.1.2-linux-installer.run --unattendedmodeui minimal --mode unattended
pwd
./jxplorer.sh >/dev/null &
echo "Done"

Wildfly / JBoss CLI X11 UI

Use for managing the standalone-full-ha.xml file via jboss-cli.sh scripts and to update values.

Ensure you have created a management user credential to access the running Wildfly/JBoss release.

config@vapp14401 VAPP-14.4.0 (192.168.2.210):~ > sudo /opt/CA/wildfly-idm/bin/add-user.sh -m -u jboss-admin -p Password01!
Added user 'jboss-admin' to file '/opt/CA/wildfly-idm/standalone/configuration/mgmt-users.properties'
Added user 'jboss-admin' to file '/opt/CA/wildfly-idm/domain/configuration/mgmt-users.properties'
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~ >
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~ > /opt/CA/wildfly-idm/bin/jboss-cli.sh   --connect  --user=jboss-admin  --password=Password01!  --gui

Next Steps

We can use the X11 functionality for the IM Workpoint Designer tool, the Connector Xpress (CX) UI tool, and any other tools, e.g. Symantec Layer7 Management UI (manager.jar)

Side Note:

The IM Workpoint Designer tool and other tools have been removed from the vApp r14.4 IAMSuite samples.

Installed IAMSuite tools only under config service ID, to determine if there is any value. Do not see any X11 client applications under this installed component.

Workpoint Designer

Extract the workpoint designer from the standalone deployment tools to a media folder.

Update the shell script files to be executable, replace the localhost entry for another host alias that will resolve to an IP address that the IM solution with Workpoint is actively listening to. Then run the designer from the virtual appliance.

config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media > unzip CA-IG_WorkpointDesigner.zip  > /dev/null
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media > cd CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin/
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > chmod 555 *.sh
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > cp -r -p ../conf/workpoint-client.properties ../conf/workpoint-client.properties.org
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > sed -i 's|localhost|caim-srv|g' ../conf/workpoint-client.properties
config@vapp14401 VAPP-14.4.0 (192.168.2.210):~/media/CA-IG_WorkpointDesigner/Workpoint/WorkPointDesigner/bin > ./Designer.sh

Ref: https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-manager/14-4/administrating/workflow/how-to-use-the-workpoint-method/configure-workpoint-administrative-tools.html

Connector Xpress

Connector Xpress only has a MS Windows installer, but we can still use this component on Linux OS. Install the CX UI on MS Windows, then zip up the installed folder with all sub-folders. Copy this compress file over to a media folder for the ‘config’ userID and extract the file.

Review the startup file of “ConnectorXpress.bat” and we will create a version for Linux OS. Copy the last line with the conxp.jar file to a new bash script file. Update the file path from MS Windows format, to Linux OS format.

We can now use CX UI from the vApp.

Restart remote IMPD DATA DSAs after long outage

“DSA is attempting to start after a long outage, perform a recovery procedure before starting”

Challenge:   The IMPD (Identity Manager Provisioning Directory) Data DSAs have been offline for a while, e.g. 7 days+ (> 1 week), and the Symantec/CA Directory solution will, to protect the data, refuse to allow the DATA DSAs to start unless there is manual intervention to prevent the possibility of production data (Live DATA DSAs) being synced with older data (Offline DATA DSAs).

If we were concern, we would follow best practices and remove the offline DATA DSAs’ *.db & *.dp files, and replace the *.db with current copies of the Live DATA DSAs’ *.db files; generate temporary time files of *.dx and allow the time files of *.dp to rebuild themselves upon startup of the offline DATA DSAs.

Example to recover from an outage: https://anapartner.com/2020/08/21/directory-backup-and-restore-dar-scenarios/

However, if we are NOT concern, or the environment is non-production we can avoid the multiple shells, multiple commands to resync by using a combinations of bash shell commands. The proposal below outlines using the Symantec/CA Identity Suite virtual appliance, where both the IMPD and IMPS (Identity Manager Provisioning Server) components reside on the same servers.

Proposal:   Use a single Linux host to send remote commands as a single user ID; sudo to the ‘dsa’ and ‘imps’ service IDs, and issue commands to address the restart process.

Pre-Work:   For the Identity Suite vApp, recommend that .ssh keys be used to avoid using a password for the ‘config’ user IDs on all vApp nodes.

Example to setup .SSH keys for ‘config’ user ID: https://anapartner.com/2020/05/01/avoid-locking-a-userid-in-a-virtual-appliance/

If using .SSH keys, do not forget to use this shortcut to cache the local session: eval `ssh-agent` && ssh-add

Steps:   Issue the following bash commands with the correct IPs or hostnames.  

If possible, wrap the remote commands in a for-loop. The below example uses the local ‘config’ user ID, to ssh to remote servers, then issues a local su to the ‘dsa’ service ID. The ‘dsa’ commands may need to be wrapped as shown below to allow multiple commands to be executed together. We have a quick hostname check, stop all IMPD DATA DSAs, find the time-stamp file that is preventing the startup of the IMPD DATA DSAs and remove it, restart all IMPD DATA DSA, and then move on to the next server with the for-loop. The ‘imps’ commands are similar with a quick hostname check, status check, stop and start process, another status check, then move on to the next server in the for-loop.

for i in {136..141}; do ssh  -t config@192.168.242.$i "su - dsa -c \"hostname;dxserver stop all;pwd;find ./data/ -type f \( -name '*.dp' \) -delete  ;dxserver start all \" "; done

for i in {136..141}; do ssh  -t config@192.168.242.$i "su - imps -c \"hostname;imps status;imps stop;imps start;imps status \" "; done

View of for-loop commands output:

Additional: Process to assist with decision to sync or not sync.

Check if the number of total entries in each individual IMPD DATA DSA match with their peers (Multi-Write groups). Goal: Avoid any deltas > 1% between peers. The IMPD “main”, “co”, “inc” DATA DSA should be 100% in sync. We may see some minor flux in the “notify” DATA DSA, as this is temporary data used by the IMPS server to store data to be sent to the IME via the IME Call Back Process.

If there are any deltas, then we may export the IMPD DATA DSAs to LDIF files and then use the Symantec/CA Directory ldifdelta process to isolate and triage the deltas.

su - dsa    OR [ sudo -iu dsa ]
export HISTIGNORE=' *'             {USE THIS LINE TO FORCE HISTORY TO IGNORE ANY COMMANDS WITH A LEADING SPACE CHARACTER}
 echo -n Password01 > .impd.pwd ; chmod 600 .impd.pwd     {USE SPACE CHARACTER IN FRONT TO AVOID HISTORY USAGE}


# NOTIFY BRANCH (TCP 20404) 

for i in {135..140}; do echo "##########  192.168.242.$i IMPD NOTIFY DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20404 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=notify,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done

# INC BRANCH (TCP 20398)

for i in {135..140}; do echo "##########  192.168.242.$i IMPD INC DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20398 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTInclusionContainerName=Inclusions,eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done

# CO BRANCH (TCP 20396)

for i in {135..140}; do echo "##########  192.168.242.$i IMPD CO DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20396 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'eTNamespaceName=CommonObjects,dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done

# MAIN BRANCH (TCP 20394)

for i in {135..140}; do echo "##########  192.168.242.$i IMPD MAIN DATA DSA ##########";LDAPTLS_REQCERT=never  dxsearch -LLL -H ldaps://192.168.242.$i:20394 -D 'eTDSAContainerName=DSAs,eTNamespaceName=CommonObjects,dc=etadb' -y .impd.pwd -s sub -b 'dc=im,dc=etadb' '(objectClass=*)' dxTotalEntryCount  |  perl -p00e 's/\r?\n //g' ; done


NOTIFY DSA is temporary data and will have deltas. This DSA is used for the IME CALL BACK process.

ADS Endpoint Configuration Challenges and Hints

  1. Ensure the hostname entry is a FQDN or alias. It can not be an IP address if MS Exchange is to be managed through this connector, due to conflict with Kerberos authentication and IP addresses. If the object was created with an IP address, it may be changed via Jxplorer for two (2) attributes: eTADSprimaryServer and eTADSServerName.

2. General Information on the ADS Endpoint Logging Tab and where this information is stored. Only two (2) the Destination have value with current deployment, e.g. Text File & System Log (MS Windows Event viewer) for Active Directory (ADS). The “Text File” will output data to two (2) files: jcs\logs\ADS\<endpoint-name>.log and ccs\logs\ADS\<endpoint-name>.log

3. Use the MS Event Viewer on the ADS Domain Controller, or use the MS Event Viewer to remotely view the transactions on the remote ADS DC. Select the event codes of 627,628,4723,4724,4738 to start with. Other codes may be added that are useful. Ref: https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/plan/appendix-l–events-to-monitor

4. Additionally, the User ID may be in one of three (3) formats: UPN (serviceid@exchange.lab), NT ( domain\serviceid ), LDAP DN ( cn=serviceid,ou=people,dc=exchange,dc=lab). We recommend UPN or NT format to allow the embedded API features for MS Exchange powershell management to correctly function. If the ID is to be changed, a password update must be done as well, since the User ID is part of the seed for the encrypted password for the service ID to be stored in CA Directory on the ADS endpoint object.

5. SASL versus TLS authentication checkboxes. We can tested the ADS authentication availability using ldapsearch binary. Ports used by Active Directory for authentication by client tools, https://docs.microsoft.com/en-us/troubleshoot/windows-server/identity/config-firewall-for-ad-domains-and-trusts

Note: SASL is encrypted traffic. If wireshark is used to intercept the traffic, the service ID may be seen during initial authentication, but NOT the password nor the payload data.

Notes on SASL validation for Active Directory. {Pro: No need to worry about TLS certificates rotation on client connections – all TLS is managed by the server}

:: Search ADS / LDAP store what is offered for SASL (use -x for simple connection)
ldapsearch -x -h dc2016.exchange.lab -p 389 -b “” -LLL -s base supportedSASLMechanisms

EXAMPLE OUTPUT

[root@oracle ~]# ldapsearch -x -h dc2016.exchange.lab -p 389 -b “” -LLL -s base supportedSASLMechanisms
dn:
supportedSASLMechanisms: GSSAPI
supportedSASLMechanisms: GSS-SPNEGO
supportedSASLMechanisms: EXTERNAL
supportedSASLMechanisms: DIGEST-MD5

:: On Linux OS, execute rpm -qa to search for SASL installed modules/libraries.
rpm -qa | grep cyrus

EXAMPLE OUTPUT

[root@oracle ~]# rpm -qa | grep cyrus
cyrus-sasl-gssapi-2.1.26-23.el7.x86_64
cyrus-sasl-lib-2.1.26-23.el7.x86_64
cyrus-sasl-md5-2.1.26-23.el7.x86_64

:: On Linux OS, install missing SASL libraries & ldapsearch (ldap-client)
yum -y install cyrus-sasl-md5 cyrus-sasl-gssapi openldap-clients

TESTING DIFFERING AUTHENTICATION MECHANISMS #### (may remove -d9 debug switch to view cleaner results)

TLS

LDAPTLS_REQCERT=never ldapsearch -d9 -LLL -H ldaps://dc2016.exchange.lab:636 -w CAdemo123 -D “CN=Administrator,CN=Users,DC=exchange,DC=lab” -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

Start TLS

LDAPTLS_REQCERT=never ldapsearch -d9 -Z -LLL -H ldap://dc2016.exchange.lab:389 -w CAdemo123 -D “CN=Administrator,CN=Users,DC=exchange,DC=lab” -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

Digest-MD5

ldapsearch -d9 -LLL -H ldap://dc2016.exchange.lab -w CAdemo123 -Y DIGEST-MD5 -U Administrator -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

Kerberos (GSS)

ldapsearch -d9 -LLL -H ldap://dc2016.exchange.lab -w CAdemo123 -Y GSSAPI -U Administrator -b “CN=Administrator,CN=Users,DC=exchange,DC=lab” -s base userAccountControl

6. TCP/UDP Ports required for Active Directory Endpoint management per CA Documentation https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-manager/14-4/reference/default-ports-for-ca-identity-manager-and-associated-components.html

SASL appears to connect on TCP 636 briefly, then use TCP 389 extensively. Other ports are 80 (Service), 135 (lsass.exe for home folders), 6405 (lsass.exe). If Kerberos authentication is defined for the service ID, then other ports will be used, e.g. 3268/3269. TCP 4104/4105 are for the legacy CAM/CAFT agents (typically not used any more).

Recommendation: Add these TCP Ports to any Firewall between the IM JCS/CCS Server and the Active Directory Domain Controllers to improve performance and avoid time-out delays.

MS Active Directory References on SASL.

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/989e0748-0953-455d-9d37-d08dfbf3998b

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-adts/a98c1f56-8246-4212-8c4e-d92da1a9563b

Parallel provisioning for Active Directory and MS Exchange mailboxes – Improve Birthright/DayOne Access

One of the challenges that IAM/IAG solutions may have is using single thread processing for select endpoints. For the CA/Symantec Identity Management solution, before IM r14.3cp2, we lived with a single-threaded connector to managed MS Active Directory endpoints.

To address this challenge, we deployed multiple connector servers. We allowed the IM Provisioning Server (IMPS) to use a built-in round-robin approach of load-balancing separate transactions to different connector servers, which would service the same Active Directory endpoints.

The IME may be running as fast as it can with its clustered deployment, but as soon as a task has MS Active Directory, and there is a bottleneck with the CCS Service. We begin to see the IME JMS queue reporting that it is stuck and the IME View Submitted Task reporting “In Progress” for all tasks. If the CCS service is restarted, all IME tasks are then reported as “Failed.”

This is/was the bottleneck for the solution for sites that have MS Active Directory for Birthright/DayOne Access.

We can now avoid this bottleneck. [*** (5/24/2021) – There is an enhancement to CP2 to address im_ccs.exe crashes during peak loads discovered using this testing process. ]

Via the newly delivered enhancement https://community.broadcom.com/participate/ideation-home/viewidea?IdeationKey=7154e15b-085d-469e-bff0-ac588ff6bd5b .

We now have full parallel provisioning to MS Active Directory from a single connector server (JCS/CCS).

The new attribute that regulates this behavior is eTADSMaxConnectionsInPool. This attribute will be applied on every existing ADS endpoint that is currently being managed by the IM Provisioning Server after CP2 is deployed. Note: The default value is 10, but we recommend after much testing, to match the value of the IMPS-> JCS and JCS->CCS to equal 200.

During testing within the IME using Bulk Tasks or the IM BLC, we can see that the CCS-> ADS traffic will reach 20-30 connections if allowed. You may set this attribute to a value of 200 via Jxplorer and/or an ldapmodify/dxmodify script.

echo "############### SET ADS MAX CONNECTIONS IN POOL SIZE ##################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
NAMESPACE=exchange2016
LDAPTLS_REQCERT=never dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD" << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'

To confirm the number of open connections is greater than one (1), we can issue a Bulk IM Task or use a performance tool like CA Directory dxsoak.

In this example, we will show case using CA Directory dxsoak to execute 100 parallel threads to create 100 ADS Accounts with MS Exchange Mailboxes. We will also enclose this script for download for others to review and use.

Performance Lab:

Pre-Steps:

  1. Leverage CA Directory samples’ dxsoak binary (performance testing). You may wish to use CA Directory on an existing IM Provisioning Server (Linux OS) or you may deploy CA Directory (MS Windows version) to the JCS/CCS connector. Examples are provided for both OSes.
  2. Create LDIF files for IM Provisioning Server and/or IM Connector Tier. This file is needed to ‘push’ the solution to-failure. The use of the IME Bulk Task and/or etautil scripts to the IM Provisioning Tier, will not provide the transaction speed we need to break the CCS service if possible.
  3. Within the IM Provisioning Manager enable the ADS Endpoint TXT Logs on the Logging TAB, for all checkboxes.
  4. Monitor the IMPS etatrans* logs, monitor the JCS ADS logs, monitor the CCS ADS logs, monitor the number of CCS-> ADS (LDAP/S – TCP 389/636) threads. [Suggest using MS Sysinternals Process Explorer and select im_ccs.exe & then TCP/IP TAB]
  5. Monitor the MS ADS Domain via MS ADUC (AD Users & Computers UI) and MS Exchange Mailbox (Mailbox UI via Browser)

Execution:

6. Perform a UNIT TEST with dxmodify/ldapmodify to confirm the LDIF file input is correct with the correct suffix.

time dxmodify -H ldap://192.168.242.135:20389 -c -x -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldif

7. Perform the PERFORMANCE TEST with dxsoak binary with the same LDIF file & correct suffix. Rate observed = 23 K ids/hr

./dxsoak -c -l 60 -t 100 -h 192.168.242.135:20389 -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldif

Observations:

8. IMPS etatrans*.log – Count the number of operations per second. Note any RACE and/or data collisions, e.g. ADS accounts deleted prior to add via 100 threads or ADS account created multiple times attempted in different threads.

9. IM CCS ADS <endpoint>.log – Will only have useful data if the ADS Endpoint Logging TAB has been checked for TXT logs.

10. Finally, validate directly in MS Active Domain with the ADUC or similar tool & MS Exchange mailboxes being created/deleted.

11. Count the number of threads from im_ccs.exe to ADS – Suggest using MS Sysinternals Process Explorer tool and/or Powershell to count the number of connections.

MS Powershell Script to count the number of LDAP (TCP 389) connection from im_ccs.exe. [Note: TCP 389 is used more if the ADS Endpoint is setup to use SASL authentication. TCP 636 is used more if the ADS Endpoint is using the older TLS authentication]

$i=1
Do {
cls
(Get-NetTCPConnection -State Established -OwningProcess (Get-Process -name im_ccs).id -RemotePort 389).count
Start-Sleep -s 1
$i++
}
while ($i -le 5)

Direct Performance Testing to JCS/CCS Service

While this testing has limited value, it can offer satisfaction and assistance to troubleshoot any challenges. We can use the prior LDIF files with a slightly different suffix, dc=etasa (instead of dc=eta), to use dxsoak to push the connector tier to failure. This step helped provide memory dumps back to CA/Symantec Engineering teams to help isolate challenges within the parallel processing. CCS Service is only exposed via localhost. If you wish to test the CCS Service remotely, then update the MS Registry key for the CCS service to use the external IP address of the JCS/CCS Server. Rate observed = 25 K ids/hr

Script to generate 100 ADS Accounts with MS Exchange Mailbox Creation

You may wish to review this script and adjust it for your ADS / MS Exchange domains for testing. You can also create a simple LDIF file with password resets or ADS group membership adds. Just remember that the IMPS Service (TCP 20389/20390) uses the suffix dc=eta, and the IM JCS/CCS Services (TCP 20410/20411) & (TCP 20402/20403) use the suffix dc=etasa. Additionally, if using CA Directory dxsoak, only use the non-TLS ports, as this binary is not equipped for using TLS certs.

#!/bin/bash
#######################################################################################################################
# Name:  Generate ADS Feed Files for IM Solution Provisioning/Connector Tiers
#
# Goal:  Validate the new parallel processes from the IM Connector Tier to Active Directory with MS Exchange
#
#
# Generate ADS User LDIF file(s) for use with unit (dxmodify) and performance testing (dxsoak) to:
#  - {Note: dxsoak will only work with non-TLS ports}
#
# IM JCS (20410)  "dc=etasa"    {Ensure MS Windows Firewall allows this port to be exposed}
# IM CCS (20402)  "dc=etasa"    {This port is localhost only, may open to network traffic via registry update}
# IMPS (20389)    "dc=eta"
#
#
# Monitor:  
#
# The IMPS etatrans*.log  {exclude searches}
# The JCS daily log
# The JCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
# The CCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
#
# Execute per the examples provided during run of this file
#
#
# ANA 05/2021
#######################################################################################################################

# Unique Variables for an ADS Domain
NAMESPACE=exchange2016
ADSDOMAIN=exchange.lab
DCDOMAIN="DC=exchange,DC=lab"
OU=People

#######################################################################################################################


MAX=100
start=00001
counter=$start
echo "###############################################################"
echo "###############################################################"
START=`/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`
echo `/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`" = Current OS UTC time stamp"
echo "###############################################################"
FILE1=ads_user_with_exchange_dc_etasa.ldif
FILE2=ads_user_with_exchange_dc_eta.ldif
echo "" > $FILE1
while [ $counter -le $MAX ]
do
    n=$((10000+counter)); n=${n#1}
    tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
   echo "Counter with leading zeros = $n   at time:  $tz"


cat << EOF >> $FILE1
dn:  eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: add
objectClass:  eTADSAccount
eTADSobjectClass:  user
eTADSAccountName:  firstname$n aaalastname$n
eTADSgivenName:  firstname$n
eTADSsn:  aaalastname$n
eTADSdisplayName:  firstname$n aaalastname$n
eTADSuserPrincipalName:  aaatestuser$n@$ADSDOMAIN
eTADSsAMAccountName:  aaatestuser$n
eTPassword:  Password01
eTADSpwdLastSet:  -1
eTSuspended:  0
eTADSuserAccountControl:  0000000512
eTADSDescription:  description $tz
eTADSphysicalDeliveryOfficeName:  office
eTADStelephoneNumber:  111-222-3333
eTADSmail:  aaatestuser$n@$ADSDOMAIN
eTADSwwwHomePage:  web.page.lab
eTADSotherTelephone:  111-222-3333
eTADSurl:  other.web.page.lab
eTADSstreetAddress:  street address line01
eTADSpostOfficeBox:  pobox 111
eTADSl:  city
eTADSst:  state
eTADSpostalCode:  11111
eTADSco:  UNITED STATES
eTADSc:  US
eTADScountryCode:  840
eTADSscriptPath:  loginscript.cmd
eTADSprofilePath:  \profile\path\here
eTADShomePhone:  111-222-3333
eTADSpager:  111-222-3333
eTADSmobile:  111-222-3333
eTADSfacsimileTelephoneNumber:  111-222-3333
eTADSipPhone:  111-222-3333
eTADSinfo:  Notes Here
eTADSotherHomePhone:  111-222-3333
eTADSotherPager:  111-222-3333
eTADSotherMobile:  111-222-3333
eTADSotherFacsimileTelephoneNumber:  111-222-3333
eTADSotherIpPhone:  111-222-3333
eTADStitle:  title
eTADSdepartment:  department
eTADScompany:  company
eTADSmanager:  CN=manager_fn manager_ln,OU=$OU,$DCDOMAIN
eTADSmemberOf:  CN=Backup Operators,CN=Builtin,$DCDOMAIN
eTADSlyncSIPAddressOption: 0000000000
eTADSdisplayNamePrintable: aaatestuser$n
eTADSmailNickname: aaatestuser$n
eTADShomeMDB: (Automatic Mailbox Distribution)
eTADShomeMTA: CN=DC001,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,$DCDOMAIN
eTAccountStatus: A
eTADSmsExchRecipientTypeDetails: 0000000001
eTADSmDBUseDefaults: TRUE
eTADSinitials: A
eTADSaccountExpires: 9223372036854775807

EOF
 counter=$(( $counter + 00001 ))
done


#  Create the delete ADS Process
start=00001
counter=$start
while [ $counter -le $MAX ]
do
    n=$((10000+counter)); n=${n#1}
    tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
   echo "Counter with leading zeros = $n   at time:  $tz"


cat << EOF >> $FILE1
dn:  eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: delete

EOF
 counter=$(( $counter + 00001 ))
done

echo ""
echo "################################### ADS USER OBJECT STATS ################################################################"
echo "Number of add objects: `grep "changetype: add" $FILE1 | wc -l`"
echo "Number of delete objects: `grep "changetype: delete" $FILE1 | wc -l`"
rm -rf $FILE2
cp -r -p $FILE1 $FILE2
sed -i 's|,dc=im,dc=etasa|,dc=im,dc=eta|g' $FILE2
ls -lart $FILE1
ls -lart $FILE2

echo ""
echo "################################### SET ADS MAX CONNECTIONS IN POOL SIZE ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
LDAPTLS_REQCERT=never dxmodify  -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD"  << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL  -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'

echo ""
echo "################################### CCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20402
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the CCS Service to test single thread with dxmodify or ldapmodify"
echo "dxmodify  -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the CCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "

echo ""
echo "################################### JCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20410
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the JCS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify  -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the JCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "


echo ""
echo "################################### IMPS UNIT & PERF TEST ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
echo "Execute this command to the IMPS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify  -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "
echo "Execute this command to the IMPS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $IMPS_HOST:$IMPS_PORT -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "




Address the new bottleneck of MS Exchange / O365 Provisioning.

After parallel provisioning has been introduced with the new im_ccs.exe service, you may noticed that the number of transactions is still being throttled during performance testing.

Out-of-the-box MS Active Directory Global Throttling Policy has the parameter of PowerShellMaxConcurrency set to a default of 18 connection. Any provisioning that uses MS Powershell for MS Exchange and/or MS O365 will be impacted by this default parameter.

To address this bottleneck, we can create a new Throttling Policy and only assign the service ID that will be managing identities, to avoid a global change.

Example: New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 & Set-Mailbox “User Name” -ThrottlingPolicy MaxPowershell

After this change has been made, restart the IM JCS/CCS Services, and retest again with your performance tools. Review the CCS ADS log for # of creations in 60 seconds, and you will be pleasantly surprise at the rate. The logs are the strong confirmation we are looking for.

Performance test (947 ADS accounts w/Exchange mailboxes in 60 seconds, 08:59:54 to  09:00:53) => Rate of 15 ids/second   (or 54 K ids/hr) with updated MaxPowershell = 100 thottlingpolicy.

The last bottleneck appears to be CPU availability to MS Exchange Supporting Services, w3wp.exe, the MS IIS Service. Which appears to be managing MS Powershell connections per its startup string of

" c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangePowerShellAppPool" -v "v4.0" -c "C:\Program Files\Microsoft\Exchange Server\V15\bin\GenericAppPoolConfigWithGCServerEnabledFalse.config" -a \.\pipe\iisipme304c50e-6b42-4b26-83a4-229ee037be5d -h "C:\inetpub\temp\apppools\MSExchangePowerShellAppPool\MSExchangePowerShellAppPool.config" -w "" -m 0"

JCS versus CCS Connector Tier Challenges

A very common challenge we see is the modification of the CA/Symantec Connector Server Service(s) startup order for the embedded C++ (CCS) Connector. This CCS connector service on MS Windows OS is marked default as “Manual” startup.

Since the solution documentation is not clear on why this is configured as manual, we will see site’ administrators that will either change this service from “Manual” to “Automatic” or will start the CCS service manually themselves upon a restart.

However, either of these processes will impact the ability of the JCS Service from managing the CCS Services cache upon startup. The JCS will NOT be able to manage the CCS service for a number of minutes until it can resolve this challenge. Unfortunately, when this occurs, the traffic to any CCS managed endpoints will be placed in a long time out within the JCS Service. The IMPS (Provisioning Service) will think that it successfully handed off the task to the JCS/CCS tier, but the task will stay in a holding pattern until either the memory of the JCS is overwhelmed or the CCS Service restarts/crashes or a timeout of the task.

TL;DR – Please do not start the CCS Service manually. Only stop/start the JCS Service, wait a full minute and you should see the CCS Service start up. If the CCS Service does NOT start, investigate why.

JCS Service’s management of the CCS Service:

To understand how the JCS Service manages the CCS Service (via localhost TCP 20402), we can review two (2) files and use MS Sysinternals Process Explorer to view the JCS Service starting the CCS Service via the command “net start im_ccs”. The JCS Service will now have access to update the CCS service’s cache with information for a managed endpoint, e.g. Active Directory.

The two (2) JCS Service configuration files for CCS Service are:

  • C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\server_osgi_ccs.xml [File contains startup properties of how the JCS will manage timeouts to the CCS Service & connections pools]
  • C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\override\server_ccs.properties [File contains the bind credentials and the service port to communicate to on localhost:20402. The password hash will be PBES or AES format depending if FIPS is enabled.]

And finally a view of the startup of the CCS Service via JCS Service using MS Sysinternals Process Explorer https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer We can see that a child process is started from the JCS Service that will call the MS Windows “net.exe” command and execute “net start im_ccs”

Keeping the JCS Service and CCS Service as-is for startup processes will help avoid confusion for the provisioning tier of the CA/Symantec solution. Please only stop/start the JCS Service. If the CCS Service does not stop after 2 minutes, kill it. But never start the CCS by itself.

A view of the data path from IMPS (IM Provisioning Server) to Active Directory (manage endpoint) via the Connector tier.

Performance Improvements

While we may not adjust the startup from manual to automatic, we can enhance the default configurations for performance and timeout improvements. The JCS Service starts up with a default of 1 GB RAM. The JCS Service is 64 bit based on using 64 bit JAVA and the memory can be increased accordingly. After testing with large data sets, we recommend increasing the JCS JVM max memory from 1 GB to 4 GB. We can confirm after startup of the JCS will use over 1 GB of RAM with MS Sysinternals Process Explorer.

Other improvement include updating the JAVA that is supporting the JCS Service. CA/Symantec now recommends using AdoptOpenJDK. The documentation now explains how this may be updated in-place. Or as we prefer to reinstall and allow the installer to update the path statements for AdoptOpenJDK.

The below image below shows in the MS Windows Registry for the JCS Service (Procrun 2.0/im_jcs) the key value pairs that are updated for AdoptOpenJDK. https://adoptopenjdk.net/ If managing Active Directory, please review your OS environmental variables to control the behavior from the CCS Service to Active Directory.

After you restart the JCS Service, open the JCS Administration Console via http://localhost:20080/main or https://localhost:20443/main right click on the “Local Connector Server” ICON and it should display that AdoptOpenJDK is in use now. Only major release 8 is supported, avoid trying later releases (11,15) until support is confirmed.

Stability Improvements

The default JCS Service configuration file has knowledge of the connection pool and timeouts, but appears to be missing the “maxWait” token defined. If we are willing to wait 5-10 minutes for the JCS Service to reset its knowledge of the CCS service, we can leave the default. However for a large environment, we have found that lowering the wait times will greatly avoid the delays in transactions when there is stoppage. We have identified two (2) configuration parameters that will assist with the long term stability of the solution. Adding the “maxWait” of 60 seconds (60000 milliseconds) to the JCS configuration file for CCS service and updating the default IM Provisioning Server domain configuration parameter of “Connections/Refresh Time” to 90 seconds.

Troubleshooting and Logging

To assist with RCA efforts, we have the following recommendations. Enable verbose logging for both the JCS Service and the managed endpoint to isolate issues. You may also need to increase logging for the API Gateway or docker logs.

Below is the example to enable verbose logging.

To monitor the JCS logs, there are several tools that will assist, but we find that the latest releases of Notepad++ allow for “tailing” the active JCS logs.

Example of verbose logs for Active Directory via the CCS’s ADS and JCS logs.

Important Logging Note: Enable the new IM r14.3cp2 feature to auto rotate your CCS ADS log. Avoid stop/start of the CCS Service yourself, that may interrupt the JCS behavior to the CCS Service (error communicating to localhost:20402 will display in JCS logs). New file(s): Connector Server\ccs\data\ADS\<Endpoint_Name>.logconfig

Ref: https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-management-and-governance-connectors/1-0/connectors/microsoft-connectors/microsoft-active-directory-exchange-and-skpye-for-business(lync)/active-directory-connector-capabilities/Active-Directory-Connector-Log-Rotation.html

CCS OS Environmental Variables

Spread throughout the documentation for the CA/Symantec IAM/IAG connector tier, is the use of MS Windows OS environmental variables for the CCS Service. The majority are used to manage behavior to Active Directory and/or MS Exchange. Please search the document for the latest updates. These may be set in MS Window OS via the System Environmental Variable section or via the command line with “setx”. https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-management-and-governance-connectors/1-0/search.html?q=Environment%20Variable&page=1

Example of setting MS Windows OS Environmental variables with “setx” and description of the value of each variable for Active Directory/MS Exchange

1[High Value. Will force AGENTLESS connection to Exchange 2010 & UP]
 setx ADS_AGENTLESS_MODE 1 /m
2[High Value. Default value = 2, Kerberos authentication for Exchange Powershell API]
 setx ADS_AGENTLESS_AUTHMETHOD 2 /m 
3[High Value. Default value = 3. Increase to 100 and ALSO have Exchange Admin create a new quota for the service account used to create mailboxes. Default Exchange Powershell Quota is 18. New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 AND Set-Mailbox ServiceAccountID -ThrottlingPolicy MaxPowershell ]
 setx ADS_AGENTLESS_MAXCONN 100 /m 
4[Monitor. Default value = 1. Error level ONLY, increase to level 3 for debugging powershell logging to MS Exchange]
 setx ADS_AGENTLESS_LOGLEVEL 1 /m 
5[Medium Value. CCS service will wait 10 minutes for single account. Exchange Powershell Mailbox Quota of 18 and BLC with 100’s of users.]
 setx ADS_CONFIRM_MAILBOX 600 /m 
6[Low Value. Mask the AD Failover List in the IM Prov Manager UI]
 setx ADS_DISABLE_DCSTATUS 1 /m 
7[Low Value. Mask the viewing the default AD Primary Group in the IM Prov Manager UI]
 setx ADS_DISABLE_PRIMARYGROUPNAME 1 /m 
8[High Value. Send the DC hostname to the Exchange server to query first instead of Exchange relying on its current pool]
 setx ADS_E2K_SEND_DC 1 /m 
9[High Value. Requires service account can view all alternatives DC. May limit failover DC via properties file.]
 setx ADS_FAILOVER 1 /m 
10[Medium Value. Performance if Terminal Services attribute are NOT being managed, e.g. changed in Account Templates or PX rules.]
 setx ADS_WTS_TIMEOUT -1 /m
11[Set “ADS_OPERATION_TIMEOUT” to -1 to disable the client side timeout functionality in the Environment Variable, otherwise 60]
 setx ADS_OPERATION_TIMEOUT 60 /m
12[The failover retry interval is the time that the Active Directory connector waits before checking the stopped server. The default retry interval is 15 minutes]
 setx ADS_RETRY   15 /m
13[To allow groups in unmanaged domains to be a part of synchronization, Defines whether the synchronization operation searches the global catalog. The value of x can be 0 or 1: 0: (Default) The synchronization operation queries the local catalog only. It does not consider universal groups in unmanaged domains. When x is set to 0, the y value has no effect. 1: Synchronization queries the global catalog to allow it to consider groups in unmanaged domains. y Defines which domains the synchronization operation considers. 0: Synchronization considers groups in both managed and unmanaged domains. 1: Synchronization considers groups in managed domains only.]
 setx ADS_MANAGE_GROUPS 01 /m
14[Monitor. Seems only valuable for debugging. Has performance hit but may assist for CCS debugging to ActiveDirectory.]
 setx ADS_FORCELOG 1 /m 
15[Low Value. The IMPS service can page with lower limits. Impact if this value is > what AD default page limit size is. ]
 setx ADS_SIZELIMIT 50000 /m 

Reinstalling the JCS Service from the Virtual Appliance

If you are using the CA/Symantec Identity Suite virtual appliance, consider after patching the solutions on the virtual appliance, to re-installing the remote JCS Services. This will avoid any confusion on which patches are deployed on the remote JCS servers. Any patches on the virtual appliance will be incorporated into the new installer. We prefer to use the JCS only on the MS Windows OS, as it can service both JCS type managed endpoints & CCS type managed endpoints together. We also have full access to adjust the behavior of these service on MS Windows OS rather than the limited access provided by the virtual appliance for the JCS service.

Hopefully some of these notes will help you avoid any challenges with the connector tier and if you do, how to isolate the issues.

Advance Review: Review how CCS Service receives IMPS data via the JCS Tier.

The below example will load the DLL for the CCS Service (pass-through), then the information to bind to the ADS endpoint will be sent, then two (2) modify operations will be executed. This process emulates the IMPS behavior with the JCS and CCS. The bind information for the ADS endpoint that is stored in the CA Provisioning User Store, and queried/decrypted by the IMPS to send to the JCS as needed. Only after this information is stored in the CCS service, will the solution be able to explore or manage the ADS endpoint accounts.

su - dsa
export HISTIGNORE=' *'
echo -n Password01 > .imps.pwd; chmod 600 .imps.pwd
HOST=192.168.242.154;LDAPTLS_REQCERT=never dxmodify -c -H ldaps://$HOST:20411 -D "cn=root,dc=etasa" -y .imps.pwd << EOF
dn: eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: add
objectClass: eTADSNamespace
eTAgentPluginDLL: W2KNamespace.dll
eTNamespaceName: ActiveDirectory

dn: eTADSDirectoryName=dc2012.exchange2012.lab,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: add
eTADSobjectCategory: CN=Domain-DNS,CN=Schema,CN=Configuration,DC=exchange2012,DC=lab
eTADSdomainFunctionality: 6
eTADSUseSSL: 1
eTLogWindowsEventSeverity: FE
eTAccountResumable: 1
eTADSnetBIOS: EXCHANGE2012
eTLogStdoutSeverity: FE
eTLog: 1
eTLogUnicenterSeverity: FE
eTADSlockoutDuration: -18000000000
objectclass: eTADSDirectory
eTLogETSeverity: FE
eTADSmsExchSystemObjectsObjectVersion: 1
eTADSsettings: 2
eTADSconfig: ExpirePwd=0: HomeDirInheritPermission=0
eTLogDestination: F
eTADSUserContainer: CN=BuiltIn;CN=Users
eTADSbackupDirs: 000;Default-First-Site-Name.Sites.Configuration.exchange2012.lab;dc2012.exchange2012.lab;0
eTADSuseFailover: 0
eTLogAuditSeverity: FE
eTADS-DefaultContext: exchange2012.lab
eTADSforestFunctionality: 6
eTADSAuthDN: Administrator
eTADSlyncMaxConnection: 5
eTADSAuthPWD: Password01!
eTLogFileSeverity: FIESW
eTADSprimaryServer: dc2012.exchange2012.lab
eTADScontainers: CN=Domain-DNS,CN=Schema,CN=Configuration,DC=exchange2012,DC=lab;exchange2012.lab;dc2012.exchange2012.lab
eTADSTimeBoundMembershipsEnabled: 0
eTADSKeepCamCaftFiles: 0
eTADSdomainControllerFunctionality: 6
eTADSexchange: 0
eTADSmsExchSchemaVersion: 1
eTADSCamCaftTimeout: 0000001800
eTADSPortNum: 636
eTADSDCDomain: DC=exchange2012,DC=lab
eTADSServerName: dc2012.exchange2012.lab
eTADSDirectoryName: dc2012.exchange2012.lab

EOF

MS Windows Firewall Rules for JCS Service

NOTE: Ensure MS Win OS F/W Port is open for 20411 on the IAMCS Server

Powershell Example:

Get-NetFirewallRule -Name jcs
New-NetFirewallRule -Name '#### IAMCS JCS TCP 20411 & 20443 #####' -DisplayName '##### IAMCS JCS TCP 20411 & 20443 #####' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 20411,20443

Win Cmd Lin Example:

netsh advfirewall firewall add rule name="##### IAMCS JCS TCP 20411 & 20433 #####" dir=in action=allow protocol=TCP localport="20411,20443"

Install MS .Net Framework 3.5 (Required for the CCS Service & ECS – enterprise common services library framework)

DISM /Online /Enable-Feature /All /FeatureName:NetFx3

Re-install or Uninstall issues

If unable to re-install, please delete the CA install/registry tracking file under C:\Windows folder, C:\Windows\vpd.properties , then reboot before attempting a re-install of the JCS/CCS component.

ECS Services

These five (5) ECS Services are typically not active used & may be changed to manual for minor CPU relief. ECS features are retained for supporting libraries.

Multi-Write HUB Model with democorp

A useful feature with CA Directory for WAN latency challenges is the HUB model. This model allows sync of the data to occur to distant peer Multi-write DATA DSA, but does NOT impact the external application that is updating its own local Router/DATA DSAs.

To assist with understanding this HUB model, we have leverage the CA Directory samples of democorp & router to build out an architecture with six (6) DATA DSAs and two (2) router DSAs, to emulate two (2) data centers across the world. These samples are included with every CA Directory deployment under $DXHOME/samples/democorp & $DXHOME/samples/router.

This lab emulates two (2) of the three (3) data centers that are displayed within the CA documentation.

Ref: https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-identity-and-access-management/directory/14-1/ca-directory-concepts/directory-replication/multiwrite-mw-groups-hubs/topology-sample-and-disaster-recovery.html

This lab may be replicated with near-real-world WAN latency within VMWARE Workstation feature.

https://www.vmware.com/products/workstation-pro.html

Below is a bash shell script used to create the lab environment that was created on a single host OS, with CA Directory samples of “democorp” and “router”. These samples were “copied” and updated via sed commands to ensure the DSA are unique for TCP Ports and naming convention. The below examples’ nomenclature will use democorpX for (A-F) and routerYY (AA or BB) .

These DATA DSA use the same suffix and are all referenced in the group knowledge file. The HUB model configuration will change the behavior for MW-DISP replication between data centers. MW-DISP replication will still be used for all local sync between DATA DSA in the same data center, and would ONLY be used between data centers for the DATA DSA that are designated as “HUB”s (aka multi-write-group-hub).

To test the value of HUB model with WAN latency, suggest that the same lab be executed on two (2) hosts, where one host has the VMWARE network latency enabled to 150/150 milli-seconds. Update the ip addresses within the $DXHOME/config/knowlege/*.dxc on both host OS, to reflect the correct hostnames for each data center in the DSAs.

The diagram below outlines the delta between various democorp DATA DSA to use the HUB model.

The changes below show the deltas within the *.dxc/*.dxg files within the knowledge folder for democorp MW HUB model.

The below image captures the only deltas in the *.dxi (startup files) for the democorp MW HUB model, located within the server folder. Note, if CA Directory management tool is deployed and used for democorp, all configurations will be in a single *.dxi file.

#!/bin/bash
##############################################
#
# Name: democorp_mw_hub_lab.sh
#
# Multi-Write HUB lab using CA Directory and the samples of
# democorp and router under DXHOME/samples
# A. Baugher, 04/2020 - ANA Technology Partner
#
# Assumptions:
#   CA Directory is deployed & dxprofile is enabled for dsa user
#   Execute script as dsa user
#
# Step 0.  Clean-Up prior deployment
#
# Step 1.  Auto deploy both democorp and router samples with: setup.sh -q
#
# Step 2.  Make common changes in democorp prior to copying
#
# Step 3.  Create six (6) copies of democorp and two (2) copies of router
#
# Step 4.  Update the six (6) copies of democorp for:
#     - name
#     - ports
#     - multi-write-group  (HUB group)
#     - DSA flags for MW & HUB-DSA
#     - Group knowledge file reference
#
#        Update the two (2) copies of router for:
#    - name
#    - ports
#    - Group knowledge file reference
#    - set write-precedence  (for HUB-DSA)
#
# Step 5. Start all DSAs
#
# Step 6. Test with dxsearch query
#
# Step 7. Execute the dxsoak command with the service account & time command
#
# Step 8. Update democorpA to force a single delta between peer members of AA and BB
#
# Step 9.  Create LDAP Export
#
# Step 10.  Create LDAP Delta & Compare the various democorp DSA to validate sync operations
#
#
##############################################
#set -xv
echo ..
echo "#############################################################"
echo "Step 0.  Clean up prior deployment of democorp and router"
echo "#############################################################"
dxserver stop all
sleep 5
kill -9 `ps -ef | grep dsa | grep democorp | grep -v grep | grep -v "democorp_mw_hub_lab" | awk '{print $2}'` >   /dev/null 2>&1
kill -9 `ps -ef | grep dsa | grep router   | grep -v grep | awk '{print $2}'` >   /dev/null 2>&1
sleep 5
rm -rf $DXHOME/data/democorp*.*
rm -rf $DXHOME/config/knowledge/democorp*.*
rm -rf $DXHOME/config/knowledge/router*.*
rm -rf $DXHOME/config/servers/democorp*.*
rm -rf $DXHOME/config/servers/router*.*
rm -rf $DXHOME/logs/democorp*.*
rm -rf $DXHOME/logs/router*.*
rm -rf $DXHOME/backup/delta*.*  > /dev/null 2>&1
rm -rf $DXHOME/backup/*.ldif > /dev/null 2>&1


echo ..
echo "#############################################################"
echo "Step 1a. Deploy clean version of democorp and router"
echo "#############################################################"
cd  $DXHOME/samples/democorp
$DXHOME/samples/democorp/setup.sh -q  > /dev/null 2>&1
cd $DXHOME/samples/router
$DXHOME/samples/router/setup.sh -q    > /dev/null 2>&1

cd
echo ..
echo "#############################################################"
echo "Step 1b. Create service ID in democorp for later use"
echo "#############################################################"
cat << EOF > $DXHOME/diradmin.ldif
version: 1
dn: cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU
objectClass: inetOrgPerson
objectClass: organizationalPerson
objectClass: person
objectClass: top
cn: diradmin
sn: diradmin
givenName: diradmin
userPassword: Password01
EOF

dxmodify -a -c -h `hostname` -p 19389 -f $DXHOME/diradmin.ldif

echo ..
echo "#############################################################"
echo "Step 1c.  Stop all running democorp & router DSAs"
echo "#############################################################"
dxserver stop all
sleep 10

echo ..
echo "#############################################################"
echo "Step 2a. Make common changes in pre-existing files before other modification"
echo "Update dsa-flags in democorp.dxc to allow Multi-Write with a HUB"
echo "#############################################################"
sed -i 's|ssl-auth|ssl-auth\n    multi-write-group = hub_group_AA\n     dsa-flags     =|g' $DXHOME/config/knowledge/democorp.dxc
sed -i 's|dsa-flags     =|dsa-flags     = multi-write, no-service-while-recovering, load-share|g' $DXHOME/config/knowledge/democorp.dxc

echo ..
echo "#############################################################"
echo "Step 2b. Update MW recovery in democorp.dxi file"
echo "#############################################################"
sed -i 's|recovery = false;|recovery = true;|g' $DXHOME/config/servers/democorp.dxi

echo ..
echo "#############################################################"
echo "Step 3a. Create six (6) copies of democorp and two (2) routers"
echo "Copy democorp data folder contents"
echo "#############################################################"
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpA.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpA.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpB.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpB.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpC.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpC.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpD.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpD.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpE.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpE.tx  > /dev/null 2>&1
cp -r -p $DXHOME/data/democorp.db $DXHOME/data/democorpF.db
cp -r -p $DXHOME/data/democorp.tx $DXHOME/data/democorpF.tx  > /dev/null 2>&1

echo ..
echo "#############################################################"
echo "Step 3b. Copy autostart folder contents"
echo "#############################################################"
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpA
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpB
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpC
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpD
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpE
cp -r -p $DXHOME/config/autostart/democorp  $DXHOME/config/autostart/democorpF
cp -r -p $DXHOME/config/autostart/router    $DXHOME/config/autostart/routerAA
cp -r -p $DXHOME/config/autostart/router    $DXHOME/config/autostart/routerBB

echo ..
echo "#############################################################"
echo "Step 3c. Copy knowledge folder contents"
echo "#############################################################"
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpA.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpB.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpC.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpD.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpE.dxc
cp -r -p $DXHOME/config/knowledge/democorp.dxc $DXHOME/config/knowledge/democorpF.dxc
cp -r -p $DXHOME/config/knowledge/router.dxc   $DXHOME/config/knowledge/routerAA.dxc
cp -r -p $DXHOME/config/knowledge/router.dxc   $DXHOME/config/knowledge/routerBB.dxc
cp -r -p $DXHOME/config/knowledge/sample.dxg   $DXHOME/config/knowledge/groupAA.dxg
cp -r -p $DXHOME/config/knowledge/sample.dxg   $DXHOME/config/knowledge/groupBB.dxg

echo ..
echo "#############################################################"
echo "Step 3d. Copy server folder contents"
echo "#############################################################"
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpA.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpB.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpC.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpD.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpE.dxi
cp -r -p $DXHOME/config/servers/democorp.dxi   $DXHOME/config/servers/democorpF.dxi
cp -r -p $DXHOME/config/servers/router.dxi     $DXHOME/config/servers/routerAA.dxi
cp -r -p $DXHOME/config/servers/router.dxi     $DXHOME/config/servers/routerBB.dxi

echo ..
echo "#############################################################"
echo "Step 4a.  Update names & ports in democorp knowledge files"
echo "#############################################################"
sed -i 's|19389|29389|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|19390|29390|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPA =|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPA>|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|19389|29489|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|19390|29490|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPB =|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPB>|g' $DXHOME/config/knowledge/democorpB.dxc
sed -i 's|19389|29589|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|19390|29590|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPC =|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPC>|g' $DXHOME/config/knowledge/democorpC.dxc
sed -i 's|19389|29689|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|19390|29690|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPD =|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPD>|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|19389|29789|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|19390|29790|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPE =|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPE>|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|19389|29889|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|19390|29890|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|dsa DEMOCORP =|dsa DEMOCORPF =|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|<c AU><o DEMOCORP><cn DXserver>|<c AU><o DEMOCORP><cn DEMOCORPF>|g' $DXHOME/config/knowledge/democorpF.dxc

echo ..
echo "#############################################################"
echo "Step 4b. Update knowledge files for router ports"
echo "#############################################################"
sed -i 's|19289|39289|g' $DXHOME/config/knowledge/routerAA.dxc
sed -i 's|19290|39290|g' $DXHOME/config/knowledge/routerAA.dxc
sed -i 's|dsa ROUTER =|dsa ROUTERAA =|g' $DXHOME/config/knowledge/routerAA.dxc
sed -i 's|19289|39389|g' $DXHOME/config/knowledge/routerBB.dxc
sed -i 's|19290|39390|g' $DXHOME/config/knowledge/routerBB.dxc
sed -i 's|dsa ROUTER =|dsa ROUTERBB =|g' $DXHOME/config/knowledge/routerBB.dxc

echo ..
echo "#############################################################"
echo "Step 4c. Update group knowledge file for three (3)MW Group HUB Peers "
echo "#############################################################"
sed -i 's|"router.dxc";|"routerAA.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
sed -i 's|"democorp.dxc";|"democorpA.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
sed -i 's|"democorpA.dxc";|"democorpA.dxc";\nsource "democorpB.dxc";\nsource "democorpC.dxc";\nsource "routerBB.dxc";\nsource "democorpD.dxc";\nsource "democorpE.dxc";\nsource "democorpF.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg
sed -i 's|source "unspsc.dxc";|#source "unspsc.dxc";|g' $DXHOME/config/knowledge/groupAA.dxg

cp -r -p $DXHOME/config/knowledge/groupAA.dxg $DXHOME/config/knowledge/groupBB.dxg

#sed -i 's|"router.dxc";|"routerBB.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|"democorp.dxc";|"democorpD.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|"democorpD.dxc";|"democorpD.dxc";\nsource "democorpE.dxc";\nsource "democorpF.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg
#sed -i 's|source "unspsc.dxc";|#source "unspsc.dxc";|g' $DXHOME/config/knowledge/groupBB.dxg

echo ..
echo "#############################################################"
echo "Step 4d.  Update Server folder contents"
echo "#############################################################"
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/democorpA.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/democorpB.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/democorpC.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/democorpD.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/democorpE.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/democorpF.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupAA.dxg";|g' $DXHOME/config/servers/routerAA.dxi
sed -i 's|/knowledge/sample.dxg";|/knowledge/groupBB.dxg";|g' $DXHOME/config/servers/routerBB.dxi


echo ..
echo "#############################################################"
echo "Step 4e.  Update HUB Configurations in DSA knowledge and DSA routers"
echo "#############################################################"
sed -i 's|load-share|load-share, multi-write-group-hub|g' $DXHOME/config/knowledge/democorpA.dxc
sed -i 's|load-share|load-share, multi-write-group-hub|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|multi-write-group = hub_group_AA|multi-write-group = hub_group_BB|g' $DXHOME/config/knowledge/democorpD.dxc
sed -i 's|multi-write-group = hub_group_AA|multi-write-group = hub_group_BB|g' $DXHOME/config/knowledge/democorpE.dxc
sed -i 's|multi-write-group = hub_group_AA|multi-write-group = hub_group_BB|g' $DXHOME/config/knowledge/democorpF.dxc
sed -i 's|/knowledge/groupAA.dxg";|/knowledge/groupAA.dxg";\nset  write-precedence = democorpA ,democorpB, democorpC;\n|g' $DXHOME/config/servers/routerAA.dxi
sed -i 's|/knowledge/groupBB.dxg";|/knowledge/groupBB.dxg";\nset  write-precedence = democorpD ,democorpE, democorpF;\n|g' $DXHOME/config/servers/routerBB.dxi

echo ..
echo "#############################################################"
echo "Step 4f.  Remove samples of router & democorp from starting "
echo "#############################################################"
rm -rf $DXHOME/config/servers/democorp.dxi
rm -rf $DXHOME/config/servers/router.dxi
rm -rf $DXHOME/config/autostart/democorp
rm -rf $DXHOME/config/autostart/router

echo ..
echo "#############################################################"
echo "Step 5. Start all DSAs"
echo "#############################################################"
dxcertgen certs > /dev/null 2>&1
dxserver start all

dxserver status

#exit

echo ..
echo "#############################################################"
echo "Step 6. Test all DSAs with dxsearch query"
echo "#############################################################"
# Comment out if too verbose
# Data DSAs
#dxsearch -h `hostname` -p 29389 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29489 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29589 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29689 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29789 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 29889 -c -x -b o=DEMOCORP,c=AU
# Router DSAs
#dxsearch -h `hostname` -p 39289 -c -x -b o=DEMOCORP,c=AU
#dxsearch -h `hostname` -p 39389 -c -x -b o=DEMOCORP,c=AU

# Data DSAs
#dxsearch -h `hostname` -p 29389 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29489 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29589 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29689 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29789 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 29889 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
# Router DSAs
#dxsearch -h `hostname` -p 39289 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01
#dxsearch -h `hostname` -p 39389 -c -x -b o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01

echo ..
echo "#############################################################"
echo "Step 7. Execute the dxsoak command with the service account & time command"
echo "allow to run for over 5 sec to monitor changes for Multi-Write"
echo "may allow for longer times (1 hour) to get better performance metrics"
echo "#############################################################"
cd $DXHOME/samples/dxsoak
echo "Update democorpA (TCP 29389) to confirm MW to from democorpA (hub_group_AA) to democorpD (hub_group_BB)"
# Create a delete file first; then re-add entries
grep dn: democorp.eldf | grep ,ou=Services > democorp-del.eldf
sed -i 's|,c=AU|,c=AU\nchangetype: del\n|g' democorp-del.eldf

echo ..
echo "#############################################################"
echo "# Delete all DN entries with ou=Services: `wc -l democorp-del.eldf` on democorpA (TCP 29389)"
time ./dxsoak -c -t 2 -q 10 -l 5 -h `hostname`:29389 -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 -f democorp-del.eldf

echo ..
echo "#############################################################"
echo "# Re-Add all DN entries with ou=Services: `wc -l democorp.eldf` on democorpD (TCP 29689)"
time ./dxsoak -c -t 2 -q 10 -l 5 -h `hostname`:29689 -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -w Password01 -f democorp.eldf


echo ..
echo "#############################################################"
echo "Step 8a. Update democorpA to force a single delta between peer members of AA and BB"
echo "#############################################################"
cd
cat << EOF > $DXHOME/diradmin_sn.ldif
dn: cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU
changetype: modify
replace: sn
sn: diradmin_AA_new_update
EOF

echo "#############################################################"
echo "# Query democorpA (TCP 29389) for sn value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29389 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 sn createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for sn value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 sn createTimestamp modifyTimestamp

echo "#############################################################"
echo "# Make update on democorpA"
echo "#############################################################"
dxmodify -a -c -h `hostname` -p 29389 -f $DXHOME/diradmin_sn.ldif

echo "#############################################################"
echo "# Query democorpA (TCP 29389) for sn value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29389 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 sn createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for sn value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 sn createTimestamp modifyTimestamp

#exit
echo ..
echo "#############################################################"
echo "Step 8b. Update democorpF to force a reverse single delta between peer members of AA and BB"
echo "#############################################################"
cd
cat << EOF > $DXHOME/diradmin_givenName.ldif
dn: cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU
changetype: modify
replace: givenName
givenName: diradmin_BB_new_update
EOF


echo "#############################################################"
echo "# Query democorpC (TCP 29589) for givenName value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29589 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 givenName createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for givenName value before change"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 givenName createTimestamp modifyTimestamp

echo "#############################################################"
echo "# Update democorpF to show replication via democorpD (HUB) to democorpA (HUB) "
echo "#############################################################"
dxmodify -a -c -h `hostname` -p 29889 -f $DXHOME/diradmin_givenName.ldif

echo "#############################################################"
echo "# Query democorpC (TCP 29589) for givenName value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29589 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 givenName createTimestamp modifyTimestamp
echo "#############################################################"
echo "# Query democorpF (TCP 29889) for givenName value after change"
echo " - May catch a fractional delta in replication"
echo "#############################################################"
dxsearch -LLL -h `hostname` -p 29889 -c -x -b cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU -D cn=diradmin,ou=Networks,ou=Support,o=DEMOCORP,c=AU  -w Password01 givenName createTimestamp modifyTimestamp




echo ..
echo "###########################################################"
echo "Step 9b. Update CA Directory DSA to allow online backup ###"
echo "###########################################################"
echo " - Configure CA Directory to provide an data dump (zdb file) while DSA are online"
cp -r -p $DXHOME/config/settings/default.dxc.org $DXHOME/config/settings/default.dxc  > /dev/null 2>&1
cp -r -p $DXHOME/config/settings/default.dxc $DXHOME/config/settings/default.dxc.org  > /dev/null 2>&1
# Edit the DSA settings file to add in one line.  dump dxgrid-db;
chmod 744 $DXHOME/config/settings/default.dxc
echo "dump dxgrid-db;" >> $DXHOME/config/settings/default.dxc



echo ..
echo "######################################################################################"
echo "Step 9c. Re-init all DSA to data dump the CA DSAs for democorp & router "
echo "######################################################################################"
echo " - This make take 5-30 seconds to complete "
dxserver init all    > /dev/null 2>&1
# View for zdb or zd? (in-progress) files
sleep 10



echo ..
echo "#################################################################"
echo "Step 9d. Export DSA backup/offline zdb data files to LDIF file ###"
echo "#################################################################"
echo " - Export will happen after the backup/offline zdb files are fully created"
echo " - This make take 5-60 seconds  to complete "
echo ..
echo "#################################################################"
echo "Step 9e. Set WHILE loop for DemocorpF DSA ###"
echo "#################################################################"
until [ -f $DXHOME/data/democorpF.zdb ]
do
     echo " - Waiting till CA Directory has completed online data dump of DemocorpF DSA"
     sleep 5
done
sleep 5
echo ..
echo "#################################################################"
echo "Step 9f. Execute dxdumbdb for Democorp DSA - FULL ###"
echo "#################################################################"
mkdir $DXHOME/backup  > /dev/null 2>&1
cd $DXHOME/backup
dxdumpdb -z -f $DXHOME/backup/democorpA.ldif democorpA   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpB.ldif democorpB   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpC.ldif democorpC   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpD.ldif democorpD   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpE.ldif democorpE   > /dev/null 2>&1
dxdumpdb -z -f $DXHOME/backup/democorpF.ldif democorpF   > /dev/null 2>&1
sleep 5

echo ..
echo "#################################################################"
echo "Step 10a. Perform LDIF DELTA compare between democorpA and democorpB within same HUB MW group"
echo "Look for any delta in the metrics > 0"
echo "#################################################################"
#ldifdelta -x -S DSANAME  OLDFILE NEWFILE DELTAFILE
ldifdelta -x -S democorpA $DXHOME/backup/democorpA.ldif  $DXHOME/backup/democorpB.ldif $DXHOME/backup/delta-between-A-and-B.ldif
echo "#################################################################"
echo "Step 10b. Perform LDIF DELTA compare between democorpD and democorpE within same HUB MW group"
echo "Look for any delta in the metrics > 0"
echo "#################################################################"
ldifdelta -x -S democorpC $DXHOME/backup/democorpD.ldif  $DXHOME/backup/democorpE.ldif $DXHOME/backup/delta-between-D-and-E.ldif
echo "#################################################################"
echo "Step 10c. Perform LDIF DELTA compare between democorpC and democorpF across different HUB MW groups"
echo "Look for any delta in the metrics > 0"
echo "#################################################################"
ldifdelta -x -S democorpC $DXHOME/backup/democorpC.ldif  $DXHOME/backup/democorpF.ldif $DXHOME/backup/delta-between-C-and-F.ldif

echo .
echo .




Ref: This HUB Model lab was built off a prior lab for MW Sync with air-gap requirements.

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?MessageKey=62ccc41d-7c37-4728-ad1e-c82e7a8acc38&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

Load Balancing Provisioning Tier

The prior releases of CA Identity Manager / Identity Suite have a bottleneck with the provisioning tier.

The top tier of the solution stack, Identity Manager Environment (IME/J2EE Application), may communicate to multiple Provisioning Servers (IMPS), but this configuration only has value for fail-over high availability.

This default deployment means we will have a “many-to-one” challenge, multiple IMEs experiencing a bottleneck with provisioning communication to a single IMPS server.

If this IMPS server is busy, then transactions for one or more IMEs are paused or may timeout. Unfortunately, the IME (J2EE) error messages or delays are not clear that this is a provisioning bottleneck challenge. Clients may attempt to resolve this challenge by increasing the number of IME and IMPS servers but will still be impacted by the provisioning bottleneck.

Two (2) prior methods used to overcome this bottleneck challenge were:


a) Pseudo hostname(s) entries, on the J2EE servers, for the Provisioning Tier, then rotate the order pseudo hostname(s) on the local J2EE host file to have their IP addresses access other IMPS. This methodology would give us a 1:1 configuration where one (1) IME is now locked to one (1) IMPS (by the pseudo hostname/IP address). This method is not perfect but ensures that all IMPS servers will be utilized if the number of IMPS servers equals IME (J2EE) servers. Noteworthy, this method is used by the CA identity Suite virtual appliance, where the pseudo hostname(s) are ca-prov-srv-01, ca-prov-srv-02, ca-prov-03, etc. (see image above)

<Connection
  host="ca-prov-srv-primary" port="20390"
  failover="ca-prov-srv-01:20390,ca-prov-srv-02:20390,ca-prov-srv-03:20390,ca-prov-srv-04:20390“
/>

b) A Router placed in-front of the IMPS (TCP 20389/20390), that contains “stickiness” to ensure that when round-robin model is used, that the same IMPS server is used for the IME that submitted a transaction, to avoid any concerns/challenges of possible”RACE” conditions, where a modify operations may occur before the create operation.


The “RACE” challenge is a concern of both of the methods above, but this risk is low, and can be managed with additional business rules that include pre-conditional checks, e.g., does the account exist before any modifications.

Ref: RACE https://en.wikipedia.org/wiki/Race_condition

Example of one type of RACE condition that may be seen.

Ref: PX Rule Engine: https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-identity-and-access-management/identity-manager/14-3/Release-Notes/Cumulative-Patches/Latest-Cumulative-Patch-14_3-CP2.html

New CP2 Loading Balance Feature – No more bottleneck.

Identity Manager can now use round-robin load balancing support, without any restrictions on either type of provisioning operations or existing runtime limitations. This load balancing approach distributes client requests across a group of Provisioning servers.

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-identity-and-access-management/identity-manager/14-3/Release-Notes/release-features-and-enhancement/Identity-Manager-14_3-CP2.html#concept.dita_b51ab03e-6e77-49be-8235-e50ee477247a_LoadBalancing

This feature is managed in the IME tier, and will also address any RACE conditions/concerns.


No configuration changes are required on the IMPS tier. After updates of CP2, we can now use the IME Management console to export the directory.xml for the IMPS servers and update the XML tag for <Connection. This configuration may also be deployed to the Virtual Appliances.

<Connection   
  host="ca-prov-srv-primary" port="20390”   
  loadbalance="ca-prov-srv-02:20390,ca-prov-srv-03:20390,ca-prov-srv-04:20390“   
  failover="ca-prov-srv-01:20390,ca-prov-srv-02:20390,ca-prov-srv-03:20390,ca-prov-srv-04:20390“ 
/>

View of CP2 to download.

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-identity-and-access-management/identity-manager/14-3/Release-Notes/Cumulative-Patches/Latest-Cumulative-Patch-14_3-CP2.html

Before applying this patch, we recommend collecting your metrics for feed operations that include multiple create operations, and modify operations to minimal of 1000 IDS, Monitor the IMPS etatrans logs as well; and the JCS/CCS logs. After the patch, run the same feed operations to determine the value of provisioning load-balance feature; and any provisioning delays that have been addressed. You may wish to increase the # of JCS/CCS servers (MS Windows) to speed up provisioning to Active Directory and other endpoints.