One of the challenges that IAM/IAG solutions may have is using single thread processing for select endpoints. For the CA/Symantec Identity Management solution, before IM r14.3cp2, we lived with a single-threaded connector to managed MS Active Directory endpoints.
To address this challenge, we deployed multiple connector servers. We allowed the IM Provisioning Server (IMPS) to use a built-in round-robin approach of load-balancing separate transactions to different connector servers, which would service the same Active Directory endpoints.
The IME may be running as fast as it can with its clustered deployment, but as soon as a task has MS Active Directory, and there is a bottleneck with the CCS Service. We begin to see the IME JMS queue reporting that it is stuck and the IME View Submitted Task reporting “In Progress” for all tasks. If the CCS service is restarted, all IME tasks are then reported as “Failed.”
This is/was the bottleneck for the solution for sites that have MS Active Directory for Birthright/DayOne Access.
We can now avoid this bottleneck. [*** (5/24/2021) – There is an enhancement to CP2 to address im_ccs.exe crashes during peak loads discovered using this testing process. ]
Via the newly delivered enhancement https://community.broadcom.com/participate/ideation-home/viewidea?IdeationKey=7154e15b-085d-469e-bff0-ac588ff6bd5b .
We now have full parallel provisioning to MS Active Directory from a single connector server (JCS/CCS).

The new attribute that regulates this behavior is eTADSMaxConnectionsInPool. This attribute will be applied on every existing ADS endpoint that is currently being managed by the IM Provisioning Server after CP2 is deployed. Note: The default value is 10, but we recommend after much testing, to match the value of the IMPS-> JCS and JCS->CCS to equal 200.

During testing within the IME using Bulk Tasks or the IM BLC, we can see that the CCS-> ADS traffic will reach 20-30 connections if allowed. You may set this attribute to a value of 200 via Jxplorer and/or an ldapmodify/dxmodify script.
echo "############### SET ADS MAX CONNECTIONS IN POOL SIZE ##################" IMPS_HOST=192.168.242.135 IMPS_PORT=20389 IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta' IMPS_PWD="Password01"
NAMESPACE=exchange2016LDAPTLS_REQCERT=never dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD" << EOF dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta changetype: modify eTADSMaxConnectionsInPool: 200 EOF LDAPTLS_REQCERT=never dxsearch -LLL -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'
To confirm the number of open connections is greater than one (1), we can issue a Bulk IM Task or use a performance tool like CA Directory dxsoak.
In this example, we will show case using CA Directory dxsoak to execute 100 parallel threads to create 100 ADS Accounts with MS Exchange Mailboxes. We will also enclose this script for download for others to review and use.
Performance Lab:
Pre-Steps:
- Leverage CA Directory samples’ dxsoak binary (performance testing). You may wish to use CA Directory on an existing IM Provisioning Server (Linux OS) or you may deploy CA Directory (MS Windows version) to the JCS/CCS connector. Examples are provided for both OSes.
- Create LDIF files for IM Provisioning Server and/or IM Connector Tier. This file is needed to ‘push’ the solution to-failure. The use of the IME Bulk Task and/or etautil scripts to the IM Provisioning Tier, will not provide the transaction speed we need to break the CCS service if possible.
- Within the IM Provisioning Manager enable the ADS Endpoint TXT Logs on the Logging TAB, for all checkboxes.
- Monitor the IMPS etatrans* logs, monitor the JCS ADS logs, monitor the CCS ADS logs, monitor the number of CCS-> ADS (LDAP/S – TCP 389/636) threads. [Suggest using MS Sysinternals Process Explorer and select im_ccs.exe & then TCP/IP TAB]
- Monitor the MS ADS Domain via MS ADUC (AD Users & Computers UI) and MS Exchange Mailbox (Mailbox UI via Browser)
Execution:
6. Perform a UNIT TEST with dxmodify/ldapmodify to confirm the LDIF file input is correct with the correct suffix.
time dxmodify -H ldap://192.168.242.135:20389 -c -x -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldif


7. Perform the PERFORMANCE TEST with dxsoak binary with the same LDIF file & correct suffix. Rate observed = 23 K ids/hr
./dxsoak -c -l 60 -t 100 -h 192.168.242.135:20389 -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldi
f


Observations:
8. IMPS etatrans*.log – Count the number of operations per second. Note any RACE and/or data collisions, e.g. ADS accounts deleted prior to add via 100 threads or ADS account created multiple times attempted in different threads.

9. IM CCS ADS <endpoint>.log – Will only have useful data if the ADS Endpoint Logging TAB has been checked for TXT logs.

10. Finally, validate directly in MS Active Domain with the ADUC or similar tool & MS Exchange mailboxes being created/deleted.

11. Count the number of threads from im_ccs.exe to ADS – Suggest using MS Sysinternals Process Explorer tool and/or Powershell to count the number of connections.

MS Powershell Script to count the number of LDAP (TCP 389) connection from im_ccs.exe. [Note: TCP 389 is used more if the ADS Endpoint is setup to use SASL authentication. TCP 636 is used more if the ADS Endpoint is using the older TLS authentication]
$i=1
Do {
cls
(Get-NetTCPConnection -State Established -OwningProcess (Get-Process -name im_ccs).id -RemotePort 389).count
Start-Sleep -s 1
$i++
}
while ($i -le 5)
Direct Performance Testing to JCS/CCS Service
While this testing has limited value, it can offer satisfaction and assistance to troubleshoot any challenges. We can use the prior LDIF files with a slightly different suffix, dc=etasa (instead of dc=eta), to use dxsoak to push the connector tier to failure. This step helped provide memory dumps back to CA/Symantec Engineering teams to help isolate challenges within the parallel processing. CCS Service is only exposed via localhost. If you wish to test the CCS Service remotely, then update the MS Registry key for the CCS service to use the external IP address of the JCS/CCS Server. Rate observed = 25 K ids/hr

Script to generate 100 ADS Accounts with MS Exchange Mailbox Creation
You may wish to review this script and adjust it for your ADS / MS Exchange domains for testing. You can also create a simple LDIF file with password resets or ADS group membership adds. Just remember that the IMPS Service (TCP 20389/20390) uses the suffix dc=eta, and the IM JCS/CCS Services (TCP 20410/20411) & (TCP 20402/20403) use the suffix dc=etasa. Additionally, if using CA Directory dxsoak, only use the non-TLS ports, as this binary is not equipped for using TLS certs.
#!/bin/bash ####################################################################################################################### # Name: Generate ADS Feed Files for IM Solution Provisioning/Connector Tiers # # Goal: Validate the new parallel processes from the IM Connector Tier to Active Directory with MS Exchange # # # Generate ADS User LDIF file(s) for use with unit (dxmodify) and performance testing (dxsoak) to: # - {Note: dxsoak will only work with non-TLS ports} # # IM JCS (20410) "dc=etasa" {Ensure MS Windows Firewall allows this port to be exposed} # IM CCS (20402) "dc=etasa" {This port is localhost only, may open to network traffic via registry update} # IMPS (20389) "dc=eta" # # # Monitor: # # The IMPS etatrans*.log {exclude searches} # The JCS daily log # The JCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes} # The CCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes} # # Execute per the examples provided during run of this file # # # ANA 05/2021 ####################################################################################################################### # Unique Variables for an ADS Domain NAMESPACE=exchange2016 ADSDOMAIN=exchange.lab DCDOMAIN="DC=exchange,DC=lab" OU=People ####################################################################################################################### MAX=100 start=00001 counter=$start echo "###############################################################" echo "###############################################################" START=`/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z` echo `/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`" = Current OS UTC time stamp" echo "###############################################################" FILE1=ads_user_with_exchange_dc_etasa.ldif FILE2=ads_user_with_exchange_dc_eta.ldif echo "" > $FILE1 while [ $counter -le $MAX ] do n=$((10000+counter)); n=${n#1} tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z` echo "Counter with leading zeros = $n at time: $tz" cat << EOF >> $FILE1 dn: eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa changetype: add objectClass: eTADSAccount eTADSobjectClass: user eTADSAccountName: firstname$n aaalastname$n eTADSgivenName: firstname$n eTADSsn: aaalastname$n eTADSdisplayName: firstname$n aaalastname$n eTADSuserPrincipalName: aaatestuser$n@$ADSDOMAIN eTADSsAMAccountName: aaatestuser$n eTPassword: Password01 eTADSpwdLastSet: -1 eTSuspended: 0 eTADSuserAccountControl: 0000000512 eTADSDescription: description $tz eTADSphysicalDeliveryOfficeName: office eTADStelephoneNumber: 111-222-3333 eTADSmail: aaatestuser$n@$ADSDOMAIN eTADSwwwHomePage: web.page.lab eTADSotherTelephone: 111-222-3333 eTADSurl: other.web.page.lab eTADSstreetAddress: street address line01 eTADSpostOfficeBox: pobox 111 eTADSl: city eTADSst: state eTADSpostalCode: 11111 eTADSco: UNITED STATES eTADSc: US eTADScountryCode: 840 eTADSscriptPath: loginscript.cmd eTADSprofilePath: \profile\path\here eTADShomePhone: 111-222-3333 eTADSpager: 111-222-3333 eTADSmobile: 111-222-3333 eTADSfacsimileTelephoneNumber: 111-222-3333 eTADSipPhone: 111-222-3333 eTADSinfo: Notes Here eTADSotherHomePhone: 111-222-3333 eTADSotherPager: 111-222-3333 eTADSotherMobile: 111-222-3333 eTADSotherFacsimileTelephoneNumber: 111-222-3333 eTADSotherIpPhone: 111-222-3333 eTADStitle: title eTADSdepartment: department eTADScompany: company eTADSmanager: CN=manager_fn manager_ln,OU=$OU,$DCDOMAIN eTADSmemberOf: CN=Backup Operators,CN=Builtin,$DCDOMAIN eTADSlyncSIPAddressOption: 0000000000 eTADSdisplayNamePrintable: aaatestuser$n eTADSmailNickname: aaatestuser$n eTADShomeMDB: (Automatic Mailbox Distribution) eTADShomeMTA: CN=DC001,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,$DCDOMAIN eTAccountStatus: A eTADSmsExchRecipientTypeDetails: 0000000001 eTADSmDBUseDefaults: TRUE eTADSinitials: A eTADSaccountExpires: 9223372036854775807 EOF counter=$(( $counter + 00001 )) done # Create the delete ADS Process start=00001 counter=$start while [ $counter -le $MAX ] do n=$((10000+counter)); n=${n#1} tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z` echo "Counter with leading zeros = $n at time: $tz" cat << EOF >> $FILE1 dn: eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa changetype: delete EOF counter=$(( $counter + 00001 )) done echo "" echo "################################### ADS USER OBJECT STATS ################################################################" echo "Number of add objects: `grep "changetype: add" $FILE1 | wc -l`" echo "Number of delete objects: `grep "changetype: delete" $FILE1 | wc -l`" rm -rf $FILE2 cp -r -p $FILE1 $FILE2 sed -i 's|,dc=im,dc=etasa|,dc=im,dc=eta|g' $FILE2 ls -lart $FILE1 ls -lart $FILE2 echo "" echo "################################### SET ADS MAX CONNECTIONS IN POOL SIZE ################################################################" IMPS_HOST=192.168.242.135 IMPS_PORT=20389 IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta' IMPS_PWD="Password01" LDAPTLS_REQCERT=never dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD" << EOF dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta changetype: modify eTADSMaxConnectionsInPool: 200 EOF LDAPTLS_REQCERT=never dxsearch -LLL -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g' echo "" echo "################################### CCS UNIT & PERF TEST ################################################################" CCS_HOST=192.168.242.80 CCS_PORT=20402 CCS_USER="cn=root,dc=etasa" CCS_PWD="Password01" echo "Execute this command to the CCS Service to test single thread with dxmodify or ldapmodify" echo "dxmodify -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 " echo "Execute this command to the CCS Service to test 100 threads with dxsoak " echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 " echo "" echo "################################### JCS UNIT & PERF TEST ################################################################" CCS_HOST=192.168.242.80 CCS_PORT=20410 CCS_USER="cn=root,dc=etasa" CCS_PWD="Password01" echo "Execute this command to the JCS Service to test single thread with dxmodify or ldapmodify " echo "dxmodify -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 " echo "Execute this command to the JCS Service to test 100 threads with dxsoak " echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 " echo "" echo "################################### IMPS UNIT & PERF TEST ################################################################" IMPS_HOST=192.168.242.135 IMPS_PORT=20389 IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta' IMPS_PWD="Password01" echo "Execute this command to the IMPS Service to test single thread with dxmodify or ldapmodify " echo "dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 " echo "Execute this command to the IMPS Service to test 100 threads with dxsoak " echo "./dxsoak -c -l 60 -t 100 -h $IMPS_HOST:$IMPS_PORT -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "
Address the new bottleneck of MS Exchange / O365 Provisioning.
After parallel provisioning has been introduced with the new im_ccs.exe service, you may noticed that the number of transactions is still being throttled during performance testing.
Out-of-the-box MS Active Directory Global Throttling Policy has the parameter of PowerShellMaxConcurrency set to a default of 18 connection. Any provisioning that uses MS Powershell for MS Exchange and/or MS O365 will be impacted by this default parameter.
To address this bottleneck, we can create a new Throttling Policy and only assign the service ID that will be managing identities, to avoid a global change.
Example: New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 & Set-Mailbox “User Name” -ThrottlingPolicy MaxPowershell


After this change has been made, restart the IM JCS/CCS Services, and retest again with your performance tools. Review the CCS ADS log for # of creations in 60 seconds, and you will be pleasantly surprise at the rate. The logs are the strong confirmation we are looking for.
Performance test (947 ADS accounts w/Exchange mailboxes in 60 seconds, 08:59:54 to 09:00:53) => Rate of 15 ids/second (or 54 K ids/hr) with updated MaxPowershell = 100 thottlingpolicy.


The last bottleneck appears to be CPU availability to MS Exchange Supporting Services, w3wp.exe, the MS IIS Service. Which appears to be managing MS Powershell connections per its startup string of
" c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangePowerShellAppPool" -v "v4.0" -c "C:\Program Files\Microsoft\Exchange Server\V15\bin\GenericAppPoolConfigWithGCServerEnabledFalse.config" -a \.\pipe\iisipme304c50e-6b42-4b26-83a4-229ee037be5d -h "C:\inetpub\temp\apppools\MSExchangePowerShellAppPool\MSExchangePowerShellAppPool.config" -w "" -m 0"
