On Linux OS, there are two (2) device drivers that provide entropy “noise” for components that require encryption, e.g. the /dev/random and the /dev/urandom device drivers. The /dev/random is a “blocking” device driver. When the “noise” is low, any component that relies on this driver will be “stalled” until enough entropy is returned. We can measure the entropy from a range of 0-4096. Where a value over 1000 is excellent. Any value in the double or single digits will impact the performance of the OS and solutions with delays. The root cause of these delays is not evident during troubleshooting, and typically there are no warning nor error messages related to entropy.
The Symantec Identity Suite solution, when deployed on Linux OS is typically deployed with the JVM switch -Djava.security.egd=file:/dev/./urandom for any component that uses Java (Oracle or AdoptOpenJDK), e.g. Wildfly (IM/IG/IP) and IAMCS (JCS). This JVM variable is sufficient for most use-cases to manage the encryption/hash needs of the solution.
However, for any component that does not provide a mechanism to use the alternative of /dev/urandom driver, the Linux OS vendors offer tools such as the “rng-tools” package. We can review what OS RNGD service is available using package tools, e.g.
dnf list installed | grep -i rng
If the Symantec Identity Suite or other solutions are deployed as standalone components, then we may adjust the Linux OS as we need with no restrictions to add the RNGD daemon as we wish. One favorite is the HAVEGED daemon over the default OS RNGD.
See prior notes on value and testing for Entropy on Linux OS (standalone deployments):
The challenge for Virtual Appliances is that we are limited to what functionality the Symantec Product Team provides for us to leverage. The RNGD service was available on the vApp r14.3, but was disabled for OS challenges with 100% utilization with CentOS 6.4. The service is still installed, but the actual binary is non-executable.
A new Virtual Appliance patch would be required to re-enable this RNGD on vApp r14.3cp2. We have access via sudo, to /sbin/chkconfig, /sbin/service to re-enable this service, but as the binary is not executable, we cannot progress any further. We can see the alias in the documentation still exist, but the OS alias was removed in the cp2 update.
However, since vApp r14.4 was release, we can focus on this Virtual Appliance which is running Centos 8 stream. The RNGD service here is disabled (masked) but can be re-enabled for our use with the sudo command. There is no current documented method for RNGD on vApp r14.4 at this time, but the steps below will show an approved way using the ‘config’ userID and sudo commands.
Confirm that the “rng-tools” package is installed and that the RNGD binary is executable. We can also see that the RNGD service is “masked”. Masked services are prevented from starting manually or automatically as an extra safety measure when we wish for tighter control over our systems.
If we test OS entropy for this vApp r14.4 server without RNGD, we can monitor how a simple BASH shell script that emulates a password being generated will impact the “entropy” of /dev/random. The below script will reduce the entropy to low numbers. This process will now impact the OS itself and any components that reference /dev/random. We can observe with “lsof /dev/random” that the java programs will still reference /dev/random; even though most activity is going to /dev/urandom.
Using the time command in the BASH shell script, we can see that the response is rapid for the first 20+ iterations, but as soon as the entropy is depleted, each execution is delayed by 10-30x times.
Now let’s see what RNGD service will do for us when it is enabled. Let’s follow the steps below to unmask, enable, and start the RNGD service as the ‘config’ userID. We have access to sudo to the Centos 8 Stream command of /sbin/systemctl.
After the RNGD service is enabled, test again with the same prior BASH shell script but bump the loops to 1000 or higher. Note using the time command we can see that each loop finishes within a fraction of a second.
Aim to keep the solution footprint small and the right-sized to solve the business’ needs. Do not accept the default performance; avoid over-purchasing to scale to your expected growth.
Use the JVM switch wherever there is a java process, e.g. BLC or home-grown ETL (extract-transform-load) processes.
-Djava.security.egd=file:/dev/./urandom
If you suspect a dependence may impact the OS or other processes on /dev/random, then enable the OS RNGD and perform your testing. Monitor with the top command to ensure RNGD service is providing value and not impacting the solution.
One of the challenges that IAM/IAG solutions may have is using single thread processing for select endpoints. For the CA/Symantec Identity Management solution, before IM r14.3cp2, we lived with a single-threaded connector to managed MS Active Directory endpoints.
To address this challenge, we deployed multiple connector servers. We allowed the IM Provisioning Server (IMPS) to use a built-in round-robin approach of load-balancing separate transactions to different connector servers, which would service the same Active Directory endpoints.
The IME may be running as fast as it can with its clustered deployment, but as soon as a task has MS Active Directory, and there is a bottleneck with the CCS Service. We begin to see the IME JMS queue reporting that it is stuck and the IME View Submitted Task reporting “In Progress” for all tasks. If the CCS service is restarted, all IME tasks are then reported as “Failed.”
This is/was the bottleneck for the solution for sites that have MS Active Directory for Birthright/DayOne Access.
We can now avoid this bottleneck. [*** (5/24/2021) – There is an enhancement to CP2 to address im_ccs.exe crashes during peak loads discovered using this testing process. ]
We now have full parallel provisioning to MS Active Directory from a single connector server (JCS/CCS).
The new attribute that regulates this behavior is eTADSMaxConnectionsInPool. This attribute will be applied on every existing ADS endpoint that is currently being managed by the IM Provisioning Server after CP2 is deployed. Note: The default value is 10, but we recommend after much testing, to match the value of the IMPS-> JCS and JCS->CCS to equal 200.
During testing within the IME using Bulk Tasks or the IM BLC, we can see that the CCS-> ADS traffic will reach20-30connections if allowed. You may set this attribute to a value of 200 via Jxplorer and/or an ldapmodify/dxmodify script.
To confirm the number of open connections is greater than one (1), we can issue a Bulk IM Task or use a performance tool like CA Directory dxsoak.
In this example, we will show case using CA Directory dxsoak to execute 100 parallel threads to create 100 ADS Accounts with MS Exchange Mailboxes. We will also enclose this script for download for others to review and use.
Performance Lab:
Pre-Steps:
Leverage CA Directory samples’ dxsoak binary (performance testing). You may wish to use CA Directory on an existing IM Provisioning Server (Linux OS) or you may deploy CA Directory (MS Windows version) to the JCS/CCS connector. Examples are provided for both OSes.
Create LDIF files for IM Provisioning Server and/or IM Connector Tier. This file is needed to ‘push’ the solution to-failure. The use of the IME Bulk Task and/or etautil scripts to the IM Provisioning Tier, will not provide the transaction speed we need to break the CCS service if possible.
Within the IM Provisioning Manager enable the ADS Endpoint TXT Logs on the Logging TAB, for all checkboxes.
Monitor the IMPS etatrans* logs, monitor the JCS ADS logs, monitor the CCS ADS logs, monitor the number of CCS-> ADS (LDAP/S – TCP 389/636) threads. [Suggest using MS Sysinternals Process Explorer and select im_ccs.exe & then TCP/IP TAB]
Monitor the MS ADS Domain via MS ADUC (AD Users & Computers UI) and MS Exchange Mailbox (Mailbox UI via Browser)
Execution:
6. Perform a UNIT TEST with dxmodify/ldapmodify to confirm the LDIF file input is correct with the correct suffix.
8. IMPS etatrans*.log – Count the number of operations per second. Note any RACE and/or data collisions, e.g. ADS accounts deleted prior to add via 100 threads or ADS account created multiple times attempted in different threads.
9. IM CCS ADS <endpoint>.log – Will only have useful data if the ADS Endpoint Logging TAB has been checked for TXT logs.
10. Finally, validate directly in MS Active Domain with the ADUC or similar tool & MS Exchange mailboxes being created/deleted.
11. Count the number of threads from im_ccs.exe to ADS – Suggest using MS Sysinternals Process Explorer tool and/or Powershell to count the number of connections.
MS Powershell Script to count the number of LDAP (TCP 389) connection from im_ccs.exe. [Note: TCP 389 is used more if the ADS Endpoint is setup to use SASL authentication. TCP 636 is used more if the ADS Endpoint is using the older TLS authentication]
$i=1
Do {
cls
(Get-NetTCPConnection -State Established -OwningProcess (Get-Process -name im_ccs).id -RemotePort 389).count
Start-Sleep -s 1
$i++
}
while ($i -le 5)
Direct Performance Testing to JCS/CCS Service
While this testing has limited value, it can offer satisfaction and assistance to troubleshoot any challenges. We can use the prior LDIF files with a slightly different suffix, dc=etasa (instead of dc=eta), to use dxsoak to push the connector tier to failure. This step helped provide memory dumps back to CA/Symantec Engineering teams to help isolate challenges within the parallel processing. CCS Service is only exposed via localhost. If you wish to test the CCS Service remotely, then update the MS Registry key for the CCS service to use the external IP address of the JCS/CCS Server. Rate observed = 25 K ids/hr
Script to generate 100 ADS Accounts with MS Exchange Mailbox Creation
You may wish to review this script and adjust it for your ADS / MS Exchange domains for testing. You can also create a simple LDIF file with password resets or ADS group membership adds. Just remember that the IMPS Service (TCP 20389/20390) uses the suffix dc=eta, and the IM JCS/CCS Services (TCP 20410/20411) & (TCP 20402/20403) use the suffix dc=etasa. Additionally, if using CA Directory dxsoak, only use the non-TLS ports, as this binary is not equipped for using TLS certs.
#!/bin/bash
#######################################################################################################################
# Name: Generate ADS Feed Files for IM Solution Provisioning/Connector Tiers
#
# Goal: Validate the new parallel processes from the IM Connector Tier to Active Directory with MS Exchange
#
#
# Generate ADS User LDIF file(s) for use with unit (dxmodify) and performance testing (dxsoak) to:
# - {Note: dxsoak will only work with non-TLS ports}
#
# IM JCS (20410) "dc=etasa" {Ensure MS Windows Firewall allows this port to be exposed}
# IM CCS (20402) "dc=etasa" {This port is localhost only, may open to network traffic via registry update}
# IMPS (20389) "dc=eta"
#
#
# Monitor:
#
# The IMPS etatrans*.log {exclude searches}
# The JCS daily log
# The JCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
# The CCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
#
# Execute per the examples provided during run of this file
#
#
# ANA 05/2021
#######################################################################################################################
# Unique Variables for an ADS Domain
NAMESPACE=exchange2016
ADSDOMAIN=exchange.lab
DCDOMAIN="DC=exchange,DC=lab"
OU=People
#######################################################################################################################
MAX=100
start=00001
counter=$start
echo "###############################################################"
echo "###############################################################"
START=`/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`
echo `/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`" = Current OS UTC time stamp"
echo "###############################################################"
FILE1=ads_user_with_exchange_dc_etasa.ldif
FILE2=ads_user_with_exchange_dc_eta.ldif
echo "" > $FILE1
while [ $counter -le $MAX ]
do
n=$((10000+counter)); n=${n#1}
tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
echo "Counter with leading zeros = $n at time: $tz"
cat << EOF >> $FILE1
dn: eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: add
objectClass: eTADSAccount
eTADSobjectClass: user
eTADSAccountName: firstname$n aaalastname$n
eTADSgivenName: firstname$n
eTADSsn: aaalastname$n
eTADSdisplayName: firstname$n aaalastname$n
eTADSuserPrincipalName: aaatestuser$n@$ADSDOMAIN
eTADSsAMAccountName: aaatestuser$n
eTPassword: Password01
eTADSpwdLastSet: -1
eTSuspended: 0
eTADSuserAccountControl: 0000000512
eTADSDescription: description $tz
eTADSphysicalDeliveryOfficeName: office
eTADStelephoneNumber: 111-222-3333
eTADSmail: aaatestuser$n@$ADSDOMAIN
eTADSwwwHomePage: web.page.lab
eTADSotherTelephone: 111-222-3333
eTADSurl: other.web.page.lab
eTADSstreetAddress: street address line01
eTADSpostOfficeBox: pobox 111
eTADSl: city
eTADSst: state
eTADSpostalCode: 11111
eTADSco: UNITED STATES
eTADSc: US
eTADScountryCode: 840
eTADSscriptPath: loginscript.cmd
eTADSprofilePath: \profile\path\here
eTADShomePhone: 111-222-3333
eTADSpager: 111-222-3333
eTADSmobile: 111-222-3333
eTADSfacsimileTelephoneNumber: 111-222-3333
eTADSipPhone: 111-222-3333
eTADSinfo: Notes Here
eTADSotherHomePhone: 111-222-3333
eTADSotherPager: 111-222-3333
eTADSotherMobile: 111-222-3333
eTADSotherFacsimileTelephoneNumber: 111-222-3333
eTADSotherIpPhone: 111-222-3333
eTADStitle: title
eTADSdepartment: department
eTADScompany: company
eTADSmanager: CN=manager_fn manager_ln,OU=$OU,$DCDOMAIN
eTADSmemberOf: CN=Backup Operators,CN=Builtin,$DCDOMAIN
eTADSlyncSIPAddressOption: 0000000000
eTADSdisplayNamePrintable: aaatestuser$n
eTADSmailNickname: aaatestuser$n
eTADShomeMDB: (Automatic Mailbox Distribution)
eTADShomeMTA: CN=DC001,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,$DCDOMAIN
eTAccountStatus: A
eTADSmsExchRecipientTypeDetails: 0000000001
eTADSmDBUseDefaults: TRUE
eTADSinitials: A
eTADSaccountExpires: 9223372036854775807
EOF
counter=$(( $counter + 00001 ))
done
# Create the delete ADS Process
start=00001
counter=$start
while [ $counter -le $MAX ]
do
n=$((10000+counter)); n=${n#1}
tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
echo "Counter with leading zeros = $n at time: $tz"
cat << EOF >> $FILE1
dn: eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: delete
EOF
counter=$(( $counter + 00001 ))
done
echo ""
echo "################################### ADS USER OBJECT STATS ################################################################"
echo "Number of add objects: `grep "changetype: add" $FILE1 | wc -l`"
echo "Number of delete objects: `grep "changetype: delete" $FILE1 | wc -l`"
rm -rf $FILE2
cp -r -p $FILE1 $FILE2
sed -i 's|,dc=im,dc=etasa|,dc=im,dc=eta|g' $FILE2
ls -lart $FILE1
ls -lart $FILE2
echo ""
echo "################################### SET ADS MAX CONNECTIONS IN POOL SIZE ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
LDAPTLS_REQCERT=never dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD" << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'
echo ""
echo "################################### CCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20402
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the CCS Service to test single thread with dxmodify or ldapmodify"
echo "dxmodify -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the CCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo ""
echo "################################### JCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20410
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the JCS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the JCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo ""
echo "################################### IMPS UNIT & PERF TEST ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
echo "Execute this command to the IMPS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "
echo "Execute this command to the IMPS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $IMPS_HOST:$IMPS_PORT -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "
Address the new bottleneck of MS Exchange / O365 Provisioning.
After parallel provisioning has been introduced with the new im_ccs.exe service, you may noticed that the number of transactions is still being throttled during performance testing.
Out-of-the-box MS Active Directory Global Throttling Policy has the parameter of PowerShellMaxConcurrency set to a default of 18 connection. Any provisioning that uses MS Powershell for MS Exchange and/or MS O365 will be impacted by this default parameter.
To address this bottleneck, we can create a new Throttling Policy and only assign the service ID that will be managing identities, to avoid a global change.
After this change has been made, restart the IM JCS/CCS Services, and retest again with your performance tools. Review the CCS ADS log for # of creations in 60 seconds, and you will be pleasantly surprise at the rate. The logs are the strong confirmation we are looking for.
Performance test (947 ADS accounts w/Exchange mailboxes in 60 seconds, 08:59:54 to 09:00:53) =>Rate of 15 ids/second (or 54 K ids/hr) with updated MaxPowershell = 100 thottlingpolicy.
The last bottleneck appears to be CPU availability to MS Exchange Supporting Services, w3wp.exe, the MS IIS Service. Which appears to be managing MS Powershell connections per its startup string of
The Symantec (CA/Broadcom) Directory solution provides a mechanism for routing LDAPv3 traffic to other solutions. This routing mechanism allows Symantec Directory to act as a virtual directory service for other directories, e.g., MS Active Directory, SunOne, Novell eDirectory, etc.
The Symantec Identity Suite solution uses the LDAP protocol for its mid-tier and connector-tier components. The Provisioning Server is exposed on TCP 20389/20390, the JCS (Java Connector Server) is exposed on TCP 20410/20411, and the CCS (C++ Connector Server) is exposed on TCP 20402/20403.
We wished to isolate provisioning data challenges within the Symantec Identity Management solution that was not fully viewable using the existing debugging logs & features of the provisioning tier & connector tiers. Using Symantec Directory, we can leverage the routing mechanism to build a MITM (man-in-the-middle) methodology to track all LDAP traffic through the Symantec Identity Manager connector tier.
We focused on the final leg of provisioning and created a process to track the JCS -> CCS LDAP traffic. We wanted to understand what and how the data was being sent from the JCS to the CCS to isolate issues to the CCS service and MS Active Directory. Using the trace level of Symantec Directory, we can capture all LDAP traffic, including binds/queries/add/modify actions.
The below steps showcase how to use Symantec Directory as an approved MITM process for troubleshooting exercises. We found this process more valuable than deploying Wireshark on the JCS/CCS Server and decoding the encrypted traffic for LDAP.
Background:
Symantec Directory documentation on routing. Please note the concept / feature of “set transparent-routing = true;” to avoid schema challenges when routing to other directory/ldap solutions.
The Symantec Identity Management connector tier may be deployed on MS Windows or Linux OS. If the CCS service is being used, then MS Windows OS is required for this MS Visual C++ component/service. As we are focused on the CCS service, we will introduce the Symantec Directory solution on the same MS Windows OS.
NOTE: We will keep the MITM process contained on a single host, and will not redirect the network traffic beyond the host.
Step 1: Deploy the latest Symantec Directory solution on MS Windows OS. This deployment is a blank slate for the next steps to follow.
Step 2: Copy the folders of schema, limits, and ssld from an existing Symantec Directory deployment of the Symantec Identity Manager solution. Using the existing schema files, references, and certificates will allow us to avoid any challenges during startup of the Router DSA due to the pre-defined provisioning/connector tier configurations. Please note when copying from a Linux OS version of Symantec Directory, we will need to update the path from Linux format to MS Windows format in the SSLD impd.dxc file for “cert-dir” and “ca-file” parameters.
Step 3: Create a new Router DSA DXI configuration file. This is the primary configuration file for Symantec Directory DSA. It will referenced the schema, knowledge, limits, and certificates for the DSA. Note the parameters for “transparent-routing” to avoid schema challenges with other solutions. Note the trace level used to trace the LDAP traffic in the Symantec Directory Router DSA trace log.
# DXserver/config/servers/admin_router_ccs_30402.dxi
# logging and tracing
close summary-log;
close trace-log;
source "../logging/default.dxc";
# schema
clear schema;
source "../schema/impd.dxg";
# access controls
clear access;
# source "../access/";
# ssld
source "../ssld/impd.dxc";
# knowledge
clear dsas;
source "../knowledge/admin_router_ccs_group.dxg";
# operational settings
source "../settings/default.dxc";
# service limits
source "../limits/impd.dxc";
# database - none - transparent router
set transparent-routing=TRUE;
# tunnel through eAdmin server error code and messages
set route-non-compliant-ldap-error-codes = true;
set trace=ldap,time,stats;
#set trace=dsa,time;
Step 4: Create the three (3) knowledge files. The “group” knowledge file will be used to redirect to the other two (2) knowledge files of the router DSA and the re-direct DSA to the CCS service.
# DXserver/config/knowledge/admin_router_ccs_group.dxg
# The admin_router_ccs_30402.dxc PORT 30402
# will be used for the IAMCS (JCS) CCS port override configuration file
# server_ccs.properties via proxyConnectionConfig.proxyServerPort=30402
source "admin_router_ccs_30402.dxc";
source "admin_ccs_server_01.dxc";
# DXserver/config/knowledge/admin_ccs_server_01.dxc
# This file is sourced by admin_router_ccs_group.dxg.
set dsa admin_ccs_server_01 =
{
prefix = <dc etasa>
dsa-name = <dc etasa><cn admin_ccs_server_01>
dsa-password = "secret"
address = ipv4 localhost port 20402
auth-levels = clear-password
dsp-idle-time = 100000
dsa-flags = load-share
trust-flags = allow-check-password, no-server-credentials, trust-conveyed-originator
link-flags = dsp-ldap
#link-flags = dsp-ldap, ssl-encryption
# Note: ssl will require update to /etc/hosts with: <IP_Address> eta_server
};
Step 5: Update the JCS configuration file that contains the TCP port that we will be redirecting to. In this example, we will declare TCP 30402 to be the new port.
Overview of all files updated and their relationship to each other.
Validation
Start up the solution in the following order. Ensure that the new Symantec Directory Router DSA is starting with no issue. If there are any syntax issues, isolate them with the command: dxserver -d start DSA_NAME.
Start the Router DSA first, then restart the im_jcs (JCS) service. The im_ccs (CCS) service will be auto-started by the JCS service. Wait one (1) minute, then check that both TCP Ports 20402 (CCS) and 30402 (Router DSA) are both in the LISTEN state. If we do not see these both ports, please stop and restart these services.
May use MS Sysinternals ProcessExplorer to monitor both services and using the TCP/IP tab, to view which ports are being used.
A view of the im_ccs.exe and dxserver.exe services and which TCP ports they are listening on.
Use a 3rd party LDAP client tool, such as Jxplorer to authenticate to both the CCS and the Router DSA ports, with the embedded service ID of “cn=root,dc=etasa”. We should see exactly the SAME data.
Use the IME or IMPS to perform a query to MS Active Directory (or any other endpoint that uses the CCS connector tier). We should now see the “cache” on the CCS service be populated with the endpoint information, and the base DN structure. We can now track all LDAP traffic through the Router DSA MITM process.
View of trace logs
We can monitor when the JCS first binds to the CCS service.
We can monitor when the IMPS via the JCS queries if the CCS is aware of the ADS endpoint.
Finally, we can view when the IMPS service decrypt its stored information on the Active Directory endpoint, and push this information to the CCS cache, to allow communication to MS Active Directory. Using Notepad++ we can tail the trace log.
Please note, this is a secure LDAP/S tunnel from the IMPS -> JCS -> CCS -> MS ADS.
We can now view how this data is pushed via this secure tunnel with the MITM process.
We can now monitor all traffic and assist with troubleshooting any CCS/MS-ADS challenges.
This same MITM methodology/process may also be used for the IMPS (TCP 20389/2039) and the JCS (TCP 20410/20411) services. We have used this process to capture the IME (JIAM) LDAP traffic to the IMPS Service, to isolate multiple queries for Child Provisioning Roles. Which has been used by the product team to enhance the solution to lower startup durations of the IME in the latest releases.
Binds/queries/add/modification all work with this approach, but we do see an issue with OID for IMPS ADS endpoint “explore process” on ADS OU object. We are reviewing how to address this last challenge that states “critical extension is unavailable” for a LDAP control property of the OU object. The OIDs captured appear to be related to SunOne/Iplanet.
A very common challenge we see is the modification of the CA/Symantec Connector Server Service(s) startup order for the embedded C++ (CCS) Connector. This CCS connector service on MS Windows OS is marked default as “Manual” startup.
Since the solution documentation is not clear on why this is configured as manual, we will see site’ administrators that will either change this service from “Manual” to “Automatic” or will start the CCS service manually themselves upon a restart.
However, either of these processes will impact the ability of the JCS Service from managing the CCS Services cache upon startup. The JCS will NOT be able to manage the CCS service for a number of minutes until it can resolve this challenge. Unfortunately, when this occurs, the traffic to any CCS managed endpoints will be placed in a long time out within the JCS Service. The IMPS (Provisioning Service) will think that it successfully handed off the task to the JCS/CCS tier, but the task will stay in a holding pattern until either the memory of the JCS is overwhelmed or the CCS Service restarts/crashes or a timeout of the task.
TL;DR – Please do not start the CCS Service manually. Only stop/start the JCS Service, wait a full minute and you should see the CCS Service start up. If the CCS Service does NOT start, investigate why.
JCS Service’s management of the CCS Service:
To understand how the JCS Service manages the CCS Service (via localhost TCP 20402), we can review two (2) files and use MS Sysinternals Process Explorer to view the JCS Service starting the CCS Service via the command “net start im_ccs”. The JCS Service will now have access to update the CCS service’s cache with information for a managed endpoint, e.g. Active Directory.
The two (2) JCS Service configuration files for CCS Service are:
C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\server_osgi_ccs.xml [File contains startup properties of how the JCS will manage timeouts to the CCS Service & connections pools]
C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\override\server_ccs.properties [File contains the bind credentials and the service port to communicate to on localhost:20402. The password hash will be PBES or AES format depending if FIPS is enabled.]
And finally a view of the startup of the CCS Service via JCS Service using MS Sysinternals Process Explorer https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer We can see that a child process is started from the JCS Service that will call the MS Windows “net.exe” command and execute “net start im_ccs”
Keeping the JCS Service and CCS Service as-is for startup processes will help avoid confusion for the provisioning tier of the CA/Symantec solution. Please only stop/start the JCS Service. If the CCS Service does not stop after 2 minutes, kill it. But never start the CCS by itself.
A view of the data path from IMPS (IM Provisioning Server) to Active Directory (manage endpoint) via the Connector tier.
Performance Improvements
While we may not adjust the startup from manual to automatic, we can enhance the default configurations for performance and timeout improvements. The JCS Service starts up with a default of 1 GB RAM. The JCS Service is 64 bit based on using 64 bit JAVA and the memory can be increased accordingly. After testing with large data sets, we recommend increasing the JCS JVM max memory from 1 GB to 4 GB. We can confirm after startup of the JCS will use over 1 GB of RAM with MS Sysinternals Process Explorer.
Other improvement include updating the JAVA that is supporting the JCS Service. CA/Symantec now recommends using AdoptOpenJDK. The documentation now explains how this may be updated in-place. Or as we prefer to reinstall and allow the installer to update the path statements for AdoptOpenJDK.
The below image below shows in the MS Windows Registry for the JCS Service (Procrun 2.0/im_jcs) the key value pairs that are updated for AdoptOpenJDK. https://adoptopenjdk.net/ If managing Active Directory, please review your OS environmental variables to control the behavior from the CCS Service to Active Directory.
After you restart the JCS Service, open the JCS Administration Console via http://localhost:20080/main or https://localhost:20443/main right click on the “Local Connector Server” ICON and it should display that AdoptOpenJDK is in use now. Only major release 8 is supported, avoid trying later releases (11,15) until support is confirmed.
Stability Improvements
The default JCS Service configuration file has knowledge of the connection pool and timeouts, but appears to be missing the “maxWait” token defined. If we are willing to wait 5-10 minutes for the JCS Service to reset its knowledge of the CCS service, we can leave the default. However for a large environment, we have found that lowering the wait times will greatly avoid the delays in transactions when there is stoppage. We have identified two (2) configuration parameters that will assist with the long term stability of the solution. Adding the “maxWait” of 60 seconds (60000 milliseconds) to the JCS configuration file for CCS service and updating the default IM Provisioning Server domain configuration parameter of “Connections/Refresh Time” to 90 seconds.
Troubleshooting and Logging
To assist with RCA efforts, we have the following recommendations. Enable verbose logging for both the JCS Service and the managed endpoint to isolate issues. You may also need to increase logging for the API Gateway or docker logs.
Below is the example to enable verbose logging.
To monitor the JCS logs, there are several tools that will assist, but we find that the latest releases of Notepad++ allow for “tailing” the active JCS logs.
Example of verbose logs for Active Directory via the CCS’s ADS and JCS logs.
Important Logging Note: Enable the new IM r14.3cp2 feature to auto rotate your CCS ADS log. Avoid stop/start of the CCS Service yourself, that may interrupt the JCS behavior to the CCS Service (error communicating to localhost:20402 will display in JCS logs). New file(s): Connector Server\ccs\data\ADS\<Endpoint_Name>.logconfig
Example of setting MS Windows OS Environmental variables with “setx” and description of the value of each variable for Active Directory/MS Exchange
1
[High Value. Will force AGENTLESS connection to Exchange 2010 & UP]
setx ADS_AGENTLESS_MODE 1 /m
2
[High Value. Default value = 2, Kerberos authentication for Exchange Powershell API]
setx ADS_AGENTLESS_AUTHMETHOD 2 /m
3
[High Value. Default value = 3. Increase to 100 and ALSO have Exchange Admin create a new quota for the service account used to create mailboxes. Default Exchange Powershell Quota is 18. New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 AND Set-Mailbox ServiceAccountID -ThrottlingPolicy MaxPowershell ]
setx ADS_AGENTLESS_MAXCONN 100 /m
4
[Monitor. Default value = 1. Error level ONLY, increase to level 3 for debugging powershell logging to MS Exchange]
setx ADS_AGENTLESS_LOGLEVEL 1 /m
5
[Medium Value. CCS service will wait 10 minutes for single account. Exchange Powershell Mailbox Quota of 18 and BLC with 100’s of users.]
setx ADS_CONFIRM_MAILBOX 600 /m
6
[Low Value. Mask the AD Failover List in the IM Prov Manager UI]
setx ADS_DISABLE_DCSTATUS 1 /m
7
[Low Value. Mask the viewing the default AD Primary Group in the IM Prov Manager UI]
setx ADS_DISABLE_PRIMARYGROUPNAME 1 /m
8
[High Value. Send the DC hostname to the Exchange server to query first instead of Exchange relying on its current pool]
setx ADS_E2K_SEND_DC 1 /m
9
[High Value. Requires service account can view all alternatives DC. May limit failover DC via properties file.]
setx ADS_FAILOVER 1 /m
10
[Medium Value. Performance if Terminal Services attribute are NOT being managed, e.g. changed in Account Templates or PX rules.]
setx ADS_WTS_TIMEOUT -1 /m
11
[Set “ADS_OPERATION_TIMEOUT” to -1 to disable the client side timeout functionality in the Environment Variable, otherwise 60]
setx ADS_OPERATION_TIMEOUT 60 /m
12
[The failover retry interval is the time that the Active Directory connector waits before checking the stopped server. The default retry interval is 15 minutes]
setx ADS_RETRY 15 /m
13
[To allow groups in unmanaged domains to be a part of synchronization, Defines whether the synchronization operation searches the global catalog. The value of x can be 0 or 1: 0: (Default) The synchronization operation queries the local catalog only. It does not consider universal groups in unmanaged domains. When x is set to 0, the y value has no effect. 1: Synchronization queries the global catalog to allow it to consider groups in unmanaged domains. y Defines which domains the synchronization operation considers. 0: Synchronization considers groups in both managed and unmanaged domains. 1: Synchronization considers groups in managed domains only.]
setx ADS_MANAGE_GROUPS 01 /m
14
[Monitor. Seems only valuable for debugging. Has performance hit but may assist for CCS debugging to ActiveDirectory.]
setx ADS_FORCELOG 1 /m
15
[Low Value. The IMPS service can page with lower limits. Impact if this value is > what AD default page limit size is. ]
setx ADS_SIZELIMIT 50000 /m
Reinstalling the JCS Service from the Virtual Appliance
If you are using the CA/Symantec Identity Suite virtual appliance, consider after patching the solutions on the virtual appliance, to re-installing the remote JCS Services. This will avoid any confusion on which patches are deployed on the remote JCS servers. Any patches on the virtual appliance will be incorporated into the new installer. We prefer to use the JCS only on the MS Windows OS, as it can service both JCS type managed endpoints & CCS type managed endpoints together. We also have full access to adjust the behavior of these service on MS Windows OS rather than the limited access provided by the virtual appliance for the JCS service.
Hopefully some of these notes will help you avoid any challenges with the connector tier and if you do, how to isolate the issues.
Advance Review: Review how CCS Service receives IMPS data via the JCS Tier.
The below example will load the DLL for the CCS Service (pass-through), then the information to bind to the ADS endpoint will be sent, then two (2) modify operations will be executed. This process emulates the IMPS behavior with the JCS and CCS. The bind information for the ADS endpoint that is stored in the CA Provisioning User Store, and queried/decrypted by the IMPS to send to the JCS as needed. Only after this information is stored in the CCS service, will the solution be able to explore or manage the ADS endpoint accounts.
If unable to re-install, please delete the CA install/registry tracking file under C:\Windows folder, C:\Windows\vpd.properties , then reboot before attempting a re-install of the JCS/CCS component.
ECS Services
These five (5) ECS Services are typically not active used & may be changed to manual for minor CPU relief. ECS features are retained for supporting libraries.
