Written by A very common challenge we see is the modification of the CA/Symantec Connector Server Service(s) startup order for the embedded C++ (CCS) Connector. This CCS connector service on MS Windows OS is marked default as “Manual” startup.
Since the solution documentation is not clear on why this is configured as manual, we will see site’ administrators that will either change this service from “Manual” to “Automatic” or will start the CCS service manually themselves upon a restart.
However, either of these processes will impact the ability of the JCS Service from managing the CCS Services cache upon startup. The JCS will NOT be able to manage the CCS service for a number of minutes until it can resolve this challenge. Unfortunately, when this occurs, the traffic to any CCS managed endpoints will be placed in a long time out within the JCS Service. The IMPS (Provisioning Service) will think that it successfully handed off the task to the JCS/CCS tier, but the task will stay in a holding pattern until either the memory of the JCS is overwhelmed or the CCS Service restarts/crashes or a timeout of the task.
TL;DR – Please do not start the CCS Service manually. Only stop/start the JCS Service, wait a full minute and you should see the CCS Service start up. If the CCS Service does NOT start, investigate why.
JCS Service’s management of the CCS Service:
To understand how the JCS Service manages the CCS Service (via localhost TCP 20402), we can review two (2) files and use MS Sysinternals Process Explorer to view the JCS Service starting the CCS Service via the command “net start im_ccs”. The JCS Service will now have access to update the CCS service’s cache with information for a managed endpoint, e.g. Active Directory.
The two (2) JCS Service configuration files for CCS Service are:
- C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\server_osgi_ccs.xml [File contains startup properties of how the JCS will manage timeouts to the CCS Service & connections pools]
- C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\override\server_ccs.properties [File contains the bind credentials and the service port to communicate to on localhost:20402. The password hash will be PBES or AES format depending if FIPS is enabled.]
And finally a view of the startup of the CCS Service via JCS Service using MS Sysinternals Process Explorer https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer We can see that a child process is started from the JCS Service that will call the MS Windows “net.exe” command and execute “net start im_ccs”
Keeping the JCS Service and CCS Service as-is for startup processes will help avoid confusion for the provisioning tier of the CA/Symantec solution. Please only stop/start the JCS Service. If the CCS Service does not stop after 2 minutes, kill it. But never start the CCS by itself.
A view of the data path from IMPS (IM Provisioning Server) to Active Directory (manage endpoint) via the Connector tier.
Performance Improvements
While we may not adjust the startup from manual to automatic, we can enhance the default configurations for performance and timeout improvements. The JCS Service starts up with a default of 1 GB RAM. The JCS Service is 64 bit based on using 64 bit JAVA and the memory can be increased accordingly. After testing with large data sets, we recommend increasing the JCS JVM max memory from 1 GB to 4 GB. We can confirm after startup of the JCS will use over 1 GB of RAM with MS Sysinternals Process Explorer.
Other improvement include updating the JAVA that is supporting the JCS Service. CA/Symantec now recommends using AdoptOpenJDK. The documentation now explains how this may be updated in-place. Or as we prefer to reinstall and allow the installer to update the path statements for AdoptOpenJDK.
The below image below shows in the MS Windows Registry for the JCS Service (Procrun 2.0/im_jcs) the key value pairs that are updated for AdoptOpenJDK. https://adoptopenjdk.net/ If managing Active Directory, please review your OS environmental variables to control the behavior from the CCS Service to Active Directory.
After you restart the JCS Service, open the JCS Administration Console via http://localhost:20080/main or https://localhost:20443/main right click on the “Local Connector Server” ICON and it should display that AdoptOpenJDK is in use now. Only major release 8 is supported, avoid trying later releases (11,15) until support is confirmed.
Stability Improvements
The default JCS Service configuration file has knowledge of the connection pool and timeouts, but appears to be missing the “maxWait” token defined. If we are willing to wait 5-10 minutes for the JCS Service to reset its knowledge of the CCS service, we can leave the default. However for a large environment, we have found that lowering the wait times will greatly avoid the delays in transactions when there is stoppage. We have identified two (2) configuration parameters that will assist with the long term stability of the solution. Adding the “maxWait” of 60 seconds (60000 milliseconds) to the JCS configuration file for CCS service and updating the default IM Provisioning Server domain configuration parameter of “Connections/Refresh Time” to 90 seconds.
Troubleshooting and Logging
To assist with RCA efforts, we have the following recommendations. Enable verbose logging for both the JCS Service and the managed endpoint to isolate issues. You may also need to increase logging for the API Gateway or docker logs.
Below is the example to enable verbose logging.
To monitor the JCS logs, there are several tools that will assist, but we find that the latest releases of Notepad++ allow for “tailing” the active JCS logs.
Example of verbose logs for Active Directory via the CCS’s ADS and JCS logs.
Important Logging Note: Enable the new IM r14.3cp2 feature to auto rotate your CCS ADS log. Avoid stop/start of the CCS Service yourself, that may interrupt the JCS behavior to the CCS Service (error communicating to localhost:20402 will display in JCS logs). New file(s): Connector Server\ccs\data\ADS\<Endpoint_Name>.logconfig
CCS OS Environmental Variables
Spread throughout the documentation for the CA/Symantec IAM/IAG connector tier, is the use of MS Windows OS environmental variables for the CCS Service. The majority are used to manage behavior to Active Directory and/or MS Exchange. Please search the document for the latest updates. These may be set in MS Window OS via the System Environmental Variable section or via the command line with “setx”. https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/identity-management-and-governance-connectors/1-0/search.html?q=Environment%20Variable&page=1
Example of setting MS Windows OS Environmental variables with “setx” and description of the value of each variable for Active Directory/MS Exchange
| 1 | [High Value. Will force AGENTLESS connection to Exchange 2010 & UP] |
| setx ADS_AGENTLESS_MODE 1 /m | |
| 2 | [High Value. Default value = 2, Kerberos authentication for Exchange Powershell API] |
| setx ADS_AGENTLESS_AUTHMETHOD 2 /m | |
| 3 | [High Value. Default value = 3. Increase to 100 and ALSO have Exchange Admin create a new quota for the service account used to create mailboxes. Default Exchange Powershell Quota is 18. New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 AND Set-Mailbox ServiceAccountID -ThrottlingPolicy MaxPowershell ] |
| setx ADS_AGENTLESS_MAXCONN 100 /m | |
| 4 | [Monitor. Default value = 1. Error level ONLY, increase to level 3 for debugging powershell logging to MS Exchange] |
| setx ADS_AGENTLESS_LOGLEVEL 1 /m | |
| 5 | [Medium Value. CCS service will wait 10 minutes for single account. Exchange Powershell Mailbox Quota of 18 and BLC with 100’s of users.] |
| setx ADS_CONFIRM_MAILBOX 600 /m | |
| 6 | [Low Value. Mask the AD Failover List in the IM Prov Manager UI] |
| setx ADS_DISABLE_DCSTATUS 1 /m | |
| 7 | [Low Value. Mask the viewing the default AD Primary Group in the IM Prov Manager UI] |
| setx ADS_DISABLE_PRIMARYGROUPNAME 1 /m | |
| 8 | [High Value. Send the DC hostname to the Exchange server to query first instead of Exchange relying on its current pool] |
| setx ADS_E2K_SEND_DC 1 /m | |
| 9 | [High Value. Requires service account can view all alternatives DC. May limit failover DC via properties file.] |
| setx ADS_FAILOVER 1 /m | |
| 10 | [Medium Value. Performance if Terminal Services attribute are NOT being managed, e.g. changed in Account Templates or PX rules.] |
| setx ADS_WTS_TIMEOUT -1 /m | |
| 11 | [Set “ADS_OPERATION_TIMEOUT” to -1 to disable the client side timeout functionality in the Environment Variable, otherwise 60] |
| setx ADS_OPERATION_TIMEOUT 60 /m | |
| 12 | [The failover retry interval is the time that the Active Directory connector waits before checking the stopped server. The default retry interval is 15 minutes] |
| setx ADS_RETRY 15 /m | |
| 13 | [To allow groups in unmanaged domains to be a part of synchronization, Defines whether the synchronization operation searches the global catalog. The value of x can be 0 or 1: 0: (Default) The synchronization operation queries the local catalog only. It does not consider universal groups in unmanaged domains. When x is set to 0, the y value has no effect. 1: Synchronization queries the global catalog to allow it to consider groups in unmanaged domains. y Defines which domains the synchronization operation considers. 0: Synchronization considers groups in both managed and unmanaged domains. 1: Synchronization considers groups in managed domains only.] |
| setx ADS_MANAGE_GROUPS 01 /m | |
| 14 | [Monitor. Seems only valuable for debugging. Has performance hit but may assist for CCS debugging to ActiveDirectory.] |
| setx ADS_FORCELOG 1 /m | |
| 15 | [Low Value. The IMPS service can page with lower limits. Impact if this value is > what AD default page limit size is. ] |
| setx ADS_SIZELIMIT 50000 /m |
Reinstalling the JCS Service from the Virtual Appliance
If you are using the CA/Symantec Identity Suite virtual appliance, consider after patching the solutions on the virtual appliance, to re-installing the remote JCS Services. This will avoid any confusion on which patches are deployed on the remote JCS servers. Any patches on the virtual appliance will be incorporated into the new installer. We prefer to use the JCS only on the MS Windows OS, as it can service both JCS type managed endpoints & CCS type managed endpoints together. We also have full access to adjust the behavior of these service on MS Windows OS rather than the limited access provided by the virtual appliance for the JCS service.
Hopefully some of these notes will help you avoid any challenges with the connector tier and if you do, how to isolate the issues.
Advance Review: Review how CCS Service receives IMPS data via the JCS Tier.
The below example will load the DLL for the CCS Service (pass-through), then the information to bind to the ADS endpoint will be sent, then two (2) modify operations will be executed. This process emulates the IMPS behavior with the JCS and CCS. The bind information for the ADS endpoint that is stored in the CA Provisioning User Store, and queried/decrypted by the IMPS to send to the JCS as needed. Only after this information is stored in the CCS service, will the solution be able to explore or manage the ADS endpoint accounts.
su - dsa export HISTIGNORE=' *' echo -n Password01 > .imps.pwd; chmod 600 .imps.pwd HOST=192.168.242.154;LDAPTLS_REQCERT=never dxmodify -c -H ldaps://$HOST:20411 -D "cn=root,dc=etasa" -y .imps.pwd << EOF dn: eTNamespaceName=ActiveDirectory,dc=im,dc=etasa changetype: add objectClass: eTADSNamespace eTAgentPluginDLL: W2KNamespace.dll eTNamespaceName: ActiveDirectory dn: eTADSDirectoryName=dc2012.exchange2012.lab,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa changetype: add eTADSobjectCategory: CN=Domain-DNS,CN=Schema,CN=Configuration,DC=exchange2012,DC=lab eTADSdomainFunctionality: 6 eTADSUseSSL: 1 eTLogWindowsEventSeverity: FE eTAccountResumable: 1 eTADSnetBIOS: EXCHANGE2012 eTLogStdoutSeverity: FE eTLog: 1 eTLogUnicenterSeverity: FE eTADSlockoutDuration: -18000000000 objectclass: eTADSDirectory eTLogETSeverity: FE eTADSmsExchSystemObjectsObjectVersion: 1 eTADSsettings: 2 eTADSconfig: ExpirePwd=0: HomeDirInheritPermission=0 eTLogDestination: F eTADSUserContainer: CN=BuiltIn;CN=Users eTADSbackupDirs: 000;Default-First-Site-Name.Sites.Configuration.exchange2012.lab;dc2012.exchange2012.lab;0 eTADSuseFailover: 0 eTLogAuditSeverity: FE eTADS-DefaultContext: exchange2012.lab eTADSforestFunctionality: 6 eTADSAuthDN: Administrator eTADSlyncMaxConnection: 5 eTADSAuthPWD: Password01! eTLogFileSeverity: FIESW eTADSprimaryServer: dc2012.exchange2012.lab eTADScontainers: CN=Domain-DNS,CN=Schema,CN=Configuration,DC=exchange2012,DC=lab;exchange2012.lab;dc2012.exchange2012.lab eTADSTimeBoundMembershipsEnabled: 0 eTADSKeepCamCaftFiles: 0 eTADSdomainControllerFunctionality: 6 eTADSexchange: 0 eTADSmsExchSchemaVersion: 1 eTADSCamCaftTimeout: 0000001800 eTADSPortNum: 636 eTADSDCDomain: DC=exchange2012,DC=lab eTADSServerName: dc2012.exchange2012.lab eTADSDirectoryName: dc2012.exchange2012.lab EOF
MS Windows Firewall Rules for JCS Service
NOTE: Ensure MS Win OS F/W Port is open for 20411 on the IAMCS Server
Powershell Example:
Get-NetFirewallRule -Name jcs New-NetFirewallRule -Name '#### IAMCS JCS TCP 20411 & 20443 #####' -DisplayName '##### IAMCS JCS TCP 20411 & 20443 #####' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 20411,20443
Win Cmd Lin Example:
netsh advfirewall firewall add rule name="##### IAMCS JCS TCP 20411 & 20433 #####" dir=in action=allow protocol=TCP localport="20411,20443"
Install MS .Net Framework 3.5 (Required for the CCS Service & ECS – enterprise common services library framework)
DISM /Online /Enable-Feature /All /FeatureName:NetFx3
Re-install or Uninstall issues
If unable to re-install, please delete the CA install/registry tracking file under C:\Windows folder, C:\Windows\vpd.properties , then reboot before attempting a re-install of the JCS/CCS component.
ECS Services
These five (5) ECS Services are typically not active used & may be changed to manual for minor CPU relief. ECS features are retained for supporting libraries.
