The Symantec (CA) Identity Suite includes the Symantec (CA) Directory. This component is installed under the ‘dsa’ service ID. On the virtual appliance, this ‘dsa’ service ID does not have a password defined, and therefore no login is allowed.
As an enhancement, we would like to add in a SSH private key to allow authentication to the ‘dsa’ service ID from other virtual appliances and desktop usage with various tools, e.g. Putty, MobaXterm, WinSCP, etc. This enhancement will allow for a streamlined process to address out-of-sync Directory DATA DSAs with scp/Rsync copies without intermediate file shares or use of other service IDs.
Challenge:
The virtual appliance of Symantec (CA) Identity Suite r14.3 is built on CentOS 6.4. The OpenSSH services on this OS apparently do not use a private key format that can be used by desktop tools or the PuttyGen (keygen conversion tool). However, the private key may be used between vApp servers if using the FQDN (full qualified domain name). We noted that during testing, that localhost is not allowed due to localhost not defined in the SSHD “AllowedUsers” property file.
On newer virtual appliances vApp r14.4 with CentOS 8 Stream, this challenge does not exist, and we can use the OpenSSH private key, id_rsa, with the desktop tools as-is.
To assist with challenge and streamlining this process we have the following three (2) options:
Option 1: On newer OS, use OpenSSH process
After creating the private key, ./ssh/id_rsa, cat this file out to notepad, and save for use with the desktop tools
Generate this OpenSSH private/public key. The final command will help to validate this private key may be used for server to server communication.
On any OS (new/old) just use Putty-Gen tool to generate the private key. Update key comment/passphrase. After the private key is created, copy the TEXT “Public Key for pasting into OpenSSH authorized_keys file”. Just like it says, and then you may use the associated private key, id_rsa.ppk, with the desktop tools for the ‘dsa’ service ID.
Option 3: Combination of processes/tools
Important: .ssh/authorized_keys is updated and not overwritten.
A very common challenge we see is the modification of the CA/Symantec Connector Server Service(s) startup order for the embedded C++ (CCS) Connector. This CCS connector service on MS Windows OS is marked default as “Manual” startup.
Since the solution documentation is not clear on why this is configured as manual, we will see site’ administrators that will either change this service from “Manual” to “Automatic” or will start the CCS service manually themselves upon a restart.
However, either of these processes will impact the ability of the JCS Service from managing the CCS Services cache upon startup. The JCS will NOT be able to manage the CCS service for a number of minutes until it can resolve this challenge. Unfortunately, when this occurs, the traffic to any CCS managed endpoints will be placed in a long time out within the JCS Service. The IMPS (Provisioning Service) will think that it successfully handed off the task to the JCS/CCS tier, but the task will stay in a holding pattern until either the memory of the JCS is overwhelmed or the CCS Service restarts/crashes or a timeout of the task.
TL;DR – Please do not start the CCS Service manually. Only stop/start the JCS Service, wait a full minute and you should see the CCS Service start up. If the CCS Service does NOT start, investigate why.
JCS Service’s management of the CCS Service:
To understand how the JCS Service manages the CCS Service (via localhost TCP 20402), we can review two (2) files and use MS Sysinternals Process Explorer to view the JCS Service starting the CCS Service via the command “net start im_ccs”. The JCS Service will now have access to update the CCS service’s cache with information for a managed endpoint, e.g. Active Directory.
The two (2) JCS Service configuration files for CCS Service are:
C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\server_osgi_ccs.xml [File contains startup properties of how the JCS will manage timeouts to the CCS Service & connections pools]
C:\Program Files (x86)\CA\Identity Manager\Connector Server\jcs\conf\override\server_ccs.properties [File contains the bind credentials and the service port to communicate to on localhost:20402. The password hash will be PBES or AES format depending if FIPS is enabled.]
And finally a view of the startup of the CCS Service via JCS Service using MS Sysinternals Process Explorer https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer We can see that a child process is started from the JCS Service that will call the MS Windows “net.exe” command and execute “net start im_ccs”
Keeping the JCS Service and CCS Service as-is for startup processes will help avoid confusion for the provisioning tier of the CA/Symantec solution. Please only stop/start the JCS Service. If the CCS Service does not stop after 2 minutes, kill it. But never start the CCS by itself.
A view of the data path from IMPS (IM Provisioning Server) to Active Directory (manage endpoint) via the Connector tier.
Performance Improvements
While we may not adjust the startup from manual to automatic, we can enhance the default configurations for performance and timeout improvements. The JCS Service starts up with a default of 1 GB RAM. The JCS Service is 64 bit based on using 64 bit JAVA and the memory can be increased accordingly. After testing with large data sets, we recommend increasing the JCS JVM max memory from 1 GB to 4 GB. We can confirm after startup of the JCS will use over 1 GB of RAM with MS Sysinternals Process Explorer.
Other improvement include updating the JAVA that is supporting the JCS Service. CA/Symantec now recommends using AdoptOpenJDK. The documentation now explains how this may be updated in-place. Or as we prefer to reinstall and allow the installer to update the path statements for AdoptOpenJDK.
The below image below shows in the MS Windows Registry for the JCS Service (Procrun 2.0/im_jcs) the key value pairs that are updated for AdoptOpenJDK. https://adoptopenjdk.net/ If managing Active Directory, please review your OS environmental variables to control the behavior from the CCS Service to Active Directory.
After you restart the JCS Service, open the JCS Administration Console via http://localhost:20080/main or https://localhost:20443/main right click on the “Local Connector Server” ICON and it should display that AdoptOpenJDK is in use now. Only major release 8 is supported, avoid trying later releases (11,15) until support is confirmed.
Stability Improvements
The default JCS Service configuration file has knowledge of the connection pool and timeouts, but appears to be missing the “maxWait” token defined. If we are willing to wait 5-10 minutes for the JCS Service to reset its knowledge of the CCS service, we can leave the default. However for a large environment, we have found that lowering the wait times will greatly avoid the delays in transactions when there is stoppage. We have identified two (2) configuration parameters that will assist with the long term stability of the solution. Adding the “maxWait” of 60 seconds (60000 milliseconds) to the JCS configuration file for CCS service and updating the default IM Provisioning Server domain configuration parameter of “Connections/Refresh Time” to 90 seconds.
Troubleshooting and Logging
To assist with RCA efforts, we have the following recommendations. Enable verbose logging for both the JCS Service and the managed endpoint to isolate issues. You may also need to increase logging for the API Gateway or docker logs.
Below is the example to enable verbose logging.
To monitor the JCS logs, there are several tools that will assist, but we find that the latest releases of Notepad++ allow for “tailing” the active JCS logs.
Example of verbose logs for Active Directory via the CCS’s ADS and JCS logs.
Important Logging Note: Enable the new IM r14.3cp2 feature to auto rotate your CCS ADS log. Avoid stop/start of the CCS Service yourself, that may interrupt the JCS behavior to the CCS Service (error communicating to localhost:20402 will display in JCS logs). New file(s): Connector Server\ccs\data\ADS\<Endpoint_Name>.logconfig
Example of setting MS Windows OS Environmental variables with “setx” and description of the value of each variable for Active Directory/MS Exchange
1
[High Value. Will force AGENTLESS connection to Exchange 2010 & UP]
setx ADS_AGENTLESS_MODE 1 /m
2
[High Value. Default value = 2, Kerberos authentication for Exchange Powershell API]
setx ADS_AGENTLESS_AUTHMETHOD 2 /m
3
[High Value. Default value = 3. Increase to 100 and ALSO have Exchange Admin create a new quota for the service account used to create mailboxes. Default Exchange Powershell Quota is 18. New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 AND Set-Mailbox ServiceAccountID -ThrottlingPolicy MaxPowershell ]
setx ADS_AGENTLESS_MAXCONN 100 /m
4
[Monitor. Default value = 1. Error level ONLY, increase to level 3 for debugging powershell logging to MS Exchange]
setx ADS_AGENTLESS_LOGLEVEL 1 /m
5
[Medium Value. CCS service will wait 10 minutes for single account. Exchange Powershell Mailbox Quota of 18 and BLC with 100’s of users.]
setx ADS_CONFIRM_MAILBOX 600 /m
6
[Low Value. Mask the AD Failover List in the IM Prov Manager UI]
setx ADS_DISABLE_DCSTATUS 1 /m
7
[Low Value. Mask the viewing the default AD Primary Group in the IM Prov Manager UI]
setx ADS_DISABLE_PRIMARYGROUPNAME 1 /m
8
[High Value. Send the DC hostname to the Exchange server to query first instead of Exchange relying on its current pool]
setx ADS_E2K_SEND_DC 1 /m
9
[High Value. Requires service account can view all alternatives DC. May limit failover DC via properties file.]
setx ADS_FAILOVER 1 /m
10
[Medium Value. Performance if Terminal Services attribute are NOT being managed, e.g. changed in Account Templates or PX rules.]
setx ADS_WTS_TIMEOUT -1 /m
11
[Set “ADS_OPERATION_TIMEOUT” to -1 to disable the client side timeout functionality in the Environment Variable, otherwise 60]
setx ADS_OPERATION_TIMEOUT 60 /m
12
[The failover retry interval is the time that the Active Directory connector waits before checking the stopped server. The default retry interval is 15 minutes]
setx ADS_RETRY 15 /m
13
[To allow groups in unmanaged domains to be a part of synchronization, Defines whether the synchronization operation searches the global catalog. The value of x can be 0 or 1: 0: (Default) The synchronization operation queries the local catalog only. It does not consider universal groups in unmanaged domains. When x is set to 0, the y value has no effect. 1: Synchronization queries the global catalog to allow it to consider groups in unmanaged domains. y Defines which domains the synchronization operation considers. 0: Synchronization considers groups in both managed and unmanaged domains. 1: Synchronization considers groups in managed domains only.]
setx ADS_MANAGE_GROUPS 01 /m
14
[Monitor. Seems only valuable for debugging. Has performance hit but may assist for CCS debugging to ActiveDirectory.]
setx ADS_FORCELOG 1 /m
15
[Low Value. The IMPS service can page with lower limits. Impact if this value is > what AD default page limit size is. ]
setx ADS_SIZELIMIT 50000 /m
Reinstalling the JCS Service from the Virtual Appliance
If you are using the CA/Symantec Identity Suite virtual appliance, consider after patching the solutions on the virtual appliance, to re-installing the remote JCS Services. This will avoid any confusion on which patches are deployed on the remote JCS servers. Any patches on the virtual appliance will be incorporated into the new installer. We prefer to use the JCS only on the MS Windows OS, as it can service both JCS type managed endpoints & CCS type managed endpoints together. We also have full access to adjust the behavior of these service on MS Windows OS rather than the limited access provided by the virtual appliance for the JCS service.
Hopefully some of these notes will help you avoid any challenges with the connector tier and if you do, how to isolate the issues.
Advance Review: Review how CCS Service receives IMPS data via the JCS Tier.
The below example will load the DLL for the CCS Service (pass-through), then the information to bind to the ADS endpoint will be sent, then two (2) modify operations will be executed. This process emulates the IMPS behavior with the JCS and CCS. The bind information for the ADS endpoint that is stored in the CA Provisioning User Store, and queried/decrypted by the IMPS to send to the JCS as needed. Only after this information is stored in the CCS service, will the solution be able to explore or manage the ADS endpoint accounts.
If unable to re-install, please delete the CA install/registry tracking file under C:\Windows folder, C:\Windows\vpd.properties , then reboot before attempting a re-install of the JCS/CCS component.
ECS Services
These five (5) ECS Services are typically not active used & may be changed to manual for minor CPU relief. ECS features are retained for supporting libraries.
One of the challenges that IAM/IAG teams may have every few months is delivery or access for internal/external auditors to validate access within the IAM/IAG system and their managed endpoints.
Usually, auditors may directly access the 100’s system/endpoints/applications and randomly select a few or export the entire directory structure to review access. This effort takes time and possible 100’s of entitlements to grant temporary/expiry access to view. Auditors also prefer Excel or CSV files to review rather than fixed documents (PDF) to allow them to filter and isolate what interests them.
One process that may have value for your team is various tools with export functionality to CSV/XLS and the ability to query the 100’s-1000’s of systems from a single entry point.
A tool that we have found valuable over the years is SoftTerra LDAP Browser.
The multiple benefits from this tool for IAM/IAG are:
It is a read-only tool, so no mistakes can be made by granting too much access.
It has the ability to save queries that are popular and can be copied from other tools.
It has the ability to export the queries to CSV/XLS formats (plus others)
It can be used to pull reports from an IAM/IAG solution via their directory ports.
It can be used to pull reports from the managed applications (on-prem or SaaS) via the IAM provisioning directory ports.
The tool is free from SoftTerra, it is a limited version of their Administration tool
Example of the SoftTerra LDAP Browser tool used to query Active Directory, LDAP user stores, and Provisioning User Store & managed endpoints/applications.
A view to export Service Now (SNOW) accounts via the CA/Symantec Identity Manager Provisioning Server/Service (TCP 20390) via the LDAP/S protocol.
Why? The provisioning server may be viewed as a virtual directory/pass-through directory to the managed endpoints via its connector tier.
The below image shows SoftTerra LDAPBrowser used to connect to the Provisioning Server (TCP 20390). Then navigate to a Service Now (SNOW) managed endpoint, to query on all accounts and their respective profiles & entitlements. This same report/extract process may be done for mainframe/AS400 and client-server applications, e.g Active Directory, Unix, Databases, etc.
Enhance this process with defense-in-depth
We will not use the primary default administration account of the provisioning tier, “etaadmin”. Since this account has full access to change data.
Within the IAM/IAG solution, create an auditor account.
In the example below we create a new Global User, with the name “auditor”, a description, password, and a local “read-only admin profile” with an expiration date. This will allow the auditors to use the account as they wish (or you may grant this “read-only admin profile directly to their existing Global User ID). The account may still follow the same password reset expiration processes. If the account is marked as “restricted” in the CA/Symantec IM solution, then this account is limited how it may be changed to avoid any unexpected sync challenges to managed endpoints (if it was correlated to other accounts).
After the new Global User is created (or existing ID is added to the Admin Profile “ReadAdministrator”), update SoftTerra Credentials for the Provisioning Service. Below the new DN with “auditor” is shown in the credentials for login ID, e.g. “eTGlobalUserName=auditor,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta”
Now, the auditors may run as many reports as they would like, and export to spreadsheets or PDF files using a read-only account with a read-only tool.
Honorable mentions for other query tools.
Jxplorer is a useful & free java-based tool for reports, but this is a full edit tool & only exports out to LDIF format. http://jxplorer.org/
Apache Directory Studio is another very useful & free java-based tool for reports. This is a full edit tool. It does have the ability to export to many different formats. Since this tool does NOT need an MS Windows installer, and if the Desktop prevent installation, this is typically our 2nd choice to use. Extract and use the current java on the MS Windows OS or download AdoptOpenJDK and extract it to use with Apache Directory Studio. https://directory.apache.org/studio/ & https://adoptopenjdk.net/
SoftTerra LDAP Administrator is a paid and full edit tool. It has the same look-n-feel of the SoftTerra LDAP Browser tool. It is typically used by administrators of various LDAP solutions. We recommend this tool for your larger sites or if you would like a fast responsive tool on MS Windows OS. https://www.ldapadministrator.com/
If you have other recommendations, please leave a response.
Bonus Feature – SoftTerra AD Authentication
Both the SoftTerra tools allow binding using your existing authentication (on your desktop/laptop) into Active Directory. No need to create additional user ID for the auditors or yourself.
Perhaps the O365 or Outlook contacts process is not robust or too slow or perhaps you wish you had a more detail view of your internal active directory to view a manager’s direct reports. You can use this feature to view the the non-privacy attributes of your domain of all accounts with a read-only tool.
Step 01: Open a command-line prompt on your desktop/workstation after you have authenticated to your Active Directory domain & type set | findstr LOGONSERVER
Step 02: Install SoftTerra LDAP Browser Tool & Create a new profile
Step 03: Type the name of the Active Directory LOGONSERVER (aka Domain Controller) into the following fields & ensure “Use Secure Connection (SSL)” is selected (to avoid query issues).
Step 04: Click Next until you see “User Authentication Information” then select the radio button for “Currently logged on user (Active Directory)”, then click Finish button.
Step 05: After the profile is built, now click on the profile and watch it expand into a tree display of Active Directory. Select the branch that you believe has the list of users you would like to view, then select an individual user account, to see the values populated.
Step 06: If you wish to export this data to a spreadsheet (CSV/XLS), right click on the left object and select export option.
Step 07: You will have a series of options to export to & the file name it will write to.
Step 08: Advance search and export process. Select the branch that holds all the users you wish to view and export. Note: If the branch has 10,000 objects, this process may take minutes to complete depending on the query.
Step 09: The follow search windows will appear to help you create, save, and export your queries. Note that if you start to type in the field name, the list of the fields will start to appear.
Step 10: Ensure the FILTER is properly formed (use google to assist), and which attribute you wish to view or export is defined, then click search. If you are satisfied with your search, use the “Save Results” to export to a spreadsheet (CSV/XLS) or other format.
