identity manager – ANA Technology Partner

In this blog entry, we will dive into the latest enhancements to the custom `rsyslog` configuration designed for managing logs on the Symantec Identity Governance and Administration (IGA) system running on the Symantec Virtual Appliance (vApp). The previous configuration has been refined to block unwanted systemd (imjournal-related) logs, adjust logging formats, and ensure reliable forwarding of critical logs to centralized collectors like Splunk or Grafana.

This setup is particularly useful when the service ID (such as `config`) has limited access to the root-level `rsyslog.conf` but can modify configurations in the `/etc/rsyslog.d/` directory. We’ll also implement a disk-based queue mechanism to handle potential overflows during log forwarding.

Introduction

The Virtual Appliance (vApp) uses centralized logging with RFC 3164 by default, but we’re updating this configuration to utilize RFC 5424, a newer standard for better log management. Moreover, since the `config` service ID has no access to `/etc/rsyslog.conf` to disable the `imjournal` module, we’ll mask these systemd-related logs at the configuration level, ensuring a cleaner log output without noise.

Let’s look at how we can customize the `rsyslog` setup to optimize logging for Symantec IGA, block unwanted logs, and ensure reliable log forwarding.

Key Changes in the Updated Configuration:

1. Blocking Duplicate or Noisy Logs:

We’ll block messages originating from systemd (via `imjournal`) and logs using facility `local1`, which is often tied to systemd logging, to avoid duplicates.

2. Efficient Log Monitoring:

The `imfile` module will be used for monitoring various critical log files from the vApp, including CA Directory, Provisioning Server, Wildfly (Identity Manager, Identity Portal, Identity Governance), ConnectorServer, and Wildfly metrics (accessed via jboss-cli.sh)

3. Reliable Log Forwarding:

Logs will be forwarded to a remote syslog collector using UDP or TCP with a built-in disk-backed queue for reliability in case of connection issues.

Example: /etc/rsyslog.d/rsyslog-custom.conf

##########################################################################
#
#  Name:  /etc/rsyslog.d/rsyslog-custom.conf
#
#  Customize rsyslog configuration file to forward syslog events
#  via the imfile module instead of the imjournal module
#
#  The vApp centralized logger is configured to use RFC 3164
#  We will use the newer default of RFC 5424
#
#  The 'config' service ID has access to stop/start the rsyslog service
#  and access to the above reference configuration file
#
#  The 'config' service ID does NOT have access to /etc/rsyslog.conf to
#  disable the imjournal module, so we will mask the use of it
#  if the Centralized Logger was deployed.
#
#
#  ANA 10/2024
#
##########################################################################

# Reduce noise and/or duplicate messages
# Block messages from systemd (imjournal-related logs) or messages with facility local1 (imjournal-centralized-logger)
if ($programname == 'systemd' or $syslogfacility-text == 'local1') then {
    stop
}

# Enable debug as needed
# $DebugFile /var/tmp/rsyslog-debug.log
# $DebugLevel 2
# Use: cat /var/tmp/rsyslog-debug.log

# Global Settings
# Increase maxMessageSize from default of 8k to 64k
# To prevent loss of data from STDErr and IMPS etatrans logs due to size
global(maxMessageSize="64k")


# Use imfile module to monitor logs efficiently with inotify
module(load="imfile" mode="inotify")

# Define log locations with a good tag name to search on
# within any centralized log collector, e.g. Splunk, Grafana, etc
# vApp Main log
input(type="imfile"
      addMetadata="on"
      File="/opt/CA/VirtualAppliance/logs/ca_vapp_main.log"
      Tag="vapp-main-log"
      Facility="local5")

# vApp Deployment log
input(type="imfile"
      addMetadata="on"
      File="/opt/CA/VirtualAppliance/logs/ca_vapp_deployment.log"
      Tag="vapp-deployment-log"
      Facility="local5")

# IM Wildfly Server log: server.log
input(type="imfile"
      addMetadata="on"
      File="/opt/CA/VirtualAppliance/logs/IDM_logs/server.log"
      Tag="im-wildfly-server-log"
      Facility="local5")

# IP Wildfly Server log: server.log
input(type="imfile"
      addMetadata="on"
      File="/opt/CA/VirtualAppliance/logs/IP_logs/server.log"
      Tag="ip-wildfly-server-log"
      Facility="local5")

# IG Wildfly Server log: server.log
input(type="imfile"
      addMetadata="on"
      File="/opt/CA/VirtualAppliance/logs/IG_logs/server.log"
      Tag="ig-wildfly-server-log"
      Facility="local5")

# JCS Server log: jcs_daily.log
input(type="imfile"
      addMetadata="on"
      File="/opt/CA/VirtualAppliance/logs/ConnectorServer_logs/jcs_daily.log"
      Tag="jcs-daily-vapp-server-log"
      Facility="local5")

# IM Prov Server logs: etatrans*.log
input(type="imfile"
      addMetadata="on"
      File="/opt/CA/VirtualAppliance/logs/ProvisioningServer_logs/etatrans*.log"
      Tag="imps-etatrans-log"
      Facility="local5")

# IM Prov Server start-up log:  im_ps.log
input(type="imfile"
      addMetadata="on"
      File="/opt/CA/VirtualAppliance/logs/ProvisioningServer_logs/im_ps.log"
      Tag="imps-im-ps-log"
      Facility="local5")

# CA Directory logs [May wish to split apart to alarm, time, stat, trace]
# *-idm-userstore-router-caim-srv-*_warn_*.log
# *-idm-userstore-router-caim-srv-*_diag_*.log
# *-idm-userstore-router-caim-srv-*_stats_*.log
# *-idm-userstore-router-caim-srv-*_summary_*.log
# *-idm-userstore-router-caim-srv-*_alarm.log
# *-idm-userstore-router-caim-srv-*_trace.log
#
input(type="imfile"
      addMetadata="on"
      File="/opt/CA/Directory/dxserver/logs/*-idm-userstore-router-caim-srv-*_*_*.log"
      Tag="dir-idm-userstore-router-log"
      Facility="local5")

input(type="imfile"
      addMetadata="on"
      File="/opt/CA/Directory/dxserver/logs/*-idm-userstore-router-caim-srv-*_trace.log"
      Tag="dir-idm-userstore-router-trace-log"
      Facility="local5")

input(type="imfile"
      addMetadata="on"
      File="/opt/CA/Directory/dxserver/logs/*UserStore_userstore-*_*_*.log"
      Tag="dir-idm-userstore-log"
      Facility="local5")

input(type="imfile"
      addMetadata="on"
      File="/opt/CA/Directory/dxserver/logs/*UserStore_userstore-*_trace.log"
      Tag="dir-idm-userstore-trace-log"
      Facility="local5")

input(type="imfile"
      addMetadata="on"
      File="/opt/CA/Directory/dxserver/logs/*ca-prov-srv-*-imps_*.log"
      Tag="dir-imps-router-log"
      Facility="local5")

input(type="imfile"
      addMetadata="on"
      File="/opt/CA/Directory/dxserver/logs/*ca-prov-srv-*-imps_trace.log"
      Tag="dir-imps-router-trace-log"
      Facility="local5")

input(type="imfile"
      addMetadata="on"
      File="/opt/CA/Directory/dxserver/logs/*ca-prov-srv-*-impd-*_*_*.log"
      Tag="dir-impd-data-dsa-log"
      Facility="local5")

input(type="imfile"
      addMetadata="on"
      File="/opt/CA/Directory/dxserver/logs/*ca-prov-srv-*-impd-*_trace.log"
      Tag="dir-impd-data-dsa-trace-log"
      Facility="local5")

# Example: Create your own metrics data (key-value pair) for any component.
# Typically, this deployment is managed by a bash shell script that queries jboss-cli.sh
# and is executed by the 'config' crontab every few minutes.
# Why? TCP 9990 jboss management console web-services access is disabled by default on vApp

input(type="imfile"
      addMetadata="on"
      File="/opt/CA/VirtualAppliance/custom/metrics/jboss_metrics_latest_kv_splunk.log"
      Tag="vapp-idm-metrics-kv-log"
      Facility="local5")


# Recommendation:  Send logs to remote syslog server using TCP (reliable transmission)
# Adjust the target (ip/fqdn), port, & protocol to the intermediate or final syslog collector.
# We adding an action queue for better reliability and buffering in case of
# connection issues with TCP (if the syslog collector was restarted)
# Added a larger queue configuration with disk-based queue if needed
# To avoid possible duplicate of events being forwarded, we only allow logs with Facility local5 to be forwarded
if ($syslogfacility-text == "local5") then {
    action(
      type="omfwd"
      #target="log-srv"
      target="192.168.2.102"
      #port="514"
      port="10514"
      #protocol="udp"
      protocol="tcp"
      action.resumeRetryCount="-1"       # Retry indefinitely if the connection is lost
      queue.type="LinkedList"            # Use a linked list for the queue
      queue.size="50000"                 # Buffer up to 50,000 log messages in memory
      queue.dequeueBatchSize="100"       # Process 100 messages at a time when the queue is being flushed
      queue.highWatermark="40000"        # Start applying flow control when the queue size reaches 40,000
      queue.lowWatermark="10000"         # Resume normal operations when the queue size drops to 10,000
      queue.timeoutEnqueue="0"           # Do not drop messages, wait indefinitely if the queue is full
      queue.workerThreads="4"            # Use multiple worker threads to improve message throughput

      # Disk-based queue settings to handle overflow when memory queue fills up
      queue.spoolDirectory="/var/tmp/rsyslog-queue"   # Directory for disk-backed queue
      queue.maxDiskSpace="10g"                        # Maximum disk space for the queue
      queue.fileName="largequeue"                     # Name for disk queue files
      queue.saveOnShutdown="on"                       # Save queue to disk on shutdown to prevent data loss
    )
    stop  # Prevent further processing of local5 logs
}


# Restart rsyslog:   sudo systemctl restart rsyslog
#
# Note: Check /var/log/messages for any typo errors via
#   sudo systemctl restart rsyslog ; tail -n 100 -F /var/log/messages | grep -i syslog
#
# Optional: You may redirect additional facility levels as needed
# local5.* /var/log/local5.log
#
# Within Splunk UI - Use sourcetype="syslog"  and the tag strings provided

Key Points:

– Blocking Duplicate Logs:

We effectively block duplicate or noisy logs from systemd by filtering on `systemd` as the program name or `local1` as the facility.

– Disk-based Queue:

We’ve set up a disk-based queue under `/var/tmp/rsyslog-queue`, which is accessible to non-root service IDs like `config`. If the intermediate or final syslog collector is stopped, we will reduce the risk of “losted” events being forwarded.

– Log Forwarding:

Logs tagged with facility `local5` are forwarded to a remote syslog server using TCP for reliability.

This configuration ensures that logs critical to Symantec IGA operations are captured efficiently, forwarded reliably, and the system remains free from duplicate or irrelevant noise. By using a combination of filters, disk-based queues, and TCP forwarding, we achieve a robust logging mechanism for the Virtual Appliance.

If TCP is not available, we can fall back to UDP.

Splunk Enterprise (syslog UDP/TCP custom ports)

A few examples of integration with a centralized log collector.

vApp_logs_via_Syslog_to_Splunk Download

One of the challenges that IAM/IAG teams may have every few months is delivery or access for internal/external auditors to validate access within the IAM/IAG system and their managed endpoints.

Usually, auditors may directly access the 100’s system/endpoints/applications and randomly select a few or export the entire directory structure to review access. This effort takes time and possible 100’s of entitlements to grant temporary/expiry access to view. Auditors also prefer Excel or CSV files to review rather than fixed documents (PDF) to allow them to filter and isolate what interests them.

One process that may have value for your team is various tools with export functionality to CSV/XLS and the ability to query the 100’s-1000’s of systems from a single entry point.

A tool that we have found valuable over the years is SoftTerra LDAP Browser.

https://www.ldapadministrator.com/softerra-ldap-browser.htm

The multiple benefits from this tool for IAM/IAG are:

It is a read-only tool, so no mistakes can be made by granting too much access.
It has the ability to save queries that are popular and can be copied from other tools.
It has the ability to export the queries to CSV/XLS formats (plus others)
It can be used to pull reports from an IAM/IAG solution via their directory ports.
It can be used to pull reports from the managed applications (on-prem or SaaS) via the IAM provisioning directory ports.
The tool is free from SoftTerra, it is a limited version of their Administration tool

Example of the SoftTerra LDAP Browser tool used to query Active Directory, LDAP user stores, and Provisioning User Store & managed endpoints/applications.

A view to export Service Now (SNOW) accounts via the CA/Symantec Identity Manager Provisioning Server/Service (TCP 20390) via the LDAP/S protocol.

Why? The provisioning server may be viewed as a virtual directory/pass-through directory to the managed endpoints via its connector tier.

The below image shows SoftTerra LDAPBrowser used to connect to the Provisioning Server (TCP 20390). Then navigate to a Service Now (SNOW) managed endpoint, to query on all accounts and their respective profiles & entitlements. This same report/extract process may be done for mainframe/AS400 and client-server applications, e.g Active Directory, Unix, Databases, etc.

Enhance this process with defense-in-depth

We will not use the primary default administration account of the provisioning tier, “etaadmin”. Since this account has full access to change data.

Within the IAM/IAG solution, create an auditor account.

In the example below we create a new Global User, with the name “auditor”, a description, password, and a local “read-only admin profile” with an expiration date. This will allow the auditors to use the account as they wish (or you may grant this “read-only admin profile directly to their existing Global User ID). The account may still follow the same password reset expiration processes. If the account is marked as “restricted” in the CA/Symantec IM solution, then this account is limited how it may be changed to avoid any unexpected sync challenges to managed endpoints (if it was correlated to other accounts).

After the new Global User is created (or existing ID is added to the Admin Profile “ReadAdministrator”), update SoftTerra Credentials for the Provisioning Service. Below the new DN with “auditor” is shown in the credentials for login ID, e.g. “eTGlobalUserName=auditor,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta”

Now, the auditors may run as many reports as they would like, and export to spreadsheets or PDF files using a read-only account with a read-only tool.

Honorable mentions for other query tools.

Jxplorer is a useful & free java-based tool for reports, but this is a full edit tool & only exports out to LDIF format. http://jxplorer.org/

Apache Directory Studio is another very useful & free java-based tool for reports. This is a full edit tool. It does have the ability to export to many different formats. Since this tool does NOT need an MS Windows installer, and if the Desktop prevent installation, this is typically our 2nd choice to use. Extract and use the current java on the MS Windows OS or download AdoptOpenJDK and extract it to use with Apache Directory Studio. https://directory.apache.org/studio/ & https://adoptopenjdk.net/

SoftTerra LDAP Administrator is a paid and full edit tool. It has the same look-n-feel of the SoftTerra LDAP Browser tool. It is typically used by administrators of various LDAP solutions. We recommend this tool for your larger sites or if you would like a fast responsive tool on MS Windows OS. https://www.ldapadministrator.com/

If you have other recommendations, please leave a response.

Bonus Feature – SoftTerra AD Authentication

Both the SoftTerra tools allow binding using your existing authentication (on your desktop/laptop) into Active Directory. No need to create additional user ID for the auditors or yourself.

Perhaps the O365 or Outlook contacts process is not robust or too slow or perhaps you wish you had a more detail view of your internal active directory to view a manager’s direct reports. You can use this feature to view the the non-privacy attributes of your domain of all accounts with a read-only tool.

Step 01: Open a command-line prompt on your desktop/workstation after you have authenticated to your Active Directory domain & type set | findstr LOGONSERVER

Step 02: Install SoftTerra LDAP Browser Tool & Create a new profile

Step 03: Type the name of the Active Directory LOGONSERVER (aka Domain Controller) into the following fields & ensure “Use Secure Connection (SSL)” is selected (to avoid query issues).

Step 04: Click Next until you see “User Authentication Information” then select the radio button for “Currently logged on user (Active Directory)”, then click Finish button.

Step 05: After the profile is built, now click on the profile and watch it expand into a tree display of Active Directory. Select the branch that you believe has the list of users you would like to view, then select an individual user account, to see the values populated.

Step 06: If you wish to export this data to a spreadsheet (CSV/XLS), right click on the left object and select export option.

Step 07: You will have a series of options to export to & the file name it will write to.

Step 08: Advance search and export process. Select the branch that holds all the users you wish to view and export. Note: If the branch has 10,000 objects, this process may take minutes to complete depending on the query.

Step 09: The follow search windows will appear to help you create, save, and export your queries. Note that if you start to type in the field name, the list of the fields will start to appear.

Step 10: Ensure the FILTER is properly formed (use google to assist), and which attribute you wish to view or export is defined, then click search. If you are satisfied with your search, use the “Save Results” to export to a spreadsheet (CSV/XLS) or other format.

Tag: identity manager

Optimizing Syslog Configuration for Symantec Virtual Appliance IGA