In this blog entry, we will dive into the latest enhancements to the custom `rsyslog` configuration designed for managing logs on the Symantec Identity Governance and Administration (IGA) system running on the Symantec Virtual Appliance (vApp). The previous configuration has been refined to block unwanted systemd (imjournal-related) logs, adjust logging formats, and ensure reliable forwarding of critical logs to centralized collectors like Splunk or Grafana.
This setup is particularly useful when the service ID (such as `config`) has limited access to the root-level `rsyslog.conf` but can modify configurations in the `/etc/rsyslog.d/` directory. We’ll also implement a disk-based queue mechanism to handle potential overflows during log forwarding.
Introduction
The Virtual Appliance (vApp) uses centralized logging with RFC 3164 by default, but we’re updating this configuration to utilize RFC 5424, a newer standard for better log management. Moreover, since the `config` service ID has no access to `/etc/rsyslog.conf` to disable the `imjournal` module, we’ll mask these systemd-related logs at the configuration level, ensuring a cleaner log output without noise.
Let’s look at how we can customize the `rsyslog` setup to optimize logging for Symantec IGA, block unwanted logs, and ensure reliable log forwarding.
Key Changes in the Updated Configuration:
1. Blocking Duplicate or Noisy Logs:
We’ll block messages originating from systemd (via `imjournal`) and logs using facility `local1`, which is often tied to systemd logging, to avoid duplicates.
2. Efficient Log Monitoring:
The `imfile` module will be used for monitoring various critical log files from the vApp, including CA Directory, Provisioning Server, Wildfly (Identity Manager, Identity Portal, Identity Governance), ConnectorServer, and Wildfly metrics (accessed via jboss-cli.sh)
3. Reliable Log Forwarding:
Logs will be forwarded to a remote syslog collector using UDP or TCP with a built-in disk-backed queue for reliability in case of connection issues.
Example: /etc/rsyslog.d/rsyslog-custom.conf
##########################################################################
#
# Name: /etc/rsyslog.d/rsyslog-custom.conf
#
# Customize rsyslog configuration file to forward syslog events
# via the imfile module instead of the imjournal module
#
# The vApp centralized logger is configured to use RFC 3164
# We will use the newer default of RFC 5424
#
# The 'config' service ID has access to stop/start the rsyslog service
# and access to the above reference configuration file
#
# The 'config' service ID does NOT have access to /etc/rsyslog.conf to
# disable the imjournal module, so we will mask the use of it
# if the Centralized Logger was deployed.
#
#
# ANA 10/2024
#
##########################################################################
# Reduce noise and/or duplicate messages
# Block messages from systemd (imjournal-related logs) or messages with facility local1 (imjournal-centralized-logger)
if ($programname == 'systemd' or $syslogfacility-text == 'local1') then {
stop
}
# Enable debug as needed
# $DebugFile /var/tmp/rsyslog-debug.log
# $DebugLevel 2
# Use: cat /var/tmp/rsyslog-debug.log
# Global Settings
# Increase maxMessageSize from default of 8k to 64k
# To prevent loss of data from STDErr and IMPS etatrans logs due to size
global(maxMessageSize="64k")
# Use imfile module to monitor logs efficiently with inotify
module(load="imfile" mode="inotify")
# Define log locations with a good tag name to search on
# within any centralized log collector, e.g. Splunk, Grafana, etc
# vApp Main log
input(type="imfile"
addMetadata="on"
File="/opt/CA/VirtualAppliance/logs/ca_vapp_main.log"
Tag="vapp-main-log"
Facility="local5")
# vApp Deployment log
input(type="imfile"
addMetadata="on"
File="/opt/CA/VirtualAppliance/logs/ca_vapp_deployment.log"
Tag="vapp-deployment-log"
Facility="local5")
# IM Wildfly Server log: server.log
input(type="imfile"
addMetadata="on"
File="/opt/CA/VirtualAppliance/logs/IDM_logs/server.log"
Tag="im-wildfly-server-log"
Facility="local5")
# IP Wildfly Server log: server.log
input(type="imfile"
addMetadata="on"
File="/opt/CA/VirtualAppliance/logs/IP_logs/server.log"
Tag="ip-wildfly-server-log"
Facility="local5")
# IG Wildfly Server log: server.log
input(type="imfile"
addMetadata="on"
File="/opt/CA/VirtualAppliance/logs/IG_logs/server.log"
Tag="ig-wildfly-server-log"
Facility="local5")
# JCS Server log: jcs_daily.log
input(type="imfile"
addMetadata="on"
File="/opt/CA/VirtualAppliance/logs/ConnectorServer_logs/jcs_daily.log"
Tag="jcs-daily-vapp-server-log"
Facility="local5")
# IM Prov Server logs: etatrans*.log
input(type="imfile"
addMetadata="on"
File="/opt/CA/VirtualAppliance/logs/ProvisioningServer_logs/etatrans*.log"
Tag="imps-etatrans-log"
Facility="local5")
# IM Prov Server start-up log: im_ps.log
input(type="imfile"
addMetadata="on"
File="/opt/CA/VirtualAppliance/logs/ProvisioningServer_logs/im_ps.log"
Tag="imps-im-ps-log"
Facility="local5")
# CA Directory logs [May wish to split apart to alarm, time, stat, trace]
# *-idm-userstore-router-caim-srv-*_warn_*.log
# *-idm-userstore-router-caim-srv-*_diag_*.log
# *-idm-userstore-router-caim-srv-*_stats_*.log
# *-idm-userstore-router-caim-srv-*_summary_*.log
# *-idm-userstore-router-caim-srv-*_alarm.log
# *-idm-userstore-router-caim-srv-*_trace.log
#
input(type="imfile"
addMetadata="on"
File="/opt/CA/Directory/dxserver/logs/*-idm-userstore-router-caim-srv-*_*_*.log"
Tag="dir-idm-userstore-router-log"
Facility="local5")
input(type="imfile"
addMetadata="on"
File="/opt/CA/Directory/dxserver/logs/*-idm-userstore-router-caim-srv-*_trace.log"
Tag="dir-idm-userstore-router-trace-log"
Facility="local5")
input(type="imfile"
addMetadata="on"
File="/opt/CA/Directory/dxserver/logs/*UserStore_userstore-*_*_*.log"
Tag="dir-idm-userstore-log"
Facility="local5")
input(type="imfile"
addMetadata="on"
File="/opt/CA/Directory/dxserver/logs/*UserStore_userstore-*_trace.log"
Tag="dir-idm-userstore-trace-log"
Facility="local5")
input(type="imfile"
addMetadata="on"
File="/opt/CA/Directory/dxserver/logs/*ca-prov-srv-*-imps_*.log"
Tag="dir-imps-router-log"
Facility="local5")
input(type="imfile"
addMetadata="on"
File="/opt/CA/Directory/dxserver/logs/*ca-prov-srv-*-imps_trace.log"
Tag="dir-imps-router-trace-log"
Facility="local5")
input(type="imfile"
addMetadata="on"
File="/opt/CA/Directory/dxserver/logs/*ca-prov-srv-*-impd-*_*_*.log"
Tag="dir-impd-data-dsa-log"
Facility="local5")
input(type="imfile"
addMetadata="on"
File="/opt/CA/Directory/dxserver/logs/*ca-prov-srv-*-impd-*_trace.log"
Tag="dir-impd-data-dsa-trace-log"
Facility="local5")
# Example: Create your own metrics data (key-value pair) for any component.
# Typically, this deployment is managed by a bash shell script that queries jboss-cli.sh
# and is executed by the 'config' crontab every few minutes.
# Why? TCP 9990 jboss management console web-services access is disabled by default on vApp
input(type="imfile"
addMetadata="on"
File="/opt/CA/VirtualAppliance/custom/metrics/jboss_metrics_latest_kv_splunk.log"
Tag="vapp-idm-metrics-kv-log"
Facility="local5")
# Recommendation: Send logs to remote syslog server using TCP (reliable transmission)
# Adjust the target (ip/fqdn), port, & protocol to the intermediate or final syslog collector.
# We adding an action queue for better reliability and buffering in case of
# connection issues with TCP (if the syslog collector was restarted)
# Added a larger queue configuration with disk-based queue if needed
# To avoid possible duplicate of events being forwarded, we only allow logs with Facility local5 to be forwarded
if ($syslogfacility-text == "local5") then {
action(
type="omfwd"
#target="log-srv"
target="192.168.2.102"
#port="514"
port="10514"
#protocol="udp"
protocol="tcp"
action.resumeRetryCount="-1" # Retry indefinitely if the connection is lost
queue.type="LinkedList" # Use a linked list for the queue
queue.size="50000" # Buffer up to 50,000 log messages in memory
queue.dequeueBatchSize="100" # Process 100 messages at a time when the queue is being flushed
queue.highWatermark="40000" # Start applying flow control when the queue size reaches 40,000
queue.lowWatermark="10000" # Resume normal operations when the queue size drops to 10,000
queue.timeoutEnqueue="0" # Do not drop messages, wait indefinitely if the queue is full
queue.workerThreads="4" # Use multiple worker threads to improve message throughput
# Disk-based queue settings to handle overflow when memory queue fills up
queue.spoolDirectory="/var/tmp/rsyslog-queue" # Directory for disk-backed queue
queue.maxDiskSpace="10g" # Maximum disk space for the queue
queue.fileName="largequeue" # Name for disk queue files
queue.saveOnShutdown="on" # Save queue to disk on shutdown to prevent data loss
)
stop # Prevent further processing of local5 logs
}
# Restart rsyslog: sudo systemctl restart rsyslog
#
# Note: Check /var/log/messages for any typo errors via
# sudo systemctl restart rsyslog ; tail -n 100 -F /var/log/messages | grep -i syslog
#
# Optional: You may redirect additional facility levels as needed
# local5.* /var/log/local5.log
#
# Within Splunk UI - Use sourcetype="syslog" and the tag strings provided
Key Points:
– Blocking Duplicate Logs:
We effectively block duplicate or noisy logs from systemd by filtering on `systemd` as the program name or `local1` as the facility.
– Disk-based Queue:
We’ve set up a disk-based queue under `/var/tmp/rsyslog-queue`, which is accessible to non-root service IDs like `config`. If the intermediate or final syslog collector is stopped, we will reduce the risk of “losted” events being forwarded.
– Log Forwarding:
Logs tagged with facility `local5` are forwarded to a remote syslog server using TCP for reliability.
This configuration ensures that logs critical to Symantec IGA operations are captured efficiently, forwarded reliably, and the system remains free from duplicate or irrelevant noise. By using a combination of filters, disk-based queues, and TCP forwarding, we achieve a robust logging mechanism for the Virtual Appliance.
If TCP is not available, we can fall back to UDP.
Splunk Enterprise (syslog UDP/TCP custom ports)
A few examples of integration with a centralized log collector.
