Organizations may rely on software appliances to meet the demands of secure, mission-critical applications. To ensure the optimal operation of these appliances, Application Performance Monitoring/Management (APM) tools have emerged as invaluable assets. In this comprehensive blog post, we’ll explore how the synergy of APM host monitoring via the SysEdge module, APM Java, and APM JBoss(Wildfly/J2EE) can significantly enhance the performance and security of applications running on software appliances like the Symantec Identity Suite Virtual Appliance (on-prem and AWS instances).

1. Value of APM Java and APM JBoss(Wildfly) Monitoring

APM tools feature specialized modules for Java-based applications and JBoss(Wildfly/J2EE) application servers, addressing the unique challenges posed by these technologies:

• Optimizing Java-Based Applications: APM Java monitoring delves deep into Java applications, tracing bottlenecks, optimizing code, and ensuring the efficient use of Java Virtual Machine (JVM) resources.
• JBoss Application Server Expertise: APM JBoss monitoring tracks the performance and stability of JBoss deployments, providing insights into JBoss-specific metrics critical for the smooth operation of JBoss-based applications.

2. The Role of APM Host Monitoring with SysEdge Module

The SysEdge module, integrated into APM tools, plays a pivotal role in enhancing the performance, security, and overall management of software appliances. This module focuses on host-based metrics, offering insights into the appliance’s performance and health.

• Resource Utilization: SysEdge monitors essential resources, such as CPU, memory, disk space, and network usage, ensuring efficient resource allocation and preventing performance bottlenecks.
• Hardware Health: It provides insights into the hardware/virtual components, crucial for maintaining the reliability of the appliance.
• Comprehensive Diagnostics: The detailed host-based metrics allow for more accurate and rapid issue diagnostics, helping administrators identify and address problems efficiently. Gain real-time insight into digital performance, user experience and behavior

3. Benefits of APM Tools for Software Appliances

The integration of host-based metrics through SysEdge, along with APM Java and APM JBoss monitoring, offers a multitude of benefits:

• Holistic Insights: APM tools provide a complete picture of the appliance’s performance, helping administrators make informed decisions by combining application-specific data with host-based metrics. Ensure flawless user experiences with analytics-driven insights
• Proactive Issue Resolution: Administrators can proactively identify and address issues that may impact both application performance and the host system, reducing downtime and increasing reliability. Eliminate alert fatigue with automated root cause analysis.
• Streamlined Management: These tools enable remote management of the appliance, even in challenging environments, allowing fine-tuning, patch application, and addressing security concerns. Empower every team. Improve every app.

4. Secure Deployment with Non-Root User ID and DevOps Automation

The utilization of non-root user IDs and DevOps automation can significantly enhance both security and operational efficiency in the deployment and management of applications on software appliances. Traditional application deployments often involved elevated privileges, exposing them to security vulnerabilities. Deploying applications with non-root user IDs offers several advantages:

• Reduced Attack Surface: Non-root users have limited permissions, reducing the potential attack surface and making it more difficult for malicious actors to compromise the system.
• Enhanced Security: By limiting application permissions, non-root deployments minimize the risk of security breaches and unauthorized access.
• Compliance: Using non-root user IDs aligns with security best practices and compliance requirements, ensuring your organization meets regulatory standards.
• Isolation: Non-root deployments prevent applications from interfering with critical system components, reducing the risk of conflicts and crashes.

5. Example of integration/deployment of APM tools (Java/JBoss/SysEdge) on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

The Symantec Identity Suite Virtual Appliance is a harden software appliance that only allow authentication for one (1) of two (2) non-root IDs (config or ec2-user). The Symantec Identity Suite does allow for APM type tools to be deployed via extraction under the path /opt/CA/VirtualAppliance/custom/profiler.

We want to walk-through how to enable the DX APM SaaS Infra Agent with HostMonitoring (SysEdge) enabled on an Amazon Linux 2 host as a non-root user ID, as well as integration with the CA Identity Suite three (3) JBoss/Wildfly instances for IM/IG/IP and the CA Identity Suite JCS Connector Server with the embedded Java agent.

Additionally, we wanted to ensure that any external configuration access was disabled, as we only wanted to allow a “push” configuration/model of data from the vApp to the APM SaaS Collection APIs. We did not wish to allow any modification of the APM agent’s configuration on the vApp, that was not defined during initial deployment.

Four (4) parameters were modified from default installation:

1. Ensure non-root id is used for sysedge 

echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf

2. Ensure only local host can pull or send data to the sysedge agent

echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf  

3. Mask low value entries - Switch to debug loglevel as needed to address configuration challenges

echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf 

4. Disable remote management via APM Command Center (ACC)
      
sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile

Leveraging the information provided by the Broadcom DX Application Performance Management and Symantec Identity Suite documentation, we were able to clarify the dependencies on JBoss logging log4j jar(s) and LogManager objects with Adopt Open JDK 8. Neither document had the exact configuration that we found viable during testing.

We had two (2) challenges deploying the HostMonitoring (sysedge) agent on the AWS Identity Suite vApp instance that we were able to identify and address. No issues were found on the on-prem edition of the Identity Suite vApp.

• Challenge(s):
  • File ownership of the sysedge PID and log file by ‘root’ instead of the non-root user due to default systemd startup script for sysedge module/binary.
  • Null entry within a file impacted startup of ‘sysedge’ binary from default OS file /etc/redhat-release

The below bash shell script utilized the APM SaaS Binary download feature of embedding the sysedge module/binary component with the primary APM Infra agent. This will include the unique token for your own APM SaaS infrastructure (or APM Enterprise infra). The APM SaaS download process provides a great automation download via wget process. The Symantec Identity Suite allows the non-root IDs to start/stop systemd process, which we leverage. Alternatively, we may use crontab for the non-root IDs to start/stop the two (2) services: apmia and sysedge as documented for DX APM agents.

#!/bin/bash
####################################################################
#
#  Install the APM SaaS Infra Agent with HostMonitoring module (sysedge) as non-root id
#  - Update variables of NON_ROOT_USER_ID and INFRA_DOWNLOAD_URL and APM_INFRA_HOME_FOLDER
#  - This script works for a host that allows minimal sudo access to systemctl
#  - Alternative startup process is crontab for the non-root-id
#
#  Goal:  Replica process for:   ./APMIACtrl.sh install user=non-root-id
#   and ./APMIACtrl.sh console_start
#
#  Methodology:  Isolate delta between folders using diff with sub-folder detection
#   diff -iry --suppress-common-lines apmia/ apmia.orginal/
#
#  Important Note: Identified RCA for sysedge binary having memory fault SEGV
#  /etc/redhat-release MUST be populated (avoid null value)
#
#
#  Crontab notes from online APM agent docs: (if needed)
#    @reboot /home/user/apmia/APMIACtrl.sh console_start > /home/user/logs/cron.log 2>&1
#    */5 * * * * /home/user/apmia/APMIACtrl.sh console_start > /home/user/logs/cron.log 2>&1
#
#  Modify default APM Infra Agent parameters with these changes
#   1. Ensure non-root id is used for sysedge
#         echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   2. Ensure only local host can pull or send data to the sysedge agent
#        echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   3. Mask low value entries - Switch to debug loglevel as needed to address configuration challenges
#        echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   4. Disable acc integration (remote management) via APM Command Center (ACC)
#        sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile
#
#
#
#  Ref.  https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/install-and-deploy-infrastructure-agent/install-the-infrastructure-agent-on-ca-digital-experience-insights.html
#        https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/Supportability-Matrix-for-Infrastructure-Agent.html
#        https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/SystemEDGE-based-Monitoring.html
#
# ANA 10/2023
#
####################################################################
#NON_ROOT_USER_ID=config
NON_ROOT_USER_ID=ec2-user

echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM Infra Agent with HostMonitoring check / enabled - This will be packaged together "
echo "################################################################################################"
INFRA_DOWNLOAD_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXXXXXXXXXXXXXXXXX_LONG_TOKEN_HERE__XXXXXXXXXXX"


APM_INFRA_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_infra
mkdir -p ${APM_INFRA_HOME_FOLDER}
cd       ${APM_INFRA_HOME_FOLDER}
pwd
ls -lart

echo "wget --no-check-certificate --content-disposition ${INFRA_DOWNLOAD_URL} -O Infrastructure_Agent_apmia.tar"
wget --no-check-certificate --content-disposition ${INFRA_DOWNLOAD_URL} -O Infrastructure_Agent_apmia.tar
APM_INFRA_FILE_NAME=$(ls -lart Infrastructure_Agent_apmia* |tail -1 | awk '{print $9}')
echo "tar -xvf ${APM_INFRA_FILE_NAME} "
#tar -xvf ${APM_INFRA_FILE_NAME}
tar -xf ${APM_INFRA_FILE_NAME}


echo ""
echo ""
echo "################################################################################################"
echo "Update APM Infra Agent startup file to use non-root user ID of ${NON_ROOT_USER_ID} "
echo "################################################################################################"
sed -i "s|#RUN_AS_USER=|RUN_AS_USER=${NON_ROOT_USER_ID}|g" ${APM_INFRA_HOME_FOLDER}/apmia/bin/APMIAgent.sh

echo ""
echo ""
echo "################################################################################################"
echo "Validate update of NON_ROOT_USER_ID"
echo "################################################################################################"
grep -C 2 -i "RUN_AS_USER=${NON_ROOT_USER_ID}"  ${APM_INFRA_HOME_FOLDER}/apmia/bin/APMIAgent.sh


echo ""
echo ""
echo "################################################################################################"
echo "Extract SystemEdge component for APM Infra Host Monitoring"
echo "################################################################################################"
export AGENTHOME=${APM_INFRA_HOME_FOLDER}/apmia
SYSEDGE_FILE_NAME=$(ls ${AGENTHOME}/casystemedge*)
echo ${SYSEDGE_FILE_NAME}
cd ${AGENTHOME}
#tar -xvf ${SYSEDGE_FILE_NAME}
tar -xf ${SYSEDGE_FILE_NAME}


echo ""
echo ""
echo "################################################################################################"
echo "Deploy and install SystemEdge component for APM Infra Host Monitoring with non-root user ID"
echo "################################################################################################"
kill $(pidof sysedge) &>/dev/null
rm -rf ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE &>/dev/null
cd ${APM_INFRA_HOME_FOLDER}/apmia/CA_SystemEDGE_Core
./ca-setup.sh install
/bin/ps -ef | grep -i sysedge | grep -v grep


echo ""
echo ""
echo "################################################################################################"
echo "Update and restart the SystemEdge component for APM Infra Host Monitoring with non-root user ID"
echo "################################################################################################"
# Ensure non-root id is used for sysedge
echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
# Ensure only local host can pull or send data to the sysedge agent
echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
# Mask low value entries - Switch to debug loglevel as needed to address configuration challenges
echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
cp -r -p ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf  ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl stop
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl start
/bin/ps -ef | grep -i sysedge | grep -v grep

echo ""
echo ""
echo "################################################################################################"
echo "Check updates to sysedge configuration file sysedge.cf "
echo "################################################################################################"
#tail -5 ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf
grep -Hin -v -e "^$" -e "^#" -e "^template" ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf


echo ""
echo ""
echo "################################################################################################"
echo "Update the APM Infra main agent and disable the acc component "
echo "################################################################################################"
sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile
grep "introscope.agent.acc.enable" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile


echo ""
echo ""
echo "################################################################################################"
echo "Create systemd startup process on vApp due to sudo systemctl process allowed on vApp for APM Infra main agent"
echo "################################################################################################"

cat << EOF > ${APM_INFRA_HOME_FOLDER}/apmia/apmia.service
# /etc/systemd/system/apmia.service
[Unit]
Description=APM Infrastructure Agent
After=syslog.target

[Service]
Type=forking
ExecStart="${APM_INFRA_HOME_FOLDER}/apmia/bin/./APMIAgent.sh" start sysd
ExecStop="${APM_INFRA_HOME_FOLDER}/apmia/bin/./APMIAgent.sh" stop sysd
User=${NON_ROOT_USER_ID}
KillMode=control-group
Environment=SYSTEMD_KILLMODE_WARNING=true

[Install]
WantedBy=multi-user.target

EOF

sudo systemctl stop          apmia.service &>/dev/null
sudo systemctl disable       apmia.service &>/dev/null
sudo systemctl enable ${APM_INFRA_HOME_FOLDER}/apmia/apmia.service
echo "################################################################################################"
sudo systemctl cat           apmia.service
echo "################################################################################################"
sudo systemctl daemon-reload
sudo systemctl start         apmia.service
sudo systemctl status        apmia.service -a -l --no-pager



echo ""
echo ""
echo "################################################################################################"
echo "Create systemd startup process on vApp due to sudo systemctl process allowed on vApp for Sysedge agent"
echo "################################################################################################"
# Stop sysedge via manual process to use the systemd process
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl stop

# Manage these two (2) systemd error with PIDs and Paths
# Refusing to accept PID outside of service control group, acquired through unsafe symlink chain
# /opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/config/port1691/sysedge.service:8] Not an absolute path
#
cat << EOF > ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service
# /etc/systemd/system/sysedge.service
[Unit]
Description=sysedge
After=syslog.target

[Service]
Type=forking
WorkingDirectory=${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691
#Environment=VAR_HERE_ENV_ENV=production PATH=PATH_HERE_IF_NEEDED
ExecStart="${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/CA-SystemEDGE" start sysd
ExecStop="${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/CA-SystemEDGE" stop sysd
PIDFile=${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.pid
User=${NON_ROOT_USER_ID}
KillMode=none

[Install]
WantedBy=multi-user.target

EOF
echo ""
echo ""
echo "################################################################################################"
cat ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service
echo "################################################################################################"
echo ""
echo ""
sudo systemctl stop          sysedge.service  &>/dev/null
sudo systemctl disable       sysedge.service  &>/dev/null
sudo systemctl enable "${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service"
echo "################################################################################################"
sudo systemctl cat           sysedge.service
echo "################################################################################################"
sudo systemctl daemon-reload
sudo systemctl start         sysedge.service
sudo systemctl status        sysedge.service -a -l --no-pager



echo ""
echo ""
echo "################################################################################################"
echo "Ensure all files are owned by the non-root id of ${NON_ROOT_USER_ID} for APM Infra Agent"
echo "Check for any error messages "
echo "################################################################################################"
sudo systemctl stop          apmia.service
sudo systemctl stop          sysedge.service
echo "chown -R ${NON_ROOT_USER_ID}:${NON_ROOT_USER_ID} ${APM_INFRA_HOME_FOLDER}/apmia "
echo "################################################################################################"
chown -R ${NON_ROOT_USER_ID}:${NON_ROOT_USER_ID} ${APM_INFRA_HOME_FOLDER}/apmia
echo ""
echo ""

echo ""
echo ""
echo "################################################################################################"
echo "View running services: APM Infra will have a parent process of wrapper and two (2) java child processes"
echo "The APM Infra HostMonitoring module will have one process name sysedge "
echo "################################################################################################"
sudo systemctl start          apmia.service
sudo systemctl start          sysedge.service
ps -ef | grep apmia | grep -v grep



echo ""
echo ""
echo "################################################################################################"
echo "Use these these commands to view / monitor / start & stop APM Infra Agent"
echo "################################################################################################"
echo "sudo systemctl status  apmia.service -a -l --no-pager"
echo "sudo systemctl stop    apmia.service"
echo "sudo systemctl start   apmia.service"
echo "sudo systemctl cat     apmia.service"
echo "journalctl          -u apmia.service -f"
echo ""
echo "sudo systemctl status  sysedge.service -a -l --no-pager"
echo "sudo systemctl stop    sysedge.service"
echo "sudo systemctl start   sysedge.service"
echo "sudo systemctl cat     sysedge.service"
echo "journalctl          -u sysedge.service -f"
echo "journalctl          -u sysedge.service -xe -f"
echo ""
echo ""

6. Example of JVM arguments for integration of APM SaaS Java Agent with Identity Suite JCS Connector Server)

The APM SaaS or APM Enterprise UI solution will provide an agent download page, that allows section of the agent to be deployed.

Instruction will be provided and please noticed the “wget” auto download link that may be leveraged for automation (dev-ops) processes. This link will include both the siteID and the download token ID.

If you haven’t generated a new credentials token, after you download the Java/Jboss agents, click “Show Agent Details” to harvest the three (3) key/value pairs.

The below bash shell script will download the APM Java Agent package. There is no embedded credential token with this download. The credentials are provided separately via the APM SaaS UI (as shown above). While we could place these three (3) parameters within the APM agent configuration file, IntroscopeAgent.profile, we decided to clarity the use of JVM switches to override any values, and allow us to automate this deployment independent of any new APM agents updates.

#!/bin/bash
######################################################################################
#
#  Automate deployment of the APM SaaS Java Agent with credentials & urls
#   to the single Identity Manager JCS Connector Server instance
#
#  Use variables for the JVM parameters.  Adjust if needed.
#  We may override the default naming convention to clarity which instance is
#  being monitoring within the APM SaaS Dashboard, to avoid confusion with any
#  JBoss agent instance from IM/IG/IP
#
#  ANA 10/2023
#
######################################################################################
echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM Java Agent "
echo "################################################################################################"
APM_CREDENTIAL_TOKEN='XXXXXXX_LONG_TOKEN_HERE_FROM_APM_SAAS_UI__XXXXXXX'
APM_URL='apmgw.dxi-na1.saas.broadcom.com'
APM_SAAS_AGENT_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXX_DOWNLOAD_TOKEN_PROVIDED_FROM_APM_SAAS_UI_WHEN_SELECTED___XXXXXXXXXXXX"
APM_AGENT_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_java
APM_AGENT_FILE_NAME="APM_SaaS_Java_Agent.tar"
mkdir -p ${APM_AGENT_HOME_FOLDER}
cd       ${APM_AGENT_HOME_FOLDER}
pwd
ls -lart


echo "wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}"
wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}
ls -lart
echo "tar -xvf ${APM_AGENT_FILE_NAME} "
#tar -xvf ${APM_AGENT_FILE_NAME}
tar -xf ${APM_AGENT_FILE_NAME}
ls -lart
#
#
tz=`/bin/date --utc +%Y%m%d%H%M%S`
APM_AGENT_NAME=IM_JCS_NODE
JVM_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/profiler/
JVM_FILE=/opt/CA/IdentityManager/ConnectorServer/data/jvm_options.conf

if [ -f ${JVM_FILE} ];then

cp -r -p ${JVM_FILE}  ${JVM_BACKUP_LOCATION}/${tz}_jvm_options.conf

echo "-server -Xms1g -Xmx2g -Djava.awt.headless=true -Dcom.sun.net.ssl.enableECC=true -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9  -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIAL_TOKEN}\"  -Dcom.wily.introscope.agent.agentName=${APM_AGENT_NAME}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${JCS_JVM_FILE}

echo "Start / Stop JCS"

echo "sudo systemctl stop im_jcs "
sudo systemctl stop im_jcs

echo "sudo systemctl start im_jcs "
sudo systemctl start im_jcs

}

7. Example of JVM arguments for integration of APM SaaS JBoss Agent with Identity Suite IM/IG/IP JBoss/Wildfly instances

The below bash shell script is for the three (3) JBoss (Wildfly) instances on the vApp. Fortunately for us, the Broadcom/Symantec Product/Engineering team kept the same log4j jar and class files version between all three (3) instances, so we were able to leverage variables for each. You may wish to adjust the JVM parameters as needed. Please note, that again we did not embed any key/value pair into the APM configuration file, as we wanted to automate this deployment independent of any new APM agents updates.

#!/bin/bash
#########################################################################
#
#  Automate deployment of the APM SaaS JBOSS Agent with credentials & urls
#   to the three JBoss/Wildfly instances
#     - IdentityManager
#     - IdentityGovernance
#     - IdentityPortal
#
#  Use variables for the JVM parameters that allow a similar configuration for
#  all three (3) instances.  Adjust if needed.  Since all three (3) JBoss are on the
#  same host, we will override the default naming convention to clarity which instance is
#  being monitoring within the APM SaaS Dashboard
#
#  ANA 10/2023
#
#########################################################################
echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM JBoss Agent "
echo "################################################################################################"
APM_SAAS_AGENT_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXX__DOWNLOAD_TOKEN_HERE"
APM_CREDENTIALS='XXXXX_LONG_CREDENTIAL_TOKEN_HERE__XXXXXX'
APM_URL='apmgw.dxi-na1.saas.broadcom.com'
APM_AGENT_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_jboss
APM_AGENT_FILE_NAME="APM_SaaS_JBOSS_Agent.tar"
mkdir -p ${APM_AGENT_HOME_FOLDER}
cd       ${APM_AGENT_HOME_FOLDER}
pwd
ls -lart


echo "wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}"
wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}
echo "tar -xvf ${APM_AGENT_FILE_NAME} "
#tar -xvf ${APM_AGENT_FILE_NAME}
tar -xf ${APM_AGENT_FILE_NAME}
ls -lart
tz=`/bin/date --utc +%Y%m%d%H%M%S`



JBOSS_INSTANCE=IdentityManager
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf

if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "
cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-idm "
sudo systemctl stop wildfly-idm

echo "sudo systemctl start wildfly-idm "
sudo systemctl start wildfly-idm

fi

JBOSS_INSTANCE=IdentityGovernance
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf
if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "

cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-ig "
sudo systemctl stop wildfly-ig
echo "sudo systemctl start wildfly-ig "
sudo systemctl start wildfly-ig

fi


JBOSS_INSTANCE=IdentityPortal
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf
if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "
cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-portal "
sudo systemctl stop wildfly-portal
echo "sudo systemctl start wildfly-portal "
sudo systemctl start wildfly-portal

fi

8. Conclusion of APM tools integration on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

We have been successful using APM tool and home-grown java monitoring processes to improve the performance of the Symantec Identity Suite solution for peak use-cases. Using these processes should allow you to peak into the ‘blackbox’ of Java/JBoss processes to understand where bottlenecks may exists.

9. View sysedge.cf configuration for proposed change to isolate behavior to single host

View of the configuration file for sysedge.cf via cat sysedge.cf | grep -i -v -e “^$” -e “^#” -e “^template”

10. Prior notes / examples of integration/deployment of APM tools (Java/JBoss) on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

We have been involved with use of APM tools for quite a bit and contributed to the inclusion of these features into various solution, including software appliances like the Symantec Identity Suite with non-root access.

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=2197&MID=773616&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

Engage with us to help accelerate your business goals