Tag: management

Elevating Application Performance and Security on Software Appliances with APM Host Monitoring, APM Java, and APM JBoss(Wildfly)

Posted on by Leave a comment

Organizations may rely on software appliances to meet the demands of secure, mission-critical applications. To ensure the optimal operation of these appliances, Application Performance Monitoring/Management (APM) tools have emerged as invaluable assets. In this comprehensive blog post, we’ll explore how the synergy of APM host monitoring via the SysEdge module, APM Java, and APM JBoss(Wildfly/J2EE) can significantly enhance the performance and security of applications running on software appliances like the Symantec Identity Suite Virtual Appliance (on-prem and AWS instances).

1. Value of APM Java and APM JBoss(Wildfly) Monitoring

APM tools feature specialized modules for Java-based applications and JBoss(Wildfly/J2EE) application servers, addressing the unique challenges posed by these technologies:

  • Optimizing Java-Based Applications: APM Java monitoring delves deep into Java applications, tracing bottlenecks, optimizing code, and ensuring the efficient use of Java Virtual Machine (JVM) resources.
  • JBoss Application Server Expertise: APM JBoss monitoring tracks the performance and stability of JBoss deployments, providing insights into JBoss-specific metrics critical for the smooth operation of JBoss-based applications.

2. The Role of APM Host Monitoring with SysEdge Module

The SysEdge module, integrated into APM tools, plays a pivotal role in enhancing the performance, security, and overall management of software appliances. This module focuses on host-based metrics, offering insights into the appliance’s performance and health.

  • Resource Utilization: SysEdge monitors essential resources, such as CPU, memory, disk space, and network usage, ensuring efficient resource allocation and preventing performance bottlenecks.
  • Hardware Health: It provides insights into the hardware/virtual components, crucial for maintaining the reliability of the appliance.
  • Comprehensive Diagnostics: The detailed host-based metrics allow for more accurate and rapid issue diagnostics, helping administrators identify and address problems efficiently. Gain real-time insight into digital performance, user experience and behavior

3. Benefits of APM Tools for Software Appliances

The integration of host-based metrics through SysEdge, along with APM Java and APM JBoss monitoring, offers a multitude of benefits:

  • Holistic Insights: APM tools provide a complete picture of the appliance’s performance, helping administrators make informed decisions by combining application-specific data with host-based metrics. Ensure flawless user experiences with analytics-driven insights
  • Proactive Issue Resolution: Administrators can proactively identify and address issues that may impact both application performance and the host system, reducing downtime and increasing reliability. Eliminate alert fatigue with automated root cause analysis.
  • Streamlined Management: These tools enable remote management of the appliance, even in challenging environments, allowing fine-tuning, patch application, and addressing security concerns. Empower every team. Improve every app.

4. Secure Deployment with Non-Root User ID and DevOps Automation

The utilization of non-root user IDs and DevOps automation can significantly enhance both security and operational efficiency in the deployment and management of applications on software appliances. Traditional application deployments often involved elevated privileges, exposing them to security vulnerabilities. Deploying applications with non-root user IDs offers several advantages:

  • Reduced Attack Surface: Non-root users have limited permissions, reducing the potential attack surface and making it more difficult for malicious actors to compromise the system.
  • Enhanced Security: By limiting application permissions, non-root deployments minimize the risk of security breaches and unauthorized access.
  • Compliance: Using non-root user IDs aligns with security best practices and compliance requirements, ensuring your organization meets regulatory standards.
  • Isolation: Non-root deployments prevent applications from interfering with critical system components, reducing the risk of conflicts and crashes.

5. Example of integration/deployment of APM tools (Java/JBoss/SysEdge) on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

The Symantec Identity Suite Virtual Appliance is a harden software appliance that only allow authentication for one (1) of two (2) non-root IDs (config or ec2-user). The Symantec Identity Suite does allow for APM type tools to be deployed via extraction under the path /opt/CA/VirtualAppliance/custom/profiler.

We want to walk-through how to enable the DX APM SaaS Infra Agent with HostMonitoring (SysEdge) enabled on an Amazon Linux 2 host as a non-root user ID, as well as integration with the CA Identity Suite three (3) JBoss/Wildfly instances for IM/IG/IP and the CA Identity Suite JCS Connector Server with the embedded Java agent.

Additionally, we wanted to ensure that any external configuration access was disabled, as we only wanted to allow a “push” configuration/model of data from the vApp to the APM SaaS Collection APIs. We did not wish to allow any modification of the APM agent’s configuration on the vApp, that was not defined during initial deployment.

Four (4) parameters were modified from default installation:

1. Ensure non-root id is used for sysedge 

echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf

2. Ensure only local host can pull or send data to the sysedge agent

echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf  

3. Mask low value entries - Switch to debug loglevel as needed to address configuration challenges

echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf 

4. Disable remote management via APM Command Center (ACC)
      
sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile

Leveraging the information provided by the Broadcom DX Application Performance Management and Symantec Identity Suite documentation, we were able to clarify the dependencies on JBoss logging log4j jar(s) and LogManager objects with Adopt Open JDK 8. Neither document had the exact configuration that we found viable during testing.

We had two (2) challenges deploying the HostMonitoring (sysedge) agent on the AWS Identity Suite vApp instance that we were able to identify and address. No issues were found on the on-prem edition of the Identity Suite vApp.

  • Challenge(s):
    • File ownership of the sysedge PID and log file by ‘root’ instead of the non-root user due to default systemd startup script for sysedge module/binary.
    • Null entry within a file impacted startup of ‘sysedge’ binary from default OS file /etc/redhat-release

The below bash shell script utilized the APM SaaS Binary download feature of embedding the sysedge module/binary component with the primary APM Infra agent. This will include the unique token for your own APM SaaS infrastructure (or APM Enterprise infra). The APM SaaS download process provides a great automation download via wget process. The Symantec Identity Suite allows the non-root IDs to start/stop systemd process, which we leverage. Alternatively, we may use crontab for the non-root IDs to start/stop the two (2) services: apmia and sysedge as documented for DX APM agents.

#!/bin/bash
####################################################################
#
#  Install the APM SaaS Infra Agent with HostMonitoring module (sysedge) as non-root id
#  - Update variables of NON_ROOT_USER_ID and INFRA_DOWNLOAD_URL and APM_INFRA_HOME_FOLDER
#  - This script works for a host that allows minimal sudo access to systemctl
#  - Alternative startup process is crontab for the non-root-id
#
#  Goal:  Replica process for:   ./APMIACtrl.sh install user=non-root-id
#   and ./APMIACtrl.sh console_start
#
#  Methodology:  Isolate delta between folders using diff with sub-folder detection
#   diff -iry --suppress-common-lines apmia/ apmia.orginal/
#
#  Important Note: Identified RCA for sysedge binary having memory fault SEGV
#  /etc/redhat-release MUST be populated (avoid null value)
#
#
#  Crontab notes from online APM agent docs: (if needed)
#    @reboot /home/user/apmia/APMIACtrl.sh console_start > /home/user/logs/cron.log 2>&1
#    */5 * * * * /home/user/apmia/APMIACtrl.sh console_start > /home/user/logs/cron.log 2>&1
#
#  Modify default APM Infra Agent parameters with these changes
#   1. Ensure non-root id is used for sysedge
#         echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   2. Ensure only local host can pull or send data to the sysedge agent
#        echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   3. Mask low value entries - Switch to debug loglevel as needed to address configuration challenges
#        echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   4. Disable acc integration (remote management) via APM Command Center (ACC)
#        sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile
#
#
#
#  Ref.  https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/install-and-deploy-infrastructure-agent/install-the-infrastructure-agent-on-ca-digital-experience-insights.html
#        https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/Supportability-Matrix-for-Infrastructure-Agent.html
#        https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/SystemEDGE-based-Monitoring.html
#
# ANA 10/2023
#
####################################################################
#NON_ROOT_USER_ID=config
NON_ROOT_USER_ID=ec2-user

echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM Infra Agent with HostMonitoring check / enabled - This will be packaged together "
echo "################################################################################################"
INFRA_DOWNLOAD_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXXXXXXXXXXXXXXXXX_LONG_TOKEN_HERE__XXXXXXXXXXX"


APM_INFRA_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_infra
mkdir -p ${APM_INFRA_HOME_FOLDER}
cd       ${APM_INFRA_HOME_FOLDER}
pwd
ls -lart

echo "wget --no-check-certificate --content-disposition ${INFRA_DOWNLOAD_URL} -O Infrastructure_Agent_apmia.tar"
wget --no-check-certificate --content-disposition ${INFRA_DOWNLOAD_URL} -O Infrastructure_Agent_apmia.tar
APM_INFRA_FILE_NAME=$(ls -lart Infrastructure_Agent_apmia* |tail -1 | awk '{print $9}')
echo "tar -xvf ${APM_INFRA_FILE_NAME} "
#tar -xvf ${APM_INFRA_FILE_NAME}
tar -xf ${APM_INFRA_FILE_NAME}


echo ""
echo ""
echo "################################################################################################"
echo "Update APM Infra Agent startup file to use non-root user ID of ${NON_ROOT_USER_ID} "
echo "################################################################################################"
sed -i "s|#RUN_AS_USER=|RUN_AS_USER=${NON_ROOT_USER_ID}|g" ${APM_INFRA_HOME_FOLDER}/apmia/bin/APMIAgent.sh

echo ""
echo ""
echo "################################################################################################"
echo "Validate update of NON_ROOT_USER_ID"
echo "################################################################################################"
grep -C 2 -i "RUN_AS_USER=${NON_ROOT_USER_ID}"  ${APM_INFRA_HOME_FOLDER}/apmia/bin/APMIAgent.sh


echo ""
echo ""
echo "################################################################################################"
echo "Extract SystemEdge component for APM Infra Host Monitoring"
echo "################################################################################################"
export AGENTHOME=${APM_INFRA_HOME_FOLDER}/apmia
SYSEDGE_FILE_NAME=$(ls ${AGENTHOME}/casystemedge*)
echo ${SYSEDGE_FILE_NAME}
cd ${AGENTHOME}
#tar -xvf ${SYSEDGE_FILE_NAME}
tar -xf ${SYSEDGE_FILE_NAME}


echo ""
echo ""
echo "################################################################################################"
echo "Deploy and install SystemEdge component for APM Infra Host Monitoring with non-root user ID"
echo "################################################################################################"
kill $(pidof sysedge) &>/dev/null
rm -rf ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE &>/dev/null
cd ${APM_INFRA_HOME_FOLDER}/apmia/CA_SystemEDGE_Core
./ca-setup.sh install
/bin/ps -ef | grep -i sysedge | grep -v grep


echo ""
echo ""
echo "################################################################################################"
echo "Update and restart the SystemEdge component for APM Infra Host Monitoring with non-root user ID"
echo "################################################################################################"
# Ensure non-root id is used for sysedge
echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
# Ensure only local host can pull or send data to the sysedge agent
echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
# Mask low value entries - Switch to debug loglevel as needed to address configuration challenges
echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
cp -r -p ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf  ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl stop
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl start
/bin/ps -ef | grep -i sysedge | grep -v grep

echo ""
echo ""
echo "################################################################################################"
echo "Check updates to sysedge configuration file sysedge.cf "
echo "################################################################################################"
#tail -5 ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf
grep -Hin -v -e "^$" -e "^#" -e "^template" ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf


echo ""
echo ""
echo "################################################################################################"
echo "Update the APM Infra main agent and disable the acc component "
echo "################################################################################################"
sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile
grep "introscope.agent.acc.enable" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile


echo ""
echo ""
echo "################################################################################################"
echo "Create systemd startup process on vApp due to sudo systemctl process allowed on vApp for APM Infra main agent"
echo "################################################################################################"

cat << EOF > ${APM_INFRA_HOME_FOLDER}/apmia/apmia.service
# /etc/systemd/system/apmia.service
[Unit]
Description=APM Infrastructure Agent
After=syslog.target

[Service]
Type=forking
ExecStart="${APM_INFRA_HOME_FOLDER}/apmia/bin/./APMIAgent.sh" start sysd
ExecStop="${APM_INFRA_HOME_FOLDER}/apmia/bin/./APMIAgent.sh" stop sysd
User=${NON_ROOT_USER_ID}
KillMode=control-group
Environment=SYSTEMD_KILLMODE_WARNING=true

[Install]
WantedBy=multi-user.target

EOF

sudo systemctl stop          apmia.service &>/dev/null
sudo systemctl disable       apmia.service &>/dev/null
sudo systemctl enable ${APM_INFRA_HOME_FOLDER}/apmia/apmia.service
echo "################################################################################################"
sudo systemctl cat           apmia.service
echo "################################################################################################"
sudo systemctl daemon-reload
sudo systemctl start         apmia.service
sudo systemctl status        apmia.service -a -l --no-pager



echo ""
echo ""
echo "################################################################################################"
echo "Create systemd startup process on vApp due to sudo systemctl process allowed on vApp for Sysedge agent"
echo "################################################################################################"
# Stop sysedge via manual process to use the systemd process
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl stop

# Manage these two (2) systemd error with PIDs and Paths
# Refusing to accept PID outside of service control group, acquired through unsafe symlink chain
# /opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/config/port1691/sysedge.service:8] Not an absolute path
#
cat << EOF > ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service
# /etc/systemd/system/sysedge.service
[Unit]
Description=sysedge
After=syslog.target

[Service]
Type=forking
WorkingDirectory=${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691
#Environment=VAR_HERE_ENV_ENV=production PATH=PATH_HERE_IF_NEEDED
ExecStart="${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/CA-SystemEDGE" start sysd
ExecStop="${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/CA-SystemEDGE" stop sysd
PIDFile=${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.pid
User=${NON_ROOT_USER_ID}
KillMode=none

[Install]
WantedBy=multi-user.target

EOF
echo ""
echo ""
echo "################################################################################################"
cat ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service
echo "################################################################################################"
echo ""
echo ""
sudo systemctl stop          sysedge.service  &>/dev/null
sudo systemctl disable       sysedge.service  &>/dev/null
sudo systemctl enable "${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service"
echo "################################################################################################"
sudo systemctl cat           sysedge.service
echo "################################################################################################"
sudo systemctl daemon-reload
sudo systemctl start         sysedge.service
sudo systemctl status        sysedge.service -a -l --no-pager



echo ""
echo ""
echo "################################################################################################"
echo "Ensure all files are owned by the non-root id of ${NON_ROOT_USER_ID} for APM Infra Agent"
echo "Check for any error messages "
echo "################################################################################################"
sudo systemctl stop          apmia.service
sudo systemctl stop          sysedge.service
echo "chown -R ${NON_ROOT_USER_ID}:${NON_ROOT_USER_ID} ${APM_INFRA_HOME_FOLDER}/apmia "
echo "################################################################################################"
chown -R ${NON_ROOT_USER_ID}:${NON_ROOT_USER_ID} ${APM_INFRA_HOME_FOLDER}/apmia
echo ""
echo ""

echo ""
echo ""
echo "################################################################################################"
echo "View running services: APM Infra will have a parent process of wrapper and two (2) java child processes"
echo "The APM Infra HostMonitoring module will have one process name sysedge "
echo "################################################################################################"
sudo systemctl start          apmia.service
sudo systemctl start          sysedge.service
ps -ef | grep apmia | grep -v grep



echo ""
echo ""
echo "################################################################################################"
echo "Use these these commands to view / monitor / start & stop APM Infra Agent"
echo "################################################################################################"
echo "sudo systemctl status  apmia.service -a -l --no-pager"
echo "sudo systemctl stop    apmia.service"
echo "sudo systemctl start   apmia.service"
echo "sudo systemctl cat     apmia.service"
echo "journalctl          -u apmia.service -f"
echo ""
echo "sudo systemctl status  sysedge.service -a -l --no-pager"
echo "sudo systemctl stop    sysedge.service"
echo "sudo systemctl start   sysedge.service"
echo "sudo systemctl cat     sysedge.service"
echo "journalctl          -u sysedge.service -f"
echo "journalctl          -u sysedge.service -xe -f"
echo ""
echo ""

6. Example of JVM arguments for integration of APM SaaS Java Agent with Identity Suite JCS Connector Server)

The APM SaaS or APM Enterprise UI solution will provide an agent download page, that allows section of the agent to be deployed.

Instruction will be provided and please noticed the “wget” auto download link that may be leveraged for automation (dev-ops) processes. This link will include both the siteID and the download token ID.

If you haven’t generated a new credentials token, after you download the Java/Jboss agents, click “Show Agent Details” to harvest the three (3) key/value pairs.

The below bash shell script will download the APM Java Agent package. There is no embedded credential token with this download. The credentials are provided separately via the APM SaaS UI (as shown above). While we could place these three (3) parameters within the APM agent configuration file, IntroscopeAgent.profile, we decided to clarity the use of JVM switches to override any values, and allow us to automate this deployment independent of any new APM agents updates.

#!/bin/bash
######################################################################################
#
#  Automate deployment of the APM SaaS Java Agent with credentials & urls
#   to the single Identity Manager JCS Connector Server instance
#
#  Use variables for the JVM parameters.  Adjust if needed.
#  We may override the default naming convention to clarity which instance is
#  being monitoring within the APM SaaS Dashboard, to avoid confusion with any
#  JBoss agent instance from IM/IG/IP
#
#  ANA 10/2023
#
######################################################################################
echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM Java Agent "
echo "################################################################################################"
APM_CREDENTIAL_TOKEN='XXXXXXX_LONG_TOKEN_HERE_FROM_APM_SAAS_UI__XXXXXXX'
APM_URL='apmgw.dxi-na1.saas.broadcom.com'
APM_SAAS_AGENT_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXX_DOWNLOAD_TOKEN_PROVIDED_FROM_APM_SAAS_UI_WHEN_SELECTED___XXXXXXXXXXXX"
APM_AGENT_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_java
APM_AGENT_FILE_NAME="APM_SaaS_Java_Agent.tar"
mkdir -p ${APM_AGENT_HOME_FOLDER}
cd       ${APM_AGENT_HOME_FOLDER}
pwd
ls -lart


echo "wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}"
wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}
ls -lart
echo "tar -xvf ${APM_AGENT_FILE_NAME} "
#tar -xvf ${APM_AGENT_FILE_NAME}
tar -xf ${APM_AGENT_FILE_NAME}
ls -lart
#
#
tz=`/bin/date --utc +%Y%m%d%H%M%S`
APM_AGENT_NAME=IM_JCS_NODE
JVM_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/profiler/
JVM_FILE=/opt/CA/IdentityManager/ConnectorServer/data/jvm_options.conf

if [ -f ${JVM_FILE} ];then

cp -r -p ${JVM_FILE}  ${JVM_BACKUP_LOCATION}/${tz}_jvm_options.conf

echo "-server -Xms1g -Xmx2g -Djava.awt.headless=true -Dcom.sun.net.ssl.enableECC=true -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9  -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIAL_TOKEN}\"  -Dcom.wily.introscope.agent.agentName=${APM_AGENT_NAME}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${JCS_JVM_FILE}

echo "Start / Stop JCS"

echo "sudo systemctl stop im_jcs "
sudo systemctl stop im_jcs

echo "sudo systemctl start im_jcs "
sudo systemctl start im_jcs

}

7. Example of JVM arguments for integration of APM SaaS JBoss Agent with Identity Suite IM/IG/IP JBoss/Wildfly instances

The below bash shell script is for the three (3) JBoss (Wildfly) instances on the vApp. Fortunately for us, the Broadcom/Symantec Product/Engineering team kept the same log4j jar and class files version between all three (3) instances, so we were able to leverage variables for each. You may wish to adjust the JVM parameters as needed. Please note, that again we did not embed any key/value pair into the APM configuration file, as we wanted to automate this deployment independent of any new APM agents updates.

#!/bin/bash
#########################################################################
#
#  Automate deployment of the APM SaaS JBOSS Agent with credentials & urls
#   to the three JBoss/Wildfly instances
#     - IdentityManager
#     - IdentityGovernance
#     - IdentityPortal
#
#  Use variables for the JVM parameters that allow a similar configuration for
#  all three (3) instances.  Adjust if needed.  Since all three (3) JBoss are on the
#  same host, we will override the default naming convention to clarity which instance is
#  being monitoring within the APM SaaS Dashboard
#
#  ANA 10/2023
#
#########################################################################
echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM JBoss Agent "
echo "################################################################################################"
APM_SAAS_AGENT_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXX__DOWNLOAD_TOKEN_HERE"
APM_CREDENTIALS='XXXXX_LONG_CREDENTIAL_TOKEN_HERE__XXXXXX'
APM_URL='apmgw.dxi-na1.saas.broadcom.com'
APM_AGENT_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_jboss
APM_AGENT_FILE_NAME="APM_SaaS_JBOSS_Agent.tar"
mkdir -p ${APM_AGENT_HOME_FOLDER}
cd       ${APM_AGENT_HOME_FOLDER}
pwd
ls -lart


echo "wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}"
wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}
echo "tar -xvf ${APM_AGENT_FILE_NAME} "
#tar -xvf ${APM_AGENT_FILE_NAME}
tar -xf ${APM_AGENT_FILE_NAME}
ls -lart
tz=`/bin/date --utc +%Y%m%d%H%M%S`



JBOSS_INSTANCE=IdentityManager
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf

if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "
cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-idm "
sudo systemctl stop wildfly-idm

echo "sudo systemctl start wildfly-idm "
sudo systemctl start wildfly-idm

fi

JBOSS_INSTANCE=IdentityGovernance
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf
if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "

cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-ig "
sudo systemctl stop wildfly-ig
echo "sudo systemctl start wildfly-ig "
sudo systemctl start wildfly-ig

fi


JBOSS_INSTANCE=IdentityPortal
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf
if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "
cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-portal "
sudo systemctl stop wildfly-portal
echo "sudo systemctl start wildfly-portal "
sudo systemctl start wildfly-portal

fi

8. Conclusion of APM tools integration on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

We have been successful using APM tool and home-grown java monitoring processes to improve the performance of the Symantec Identity Suite solution for peak use-cases. Using these processes should allow you to peak into the ‘blackbox’ of Java/JBoss processes to understand where bottlenecks may exists.

9. View sysedge.cf configuration for proposed change to isolate behavior to single host

View of the configuration file for sysedge.cf via cat sysedge.cf | grep -i -v -e “^$” -e “^#” -e “^template”

Token / ParameterValueCommentary
version 6.0
sysedge_loglevelfatalProposed change: Switch from default log level of “info” to “fatal” to avoid bloat in sysedge.log file due to embedded function in sysedge binary to copy configuration file to /etc folder for non-root ID. Switch to debug if there are any configurations challenges.
bind_address127.0.0.1Proposed change: Prevent external updates to SNMP trap of SysEdge (allow only localhost) – sudo systemctl start sysedge.service

udp 0 0 127.0.0.1:1691 0.0.0.0:* 117588/sysedge

Validate: ./walktree -h 127.0.0.1 -p 1691 -c apmmonitor -r 3 -o 1.3.6.1.2.1.1
communityapmmonitor read-onlySet during installation. May be set via scripts for manual deployment
no_authen_traps
no_process_sets
no_remoteshell_group
no_actions
no_stat_floppy
no_warmstart_discovery
bulk_read_response_soft_limit60000
monitoroid memCapacity.0 11 0x300200 300 absolute >= 95 ‘Physical Memory Usage (percentage)’ ” ‘Memory’ ‘Total’ ‘PercentUsed’ warning
monitoroid memCapacity.0 12 0x300200 300 absolute >= 98 ‘Physical Memory Usage (percentage)’ ” ‘Memory’ ‘Total’ ‘PercentUsed’ minor
monitoroid cpuTotalIdlePercent.0 13 0x600200 120 absolute <= 10 ‘CPU Idle Time (percentage)’ ” ‘CPU’ ‘Total’ ‘PercentIdle’ warning
monitoroid cpuTotalIdlePercent.0 14 0x600200 120 absolute <= 5 ‘CPU Idle Time (percentage)’ ” ‘CPU’ ‘Total’ ‘PercentIdle’ minor
monitoroid diskStatsUtilization.1 15 0x300200 300 absolute >= 90 ‘Disk Busy Time (percentage)’ ” ‘Disk’ ‘First’ ‘PercentBusy’ warning
monitoroid diskStatsUtilization.1 16 0x300200 300 absolute >= 95 ‘Disk Busy Time (percentage)’ ” ‘Disk’ ‘First’ ‘PercentBusy’ minor
monitordevTableEntry ‘C:’ devCapacity 17 0x0 300 absolute >= 95 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘C:’ ‘devCapacity’ warning
monitordevTableEntry ‘C:’ devCapacity 18 0x0 300 absolute >= 98 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘C:’ ‘devCapacity’ minor
monitordevTableEntry ‘/’ devCapacity 19 0x0 300 absolute >= 95 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘Root’ ‘devCapacity’ warning
monitordevTableEntry ‘/’ devCapacity 20 0x0 300 absolute >= 98 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘Root’ ‘devCapacity’ minor
autowatch-index=11 -name=’All FileSystems – Warning’ -watchtype=generic -table=devTableEntry -attribute=devCapacity -interval=300 -criteria=’.*’ -op=ge -value=95 -severity=warning -desc=’FileSystem Usage (percentage)’ -objclass=FileSystem -objattr=PercentUsed
autowatch-index=12 -name=’All FileSystems – Minor’ -watchtype=generic -table=devTableEntry -attribute=devCapacity -interval=300 -criteria=’.*’ -op=ge -value=98 -severity=minor -desc=’FileSystem Usage (percentage)’ -objclass=FileSystem -objattr=PercentUsed
autowatch-index=13 -name=’All Disks – Warning’ -watchtype=generic -table=diskStatsEntry -attribute=diskStatsUtilization -interval=300 -criteria=’.*’ -op=ge -value=90 -severity=warning -monflags=0x300200 -desc=’Disk Busy Time (percentage)’ -objclass=Disk -objattr=PercentBusy
autowatch-index=14 -name=’All Disks – Minor’ -watchtype=generic -table=diskStatsEntry -attribute=diskStatsUtilization -interval=300 -criteria=’.*’ -op=ge -value=95 -severity=minor -monflags=0x300200 -desc=’Disk Busy Time (percentage)’ -objclass=Disk -objattr=PercentBusy
sysedge_plugin*
data_directory/opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/config/’Set during installation. May be set via scripts for manual deployment
default_port1691Set during installation. May be set via scripts for manual deployment
no_proc_monitor
no_procgroup_monitor
privilege_separation_userconfig OR ec2-userProposed change: Add new required parameter to enforce non-root access for a local host account. Confirm ownership when stopping/starting the service via systedgectl or sudo systemctl start sysedge.service. Confirm file ownership for PID & log files.

config 117588 1 6 14:41 ? 00:00:16 /opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/bin/sysedge -b

10. Prior notes / examples of integration/deployment of APM tools (Java/JBoss) on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

We have been involved with use of APM tools for quite a bit and contributed to the inclusion of these features into various solution, including software appliances like the Symantec Identity Suite with non-root access.

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=2197&MID=773616&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

Engage with us to help accelerate your business goals

Accelerate PAM DevOps and Automation: A Unified Approach with Operations Teams and Business Analysts

Posted on by Leave a comment

Bridging the Gap Between Development and Operations

Collaborative platforms of PostMan and Thunderclient provide a means to address the traditional divide between development and operations teams can often lead to bottlenecks and communication breakdowns.

These collaborative platforms lower the learning curve to demonstrate and validate use-cases via REST APIs may also be used for project management, version control, and continuous integration, development and operations teams can work together seamlessly. This unified approach streamlines processes, enabling both teams to plan, develop, test, and deploy software efficiently and cohesively for the PAM (Privileged Access Management) solution.

Automating Processes with REST APIs

Automation, driven by REST APIs, is a powerful way to eliminate manual, repetitive tasks and increase overall efficiency. When operations and business analysts are involved early in automation projects, they can identify which processes are best suited for automation, optimizing productivity and reducing errors.

Common workspace tools, integrated with REST API functionality, enable the sharing of API endpoints, documentation, and workflows. This allows for close collaboration between all stakeholders, ensuring that automation projects align with the organization’s specific requirements and objectives.

Ensuring Business Goals with REST APIs

Incorporating operations and business analysts into the early stages of DevOps and automation initiatives is vital for ensuring alignment with business goals. These professionals have a deep understanding of operational challenges, customer needs, and market dynamics. Their involvement in the process helps to define clear project objectives and metrics for success.

Additionally, common workspace tools equipped with REST API integration support the creation of comprehensive project roadmaps and offer real-time tracking of project progress. This transparency is essential for all stakeholders, guaranteeing that development, operations, and business teams stay coordinated throughout the project’s lifecycle.

Example

To show-case the value would require a lengthy blog entry, so we are enclosing a detailed walk-through that validates a business automation use-case via REST APIs for the Symantec PAM solution. While the primary goal was to reduce effort and automate a process, strong value was seen with the business team collaboration and unmasking the “black-box” of API development effort with a 3rd party API UI development tool.

Import API collections into tools

  1. Use the PAM appliance’s API docs URL to download PAM API to json file {you may use your browser to see the JSON files to import}
  2. https://<FQDN>/cspm/ext/rest/swagger.json
  3. Edit the downloaded swagger.json file
  4. Added “openapi”: “3.0”, string prior to the string “swagger”: “2.0”, to import the json into PostMan
  5. Import updated json into PostMan or Thunderclient
  6. Export from PostMan/Thunderclient to new json file with unique “item” entries (for folders)
  7. Edit json file, to move all entries under a new subfolder of REST via a parent “item” entry
  8. Import new PostMan/Thunderclient json
  9. Add in the other PAM REST APIs via api.php and cspm entries manually under their own folders
  10. Adjust values for four (4) objects that did not fit REST folder structure (due to mix with api.php on PAM API Doc)

Conclusion

REST APIs are at the heart of efficient DevOps and automation, facilitating the smooth flow of data and actions between systems. To make the most of these capabilities, involving operations and business analysts from the start is essential. By using REST APIs and common workspace tools, organizations can break down silos, improve communication, and ensure their DevOps and automation efforts are in perfect harmony with their strategic business objectives.

Embracing this collaborative approach not only accelerates project timelines but also encourages innovation and continuous improvement. It’s the pathway to long-term success in our modern digital landscape. So, whether you’re a seasoned professional or just beginning your DevOps and automation journey, remember that integrating REST APIs and early engagement of operations and business analysts can make all the difference in maximizing the potential of your initiatives.

Engage with us to help accelerate your business goals