Elevating Application Performance and Security on Software Appliances with APM Host Monitoring, APM Java, and APM JBoss(Wildfly)

Organizations may rely on software appliances to meet the demands of secure, mission-critical applications. To ensure the optimal operation of these appliances, Application Performance Monitoring/Management (APM) tools have emerged as invaluable assets. In this comprehensive blog post, we’ll explore how the synergy of APM host monitoring via the SysEdge module, APM Java, and APM JBoss(Wildfly/J2EE) can significantly enhance the performance and security of applications running on software appliances like the Symantec Identity Suite Virtual Appliance (on-prem and AWS instances).

1. Value of APM Java and APM JBoss(Wildfly) Monitoring

APM tools feature specialized modules for Java-based applications and JBoss(Wildfly/J2EE) application servers, addressing the unique challenges posed by these technologies:

  • Optimizing Java-Based Applications: APM Java monitoring delves deep into Java applications, tracing bottlenecks, optimizing code, and ensuring the efficient use of Java Virtual Machine (JVM) resources.
  • JBoss Application Server Expertise: APM JBoss monitoring tracks the performance and stability of JBoss deployments, providing insights into JBoss-specific metrics critical for the smooth operation of JBoss-based applications.

2. The Role of APM Host Monitoring with SysEdge Module

The SysEdge module, integrated into APM tools, plays a pivotal role in enhancing the performance, security, and overall management of software appliances. This module focuses on host-based metrics, offering insights into the appliance’s performance and health.

  • Resource Utilization: SysEdge monitors essential resources, such as CPU, memory, disk space, and network usage, ensuring efficient resource allocation and preventing performance bottlenecks.
  • Hardware Health: It provides insights into the hardware/virtual components, crucial for maintaining the reliability of the appliance.
  • Comprehensive Diagnostics: The detailed host-based metrics allow for more accurate and rapid issue diagnostics, helping administrators identify and address problems efficiently. Gain real-time insight into digital performance, user experience and behavior

3. Benefits of APM Tools for Software Appliances

The integration of host-based metrics through SysEdge, along with APM Java and APM JBoss monitoring, offers a multitude of benefits:

  • Holistic Insights: APM tools provide a complete picture of the appliance’s performance, helping administrators make informed decisions by combining application-specific data with host-based metrics. Ensure flawless user experiences with analytics-driven insights
  • Proactive Issue Resolution: Administrators can proactively identify and address issues that may impact both application performance and the host system, reducing downtime and increasing reliability. Eliminate alert fatigue with automated root cause analysis.
  • Streamlined Management: These tools enable remote management of the appliance, even in challenging environments, allowing fine-tuning, patch application, and addressing security concerns. Empower every team. Improve every app.

4. Secure Deployment with Non-Root User ID and DevOps Automation

The utilization of non-root user IDs and DevOps automation can significantly enhance both security and operational efficiency in the deployment and management of applications on software appliances. Traditional application deployments often involved elevated privileges, exposing them to security vulnerabilities. Deploying applications with non-root user IDs offers several advantages:

  • Reduced Attack Surface: Non-root users have limited permissions, reducing the potential attack surface and making it more difficult for malicious actors to compromise the system.
  • Enhanced Security: By limiting application permissions, non-root deployments minimize the risk of security breaches and unauthorized access.
  • Compliance: Using non-root user IDs aligns with security best practices and compliance requirements, ensuring your organization meets regulatory standards.
  • Isolation: Non-root deployments prevent applications from interfering with critical system components, reducing the risk of conflicts and crashes.

5. Example of integration/deployment of APM tools (Java/JBoss/SysEdge) on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

The Symantec Identity Suite Virtual Appliance is a harden software appliance that only allow authentication for one (1) of two (2) non-root IDs (config or ec2-user). The Symantec Identity Suite does allow for APM type tools to be deployed via extraction under the path /opt/CA/VirtualAppliance/custom/profiler.

We want to walk-through how to enable the DX APM SaaS Infra Agent with HostMonitoring (SysEdge) enabled on an Amazon Linux 2 host as a non-root user ID, as well as integration with the CA Identity Suite three (3) JBoss/Wildfly instances for IM/IG/IP and the CA Identity Suite JCS Connector Server with the embedded Java agent.

Additionally, we wanted to ensure that any external configuration access was disabled, as we only wanted to allow a “push” configuration/model of data from the vApp to the APM SaaS Collection APIs. We did not wish to allow any modification of the APM agent’s configuration on the vApp, that was not defined during initial deployment.

Four (4) parameters were modified from default installation:

1. Ensure non-root id is used for sysedge 

echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf

2. Ensure only local host can pull or send data to the sysedge agent

echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf  

3. Mask low value entries - Switch to debug loglevel as needed to address configuration challenges

echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf 

4. Disable remote management via APM Command Center (ACC)
      
sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile

Leveraging the information provided by the Broadcom DX Application Performance Management and Symantec Identity Suite documentation, we were able to clarify the dependencies on JBoss logging log4j jar(s) and LogManager objects with Adopt Open JDK 8. Neither document had the exact configuration that we found viable during testing.

We had two (2) challenges deploying the HostMonitoring (sysedge) agent on the AWS Identity Suite vApp instance that we were able to identify and address. No issues were found on the on-prem edition of the Identity Suite vApp.

  • Challenge(s):
    • File ownership of the sysedge PID and log file by ‘root’ instead of the non-root user due to default systemd startup script for sysedge module/binary.
    • Null entry within a file impacted startup of ‘sysedge’ binary from default OS file /etc/redhat-release

The below bash shell script utilized the APM SaaS Binary download feature of embedding the sysedge module/binary component with the primary APM Infra agent. This will include the unique token for your own APM SaaS infrastructure (or APM Enterprise infra). The APM SaaS download process provides a great automation download via wget process. The Symantec Identity Suite allows the non-root IDs to start/stop systemd process, which we leverage. Alternatively, we may use crontab for the non-root IDs to start/stop the two (2) services: apmia and sysedge as documented for DX APM agents.

#!/bin/bash
####################################################################
#
#  Install the APM SaaS Infra Agent with HostMonitoring module (sysedge) as non-root id
#  - Update variables of NON_ROOT_USER_ID and INFRA_DOWNLOAD_URL and APM_INFRA_HOME_FOLDER
#  - This script works for a host that allows minimal sudo access to systemctl
#  - Alternative startup process is crontab for the non-root-id
#
#  Goal:  Replica process for:   ./APMIACtrl.sh install user=non-root-id
#   and ./APMIACtrl.sh console_start
#
#  Methodology:  Isolate delta between folders using diff with sub-folder detection
#   diff -iry --suppress-common-lines apmia/ apmia.orginal/
#
#  Important Note: Identified RCA for sysedge binary having memory fault SEGV
#  /etc/redhat-release MUST be populated (avoid null value)
#
#
#  Crontab notes from online APM agent docs: (if needed)
#    @reboot /home/user/apmia/APMIACtrl.sh console_start > /home/user/logs/cron.log 2>&1
#    */5 * * * * /home/user/apmia/APMIACtrl.sh console_start > /home/user/logs/cron.log 2>&1
#
#  Modify default APM Infra Agent parameters with these changes
#   1. Ensure non-root id is used for sysedge
#         echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   2. Ensure only local host can pull or send data to the sysedge agent
#        echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   3. Mask low value entries - Switch to debug loglevel as needed to address configuration challenges
#        echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   4. Disable acc integration (remote management) via APM Command Center (ACC)
#        sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile
#
#
#
#  Ref.  https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/install-and-deploy-infrastructure-agent/install-the-infrastructure-agent-on-ca-digital-experience-insights.html
#        https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/Supportability-Matrix-for-Infrastructure-Agent.html
#        https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/SystemEDGE-based-Monitoring.html
#
# ANA 10/2023
#
####################################################################
#NON_ROOT_USER_ID=config
NON_ROOT_USER_ID=ec2-user

echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM Infra Agent with HostMonitoring check / enabled - This will be packaged together "
echo "################################################################################################"
INFRA_DOWNLOAD_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXXXXXXXXXXXXXXXXX_LONG_TOKEN_HERE__XXXXXXXXXXX"


APM_INFRA_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_infra
mkdir -p ${APM_INFRA_HOME_FOLDER}
cd       ${APM_INFRA_HOME_FOLDER}
pwd
ls -lart

echo "wget --no-check-certificate --content-disposition ${INFRA_DOWNLOAD_URL} -O Infrastructure_Agent_apmia.tar"
wget --no-check-certificate --content-disposition ${INFRA_DOWNLOAD_URL} -O Infrastructure_Agent_apmia.tar
APM_INFRA_FILE_NAME=$(ls -lart Infrastructure_Agent_apmia* |tail -1 | awk '{print $9}')
echo "tar -xvf ${APM_INFRA_FILE_NAME} "
#tar -xvf ${APM_INFRA_FILE_NAME}
tar -xf ${APM_INFRA_FILE_NAME}


echo ""
echo ""
echo "################################################################################################"
echo "Update APM Infra Agent startup file to use non-root user ID of ${NON_ROOT_USER_ID} "
echo "################################################################################################"
sed -i "s|#RUN_AS_USER=|RUN_AS_USER=${NON_ROOT_USER_ID}|g" ${APM_INFRA_HOME_FOLDER}/apmia/bin/APMIAgent.sh

echo ""
echo ""
echo "################################################################################################"
echo "Validate update of NON_ROOT_USER_ID"
echo "################################################################################################"
grep -C 2 -i "RUN_AS_USER=${NON_ROOT_USER_ID}"  ${APM_INFRA_HOME_FOLDER}/apmia/bin/APMIAgent.sh


echo ""
echo ""
echo "################################################################################################"
echo "Extract SystemEdge component for APM Infra Host Monitoring"
echo "################################################################################################"
export AGENTHOME=${APM_INFRA_HOME_FOLDER}/apmia
SYSEDGE_FILE_NAME=$(ls ${AGENTHOME}/casystemedge*)
echo ${SYSEDGE_FILE_NAME}
cd ${AGENTHOME}
#tar -xvf ${SYSEDGE_FILE_NAME}
tar -xf ${SYSEDGE_FILE_NAME}


echo ""
echo ""
echo "################################################################################################"
echo "Deploy and install SystemEdge component for APM Infra Host Monitoring with non-root user ID"
echo "################################################################################################"
kill $(pidof sysedge) &>/dev/null
rm -rf ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE &>/dev/null
cd ${APM_INFRA_HOME_FOLDER}/apmia/CA_SystemEDGE_Core
./ca-setup.sh install
/bin/ps -ef | grep -i sysedge | grep -v grep


echo ""
echo ""
echo "################################################################################################"
echo "Update and restart the SystemEdge component for APM Infra Host Monitoring with non-root user ID"
echo "################################################################################################"
# Ensure non-root id is used for sysedge
echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
# Ensure only local host can pull or send data to the sysedge agent
echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
# Mask low value entries - Switch to debug loglevel as needed to address configuration challenges
echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
cp -r -p ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf  ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl stop
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl start
/bin/ps -ef | grep -i sysedge | grep -v grep

echo ""
echo ""
echo "################################################################################################"
echo "Check updates to sysedge configuration file sysedge.cf "
echo "################################################################################################"
#tail -5 ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf
grep -Hin -v -e "^$" -e "^#" -e "^template" ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf


echo ""
echo ""
echo "################################################################################################"
echo "Update the APM Infra main agent and disable the acc component "
echo "################################################################################################"
sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile
grep "introscope.agent.acc.enable" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile


echo ""
echo ""
echo "################################################################################################"
echo "Create systemd startup process on vApp due to sudo systemctl process allowed on vApp for APM Infra main agent"
echo "################################################################################################"

cat << EOF > ${APM_INFRA_HOME_FOLDER}/apmia/apmia.service
# /etc/systemd/system/apmia.service
[Unit]
Description=APM Infrastructure Agent
After=syslog.target

[Service]
Type=forking
ExecStart="${APM_INFRA_HOME_FOLDER}/apmia/bin/./APMIAgent.sh" start sysd
ExecStop="${APM_INFRA_HOME_FOLDER}/apmia/bin/./APMIAgent.sh" stop sysd
User=${NON_ROOT_USER_ID}
KillMode=control-group
Environment=SYSTEMD_KILLMODE_WARNING=true

[Install]
WantedBy=multi-user.target

EOF

sudo systemctl stop          apmia.service &>/dev/null
sudo systemctl disable       apmia.service &>/dev/null
sudo systemctl enable ${APM_INFRA_HOME_FOLDER}/apmia/apmia.service
echo "################################################################################################"
sudo systemctl cat           apmia.service
echo "################################################################################################"
sudo systemctl daemon-reload
sudo systemctl start         apmia.service
sudo systemctl status        apmia.service -a -l --no-pager



echo ""
echo ""
echo "################################################################################################"
echo "Create systemd startup process on vApp due to sudo systemctl process allowed on vApp for Sysedge agent"
echo "################################################################################################"
# Stop sysedge via manual process to use the systemd process
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl stop

# Manage these two (2) systemd error with PIDs and Paths
# Refusing to accept PID outside of service control group, acquired through unsafe symlink chain
# /opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/config/port1691/sysedge.service:8] Not an absolute path
#
cat << EOF > ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service
# /etc/systemd/system/sysedge.service
[Unit]
Description=sysedge
After=syslog.target

[Service]
Type=forking
WorkingDirectory=${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691
#Environment=VAR_HERE_ENV_ENV=production PATH=PATH_HERE_IF_NEEDED
ExecStart="${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/CA-SystemEDGE" start sysd
ExecStop="${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/CA-SystemEDGE" stop sysd
PIDFile=${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.pid
User=${NON_ROOT_USER_ID}
KillMode=none

[Install]
WantedBy=multi-user.target

EOF
echo ""
echo ""
echo "################################################################################################"
cat ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service
echo "################################################################################################"
echo ""
echo ""
sudo systemctl stop          sysedge.service  &>/dev/null
sudo systemctl disable       sysedge.service  &>/dev/null
sudo systemctl enable "${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service"
echo "################################################################################################"
sudo systemctl cat           sysedge.service
echo "################################################################################################"
sudo systemctl daemon-reload
sudo systemctl start         sysedge.service
sudo systemctl status        sysedge.service -a -l --no-pager



echo ""
echo ""
echo "################################################################################################"
echo "Ensure all files are owned by the non-root id of ${NON_ROOT_USER_ID} for APM Infra Agent"
echo "Check for any error messages "
echo "################################################################################################"
sudo systemctl stop          apmia.service
sudo systemctl stop          sysedge.service
echo "chown -R ${NON_ROOT_USER_ID}:${NON_ROOT_USER_ID} ${APM_INFRA_HOME_FOLDER}/apmia "
echo "################################################################################################"
chown -R ${NON_ROOT_USER_ID}:${NON_ROOT_USER_ID} ${APM_INFRA_HOME_FOLDER}/apmia
echo ""
echo ""

echo ""
echo ""
echo "################################################################################################"
echo "View running services: APM Infra will have a parent process of wrapper and two (2) java child processes"
echo "The APM Infra HostMonitoring module will have one process name sysedge "
echo "################################################################################################"
sudo systemctl start          apmia.service
sudo systemctl start          sysedge.service
ps -ef | grep apmia | grep -v grep



echo ""
echo ""
echo "################################################################################################"
echo "Use these these commands to view / monitor / start & stop APM Infra Agent"
echo "################################################################################################"
echo "sudo systemctl status  apmia.service -a -l --no-pager"
echo "sudo systemctl stop    apmia.service"
echo "sudo systemctl start   apmia.service"
echo "sudo systemctl cat     apmia.service"
echo "journalctl          -u apmia.service -f"
echo ""
echo "sudo systemctl status  sysedge.service -a -l --no-pager"
echo "sudo systemctl stop    sysedge.service"
echo "sudo systemctl start   sysedge.service"
echo "sudo systemctl cat     sysedge.service"
echo "journalctl          -u sysedge.service -f"
echo "journalctl          -u sysedge.service -xe -f"
echo ""
echo ""


6. Example of JVM arguments for integration of APM SaaS Java Agent with Identity Suite JCS Connector Server)

The APM SaaS or APM Enterprise UI solution will provide an agent download page, that allows section of the agent to be deployed.

Instruction will be provided and please noticed the “wget” auto download link that may be leveraged for automation (dev-ops) processes. This link will include both the siteID and the download token ID.

If you haven’t generated a new credentials token, after you download the Java/Jboss agents, click “Show Agent Details” to harvest the three (3) key/value pairs.

The below bash shell script will download the APM Java Agent package. There is no embedded credential token with this download. The credentials are provided separately via the APM SaaS UI (as shown above). While we could place these three (3) parameters within the APM agent configuration file, IntroscopeAgent.profile, we decided to clarity the use of JVM switches to override any values, and allow us to automate this deployment independent of any new APM agents updates.

#!/bin/bash
######################################################################################
#
#  Automate deployment of the APM SaaS Java Agent with credentials & urls
#   to the single Identity Manager JCS Connector Server instance
#
#  Use variables for the JVM parameters.  Adjust if needed.
#  We may override the default naming convention to clarity which instance is
#  being monitoring within the APM SaaS Dashboard, to avoid confusion with any
#  JBoss agent instance from IM/IG/IP
#
#  ANA 10/2023
#
######################################################################################
echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM Java Agent "
echo "################################################################################################"
APM_CREDENTIAL_TOKEN='XXXXXXX_LONG_TOKEN_HERE_FROM_APM_SAAS_UI__XXXXXXX'
APM_URL='apmgw.dxi-na1.saas.broadcom.com'
APM_SAAS_AGENT_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXX_DOWNLOAD_TOKEN_PROVIDED_FROM_APM_SAAS_UI_WHEN_SELECTED___XXXXXXXXXXXX"
APM_AGENT_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_java
APM_AGENT_FILE_NAME="APM_SaaS_Java_Agent.tar"
mkdir -p ${APM_AGENT_HOME_FOLDER}
cd       ${APM_AGENT_HOME_FOLDER}
pwd
ls -lart


echo "wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}"
wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}
ls -lart
echo "tar -xvf ${APM_AGENT_FILE_NAME} "
#tar -xvf ${APM_AGENT_FILE_NAME}
tar -xf ${APM_AGENT_FILE_NAME}
ls -lart
#
#
tz=`/bin/date --utc +%Y%m%d%H%M%S`
APM_AGENT_NAME=IM_JCS_NODE
JVM_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/profiler/
JVM_FILE=/opt/CA/IdentityManager/ConnectorServer/data/jvm_options.conf

if [ -f ${JVM_FILE} ];then

cp -r -p ${JVM_FILE}  ${JVM_BACKUP_LOCATION}/${tz}_jvm_options.conf

echo "-server -Xms1g -Xmx2g -Djava.awt.headless=true -Dcom.sun.net.ssl.enableECC=true -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9  -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIAL_TOKEN}\"  -Dcom.wily.introscope.agent.agentName=${APM_AGENT_NAME}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${JCS_JVM_FILE}

echo "Start / Stop JCS"

echo "sudo systemctl stop im_jcs "
sudo systemctl stop im_jcs

echo "sudo systemctl start im_jcs "
sudo systemctl start im_jcs

}

7. Example of JVM arguments for integration of APM SaaS JBoss Agent with Identity Suite IM/IG/IP JBoss/Wildfly instances

The below bash shell script is for the three (3) JBoss (Wildfly) instances on the vApp. Fortunately for us, the Broadcom/Symantec Product/Engineering team kept the same log4j jar and class files version between all three (3) instances, so we were able to leverage variables for each. You may wish to adjust the JVM parameters as needed. Please note, that again we did not embed any key/value pair into the APM configuration file, as we wanted to automate this deployment independent of any new APM agents updates.

#!/bin/bash
#########################################################################
#
#  Automate deployment of the APM SaaS JBOSS Agent with credentials & urls
#   to the three JBoss/Wildfly instances
#     - IdentityManager
#     - IdentityGovernance
#     - IdentityPortal
#
#  Use variables for the JVM parameters that allow a similar configuration for
#  all three (3) instances.  Adjust if needed.  Since all three (3) JBoss are on the
#  same host, we will override the default naming convention to clarity which instance is
#  being monitoring within the APM SaaS Dashboard
#
#  ANA 10/2023
#
#########################################################################
echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM JBoss Agent "
echo "################################################################################################"
APM_SAAS_AGENT_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXX__DOWNLOAD_TOKEN_HERE"
APM_CREDENTIALS='XXXXX_LONG_CREDENTIAL_TOKEN_HERE__XXXXXX'
APM_URL='apmgw.dxi-na1.saas.broadcom.com'
APM_AGENT_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_jboss
APM_AGENT_FILE_NAME="APM_SaaS_JBOSS_Agent.tar"
mkdir -p ${APM_AGENT_HOME_FOLDER}
cd       ${APM_AGENT_HOME_FOLDER}
pwd
ls -lart


echo "wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}"
wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}
echo "tar -xvf ${APM_AGENT_FILE_NAME} "
#tar -xvf ${APM_AGENT_FILE_NAME}
tar -xf ${APM_AGENT_FILE_NAME}
ls -lart
tz=`/bin/date --utc +%Y%m%d%H%M%S`



JBOSS_INSTANCE=IdentityManager
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf

if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "
cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-idm "
sudo systemctl stop wildfly-idm

echo "sudo systemctl start wildfly-idm "
sudo systemctl start wildfly-idm

fi

JBOSS_INSTANCE=IdentityGovernance
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf
if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "

cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-ig "
sudo systemctl stop wildfly-ig
echo "sudo systemctl start wildfly-ig "
sudo systemctl start wildfly-ig

fi


JBOSS_INSTANCE=IdentityPortal
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf
if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "
cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-portal "
sudo systemctl stop wildfly-portal
echo "sudo systemctl start wildfly-portal "
sudo systemctl start wildfly-portal

fi

8. Conclusion of APM tools integration on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

We have been successful using APM tool and home-grown java monitoring processes to improve the performance of the Symantec Identity Suite solution for peak use-cases. Using these processes should allow you to peak into the ‘blackbox’ of Java/JBoss processes to understand where bottlenecks may exists.

9. View sysedge.cf configuration for proposed change to isolate behavior to single host

View of the configuration file for sysedge.cf via cat sysedge.cf | grep -i -v -e “^$” -e “^#” -e “^template”

Token / ParameterValueCommentary
version 6.0
sysedge_loglevelfatalProposed change: Switch from default log level of “info” to “fatal” to avoid bloat in sysedge.log file due to embedded function in sysedge binary to copy configuration file to /etc folder for non-root ID. Switch to debug if there are any configurations challenges.
bind_address127.0.0.1Proposed change: Prevent external updates to SNMP trap of SysEdge (allow only localhost) – sudo systemctl start sysedge.service

udp 0 0 127.0.0.1:1691 0.0.0.0:* 117588/sysedge

Validate: ./walktree -h 127.0.0.1 -p 1691 -c apmmonitor -r 3 -o 1.3.6.1.2.1.1
communityapmmonitor read-onlySet during installation. May be set via scripts for manual deployment
no_authen_traps
no_process_sets
no_remoteshell_group
no_actions
no_stat_floppy
no_warmstart_discovery
bulk_read_response_soft_limit60000
monitoroid memCapacity.0 11 0x300200 300 absolute >= 95 ‘Physical Memory Usage (percentage)’ ” ‘Memory’ ‘Total’ ‘PercentUsed’ warning
monitoroid memCapacity.0 12 0x300200 300 absolute >= 98 ‘Physical Memory Usage (percentage)’ ” ‘Memory’ ‘Total’ ‘PercentUsed’ minor
monitoroid cpuTotalIdlePercent.0 13 0x600200 120 absolute <= 10 ‘CPU Idle Time (percentage)’ ” ‘CPU’ ‘Total’ ‘PercentIdle’ warning
monitoroid cpuTotalIdlePercent.0 14 0x600200 120 absolute <= 5 ‘CPU Idle Time (percentage)’ ” ‘CPU’ ‘Total’ ‘PercentIdle’ minor
monitoroid diskStatsUtilization.1 15 0x300200 300 absolute >= 90 ‘Disk Busy Time (percentage)’ ” ‘Disk’ ‘First’ ‘PercentBusy’ warning
monitoroid diskStatsUtilization.1 16 0x300200 300 absolute >= 95 ‘Disk Busy Time (percentage)’ ” ‘Disk’ ‘First’ ‘PercentBusy’ minor
monitordevTableEntry ‘C:’ devCapacity 17 0x0 300 absolute >= 95 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘C:’ ‘devCapacity’ warning
monitordevTableEntry ‘C:’ devCapacity 18 0x0 300 absolute >= 98 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘C:’ ‘devCapacity’ minor
monitordevTableEntry ‘/’ devCapacity 19 0x0 300 absolute >= 95 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘Root’ ‘devCapacity’ warning
monitordevTableEntry ‘/’ devCapacity 20 0x0 300 absolute >= 98 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘Root’ ‘devCapacity’ minor
autowatch-index=11 -name=’All FileSystems – Warning’ -watchtype=generic -table=devTableEntry -attribute=devCapacity -interval=300 -criteria=’.*’ -op=ge -value=95 -severity=warning -desc=’FileSystem Usage (percentage)’ -objclass=FileSystem -objattr=PercentUsed
autowatch-index=12 -name=’All FileSystems – Minor’ -watchtype=generic -table=devTableEntry -attribute=devCapacity -interval=300 -criteria=’.*’ -op=ge -value=98 -severity=minor -desc=’FileSystem Usage (percentage)’ -objclass=FileSystem -objattr=PercentUsed
autowatch-index=13 -name=’All Disks – Warning’ -watchtype=generic -table=diskStatsEntry -attribute=diskStatsUtilization -interval=300 -criteria=’.*’ -op=ge -value=90 -severity=warning -monflags=0x300200 -desc=’Disk Busy Time (percentage)’ -objclass=Disk -objattr=PercentBusy
autowatch-index=14 -name=’All Disks – Minor’ -watchtype=generic -table=diskStatsEntry -attribute=diskStatsUtilization -interval=300 -criteria=’.*’ -op=ge -value=95 -severity=minor -monflags=0x300200 -desc=’Disk Busy Time (percentage)’ -objclass=Disk -objattr=PercentBusy
sysedge_plugin*
data_directory/opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/config/’Set during installation. May be set via scripts for manual deployment
default_port1691Set during installation. May be set via scripts for manual deployment
no_proc_monitor
no_procgroup_monitor
privilege_separation_userconfig OR ec2-userProposed change: Add new required parameter to enforce non-root access for a local host account. Confirm ownership when stopping/starting the service via systedgectl or sudo systemctl start sysedge.service. Confirm file ownership for PID & log files.

config 117588 1 6 14:41 ? 00:00:16 /opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/bin/sysedge -b

10. Prior notes / examples of integration/deployment of APM tools (Java/JBoss) on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

We have been involved with use of APM tools for quite a bit and contributed to the inclusion of these features into various solution, including software appliances like the Symantec Identity Suite with non-root access.

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=2197&MID=773616&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

Engage with us to help accelerate your business goals

View the JMS HornetQ Queue

Typically, we may use various tools to view JMS queue(s) related metrics for trends and stale/stuck activity. During issues with J2EE JMS Queue, though, it would be helpful to be able to view and trace transactions to assist with a resolution. With proper logging levels enabled, Wildfly/JBOSS logs show detailed information containing the JMS IDs associated with each transaction. These JMS transactions we see in the logs are already ‘in-flight’ and are being processed by a message handler.

On the Symantec Identity Suite Virtual Appliance, the Wildfly & HornetQ processes are run under the ‘wildfly’ service ID. The wildfly journals are located in the wildfly data folder and stored in a format that is efficient for processing. To perform analysis on the data within these journals, though, we noticed a challenge with read-permissions for the HornetQ files even when Wildfly/Java process is not actively running.

To avoid this issue on the Virtual Appliance, copy the HornetQ files to a temporary folder. Remember to copy the entire folder, including sub-folders.

mkdir -p /tmp/hornetq; cd /tmp/hornetq

cp -r -p /opt/CA/wildfly-idm/standalone/data/live-hornetq ./

java -cp "/opt/CA/wildfly-idm/modules/system/layers/base/io/netty/main/*:/opt/CA/wildfly-idm/modules/system/layers/base/org/hornetq/main/*:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/logging/main/*" org.hornetq.tools.Main print-data /tmp/hornetq/live-hornetq/bindings  /tmp/hornetq/live-hornetq/journal

Once the live-hornetq folder is available in a tmp location, execute the below process for printing Journal content.

Print HornetQ Journal and Bindings

To export the HornetQ Journal Files to XML, the Java module of “org.hornetq.core.journal.impl.ExportJournal” requires the journal sub-folder with the prefix of “hornetq-data”, the file extension (hq), the file sizes, and where to export the XML file (export.dat). The prefix and file extension (hq) are unique to the Identity Suite vApp.

mkdir -p /tmp/hornetq; cd /tmp/hornetq

cp -r -p /opt/CA/wildfly-idm/standalone/data/live-hornetq ./

java -cp "/opt/CA/wildfly-idm/modules/system/layers/base/io/netty/main/*:/opt/CA/wildfly-idm/modules/system/layers/base/org/hornetq/main/*:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/logging/main/*" org.hornetq.core.journal.impl.ExportJournal  /tmp/hornetq/live-hornetq/journal hornetq-data hq  25485760  /tmp/hornetq/export.dat
Export HornetQ Journal

The body/rows of the JMS export is partially base64. You may parse through this information as you wish.

Use this information to trace through transactions in the JMS queue.

For Cleanup, within the Symantec Identity Suite vApp, there are a few options. The first is deleting the JMS queue journals before starting the Wildfly service. This can be accomplished using the build-in alias ‘deleteIDMJMSqueue’.

alias deleteIDMJMSqueue='sudo /opt/CA/VirtualAppliance/scripts/.firstrun/deleteIDMJMSqueue.sh'

Another option is to remove a select JMS entry from the queue using /opt/CA/wildfly-idm/bin/jboss-cli.sh process. If created with an input script, escape the colons in the GUID.

/subsystem=transactions/log-store=log-store/:probe()

ls /subsystem=transactions/log-store=log-store/transactions

/subsystem=transactions/log-store=log-store/transactions=0:ffffa409cc8a:1c01b1ff:5c7e95ac:eb:delete() 

View a description of the JMS Processing from Broadcom Engineering/Support Teams (see below video)

This write-up provides the tools required for a deeper analysis. Debugging issues with JMS may test one’s patience, stay the course, stay persistent, and have fun!

References: (Delete JMS queue and remove a single entry)

https://knowledge.broadcom.com/external/article/233003/inprogress-task-issues-a-clients-guide.html

https://knowledge.broadcom.com/external/article/129101/arjuna016037-could-not-find-new-xaresour.html

Build an eight (8) node Wildfly cluster on a single server

The follow methodology was used to isolate performance challenges with the increase number of cluster nodes for a common database, the Jgroup/JTS/JMS communication, database pools values for each “instance” in the wildfly/JBOSS configuration file.

Note: The individual nodes name are generated with a port offset of 100-800 for each of the eight (8) nodes; any hard-coded values are updated as well (via addition or multiplication).

To ensure the hornetq and Jgroup names are correctly defined for the chain cluster, a case statement is used to ensure that each node’s standalone-full-ha.xml configuration file is updated accordingly, if # of nodes are changed (this is offered as a variable at the top of the script.)

The below example also shows how to leverage CA APM / Wily agent for each J2EE/Wildfly node.

#!/bin/bash
###############################################################################################
#
#  Goal:  Create a N node J2EE Cluster using Wildfly 8.x.x for CA Identity Manager on a single host
#         Use for sandbox testing and validation of performance I/O parameters
#
#  Notes:  Tested for 2-8 nodes and with the CA APM (Wily) agent enabled for each node
#
#
#  Author:  A. Baugher, ANA, 8/2019
#
#
###############################################################################################
#set -vx
tz=`/bin/date --utc +%Y%m%d%H%M%S.3%N.0Z`
MAX=5
counter=1
JBOSS_HOME=/opt/CA/wildfly-idm


echo "######  STEP 00:  Stop all prior work with cluster testing ######"  > /dev/null 2>&1
kill -9 `ps -ef | grep java | grep -v grep | grep UseString | awk '{print $2}'`

echo "######  STEP 01:  Copy the current IME (Wildfly) folder to a new folder & with new port offset ######"
echo "Create this many cluster nodes:  $MAX"
echo "Current TimeStamp:  $tz"
echo ""
while [ $counter -le $MAX ]
do
  c=$counter
  n=$((100+counter)); n=${n#1}
  o=$((100*counter))
  nettyo=$((5456+o))
  jgrpo=$((7600+o))
  cli=$((9990+o))

 echo "Current counter is: $counter and the jboss number is:  $n  with a port offset of: $o"
 echo ""
 if [ -d $JBOSS_HOME$n ]; then
   echo "Prior directory exists for $JBOSS_HOME$n"
   kill -9 `ps -ef | grep "wildfly-idm$n" | grep -v grep | awk '{print $2}'` >   /dev/null 2>&1
   echo "Remove any running processes then sleep 5 seconds before removing directory: $JBOSS_HOME$n "
   sleep 5
   rm -rf /opt/CA/wildfly-idm$n
 fi

 cp -r -p /opt/CA/wildfly-idm /opt/CA/wildfly-idm$n
 cd $JBOSS_HOME$n/standalone
 echo "Current Folder is: `pwd`"
 ls -rt
 echo "Remove data tmp log folders for new node"
 rm -rf data tmp log
 ls -rt
 echo ""
 echo ""


 echo "Update standalone-full-ha.xml for hardcoded port 5456 with offset $o"
 cd $JBOSS_HOME$n/standalone/configuration
 echo "Current Folder is: `pwd`"
 cp -r -p ca-standalone-full-ha.xml ca-standalone-full-ha.xml.$tz
 sed -i "s|5456|$nettyo|g"  ca-standalone-full-ha.xml
 echo "Updated Jgroup netty connector port:  $nettyo"
 grep  $nettyo ca-standalone-full-ha.xml
 echo ""
 echo ""

 echo "Update standalone.conf (wildfly.conf) & jboss-cli.xml for port offset by $o"
 cd $JBOSS_HOME$n/bin
 echo "Current Folder is: `pwd`"
 ls -lart standalone.conf
 ls -lart jboss-cli.xml
 cp -r -p ./init.d/wildfly.conf ./init.d/wildfly.conf.conf.$tz
 cp -r -p jboss-cli.xml jboss-cli.xml.$tz
 sed -i "s|/opt/CA/wildfly-idm|/opt/CA/wildfly-idm$n|g" ./init.d/wildfly.conf
 sed -i "s|9990|$cli|g" jboss-cli.xml
 unlink standalone.conf
 ln -s $JBOSS_HOME$n/bin/init.d/wildfly.conf standalone.conf
 echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.socket.binding.port-offset=$o\""  >> standalone.conf
 ls -lart standalone.conf
 ls -lart jboss-cli.xml
 grep "port-offset" standalone.conf
 grep "$cli" jboss-cli.xml
 echo ""
 echo ""



 echo "Update standalone.sh for node name & tcp group port"
cd $JBOSS_HOME$n/bin
pwd
cp -r -p standalone.sh   standalone.sh.$tz
ls -larth standalone.sh
sed -i "s|iamnode1|iamnode$n|g"  standalone.sh


case "$MAX" in

1)  echo "Creating JGroups for one node with port offset of $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ;;
2)  echo "Creating JGroups for two nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###################
    ;;
3)  echo "Creating JGroups for three nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###################
    ;;
4)  echo "Creating JGroups for four nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###########################
    ;;
5)  echo "Creating JGroups for five nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\],caim-srv-01\[8100\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node5_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node5_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 5 ]
        then
    sed -i '684s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node5_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node5_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###########################
    ;;
6)  echo "Creating JGroups for six nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\],caim-srv-01\[8100\],caim-srv-01\[8200\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node6_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node6_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 5 ]
        then
    sed -i '684s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 6 ]
        then
    sed -i '684s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node6_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node6_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    sed -i '682s|<journal-max-io>5000</journal-max-io>|<journal-max-io>4000</journal-max-io>|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '901s|<journal-max-io>5000</journal-max-io>|<journal-max-io>4000</journal-max-io>|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml

    ###########################
    ;;
7)  echo "Creating JGroups for seven nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\],caim-srv-01\[8100\],caim-srv-01\[8200\],caim-srv-01\[8300\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node7_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node7_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 5 ]
        then
    sed -i '684s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 6 ]
        then
    sed -i '684s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node7|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 7 ]
        then
    sed -i '684s|node1|node7|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node7_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node7_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###########################
    sed -i '682s|<journal-max-io>5000</journal-max-io>|<journal-max-io>3300</journal-max-io>|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '901s|<journal-max-io>5000</journal-max-io>|<journal-max-io>3300</journal-max-io>|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    ;;
8)  echo "Creating JGroups for eight nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\],caim-srv-01\[8100\],caim-srv-01\[8200\],caim-srv-01\[8300\],caim-srv-01\[8400\]|g"  $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node8_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node8_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 5 ]
        then
    sed -i '684s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 6 ]
        then
    sed -i '684s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node7|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 7 ]
        then
    sed -i '684s|node1|node7|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node7_live_to_node8_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node7_live_to_node8_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node8|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 8 ]
        then
    sed -i '684s|node1|node8|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node8_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node8_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node7_live_to_node8_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node7_live_to_node8_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###########################
    sed -i '682s|<journal-max-io>5000</journal-max-io>|<journal-max-io>3000</journal-max-io>|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '901s|<journal-max-io>5000</journal-max-io>|<journal-max-io>3000</journal-max-io>|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    ;;
esac

ls -lart $JBOSS_HOME$n/bin/standalone.sh
grep caim-srv $JBOSS_HOME$n/bin/standalone.sh
echo ""
echo "For Node: $n"
echo ""
grep node $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
echo ""
echo ""
echo ""


echo ""
echo ""
echo "Update CA APM / Wily Information / Agent for this instance"
cp -r -p /opt/CA/VirtualAppliance/custom/apm/wily_im $JBOSS_HOME$n/standalone/wily_im
chown -R wildfly:wildfly $JBOSS_HOME$n/standalone/wily_im
echo "JAVA_OPTS=\"\$JAVA_OPTS -Dcom.wily.introscope.agent.jmx.enable=true -Dcom.wily.introscope.agent.agentManager.url.1=localhost:5001 -Djboss.modules.system.pkgs=com.wily,com.wily.*,org.jboss.byteman,org.jboss.logmanager -Xbootclasspath/p:$JBOSS_HOME$n/modules/system/layers/base/org/jboss/logging/main/jboss-logging-3.1.4.GA.jar:$JBOSS_HOME$n/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.0.Final.jar:$JBOSS_HOME$n/modules/system/layers/base/org/jboss/as/logging/main/wildfly-logging-8.2.0.Final.jar:$JBOSS_HOME$n/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.2.Final.jar\""  >> standalone.conf
echo "JAVA_OPTS=\"\$JAVA_OPTS -Dcom.wily.introscope.agent.agentName=iamnode$n  -Dcom.wily.introscope.agentProfile=$JBOSS_HOME$n/standalone/wily_im/core/config/IntroscopeAgent.profile -javaagent:$JBOSS_HOME$n/standalone/wily_im/Agent.jar    \""  >> standalone.conf
echo ""
echo ""

 counter=$(( $counter + 00001 ))
done






counter=1
while [ $counter -le $MAX ]
do
  echo "Reset ownership permissions for $JBOSS_HOME$n to wildfly userID"
  chown -R wildfly:wildfly $JBOSS_HOME$n
  echo "Start up node: $n of $MAX Wildfly cluster"
  n=$((100+counter)); n=${n#1}


  if [ "$(whoami)" != "wildfly" ]; then
       echo "Run this process under the wildfly userid to avoid permissions issue with root"
       su - wildfly -c "$JBOSS_HOME$n/bin/standalone.sh &"
       chown -R wildfly:wildfly $JBOSS_HOME$n
  else
  $JBOSS_HOME$n/bin/standalone.sh &
  fi

  counter=$(( $counter + 00001 ))
done

Reduce log duplication: Avoid nohup.out

If you plan on starting your J2EE services manually, and wish to keep them running after you log out, a common method is to use nohup ./command.sh &.

The challenge with the above process, is it will create its own output file nohup.out in the folder that the command was executed in.

Additionally, this nohup.out would be a 2nd I/O operation that would recreate the server.log file for the J2EE service.

To avoid this 2nd I/O operation, review leveraging a redirection of the nohup to /dev/null or determine if this J2EE service can be enabled as a RC/init.d or systemd service.

Example to update the wildfly .profile to allow an “alias” using a bash shell function, to start up the wildfly service; and avoid the creation of the nohup.out file.

echo "Enable alias (or function)  to start and stop wildfly"

#Example of function - Use this to avoid double I/O for nohup process (nohup.out file)
function start_im01 () {
     echo "Starting IM 01 node with nohup process"
     cd /opt/CA/wildfly-idm01/bin/
     pwd
     nohup ./standalone.sh  >/dev/null 2>&1 &
     sleep 1
     /bin/ps -ef | grep wildfly-idm01 | grep -v grep
}
export -f start_im01

function stop_im01 () {
     echo "Stopping IM 01 node"
     echo "This may take 30-120 seconds"
     cd /opt/CA/wildfly-idm01/bin/
     pwd
     ./jboss-cli.sh --connect  --command=":shutdown"
     sleep 5
     /bin/kill -9 `/bin/ps -ef | grep wildfly-idm01 | grep -v grep | awk '{print $2}'` >/dev/null 2>&1
}
export -f stop_im01

You may now start and stop your J2EE Wildfly service with the new “aliases” of start_im01 and stop_im01

You may note that stop_im01 attempts to cleanly stop the Wildfly service via the JBOSS/Wildfly management console port ; and if that fails, we will search and kill the associated java service. If you did “kill” a service, and have startup issues suggest removing the $JBOSS_HOME/standalone/tmp & /data folders before restart.