Tag: performance

Elevating Application Performance and Security on Software Appliances with APM Host Monitoring, APM Java, and APM JBoss(Wildfly)

October 31, 2023October 31, 2023by Alan BaugherLeave a comment

Organizations may rely on software appliances to meet the demands of secure, mission-critical applications. To ensure the optimal operation of these appliances, Application Performance Monitoring/Management (APM) tools have emerged as invaluable assets. In this comprehensive blog post, we’ll explore how the synergy of APM host monitoring via the SysEdge module, APM Java, and APM JBoss(Wildfly/J2EE) can significantly enhance the performance and security of applications running on software appliances like the Symantec Identity Suite Virtual Appliance (on-prem and AWS instances).

1. Value of APM Java and APM JBoss(Wildfly) Monitoring

APM tools feature specialized modules for Java-based applications and JBoss(Wildfly/J2EE) application servers, addressing the unique challenges posed by these technologies:

Optimizing Java-Based Applications: APM Java monitoring delves deep into Java applications, tracing bottlenecks, optimizing code, and ensuring the efficient use of Java Virtual Machine (JVM) resources.
JBoss Application Server Expertise: APM JBoss monitoring tracks the performance and stability of JBoss deployments, providing insights into JBoss-specific metrics critical for the smooth operation of JBoss-based applications.

2. The Role of APM Host Monitoring with SysEdge Module

The SysEdge module, integrated into APM tools, plays a pivotal role in enhancing the performance, security, and overall management of software appliances. This module focuses on host-based metrics, offering insights into the appliance’s performance and health.

Resource Utilization: SysEdge monitors essential resources, such as CPU, memory, disk space, and network usage, ensuring efficient resource allocation and preventing performance bottlenecks.
Hardware Health: It provides insights into the hardware/virtual components, crucial for maintaining the reliability of the appliance.
Comprehensive Diagnostics: The detailed host-based metrics allow for more accurate and rapid issue diagnostics, helping administrators identify and address problems efficiently. Gain real-time insight into digital performance, user experience and behavior

3. Benefits of APM Tools for Software Appliances

The integration of host-based metrics through SysEdge, along with APM Java and APM JBoss monitoring, offers a multitude of benefits:

Holistic Insights: APM tools provide a complete picture of the appliance’s performance, helping administrators make informed decisions by combining application-specific data with host-based metrics. Ensure flawless user experiences with analytics-driven insights
Proactive Issue Resolution: Administrators can proactively identify and address issues that may impact both application performance and the host system, reducing downtime and increasing reliability. Eliminate alert fatigue with automated root cause analysis.
Streamlined Management: These tools enable remote management of the appliance, even in challenging environments, allowing fine-tuning, patch application, and addressing security concerns. Empower every team. Improve every app.

4. Secure Deployment with Non-Root User ID and DevOps Automation

The utilization of non-root user IDs and DevOps automation can significantly enhance both security and operational efficiency in the deployment and management of applications on software appliances. Traditional application deployments often involved elevated privileges, exposing them to security vulnerabilities. Deploying applications with non-root user IDs offers several advantages:

Reduced Attack Surface: Non-root users have limited permissions, reducing the potential attack surface and making it more difficult for malicious actors to compromise the system.
Enhanced Security: By limiting application permissions, non-root deployments minimize the risk of security breaches and unauthorized access.
Compliance: Using non-root user IDs aligns with security best practices and compliance requirements, ensuring your organization meets regulatory standards.
Isolation: Non-root deployments prevent applications from interfering with critical system components, reducing the risk of conflicts and crashes.

5. Example of integration/deployment of APM tools (Java/JBoss/SysEdge) on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

The Symantec Identity Suite Virtual Appliance is a harden software appliance that only allow authentication for one (1) of two (2) non-root IDs (config or ec2-user). The Symantec Identity Suite does allow for APM type tools to be deployed via extraction under the path /opt/CA/VirtualAppliance/custom/profiler.

We want to walk-through how to enable the DX APM SaaS Infra Agent with HostMonitoring (SysEdge) enabled on an Amazon Linux 2 host as a non-root user ID, as well as integration with the CA Identity Suite three (3) JBoss/Wildfly instances for IM/IG/IP and the CA Identity Suite JCS Connector Server with the embedded Java agent.

Additionally, we wanted to ensure that any external configuration access was disabled, as we only wanted to allow a “push” configuration/model of data from the vApp to the APM SaaS Collection APIs. We did not wish to allow any modification of the APM agent’s configuration on the vApp, that was not defined during initial deployment.

Four (4) parameters were modified from default installation:

1. Ensure non-root id is used for sysedge 

echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf

2. Ensure only local host can pull or send data to the sysedge agent

echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf  

3. Mask low value entries - Switch to debug loglevel as needed to address configuration challenges

echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf 

4. Disable remote management via APM Command Center (ACC)
      
sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile

Leveraging the information provided by the Broadcom DX Application Performance Management and Symantec Identity Suite documentation, we were able to clarify the dependencies on JBoss logging log4j jar(s) and LogManager objects with Adopt Open JDK 8. Neither document had the exact configuration that we found viable during testing.

We had two (2) challenges deploying the HostMonitoring (sysedge) agent on the AWS Identity Suite vApp instance that we were able to identify and address. No issues were found on the on-prem edition of the Identity Suite vApp.

Challenge(s):
- File ownership of the sysedge PID and log file by ‘root’ instead of the non-root user due to default systemd startup script for sysedge module/binary.
- Null entry within a file impacted startup of ‘sysedge’ binary from default OS file /etc/redhat-release

The below bash shell script utilized the APM SaaS Binary download feature of embedding the sysedge module/binary component with the primary APM Infra agent. This will include the unique token for your own APM SaaS infrastructure (or APM Enterprise infra). The APM SaaS download process provides a great automation download via wget process. The Symantec Identity Suite allows the non-root IDs to start/stop systemd process, which we leverage. Alternatively, we may use crontab for the non-root IDs to start/stop the two (2) services: apmia and sysedge as documented for DX APM agents.

#!/bin/bash
####################################################################
#
#  Install the APM SaaS Infra Agent with HostMonitoring module (sysedge) as non-root id
#  - Update variables of NON_ROOT_USER_ID and INFRA_DOWNLOAD_URL and APM_INFRA_HOME_FOLDER
#  - This script works for a host that allows minimal sudo access to systemctl
#  - Alternative startup process is crontab for the non-root-id
#
#  Goal:  Replica process for:   ./APMIACtrl.sh install user=non-root-id
#   and ./APMIACtrl.sh console_start
#
#  Methodology:  Isolate delta between folders using diff with sub-folder detection
#   diff -iry --suppress-common-lines apmia/ apmia.orginal/
#
#  Important Note: Identified RCA for sysedge binary having memory fault SEGV
#  /etc/redhat-release MUST be populated (avoid null value)
#
#
#  Crontab notes from online APM agent docs: (if needed)
#    @reboot /home/user/apmia/APMIACtrl.sh console_start > /home/user/logs/cron.log 2>&1
#    */5 * * * * /home/user/apmia/APMIACtrl.sh console_start > /home/user/logs/cron.log 2>&1
#
#  Modify default APM Infra Agent parameters with these changes
#   1. Ensure non-root id is used for sysedge
#         echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   2. Ensure only local host can pull or send data to the sysedge agent
#        echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   3. Mask low value entries - Switch to debug loglevel as needed to address configuration challenges
#        echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
#   4. Disable acc integration (remote management) via APM Command Center (ACC)
#        sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile
#
#
#
#  Ref.  https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/install-and-deploy-infrastructure-agent/install-the-infrastructure-agent-on-ca-digital-experience-insights.html
#        https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/infrastructure-agent/Supportability-Matrix-for-Infrastructure-Agent.html
#        https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-apm-agents/SaaS/SystemEDGE-based-Monitoring.html
#
# ANA 10/2023
#
####################################################################
#NON_ROOT_USER_ID=config
NON_ROOT_USER_ID=ec2-user

echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM Infra Agent with HostMonitoring check / enabled - This will be packaged together "
echo "################################################################################################"
INFRA_DOWNLOAD_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXXXXXXXXXXXXXXXXX_LONG_TOKEN_HERE__XXXXXXXXXXX"


APM_INFRA_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_infra
mkdir -p ${APM_INFRA_HOME_FOLDER}
cd       ${APM_INFRA_HOME_FOLDER}
pwd
ls -lart

echo "wget --no-check-certificate --content-disposition ${INFRA_DOWNLOAD_URL} -O Infrastructure_Agent_apmia.tar"
wget --no-check-certificate --content-disposition ${INFRA_DOWNLOAD_URL} -O Infrastructure_Agent_apmia.tar
APM_INFRA_FILE_NAME=$(ls -lart Infrastructure_Agent_apmia* |tail -1 | awk '{print $9}')
echo "tar -xvf ${APM_INFRA_FILE_NAME} "
#tar -xvf ${APM_INFRA_FILE_NAME}
tar -xf ${APM_INFRA_FILE_NAME}


echo ""
echo ""
echo "################################################################################################"
echo "Update APM Infra Agent startup file to use non-root user ID of ${NON_ROOT_USER_ID} "
echo "################################################################################################"
sed -i "s|#RUN_AS_USER=|RUN_AS_USER=${NON_ROOT_USER_ID}|g" ${APM_INFRA_HOME_FOLDER}/apmia/bin/APMIAgent.sh

echo ""
echo ""
echo "################################################################################################"
echo "Validate update of NON_ROOT_USER_ID"
echo "################################################################################################"
grep -C 2 -i "RUN_AS_USER=${NON_ROOT_USER_ID}"  ${APM_INFRA_HOME_FOLDER}/apmia/bin/APMIAgent.sh


echo ""
echo ""
echo "################################################################################################"
echo "Extract SystemEdge component for APM Infra Host Monitoring"
echo "################################################################################################"
export AGENTHOME=${APM_INFRA_HOME_FOLDER}/apmia
SYSEDGE_FILE_NAME=$(ls ${AGENTHOME}/casystemedge*)
echo ${SYSEDGE_FILE_NAME}
cd ${AGENTHOME}
#tar -xvf ${SYSEDGE_FILE_NAME}
tar -xf ${SYSEDGE_FILE_NAME}


echo ""
echo ""
echo "################################################################################################"
echo "Deploy and install SystemEdge component for APM Infra Host Monitoring with non-root user ID"
echo "################################################################################################"
kill $(pidof sysedge) &>/dev/null
rm -rf ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE &>/dev/null
cd ${APM_INFRA_HOME_FOLDER}/apmia/CA_SystemEDGE_Core
./ca-setup.sh install
/bin/ps -ef | grep -i sysedge | grep -v grep


echo ""
echo ""
echo "################################################################################################"
echo "Update and restart the SystemEdge component for APM Infra Host Monitoring with non-root user ID"
echo "################################################################################################"
# Ensure non-root id is used for sysedge
echo "privilege_separation_user ${NON_ROOT_USER_ID}" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
# Ensure only local host can pull or send data to the sysedge agent
echo "bind_address 127.0.0.1" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
# Mask low value entries - Switch to debug loglevel as needed to address configuration challenges
echo "sysedge_loglevel fatal" >> ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf
cp -r -p ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/sysedge.cf  ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl stop
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl start
/bin/ps -ef | grep -i sysedge | grep -v grep

echo ""
echo ""
echo "################################################################################################"
echo "Check updates to sysedge configuration file sysedge.cf "
echo "################################################################################################"
#tail -5 ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf
grep -Hin -v -e "^$" -e "^#" -e "^template" ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.cf


echo ""
echo ""
echo "################################################################################################"
echo "Update the APM Infra main agent and disable the acc component "
echo "################################################################################################"
sed -i "s|introscope.agent.acc.enable=true|introscope.agent.acc.enable=false|g" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile
grep "introscope.agent.acc.enable" ${APM_INFRA_HOME_FOLDER}/apmia/core/config/IntroscopeAgent.profile


echo ""
echo ""
echo "################################################################################################"
echo "Create systemd startup process on vApp due to sudo systemctl process allowed on vApp for APM Infra main agent"
echo "################################################################################################"

cat << EOF > ${APM_INFRA_HOME_FOLDER}/apmia/apmia.service
# /etc/systemd/system/apmia.service
[Unit]
Description=APM Infrastructure Agent
After=syslog.target

[Service]
Type=forking
ExecStart="${APM_INFRA_HOME_FOLDER}/apmia/bin/./APMIAgent.sh" start sysd
ExecStop="${APM_INFRA_HOME_FOLDER}/apmia/bin/./APMIAgent.sh" stop sysd
User=${NON_ROOT_USER_ID}
KillMode=control-group
Environment=SYSTEMD_KILLMODE_WARNING=true

[Install]
WantedBy=multi-user.target

EOF

sudo systemctl stop          apmia.service &>/dev/null
sudo systemctl disable       apmia.service &>/dev/null
sudo systemctl enable ${APM_INFRA_HOME_FOLDER}/apmia/apmia.service
echo "################################################################################################"
sudo systemctl cat           apmia.service
echo "################################################################################################"
sudo systemctl daemon-reload
sudo systemctl start         apmia.service
sudo systemctl status        apmia.service -a -l --no-pager



echo ""
echo ""
echo "################################################################################################"
echo "Create systemd startup process on vApp due to sudo systemctl process allowed on vApp for Sysedge agent"
echo "################################################################################################"
# Stop sysedge via manual process to use the systemd process
${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/sysedgectl stop

# Manage these two (2) systemd error with PIDs and Paths
# Refusing to accept PID outside of service control group, acquired through unsafe symlink chain
# /opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/config/port1691/sysedge.service:8] Not an absolute path
#
cat << EOF > ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service
# /etc/systemd/system/sysedge.service
[Unit]
Description=sysedge
After=syslog.target

[Service]
Type=forking
WorkingDirectory=${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691
#Environment=VAR_HERE_ENV_ENV=production PATH=PATH_HERE_IF_NEEDED
ExecStart="${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/CA-SystemEDGE" start sysd
ExecStop="${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/bin/CA-SystemEDGE" stop sysd
PIDFile=${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.pid
User=${NON_ROOT_USER_ID}
KillMode=none

[Install]
WantedBy=multi-user.target

EOF
echo ""
echo ""
echo "################################################################################################"
cat ${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service
echo "################################################################################################"
echo ""
echo ""
sudo systemctl stop          sysedge.service  &>/dev/null
sudo systemctl disable       sysedge.service  &>/dev/null
sudo systemctl enable "${APM_INFRA_HOME_FOLDER}/apmia/SystemEDGE/config/port1691/sysedge.service"
echo "################################################################################################"
sudo systemctl cat           sysedge.service
echo "################################################################################################"
sudo systemctl daemon-reload
sudo systemctl start         sysedge.service
sudo systemctl status        sysedge.service -a -l --no-pager



echo ""
echo ""
echo "################################################################################################"
echo "Ensure all files are owned by the non-root id of ${NON_ROOT_USER_ID} for APM Infra Agent"
echo "Check for any error messages "
echo "################################################################################################"
sudo systemctl stop          apmia.service
sudo systemctl stop          sysedge.service
echo "chown -R ${NON_ROOT_USER_ID}:${NON_ROOT_USER_ID} ${APM_INFRA_HOME_FOLDER}/apmia "
echo "################################################################################################"
chown -R ${NON_ROOT_USER_ID}:${NON_ROOT_USER_ID} ${APM_INFRA_HOME_FOLDER}/apmia
echo ""
echo ""

echo ""
echo ""
echo "################################################################################################"
echo "View running services: APM Infra will have a parent process of wrapper and two (2) java child processes"
echo "The APM Infra HostMonitoring module will have one process name sysedge "
echo "################################################################################################"
sudo systemctl start          apmia.service
sudo systemctl start          sysedge.service
ps -ef | grep apmia | grep -v grep



echo ""
echo ""
echo "################################################################################################"
echo "Use these these commands to view / monitor / start & stop APM Infra Agent"
echo "################################################################################################"
echo "sudo systemctl status  apmia.service -a -l --no-pager"
echo "sudo systemctl stop    apmia.service"
echo "sudo systemctl start   apmia.service"
echo "sudo systemctl cat     apmia.service"
echo "journalctl          -u apmia.service -f"
echo ""
echo "sudo systemctl status  sysedge.service -a -l --no-pager"
echo "sudo systemctl stop    sysedge.service"
echo "sudo systemctl start   sysedge.service"
echo "sudo systemctl cat     sysedge.service"
echo "journalctl          -u sysedge.service -f"
echo "journalctl          -u sysedge.service -xe -f"
echo ""
echo ""

6. Example of JVM arguments for integration of APM SaaS Java Agent with Identity Suite JCS Connector Server)

The APM SaaS or APM Enterprise UI solution will provide an agent download page, that allows section of the agent to be deployed.

Instruction will be provided and please noticed the “wget” auto download link that may be leveraged for automation (dev-ops) processes. This link will include both the siteID and the download token ID.

If you haven’t generated a new credentials token, after you download the Java/Jboss agents, click “Show Agent Details” to harvest the three (3) key/value pairs.

The below bash shell script will download the APM Java Agent package. There is no embedded credential token with this download. The credentials are provided separately via the APM SaaS UI (as shown above). While we could place these three (3) parameters within the APM agent configuration file, IntroscopeAgent.profile, we decided to clarity the use of JVM switches to override any values, and allow us to automate this deployment independent of any new APM agents updates.

#!/bin/bash
######################################################################################
#
#  Automate deployment of the APM SaaS Java Agent with credentials & urls
#   to the single Identity Manager JCS Connector Server instance
#
#  Use variables for the JVM parameters.  Adjust if needed.
#  We may override the default naming convention to clarity which instance is
#  being monitoring within the APM SaaS Dashboard, to avoid confusion with any
#  JBoss agent instance from IM/IG/IP
#
#  ANA 10/2023
#
######################################################################################
echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM Java Agent "
echo "################################################################################################"
APM_CREDENTIAL_TOKEN='XXXXXXX_LONG_TOKEN_HERE_FROM_APM_SAAS_UI__XXXXXXX'
APM_URL='apmgw.dxi-na1.saas.broadcom.com'
APM_SAAS_AGENT_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXX_DOWNLOAD_TOKEN_PROVIDED_FROM_APM_SAAS_UI_WHEN_SELECTED___XXXXXXXXXXXX"
APM_AGENT_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_java
APM_AGENT_FILE_NAME="APM_SaaS_Java_Agent.tar"
mkdir -p ${APM_AGENT_HOME_FOLDER}
cd       ${APM_AGENT_HOME_FOLDER}
pwd
ls -lart


echo "wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}"
wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}
ls -lart
echo "tar -xvf ${APM_AGENT_FILE_NAME} "
#tar -xvf ${APM_AGENT_FILE_NAME}
tar -xf ${APM_AGENT_FILE_NAME}
ls -lart
#
#
tz=`/bin/date --utc +%Y%m%d%H%M%S`
APM_AGENT_NAME=IM_JCS_NODE
JVM_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/profiler/
JVM_FILE=/opt/CA/IdentityManager/ConnectorServer/data/jvm_options.conf

if [ -f ${JVM_FILE} ];then

cp -r -p ${JVM_FILE}  ${JVM_BACKUP_LOCATION}/${tz}_jvm_options.conf

echo "-server -Xms1g -Xmx2g -Djava.awt.headless=true -Dcom.sun.net.ssl.enableECC=true -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9  -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIAL_TOKEN}\"  -Dcom.wily.introscope.agent.agentName=${APM_AGENT_NAME}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${JCS_JVM_FILE}

echo "Start / Stop JCS"

echo "sudo systemctl stop im_jcs "
sudo systemctl stop im_jcs

echo "sudo systemctl start im_jcs "
sudo systemctl start im_jcs

}

7. Example of JVM arguments for integration of APM SaaS JBoss Agent with Identity Suite IM/IG/IP JBoss/Wildfly instances

The below bash shell script is for the three (3) JBoss (Wildfly) instances on the vApp. Fortunately for us, the Broadcom/Symantec Product/Engineering team kept the same log4j jar and class files version between all three (3) instances, so we were able to leverage variables for each. You may wish to adjust the JVM parameters as needed. Please note, that again we did not embed any key/value pair into the APM configuration file, as we wanted to automate this deployment independent of any new APM agents updates.

#!/bin/bash
#########################################################################
#
#  Automate deployment of the APM SaaS JBOSS Agent with credentials & urls
#   to the three JBoss/Wildfly instances
#     - IdentityManager
#     - IdentityGovernance
#     - IdentityPortal
#
#  Use variables for the JVM parameters that allow a similar configuration for
#  all three (3) instances.  Adjust if needed.  Since all three (3) JBoss are on the
#  same host, we will override the default naming convention to clarity which instance is
#  being monitoring within the APM SaaS Dashboard
#
#  ANA 10/2023
#
#########################################################################
echo ""
echo ""
echo "################################################################################################"
echo "# Ensure the download url has the APM JBoss Agent "
echo "################################################################################################"
APM_SAAS_AGENT_URL="https://apmgw.dxi-na1.saas.broadcom.com/acc/apm/acc/downloadpackage/XXX_SITE_ID_STRING_XXXX?format=archive&layout=bootstrap_preferred&packageDownloadSecurityToken=XXXX__DOWNLOAD_TOKEN_HERE"
APM_CREDENTIALS='XXXXX_LONG_CREDENTIAL_TOKEN_HERE__XXXXXX'
APM_URL='apmgw.dxi-na1.saas.broadcom.com'
APM_AGENT_HOME_FOLDER=/opt/CA/VirtualAppliance/custom/profiler/apm_jboss
APM_AGENT_FILE_NAME="APM_SaaS_JBOSS_Agent.tar"
mkdir -p ${APM_AGENT_HOME_FOLDER}
cd       ${APM_AGENT_HOME_FOLDER}
pwd
ls -lart


echo "wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}"
wget --no-check-certificate --content-disposition ${APM_SAAS_AGENT_URL} -O ${APM_AGENT_FILE_NAME}
echo "tar -xvf ${APM_AGENT_FILE_NAME} "
#tar -xvf ${APM_AGENT_FILE_NAME}
tar -xf ${APM_AGENT_FILE_NAME}
ls -lart
tz=`/bin/date --utc +%Y%m%d%H%M%S`



JBOSS_INSTANCE=IdentityManager
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf

if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "
cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-idm "
sudo systemctl stop wildfly-idm

echo "sudo systemctl start wildfly-idm "
sudo systemctl start wildfly-idm

fi

JBOSS_INSTANCE=IdentityGovernance
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf
if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "

cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-ig "
sudo systemctl stop wildfly-ig
echo "sudo systemctl start wildfly-ig "
sudo systemctl start wildfly-ig

fi


JBOSS_INSTANCE=IdentityPortal
FILE_BACKUP_LOCATION=/opt/CA/VirtualAppliance/custom/${JBOSS_INSTANCE}
FILE_JVM_FILE=${FILE_BACKUP_LOCATION}/jvm-args.conf
if [ -f ${FILE_JVM_FILE} ]; then

echo "cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf "
cp -r -p ${FILE_JVM_FILE}  ${FILE_BACKUP_LOCATION}/${tz}_jvm-args.conf
echo "JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -javaagent:${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/Agent.jar  -Dcom.wily.introscope.agentProfile=${APM_AGENT_HOME_FOLDER}/wily/releases/2023.9/core/config/IntroscopeAgent.profile    -Djboss.modules.system.pkgs=org.jboss.logmanager,org.jboss.byteman,com.wily,com.wily.* -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Xbootclasspath/p:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.1.5.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-1.4.0.Final.jar:\${JBOSS_HOME}/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.6.Final.jar -Dcom.wily.org.apache.commons.logging.Log=com.wily.org.apache.commons.logging.impl.NoOpLog  -Dintroscope.agent.bootstrap.home=${APM_AGENT_HOME_FOLDER}/wily -Dintroscope.agent.bootstrap.release.version=2023.9 -Dintroscope.agent.bootstrap.version.loaded=2023.9 -Dcom.wily.introscope.agentManager.url.1=wss://${APM_URL} -Dcom.wily.introscope.agentManager.url.2=https://${APM_URL} -Dcom.wily.introscope.agentManager.credential=\"${APM_CREDENTIALS}\"  -Dcom.wily.introscope.agent.agentName=${JBOSS_INSTANCE}  -XX:+PrintFlagsFinal -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector" > ${FILE_JVM_FILE}

echo "sudo systemctl stop wildfly-portal "
sudo systemctl stop wildfly-portal
echo "sudo systemctl start wildfly-portal "
sudo systemctl start wildfly-portal

fi

8. Conclusion of APM tools integration on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

We have been successful using APM tool and home-grown java monitoring processes to improve the performance of the Symantec Identity Suite solution for peak use-cases. Using these processes should allow you to peak into the ‘blackbox’ of Java/JBoss processes to understand where bottlenecks may exists.

9. View sysedge.cf configuration for proposed change to isolate behavior to single host

View of the configuration file for sysedge.cf via cat sysedge.cf | grep -i -v -e “^$” -e “^#” -e “^template”

Token / Parameter	Value	Commentary
version 6.0
sysedge_loglevel	fatal	Proposed change: Switch from default log level of “info” to “fatal” to avoid bloat in sysedge.log file due to embedded function in sysedge binary to copy configuration file to /etc folder for non-root ID. Switch to debug if there are any configurations challenges.
bind_address	127.0.0.1	Proposed change: Prevent external updates to SNMP trap of SysEdge (allow only localhost) – sudo systemctl start sysedge.service udp 0 0 127.0.0.1:1691 0.0.0.0:* 117588/sysedge Validate: ./walktree -h 127.0.0.1 -p 1691 -c apmmonitor -r 3 -o 1.3.6.1.2.1.1
community	apmmonitor read-only	Set during installation. May be set via scripts for manual deployment
no_authen_traps
no_process_sets
no_remoteshell_group
no_actions
no_stat_floppy
no_warmstart_discovery
bulk_read_response_soft_limit	60000
monitor	oid memCapacity.0 11 0x300200 300 absolute >= 95 ‘Physical Memory Usage (percentage)’ ” ‘Memory’ ‘Total’ ‘PercentUsed’ warning
monitor	oid memCapacity.0 12 0x300200 300 absolute >= 98 ‘Physical Memory Usage (percentage)’ ” ‘Memory’ ‘Total’ ‘PercentUsed’ minor
monitor	oid cpuTotalIdlePercent.0 13 0x600200 120 absolute <= 10 ‘CPU Idle Time (percentage)’ ” ‘CPU’ ‘Total’ ‘PercentIdle’ warning
monitor	oid cpuTotalIdlePercent.0 14 0x600200 120 absolute <= 5 ‘CPU Idle Time (percentage)’ ” ‘CPU’ ‘Total’ ‘PercentIdle’ minor
monitor	oid diskStatsUtilization.1 15 0x300200 300 absolute >= 90 ‘Disk Busy Time (percentage)’ ” ‘Disk’ ‘First’ ‘PercentBusy’ warning
monitor	oid diskStatsUtilization.1 16 0x300200 300 absolute >= 95 ‘Disk Busy Time (percentage)’ ” ‘Disk’ ‘First’ ‘PercentBusy’ minor
monitor	devTableEntry ‘C:’ devCapacity 17 0x0 300 absolute >= 95 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘C:’ ‘devCapacity’ warning
monitor	devTableEntry ‘C:’ devCapacity 18 0x0 300 absolute >= 98 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘C:’ ‘devCapacity’ minor
monitor	devTableEntry ‘/’ devCapacity 19 0x0 300 absolute >= 95 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘Root’ ‘devCapacity’ warning
monitor	devTableEntry ‘/’ devCapacity 20 0x0 300 absolute >= 98 ‘File System Usage (percentage)’ ” ‘devTableEntry’ ‘Root’ ‘devCapacity’ minor
autowatch	-index=11 -name=’All FileSystems – Warning’ -watchtype=generic -table=devTableEntry -attribute=devCapacity -interval=300 -criteria=’.*’ -op=ge -value=95 -severity=warning -desc=’FileSystem Usage (percentage)’ -objclass=FileSystem -objattr=PercentUsed
autowatch	-index=12 -name=’All FileSystems – Minor’ -watchtype=generic -table=devTableEntry -attribute=devCapacity -interval=300 -criteria=’.*’ -op=ge -value=98 -severity=minor -desc=’FileSystem Usage (percentage)’ -objclass=FileSystem -objattr=PercentUsed
autowatch	-index=13 -name=’All Disks – Warning’ -watchtype=generic -table=diskStatsEntry -attribute=diskStatsUtilization -interval=300 -criteria=’.*’ -op=ge -value=90 -severity=warning -monflags=0x300200 -desc=’Disk Busy Time (percentage)’ -objclass=Disk -objattr=PercentBusy
autowatch	-index=14 -name=’All Disks – Minor’ -watchtype=generic -table=diskStatsEntry -attribute=diskStatsUtilization -interval=300 -criteria=’.*’ -op=ge -value=95 -severity=minor -monflags=0x300200 -desc=’Disk Busy Time (percentage)’ -objclass=Disk -objattr=PercentBusy
sysedge_plugin	*
data_directory	/opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/config/’	Set during installation. May be set via scripts for manual deployment
default_port	1691	Set during installation. May be set via scripts for manual deployment
no_proc_monitor
no_procgroup_monitor
privilege_separation_user	config OR ec2-user	Proposed change: Add new required parameter to enforce non-root access for a local host account. Confirm ownership when stopping/starting the service via systedgectl or sudo systemctl start sysedge.service. Confirm file ownership for PID & log files. config 117588 1 6 14:41 ? 00:00:16 /opt/CA/VirtualAppliance/custom/profiler/apm_infra/apmia/SystemEDGE/bin/sysedge -b

10. Prior notes / examples of integration/deployment of APM tools (Java/JBoss) on the Symantec Identity Suite Virtual Appliance with non-root Id (config/ec2-user)

We have been involved with use of APM tools for quite a bit and contributed to the inclusion of these features into various solution, including software appliances like the Symantec Identity Suite with non-root access.

https://community.broadcom.com/enterprisesoftware/communities/community-home/digestviewer/viewthread?GroupId=2197&MID=773616&CommunityKey=f9d65308-ca9b-48b7-915c-7e9cb8fc3295&tab=digestviewer

Engage with us to help accelerate your business goals

Parallel provisioning for Active Directory and MS Exchange mailboxes – Improve Birthright/DayOne Access

May 24, 2021May 30, 2021by Alan Baugher1 Comment

One of the challenges that IAM/IAG solutions may have is using single thread processing for select endpoints. For the CA/Symantec Identity Management solution, before IM r14.3cp2, we lived with a single-threaded connector to managed MS Active Directory endpoints.

To address this challenge, we deployed multiple connector servers. We allowed the IM Provisioning Server (IMPS) to use a built-in round-robin approach of load-balancing separate transactions to different connector servers, which would service the same Active Directory endpoints.

The IME may be running as fast as it can with its clustered deployment, but as soon as a task has MS Active Directory, and there is a bottleneck with the CCS Service. We begin to see the IME JMS queue reporting that it is stuck and the IME View Submitted Task reporting “In Progress” for all tasks. If the CCS service is restarted, all IME tasks are then reported as “Failed.”

This is/was the bottleneck for the solution for sites that have MS Active Directory for Birthright/DayOne Access.

We can now avoid this bottleneck. [*** (5/24/2021) – There is an enhancement to CP2 to address im_ccs.exe crashes during peak loads discovered using this testing process. ]

Via the newly delivered enhancement https://community.broadcom.com/participate/ideation-home/viewidea?IdeationKey=7154e15b-085d-469e-bff0-ac588ff6bd5b .

We now have full parallel provisioning to MS Active Directory from a single connector server (JCS/CCS).

The new attribute that regulates this behavior is eTADSMaxConnectionsInPool. This attribute will be applied on every existing ADS endpoint that is currently being managed by the IM Provisioning Server after CP2 is deployed. Note: The default value is 10, but we recommend after much testing, to match the value of the IMPS-> JCS and JCS->CCS to equal 200.

During testing within the IME using Bulk Tasks or the IM BLC, we can see that the CCS-> ADS traffic will reach 20-30 connections if allowed. You may set this attribute to a value of 200 via Jxplorer and/or an ldapmodify/dxmodify script.

echo "############### SET ADS MAX CONNECTIONS IN POOL SIZE ##################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
NAMESPACE=exchange2016
LDAPTLS_REQCERT=never dxmodify -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD" << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'

To confirm the number of open connections is greater than one (1), we can issue a Bulk IM Task or use a performance tool like CA Directory dxsoak.

In this example, we will show case using CA Directory dxsoak to execute 100 parallel threads to create 100 ADS Accounts with MS Exchange Mailboxes. We will also enclose this script for download for others to review and use.

Performance Lab:

Pre-Steps:

Leverage CA Directory samples’ dxsoak binary (performance testing). You may wish to use CA Directory on an existing IM Provisioning Server (Linux OS) or you may deploy CA Directory (MS Windows version) to the JCS/CCS connector. Examples are provided for both OSes.
Create LDIF files for IM Provisioning Server and/or IM Connector Tier. This file is needed to ‘push’ the solution to-failure. The use of the IME Bulk Task and/or etautil scripts to the IM Provisioning Tier, will not provide the transaction speed we need to break the CCS service if possible.
Within the IM Provisioning Manager enable the ADS Endpoint TXT Logs on the Logging TAB, for all checkboxes.
Monitor the IMPS etatrans* logs, monitor the JCS ADS logs, monitor the CCS ADS logs, monitor the number of CCS-> ADS (LDAP/S – TCP 389/636) threads. [Suggest using MS Sysinternals Process Explorer and select im_ccs.exe & then TCP/IP TAB]
Monitor the MS ADS Domain via MS ADUC (AD Users & Computers UI) and MS Exchange Mailbox (Mailbox UI via Browser)

Execution:

6. Perform a UNIT TEST with dxmodify/ldapmodify to confirm the LDIF file input is correct with the correct suffix.

time dxmodify -H ldap://192.168.242.135:20389 -c -x -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldif

7. Perform the PERFORMANCE TEST with dxsoak binary with the same LDIF file & correct suffix. Rate observed = 23 K ids/hr

./dxsoak -c -l 60 -t 100 -h 192.168.242.135:20389 -D "eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta" -w Password01 -f ads_user_with_exchange_dc_eta.ldif

Observations:

8. IMPS etatrans*.log – Count the number of operations per second. Note any RACE and/or data collisions, e.g. ADS accounts deleted prior to add via 100 threads or ADS account created multiple times attempted in different threads.

9. IM CCS ADS <endpoint>.log – Will only have useful data if the ADS Endpoint Logging TAB has been checked for TXT logs.

10. Finally, validate directly in MS Active Domain with the ADUC or similar tool & MS Exchange mailboxes being created/deleted.

11. Count the number of threads from im_ccs.exe to ADS – Suggest using MS Sysinternals Process Explorer tool and/or Powershell to count the number of connections.

MS Powershell Script to count the number of LDAP (TCP 389) connection from im_ccs.exe. [Note: TCP 389 is used more if the ADS Endpoint is setup to use SASL authentication. TCP 636 is used more if the ADS Endpoint is using the older TLS authentication]

$i=1
Do {
cls
(Get-NetTCPConnection -State Established -OwningProcess (Get-Process -name im_ccs).id -RemotePort 389).count
Start-Sleep -s 1
$i++
}
while ($i -le 5)

Direct Performance Testing to JCS/CCS Service

While this testing has limited value, it can offer satisfaction and assistance to troubleshoot any challenges. We can use the prior LDIF files with a slightly different suffix, dc=etasa (instead of dc=eta), to use dxsoak to push the connector tier to failure. This step helped provide memory dumps back to CA/Symantec Engineering teams to help isolate challenges within the parallel processing. CCS Service is only exposed via localhost. If you wish to test the CCS Service remotely, then update the MS Registry key for the CCS service to use the external IP address of the JCS/CCS Server. Rate observed = 25 K ids/hr

Script to generate 100 ADS Accounts with MS Exchange Mailbox Creation

You may wish to review this script and adjust it for your ADS / MS Exchange domains for testing. You can also create a simple LDIF file with password resets or ADS group membership adds. Just remember that the IMPS Service (TCP 20389/20390) uses the suffix dc=eta, and the IM JCS/CCS Services (TCP 20410/20411) & (TCP 20402/20403) use the suffix dc=etasa. Additionally, if using CA Directory dxsoak, only use the non-TLS ports, as this binary is not equipped for using TLS certs.

generate_ads_users_ldif.sh_Download

#!/bin/bash
#######################################################################################################################
# Name:  Generate ADS Feed Files for IM Solution Provisioning/Connector Tiers
#
# Goal:  Validate the new parallel processes from the IM Connector Tier to Active Directory with MS Exchange
#
#
# Generate ADS User LDIF file(s) for use with unit (dxmodify) and performance testing (dxsoak) to:
#  - {Note: dxsoak will only work with non-TLS ports}
#
# IM JCS (20410)  "dc=etasa"    {Ensure MS Windows Firewall allows this port to be exposed}
# IM CCS (20402)  "dc=etasa"    {This port is localhost only, may open to network traffic via registry update}
# IMPS (20389)    "dc=eta"
#
#
# Monitor:  
#
# The IMPS etatrans*.log  {exclude searches}
# The JCS daily log
# The JCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
# The CCS ADS log {Enable the ADS Endpoint TXT logging for all checkboxes}
#
# Execute per the examples provided during run of this file
#
#
# ANA 05/2021
#######################################################################################################################

# Unique Variables for an ADS Domain
NAMESPACE=exchange2016
ADSDOMAIN=exchange.lab
DCDOMAIN="DC=exchange,DC=lab"
OU=People

#######################################################################################################################


MAX=100
start=00001
counter=$start
echo "###############################################################"
echo "###############################################################"
START=`/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`
echo `/bin/date --utc +%Y%m%d%H%M%S,%3N.0Z`" = Current OS UTC time stamp"
echo "###############################################################"
FILE1=ads_user_with_exchange_dc_etasa.ldif
FILE2=ads_user_with_exchange_dc_eta.ldif
echo "" > $FILE1
while [ $counter -le $MAX ]
do
    n=$((10000+counter)); n=${n#1}
    tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
   echo "Counter with leading zeros = $n   at time:  $tz"


cat << EOF >> $FILE1
dn:  eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: add
objectClass:  eTADSAccount
eTADSobjectClass:  user
eTADSAccountName:  firstname$n aaalastname$n
eTADSgivenName:  firstname$n
eTADSsn:  aaalastname$n
eTADSdisplayName:  firstname$n aaalastname$n
eTADSuserPrincipalName:  aaatestuser$n@$ADSDOMAIN
eTADSsAMAccountName:  aaatestuser$n
eTPassword:  Password01
eTADSpwdLastSet:  -1
eTSuspended:  0
eTADSuserAccountControl:  0000000512
eTADSDescription:  description $tz
eTADSphysicalDeliveryOfficeName:  office
eTADStelephoneNumber:  111-222-3333
eTADSmail:  aaatestuser$n@$ADSDOMAIN
eTADSwwwHomePage:  web.page.lab
eTADSotherTelephone:  111-222-3333
eTADSurl:  other.web.page.lab
eTADSstreetAddress:  street address line01
eTADSpostOfficeBox:  pobox 111
eTADSl:  city
eTADSst:  state
eTADSpostalCode:  11111
eTADSco:  UNITED STATES
eTADSc:  US
eTADScountryCode:  840
eTADSscriptPath:  loginscript.cmd
eTADSprofilePath:  \profile\path\here
eTADShomePhone:  111-222-3333
eTADSpager:  111-222-3333
eTADSmobile:  111-222-3333
eTADSfacsimileTelephoneNumber:  111-222-3333
eTADSipPhone:  111-222-3333
eTADSinfo:  Notes Here
eTADSotherHomePhone:  111-222-3333
eTADSotherPager:  111-222-3333
eTADSotherMobile:  111-222-3333
eTADSotherFacsimileTelephoneNumber:  111-222-3333
eTADSotherIpPhone:  111-222-3333
eTADStitle:  title
eTADSdepartment:  department
eTADScompany:  company
eTADSmanager:  CN=manager_fn manager_ln,OU=$OU,$DCDOMAIN
eTADSmemberOf:  CN=Backup Operators,CN=Builtin,$DCDOMAIN
eTADSlyncSIPAddressOption: 0000000000
eTADSdisplayNamePrintable: aaatestuser$n
eTADSmailNickname: aaatestuser$n
eTADShomeMDB: (Automatic Mailbox Distribution)
eTADShomeMTA: CN=DC001,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=First Organization,CN=Microsoft Exchange,CN=Services,CN=Configuration,$DCDOMAIN
eTAccountStatus: A
eTADSmsExchRecipientTypeDetails: 0000000001
eTADSmDBUseDefaults: TRUE
eTADSinitials: A
eTADSaccountExpires: 9223372036854775807

EOF
 counter=$(( $counter + 00001 ))
done


#  Create the delete ADS Process
start=00001
counter=$start
while [ $counter -le $MAX ]
do
    n=$((10000+counter)); n=${n#1}
    tz=`/bin/date --utc +%Y%m%d%H%M%S,3%N.0Z`
   echo "Counter with leading zeros = $n   at time:  $tz"


cat << EOF >> $FILE1
dn:  eTADSAccountName=firstname$n aaalastname$n,eTADSOrgUnitName=$OU,eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=etasa
changetype: delete

EOF
 counter=$(( $counter + 00001 ))
done

echo ""
echo "################################### ADS USER OBJECT STATS ################################################################"
echo "Number of add objects: `grep "changetype: add" $FILE1 | wc -l`"
echo "Number of delete objects: `grep "changetype: delete" $FILE1 | wc -l`"
rm -rf $FILE2
cp -r -p $FILE1 $FILE2
sed -i 's|,dc=im,dc=etasa|,dc=im,dc=eta|g' $FILE2
ls -lart $FILE1
ls -lart $FILE2

echo ""
echo "################################### SET ADS MAX CONNECTIONS IN POOL SIZE ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
LDAPTLS_REQCERT=never dxmodify  -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D "$IMPS_USER" -w "$IMPS_PWD"  << EOF
dn: eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta
changetype: modify
eTADSMaxConnectionsInPool: 200
EOF
LDAPTLS_REQCERT=never dxsearch -LLL  -H ldap://$IMPS_HOST:$IMPS_PORT -x -D "$IMPS_USER" -w "$IMPS_PWD" -b "eTADSDirectoryName=$NAMESPACE,eTNamespaceName=ActiveDirectory,dc=im,dc=eta" -s base eTADSMaxConnectionsInPool | perl -p00e 's/\r?\n //g'

echo ""
echo "################################### CCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20402
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the CCS Service to test single thread with dxmodify or ldapmodify"
echo "dxmodify  -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the CCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "

echo ""
echo "################################### JCS UNIT & PERF TEST ################################################################"
CCS_HOST=192.168.242.80
CCS_PORT=20410
CCS_USER="cn=root,dc=etasa"
CCS_PWD="Password01"
echo "Execute this command to the JCS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify  -H ldap://$CCS_HOST:$CCS_PORT -c -x -D $CCS_USER -w $CCS_PWD -f $FILE1 "
echo "Execute this command to the JCS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $CCS_HOST:$CCS_PORT -D $CCS_USER -w $CCS_PWD -f $FILE1 "


echo ""
echo "################################### IMPS UNIT & PERF TEST ################################################################"
IMPS_HOST=192.168.242.135
IMPS_PORT=20389
IMPS_USER='eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=CommonObjects,dc=im,dc=eta'
IMPS_PWD="Password01"
echo "Execute this command to the IMPS Service to test single thread with dxmodify or ldapmodify "
echo "dxmodify  -H ldap://$IMPS_HOST:$IMPS_PORT -c -x -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "
echo "Execute this command to the IMPS Service to test 100 threads with dxsoak "
echo "./dxsoak -c -l 60 -t 100 -h $IMPS_HOST:$IMPS_PORT -D \"$IMPS_USER\" -w $IMPS_PWD -f $FILE2 "

Address the new bottleneck of MS Exchange / O365 Provisioning.

After parallel provisioning has been introduced with the new im_ccs.exe service, you may noticed that the number of transactions is still being throttled during performance testing.

Out-of-the-box MS Active Directory Global Throttling Policy has the parameter of PowerShellMaxConcurrency set to a default of 18 connection. Any provisioning that uses MS Powershell for MS Exchange and/or MS O365 will be impacted by this default parameter.

To address this bottleneck, we can create a new Throttling Policy and only assign the service ID that will be managing identities, to avoid a global change.

Example: New-ThrottlingPolicy MaxPowershell -PowerShellMaxConcurrency 100 & Set-Mailbox “User Name” -ThrottlingPolicy MaxPowershell

After this change has been made, restart the IM JCS/CCS Services, and retest again with your performance tools. Review the CCS ADS log for # of creations in 60 seconds, and you will be pleasantly surprise at the rate. The logs are the strong confirmation we are looking for.

Performance test (947 ADS accounts w/Exchange mailboxes in 60 seconds, 08:59:54 to 09:00:53) => Rate of 15 ids/second (or 54 K ids/hr) with updated MaxPowershell = 100 thottlingpolicy.

The last bottleneck appears to be CPU availability to MS Exchange Supporting Services, w3wp.exe, the MS IIS Service. Which appears to be managing MS Powershell connections per its startup string of

" c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangePowerShellAppPool" -v "v4.0" -c "C:\Program Files\Microsoft\Exchange Server\V15\bin\GenericAppPoolConfigWithGCServerEnabledFalse.config" -a \.\pipe\iisipme304c50e-6b42-4b26-83a4-229ee037be5d -h "C:\inetpub\temp\apppools\MSExchangePowerShellAppPool\MSExchangePowerShellAppPool.config" -w "" -m 0"