API Gateway and Docker Lab

While assisting a site with their upgrade process from CA API Gateway 9.2 (docker) to the latest CA API Gateway 9.4 image, we needed to clarify the steps. In this blog entry, we have capture our validation processes of the documented and undocumented features of API Gateway docker deployment ( https://hub.docker.com/r/caapim/gateway/ ), pedantic verbose steps to assist with training of staff resources; and enhanced the external checks for a DAR (disaster and recovery) scenario using docker & docker-compose tools.

Please use this lab to jump start your knowledge of the tools: ‘docker’, ‘docker-compose’ and the API Gateway. We have added many checks and the use of bash shell to view the contents of the API Gateway containers. If you have additional notes/tips, please leave a comment.

To lower business risk during this exercise, we made the follow decisions:

1) Avoid use of default naming conventions, to prevent accidental deletion of the supporting MySQL database for CA API Gateway. The default ‘docker-compose.yml’ was renamed as appropriate for each API Gateway version.

2) Instead of using different folders to host configuration files, we defined project names as part of the startup process for docker-compose.

3) Any docker container updates would reference the BASH shell directly instead of a soft-link, to avoid different behaviors between the API GW container and the MySQL container.

Challenges:

Challenge #1: Both the API Gateway 9.2 and 9.4 docker container have defects with regards to using the standardized ‘docker stop/start containerID‘ process. API Gateway 9.2 would not restart cleanly; and API Gateway 9.4 container would not update the embedded health check process, e.g. docker ps -a OR docker inspect containerID

Resolution #1: Both challenges were addressed in the enclosed testing scripts. Docker-compose is used exclusively for API Gateway 9.2 container, and touching an internal file in the API Gateway 9.4 container.

Challenge #2: The docker parameters between API Gateway 9.2 and API Gateway 9.4 had changed.

Resolution #2: Identify the missing parameters with ‘docker logs containerID’ and review of the embedded deployment script of ‘entrypoint.sh’

Infrastructure: Seven (7) files were used for this lab on CentOS 7.x (/opt/docker/api)

  1. ssg_license.xml (required from Broadcom/CA Sales Team – ask for 90 day trial if a current one is not available)
  2. docker-compose-ssg94.yml (the primary install configuration file for API GW 9.4)
  3. docker-compose-ssg92.yml (the primary install configuration file for API GW 9.2)
  4. docker-compose-ssg94-join-db.xml (the restart configuration file – use as needed)
  5. docker-compose-ssg92-join-db.xml (the restart configuration file – use as needed)
  6. 01_create_both_ssg92_and_ssg94_docker_deployments.sh (The installation of ‘docker’ and ‘docker-compose’ with the deployment of API GW 9.2 [with MySQL 5.5] and API GW 9.4 [with MySQL 5.7] ; with some additional updates)
  7. 02_backup_and_migrate_mysql_ssg_data_ from_ssg92_to_ssg94_db.sh (The export/import process from API GW 9.2 to API GW 9.4 and some additional checks)

Example of the seven (7) lab files’ contents:

  1. ssg_license.xml ( a view of the header only )
<?xml version="1.0" encoding="UTF-8"?&gt;
<license Id="5774266080443298199" xmlns="http://l7tech.com/license"&gt;
    <description&gt;LIC-PRODUCTION</description&gt;
    <licenseAttributes/&gt;
    <valid&gt;2018-12-10T19:32:31.000Z</valid&gt;
    <expires&gt;2019-12-11T19:32:31.000Z</expires&gt;
    <host name=""/&gt;
    <ip address=""/&gt;
    <product name="Layer 7 SecureSpan Suite"&gt;
        <version major="9" minor=""/&gt;
        <featureset name="set:Profile:EnterpriseGateway"/&gt;
    </product&gt;

2. docker-compose-ssg94.yml

version: "2.2"
services:
    ssg94:
      container_name: ssg94
      image: caapim/gateway:latest
      mem_limit: 4g
      volumes:
         - /opt/docker/api/ssg_license.xml:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/license/license.xml
      expose:
      - "8777"
      - "2142"
      ports:
        - "8443:8443"
        - "9443:9443"
      environment:
        ACCEPT_LICENSE: "true"
        SSG_CLUSTER_COMMAND: "create"
        SSG_CLUSTER_HOST: "localhost"
        SSG_CLUSTER_PASSWORD: "7layer"
        SSG_DATABASE_TYPE: "mysql"
        SSG_DATABASE_HOST: "mysql57"
        SSG_DATABASE_PORT: "3306"
        SSG_DATABASE_NAME: "ssg"
        SSG_DATABASE_USER: "gateway"
        SSG_DATABASE_PASSWORD: "7layer"
        SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql57:3306/ssg?useSSL=false"
        SSG_DATABASE_WAIT_TIMEOUT: "120"
        SSG_DATABASE_ADMIN_USER: "root"
        SSG_DATABASE_ADMIN_PASS: "7layer"
        SSG_ADMIN_USERNAME: "pmadmin"
        SSG_ADMIN_PASSWORD: "7layer"
        SSG_INTERNAL_SERVICES: "restman wsman"
        EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=false -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false  -Dcom.l7tech.service.metrics.enabled=false -Dcom.l7tech.server.disableFileLogsinks=false "
      links:
        - mysql57
    mysql57:
      container_name: ssg94_mysql57
      image: mysql:5.7
      restart: always
      mem_limit: 2g
      ports:
       - "3306:3306"
      environment:
         - MYSQL_ROOT_PASSWORD=7layer
         - MYSQL_USER=gateway
         - MYSQL_PASSWORD=7layer
         - MYSQL_DATABASE=ssg

3. docker-compose-ssg92.yml

version: "2.2"
services:
    ssg92:
      container_name: ssg92
      image: caapim/gateway:9.2.00-9087_CR10
      mem_limit: 4g
      expose:
      - "8778"
      - "2143"
      ports:
        - "8444:8443"
        - "9444:9443"
      environment:
        SKIP_CONFIG_SERVER_CHECK: "true"
        ACCEPT_LICENSE: "true"
        SSG_CLUSTER_COMMAND: "create"
        SSG_CLUSTER_HOST: "localhost"
        SSG_CLUSTER_PASSWORD: "7layer"
        SSG_DATABASE_TYPE: "mysql"
        SSG_DATABASE_HOST: "mysql55"
        SSG_DATABASE_PORT: "3306"
        SSG_DATABASE_NAME: "ssg"
        SSG_DATABASE_USER: "root"
        SSG_DATABASE_PASSWORD: "7layer"
        SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql55:3306/ssg?useSSL=false"
        SSG_DATABASE_WAIT_TIMEOUT: "120"
        SSG_DATABASE_ADMIN_USER: "root"
        SSG_DATABASE_ADMIN_PASS: "7layer"
        SSG_ADMIN_USERNAME: "pmadmin"
        SSG_ADMIN_PASSWORD: "7layer"
        SSG_ADMIN_USER: "pmadmin"
        SSG_ADMIN_PASS: "7layer"
        SSG_INTERNAL_SERVICES: "restman wsman"
        EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=true -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false  -Dcom.l7tech.service.metrics.enabled=false "
        SSG_LICENSE: "$SSG_LICENSE_ENV"
      links:
        - mysql55
    mysql55:
      container_name: ssg92_mysql55
      image: mysql:5.5
      restart: always
      mem_limit: 2g
      ports:
      - "3307:3306"
      environment:
        - MYSQL_ROOT_PASSWORD=7layer

4. docker-compose-ssg94-join-db.yml

version: "2.2"
services:
    ssg94:
      container_name: ssg94
      image: caapim/gateway:latest
      mem_limit: 4g
      volumes:
         - /opt/docker/api/ssg_license.xml:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/license/license.xml
      expose:
      - "8777"
      - "2142"
      ports:
        - "8443:8443"
        - "9443:9443"
      environment:
        ACCEPT_LICENSE: "true"
        #SSG_CLUSTER_COMMAND: "create"
        SSG_CLUSTER_COMMAND: "join"
        SSG_CLUSTER_HOST: "localhost"
        SSG_CLUSTER_PASSWORD: "7layer"
        SSG_DATABASE_TYPE: "mysql"
        SSG_DATABASE_HOST: "mysql57"
        SSG_DATABASE_PORT: "3306"
        SSG_DATABASE_NAME: "ssg"
        SSG_DATABASE_USER: "gateway"
        SSG_DATABASE_PASSWORD: "7layer"
        SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql57:3306/ssg?useSSL=false"
        SSG_DATABASE_WAIT_TIMEOUT: "120"
        SSG_DATABASE_ADMIN_USER: "root"
        SSG_DATABASE_ADMIN_PASS: "7layer"
        SSG_ADMIN_USERNAME: "pmadmin"
        SSG_ADMIN_PASSWORD: "7layer"
        SSG_INTERNAL_SERVICES: "restman wsman"
        EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=false -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false  -Dcom.l7tech.service.metrics.enabled=false -Dcom.l7tech.server.disableFileLogsinks=false "
      links:
        - mysql57
    mysql57:
      container_name: ssg94_mysql57
      image: mysql:5.7
      restart: always
      mem_limit: 2g
      ports:
       - "3306:3306"
      environment:
         - MYSQL_ROOT_PASSWORD=7layer
         - MYSQL_USER=gateway
         - MYSQL_PASSWORD=7layer
         - MYSQL_DATABASE=ssg

5. docker-compose-ssg92-join-db.yml

version: "2.2"
services:
    ssg92:
      container_name: ssg92
      image: caapim/gateway:9.2.00-9087_CR10
      mem_limit: 4g
      expose:
      - "8778"
      - "2143"
      ports:
        - "8444:8443"
        - "9444:9443"
      environment:
        SKIP_CONFIG_SERVER_CHECK: "true"
        ACCEPT_LICENSE: "true"
        SSG_CLUSTER_COMMAND: "join"
        SSG_CLUSTER_HOST: "localhost"
        SSG_CLUSTER_PASSWORD: "7layer"
        SSG_DATABASE_TYPE: "mysql"
        SSG_DATABASE_HOST: "mysql55"
        SSG_DATABASE_PORT: "3306"
        SSG_DATABASE_NAME: "ssg"
        SSG_DATABASE_USER: "root"
        SSG_DATABASE_PASSWORD: "7layer"
        SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql55:3306/ssg?useSSL=false"
        SSG_DATABASE_WAIT_TIMEOUT: "120"
        SSG_DATABASE_ADMIN_USER: "root"
        SSG_DATABASE_ADMIN_PASS: "7layer"
        SSG_ADMIN_USERNAME: "pmadmin"
        SSG_ADMIN_PASSWORD: "7layer"
        SSG_ADMIN_USER: "pmadmin"
        SSG_ADMIN_PASS: "7layer"
        SSG_INTERNAL_SERVICES: "restman wsman"
        EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=true -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false  -Dcom.l7tech.service.metrics.enabled=false "
        SSG_LICENSE: "$SSG_LICENSE_ENV"
      links:
        - mysql55
    mysql55:
      container_name: ssg92_mysql55
      image: mysql:5.5
      restart: always
      mem_limit: 2g
      ports:
      - "3307:3306"
      environment:
        - MYSQL_ROOT_PASSWORD=7layer

6. 01_create_both_ssg92_and_ssg94_docker_deployments.sh

#!/bin/bash
##################################################################
#
# Script to validate upgrade process from CA API GW 9.2 to 9.4 with docker
#  - Avoid using default of 'docker-compose.yml'
#  - Define different project names for API GW 9.2 and 9.4 to avoid conflict
#  - Explictly use bash shell  /bin/bash  instead of soft-link
#
# 1. Use docker with docker-compose to download & start
#      CA API GW 9.4 (with MySQL 5.7) &
#      CA API GW 9.2 (with MySQL 5.5)
#
# 2. Configure CA API GW 9.4 with TCP 8443/9443
#              CA API GW 9.2 with TCP 8444/9444 (redirect to 8443/9443)
#
# 3. Configure MySQL 5.7 to be externally exposed on TCP 3306
#              MySQL 5.5 to be externally exposed on TCP 3307
#  - Adjust 'grant' token on MySQL configuration file for root account
#
# 4. Validate authentication credentials to the above web services with curl
#
#
# 5. Add network modules via yum to API GW 9.4 container
#   - To assist with troubleshooting / debug exercises
#
# 6. Enable system to use API GW GUI to perform final validation
#   - Appears to be an issue to use browers to access the API GW UI TCP 8443/8444
#
#
# Alan Baugher, ANA, 10/19
#
##################################################################


echo ""
echo ""
echo "################################"
echo "Install docker and docker-compose via yum if missing"
echo "Watch for message:  Nothing to do "
echo ""
echo "yum -y install docker docker-compose "
yum -y install docker docker-compose
echo "################################"
echo ""


echo "################################"
echo "Shut down any prior docker container running for API GW 9.2 and 9.4"
cd /opt/docker/api
pwd
echo "Issue this command if script fails:  docker stop \$(docker ps -a -q)  && docker rm \$(docker ps -a -q)   "
echo "################################"
echo ""


echo "################################"
export SSG_LICENSE_ENV=$(cat ./ssg_license.xml | gzip | base64 --wrap=0)
echo "Execute  'docker-compose down'  to ensure no prior data or containers for API GW 9.4"
docker-compose -p ssg94 -f /opt/docker/api/docker-compose-ssg94.yml down
echo "################################"
echo "Execute  'docker-compose down'  to ensure no prior data or containers for API GW 9.2"
docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92.yml down
echo "################################"
echo ""


echo "################################"
echo "Execute  'docker ps -a'   to validate no running docker containers for API GW 9.2 nor 9.4"
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "################################"
echo ""


echo "################################"
echo "Change folder to execute docker-compose script for API GW 9.4 with MySql 5.7 with TCP 8443/9443"
echo "Execute  'docker-compose up -d'  to start docker containers for API GW 9.4 with MySql 5.7 with TCP 8443/9443"
docker-compose -p ssg94 -f /opt/docker/api/docker-compose-ssg94.yml up -d
echo "################################"
echo "Change folder to execute docker-compose script for API GW 9.2 with MySql 5.5 with TCP 8444/9444"
echo "Execute  'docker-compose up -d'  to start docker containers for API GW 9.2 with MySql 5.5 with TCP 8444/9444"
docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92.yml up -d
echo "################################"
echo ""


echo "################################"
echo "Backup current API GW 9.4 running container for future analysis"
echo "docker export ssg94 > ssg94.export.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar "
docker export ssg94 > ssg94.export.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar
echo "################################"
echo ""


echo "################################"
echo "Update API GW 9.4 running container with additional supporting tools with yum"
echo "docker exec -it -u root -e TERM=xterm ssg94 /bin/sh -c \"yum install -y -q net-tools iproute unzip vi --nogpgcheck\" "
docker exec -it -u root -e TERM=xterm ssg94 /bin/sh -c "yum install -y -q net-tools iproute unzip vi --nogpgcheck "
echo "Export API GW 9.4 running container after supporting tools are added"
echo "docker export ssg94 > ssg94.export.tools.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar "
docker export ssg94 > ssg94.export.tools.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar
echo "################################"
echo ""


echo "################################"
echo "Validate network ports are exposed for API GW Manager UI "
netstat -anpeW | grep -e docker -e "Local" | grep -e "tcp" -e "Local"
echo "################################"
echo ""

echo "################################"
echo "Sleep 70 seconds for both API GW to be ready"
echo "################################"
sleep 70
echo ""


echo ""
echo "################################"
echo "Extra:  Open TCP 3306 for mysql remote access "
docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /bin/bash -c "echo -e '\0041includedir /etc/mysql/conf.d/\n\0041includedir /etc/mysql/mysql.conf.d/\n[mysqld]\nskip-grant-tables' > /etc/mysql/mysql.cnf && cat /etc/mysql/mysql.cnf "
#docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /bin/bash -c "/etc/init.d/mysql restart"
#docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /bin/bash -c "/etc/init.d/mysql status && echo -n"
echo "################################"
docker restart ssg94_mysql57
echo ""



echo "################################"
echo "Execute  'docker ps -a'   to validate running docker containers for API GW 9.2 and 9.4 with their correct ports"
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "################################"
echo ""


echo "################################"
echo "Test authentication with the SSG backup URL for API 9.2 TCP 8444 - should see six (6) lines"
echo "curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8444/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "#########           ############"
curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8444/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""


echo "################################"
echo "Test authentication with the SSG backup URL for API 9.4 TCP 8443 - should see six (6) lines"
echo "curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8443/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "#########           ############"
curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8443/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""


echo "################################"
echo "Next Steps:"
echo "       Open the API GW UI for 9.2 and create a new entry in the lower left panel"
echo ""
echo "Example: "
echo "       Right click on hostname entry and select 'Publish RESTful Service Proxy with WADL' "
echo "       Select Manual Entry, then click Next"
echo "       Enter data for two (2) fields:"
echo "                  Service Name:  Alan "
echo "                  Resource Base URL:  http://www.anapartner.com/alan "
echo "       Then select Finish Button "
echo "################################"
echo ""

7. 02_backup_and_migrate_mysql_ssg_data_from_ssg92_to_ssg94_db.sh

#!/bin/bash
#######################################################################
#
# Script to validate upgrade process from CA API 9.2 to 9.4 with docker
#  - Avoid using default of 'docker-compose.yml'
#  - Define different project names for API GW 9.2 and 9.4 to avoid conflict
#  - Explictly use bash shell  /bin/bash  instead of soft-link /bin/sh
#
# 1. Stop docker containers for CA API GW 9.2 &amp; 9.4 (leave mysql containers running)
#    - To prevent any updates to mysql db during migration
#
# 2. Use mysqldump command to export CA API 9.2 MySQL 5.5 ssg database with stored procedures (aka routines)
#   - Review excluding the audit tables to avoid carrying over excessive data
#
# 3. Use mysql command to import sql file to CA API 9.4 MySQL 5.7 ssg database
#   - Review if dropping / recreate the ssg database will avoid any install issues
#   - Keep eye on table cluster_info {as this has the Gateway1 defination with the host IP address}
#
# 4. Restart CA API GW 9.2 &amp; 9.4 containers
#
#    - Challenge 1: CA API GW 9.2 docker image has issue with docker stop/start process
#    the reference /root/entrypoint.sh will loop with creation of a license folder
#    - Addressed with custom docker-compose file to recreate image to join existing MySQL 5.5 container
#
#    - Challenge 2: CA API GW 9.4 docker image has issue with docker stop/start process
#    The new heathcheck.sh process calls a base.sh script that compare the date-time stamp for two files
#    , the datestamp for one file is not updated correctly upon docker start process.
#    - Addressed with custom docker bash script to "touch" the primary file to allow date stamp to be updated.  Validate with: docker logs ssg94
#      WARNING 1      com.l7tech.server.boot.GatewayBoot: Unable to touch /opt/SecureSpan/Gateway/node/default/var/started:
#                  /opt/SecureSpan/Gateway/node/default/var/started (Permission denied)
#
#    - Challenge 3: CA API GW 9.4 docker image appears to have similar issue for hazelcast startup
#    The container may hold for 300 seconds due to hazelcast configuration not completing correctly
#     SEVERE  1      com.hazelcast.instance.Node: [172.17.0.3]:8777 [gateway] [3.10.2] Could not join cluster in 300000 ms. Shutting down now!
#     Unable to start the server: Error starting server : Lifecycle error: Could not initialize Hazelcast cluster
#     WARNING 107    com.hazelcast.cluster.impl.TcpIpJoiner: [172.17.0.3]:8777 [gateway] [3.10.2] Error during join check!
#    - Addessed with different project names to avoid conflict between API GW 9.2 broadcast to API GW 9.4
#
#    - Challenge 4: CA API GW 9.2 appears to be stuck in a while loop for /opt/docker/entrypoint.sh
#      apim-provisioning: INFO: waiting for the shutdown file at "/userdata/shutdown" to be created
#    - Addressed:  Does not seem to have impact with current testing.  Ignore.  Validate with:  docker logs ssg92
#
# 5. Important Note: Ensure that the SSG_CLUSTER_HOST and SSG_CLUSTER_PASSWORD values for CA API GW 9.4 docker-compose file
#    match those set in the configured MySQL database.
#    After CA API GW 9.4 container connects to the existing Gateway database, the Container Gateway will automatically upgrades
#    the ssg database if the ssg database version is lower than the version of the Container Gateway.
#    - Ensure the jdbc hostname
#
#    - Ref:  https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/9-4/other-gateway-form-factors/using-the-container-gateway/getting-started-with-the-container-gateway/connect-the-container-gateway-to-an-existing-mysql-database.html
#
#
# Alan Baugher, ANA, 10/19
#
#######################################################################

echo ""
echo "####################################"
echo "Early check: Address a file permission issue with the API GW 9.4 container"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c 'chmod 666 /opt/SecureSpan/Gateway/node/default/var/started' "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "chmod 666 /opt/SecureSpan/Gateway/node/default/var/started"
echo "May validate issue with:  docker logs ssg94 "
echo "####################################"


echo ""
echo "####################################"
echo "Temporarily shutdown the API GW containers for 9.2 and 9.4 to avoid any updates to the mysql db during export &amp; import"
echo "docker stop ssg92 ssg94 "
docker stop ssg92 ssg94
echo "####################################"
echo ""


echo "####################################"
echo "Validate API GW container are down and the MySQL db containers are up and working"
echo "Pause for 5 seconds:  sleep 5"
sleep 5
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "####################################"
echo ""


echo "####################################"
echo "Export the API GW 9.2 MySQL 5.5 ssg db with stored procedures (aka routines)"
echo "time docker exec -i `docker ps -a | grep mysql:5.5 | awk '{print $1}'` mysqldump -u root --password=7layer ssg  --routines &gt;  ssg92.backup_with_routines.sql  2&gt; /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.5 | awk '{print $1}'` mysqldump -u root --password=7layer ssg  --routines &gt;  ssg92.backup_with_routines.sql  2&gt; /dev/null
echo "View the size of the MySQL 5.5. ssg db for API GW 9.2"
ls -lart | grep ssg92.backup_with_routines.sql
echo "####################################"
echo ""


echo "####################################"
echo "Export the API GW 9.4 MySQL 5.7 ssg db with stored procedures (aka routines) as a 'before' reference file"
echo "time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines &gt;  ssg94.before.backup_with_routines.sql  2&gt; /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines &gt;  ssg94.before.backup_with_routines.sql  2&gt; /dev/null
echo "View the size of the MySQL 5.7. ssg db for API GW 9.4 as the 'before' reference file"
ls -lart | grep ssg94.before.backup_with_routines.sql
echo "####################################"
echo ""


echo "####################################"
echo "Import the MySQL 5.5 ssg db with stored procedures (aka routines) into MySQL 5.7 for API GW 9.4"
echo "time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysql -u root --password=7layer ssg    < ssg92.backup_with_routines.sql 2&gt; /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysql -u root --password=7layer ssg    < ssg92.backup_with_routines.sql 2&gt; /dev/null
echo "####################################"
echo ""


echo "####################################"
echo "Export the API GW 9.4 MySQL 5.7 ssg db wht stored procedures (aka routines) as a 'after' import reference file"
echo "time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines &gt;  ssg94.after.backup_with_routines.sql 2&gt; /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines &gt;  ssg94.after.backup_with_routines.sql 2&gt; /dev/null
echo "View the size of the MySQL 5.7. ssg db for API GW 9.4 as the 'after' reference file"
ls -lart | grep ssg94.after.backup_with_routines.sql
echo "####################################"
echo ""


echo "####################################"
echo "Restart the API GW containers for 9.2 and 9.4 "
# Note: Restart of the ssg94 container will 'auto' upgrade the ssg database to 9.4 tags
echo "docker restart ssg94 "
docker restart ssg94
#docker rm ssg94
#docker-compose -p ssg94 -f /opt/docker/api/docker-compose-ssg94-join-db.yml up -d
echo "####################################"
# Note:  API GW 9.2 docker image was not designed for stop/start correctly; rm then redeploy
export SSG_LICENSE_ENV=$(cat ssg_license.xml | gzip | base64 --wrap=0)
echo "Remove the API GW 9.2 container via:  docker rm ssg92"
docker rm ssg92
echo "Redeploy the API GW 9.2 container "
echo "docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92-join-db.yml up -d "
docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92-join-db.yml up -d
echo "####################################"
echo ""



echo "####################################"
echo "Validate API GW container are up and the mysql db containers are working"
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "####################################"
echo ""


echo "####################################"
echo "Export the API GW 9.4 MySQL 5.7 ssg db after import &amp; after the 'auto' upgrade as an 'after' auto upgrade reference file"
docker stop ssg94
echo "time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines &gt;  ssg94.auto.after.backup_with_routines.sql 2&gt; /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines &gt;  ssg94.auto.after.backup_with_routines.sql 2&gt; /dev/null
echo "View all the exported MySQL files to compare process flow"
ls -lart ssg*.sql
docker start ssg94
echo "View the auto upgrade from version 9.2 to version 9.4 with a delta compare of the exported sql files"
echo "diff ssg94.after.backup_with_routines.sql  ssg94.before.backup_with_routines.sql  | grep -i \"INSERT INTO .ssg_version.\" "
diff ssg94.after.backup_with_routines.sql  ssg94.before.backup_with_routines.sql  | grep -i "INSERT INTO .ssg_version."
echo "####################################"
echo ""


echo "####################################"
echo "Execute  'docker ps -a'   to validate running docker containers for API GW 9.4 and 9.2"
echo "docker ps --format \"table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}\" "
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "####################################"
echo ""


echo "####################################"
echo "Show current API GW 9.4 MySQL 5.7 databases"
echo "Validate that 'ssg' database exists "
echo "docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  -e \"show databases;\" "
docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  -e "show databases;"
echo "####################################"
echo ""


echo "####################################"
echo "Review for any delta of the MySQL ssg database after import"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.5 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e \"show tables;\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.5 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e "show tables;" &gt; ssg92.tables.txt
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e \"show tables;\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e "show tables;" &gt; ssg94.tables.txt
echo "Observer for any delta from the below command"
echo "diff ssg92.tables.txt ssg94.tables.txt"
diff ssg92.tables.txt ssg94.tables.txt
echo "####################################"
echo ""


echo "####################################"
echo "Show current API GW 9.4 admin user in the MySQL 5.7 ssg database"
echo "docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer ssg -e \"SELECT name,login,password,enabled,expiration,password_expiry FROM internal_user;\" "
docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer ssg -e "SELECT name,login,password,enabled,expiration,password_expiry FROM internal_user;"
echo "####################################"
echo ""


echo "####################################"
echo "Show current API GW 9.4 admin user in the intermediate configuration file on the AIP GW 9.4 container"
echo "docker exec -it -u root -e TERM=xterm ssg94 /bin/bash -c \"grep -i -e l7.login -e l7.password /opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle/001_update_admin_user.xml.req.bundle\" "
docker exec -it -u root -e TERM=xterm ssg94 /bin/bash -c "grep -i -e l7.login -e l7.password /opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle/001_update_admin_user.xml.req.bundle"
echo "####################################"
echo ""


echo "####################################"
echo "Show all 'new' files created or linked in API GW 9.4 container with mtime of 1 day. Excluding lock (LCK) files"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c \"find /opt -type f -mtime -1 -ls | grep -i -v -e '.LCK'\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "find /opt -type f -mtime -1 -ls | grep -i -v -e '.LCK'"
echo "####################################"
echo ""


echo "####################################"
echo " View the license.xml file that was copied to the API GW 9.4 container bootstrap folder before copied to the MySQL 5.7 ssg db table "
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c \"ls -lart  /opt/SecureSpan/Gateway/node/default/etc/bootstrap/license \" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "ls -lart  /opt/SecureSpan/Gateway/node/default/etc/bootstrap/license "
echo "####################################"
echo ""


echo "####################################"
echo "View logon count for the API GW 9.4 admin user via MySQL 5.7 ssg db"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=gateway --password=7layer ssg -e \"select hex(goid), version, hex(provider_goid), login, fail_count, last_attempted, last_activity, state from logon_info;\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=gateway --password=7layer ssg -e "select hex(goid), version, hex(provider_goid), login, fail_count, last_attempted, last_activity, state from logon_info;"
echo "####################################"
echo ""


echo "####################################"
echo "View the API GW 9.4 MySQL 5.7 mysql.user table"
### docker logs `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  2&gt;&amp;1 | grep  "GENERATED ROOT PASSWORD"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e \"SELECT User,account_locked,password_expired,password_last_changed,authentication_string FROM mysql.user;\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e "SELECT User,account_locked,password_expired,password_last_changed,authentication_string FROM mysql.user;"
echo "####################################"
echo ""


echo "####################################"
echo "To remove any locked account (including pmadmin SSG Admin User ID) from the MySQL 5.7 ssg logon_info table  {or any account}"
echo "docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer ssg -e \"delete from logon_info where login ='pmadmin';\" "
echo "docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer ssg -e \"truncate logon_info;\"  "
echo "####################################"
echo ""


echo "####################################"
echo "To change root password for MySQL 5.7 mysql.user db"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=OLD_PASSWORD -e  \"SET PASSWORD FOR 'root'@'localhost' = PASSWORD('7layer');\" "
echo "####################################"
echo ""


echo "####################################"
echo "Sleep 30 seconds to address restart health check time-stamp issue with API GW 9.4"
sleep 30
echo "####################################"
echo ""


echo "####################################"
echo "Address API GW 9.4 container health check upon stop/start or restart gap.  (base.sh script)"
echo "docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c \"date +%s -r /opt/SecureSpan/Gateway/node/default/var/started  &amp;&amp; date +%s -r /opt/SecureSpan/Gateway/node/default/var/preboot\" "
docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "date +%s -r /opt/SecureSpan/Gateway/node/default/var/started  &amp;&amp; date +%s -r /opt/SecureSpan/Gateway/node/default/var/preboot"
echo "Touch to update date-time stamp for one file: /opt/SecureSpan/Gateway/node/default/var/started"
echo "docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c \"touch /opt/SecureSpan/Gateway/node/default/var/started\" "
docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "touch /opt/SecureSpan/Gateway/node/default/var/started"
docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "date +%s -r /opt/SecureSpan/Gateway/node/default/var/started  &amp;&amp; date +%s -r /opt/SecureSpan/Gateway/node/default/var/preboot"
echo "####################################"
echo ""


echo "####################################"
echo "Sleep 30 seconds to allow health check status to update for API GW 9.4"
echo "May also monitor health and overall status with:   docker inspect ssg94 "
sleep 30
echo "####################################"
echo ""


echo "####################################"
echo "Execute  'docker ps -a'   to validate running docker containers for API GW 9.4 and 9.2"
echo "docker ps --format \"table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}\" "
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "####################################"
echo ""


echo "################################"
echo "Test authentication with the SSG backup URL for API 9.2 TCP 8444 - should see minimal of six (6) lines"
echo "curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8444/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "#########           ############"
curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8444/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""


echo "################################"
echo "Test authentication with the SSG backup URL for API 9.4 TCP 8443 - should see minimal of six (6) lines"
echo "curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8443/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "#########           ############"
curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8443/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""

View of the API Gateway via the MS Windows API GW UI for both API GW 9.2 (using the 9.3 UI) and API 9.4 (using the 9.4 UI). The API GW Policies will be migrated from API 9.2 to API 9.4 via the export/import of MySQL ssg database. After import, the API GW 9.4 docker image will ‘auto’ upgrade the ssg database to the 9.4 version.

Interesting view of the API GW 9.4 MySQL database ‘ssg’ after import and a restart (that will ‘auto’ upgrade the ssg database version). Note multiple Gateway “nodes” that will appear after each ‘docker restart containerID’

Build an eight (8) node Wildfly cluster on a single server

The follow methodology was used to isolate performance challenges with the increase number of cluster nodes for a common database, the Jgroup/JTS/JMS communication, database pools values for each “instance” in the wildfly/JBOSS configuration file.

Note: The individual nodes name are generated with a port offset of 100-800 for each of the eight (8) nodes; any hard-coded values are updated as well (via addition or multiplication).

To ensure the hornetq and Jgroup names are correctly defined for the chain cluster, a case statement is used to ensure that each node’s standalone-full-ha.xml configuration file is updated accordingly, if # of nodes are changed (this is offered as a variable at the top of the script.)

The below example also shows how to leverage CA APM / Wily agent for each J2EE/Wildfly node.

#!/bin/bash
###############################################################################################
#
#  Goal:  Create a N node J2EE Cluster using Wildfly 8.x.x for CA Identity Manager on a single host
#         Use for sandbox testing and validation of performance I/O parameters
#
#  Notes:  Tested for 2-8 nodes and with the CA APM (Wily) agent enabled for each node
#
#
#  Author:  A. Baugher, ANA, 8/2019
#
#
###############################################################################################
#set -vx
tz=`/bin/date --utc +%Y%m%d%H%M%S.3%N.0Z`
MAX=5
counter=1
JBOSS_HOME=/opt/CA/wildfly-idm


echo "######  STEP 00:  Stop all prior work with cluster testing ######"  &gt; /dev/null 2&gt;&amp;1
kill -9 `ps -ef | grep java | grep -v grep | grep UseString | awk '{print $2}'`

echo "######  STEP 01:  Copy the current IME (Wildfly) folder to a new folder &amp; with new port offset ######"
echo "Create this many cluster nodes:  $MAX"
echo "Current TimeStamp:  $tz"
echo ""
while [ $counter -le $MAX ]
do
  c=$counter
  n=$((100+counter)); n=${n#1}
  o=$((100*counter))
  nettyo=$((5456+o))
  jgrpo=$((7600+o))
  cli=$((9990+o))

 echo "Current counter is: $counter and the jboss number is:  $n  with a port offset of: $o"
 echo ""
 if [ -d $JBOSS_HOME$n ]; then
   echo "Prior directory exists for $JBOSS_HOME$n"
   kill -9 `ps -ef | grep "wildfly-idm$n" | grep -v grep | awk '{print $2}'` &gt;   /dev/null 2&gt;&amp;1
   echo "Remove any running processes then sleep 5 seconds before removing directory: $JBOSS_HOME$n "
   sleep 5
   rm -rf /opt/CA/wildfly-idm$n
 fi

 cp -r -p /opt/CA/wildfly-idm /opt/CA/wildfly-idm$n
 cd $JBOSS_HOME$n/standalone
 echo "Current Folder is: `pwd`"
 ls -rt
 echo "Remove data tmp log folders for new node"
 rm -rf data tmp log
 ls -rt
 echo ""
 echo ""


 echo "Update standalone-full-ha.xml for hardcoded port 5456 with offset $o"
 cd $JBOSS_HOME$n/standalone/configuration
 echo "Current Folder is: `pwd`"
 cp -r -p ca-standalone-full-ha.xml ca-standalone-full-ha.xml.$tz
 sed -i "s|5456|$nettyo|g"  ca-standalone-full-ha.xml
 echo "Updated Jgroup netty connector port:  $nettyo"
 grep  $nettyo ca-standalone-full-ha.xml
 echo ""
 echo ""

 echo "Update standalone.conf (wildfly.conf) &amp; jboss-cli.xml for port offset by $o"
 cd $JBOSS_HOME$n/bin
 echo "Current Folder is: `pwd`"
 ls -lart standalone.conf
 ls -lart jboss-cli.xml
 cp -r -p ./init.d/wildfly.conf ./init.d/wildfly.conf.conf.$tz
 cp -r -p jboss-cli.xml jboss-cli.xml.$tz
 sed -i "s|/opt/CA/wildfly-idm|/opt/CA/wildfly-idm$n|g" ./init.d/wildfly.conf
 sed -i "s|9990|$cli|g" jboss-cli.xml
 unlink standalone.conf
 ln -s $JBOSS_HOME$n/bin/init.d/wildfly.conf standalone.conf
 echo "JAVA_OPTS=\"\$JAVA_OPTS -Djboss.socket.binding.port-offset=$o\""  &gt;&gt; standalone.conf
 ls -lart standalone.conf
 ls -lart jboss-cli.xml
 grep "port-offset" standalone.conf
 grep "$cli" jboss-cli.xml
 echo ""
 echo ""



 echo "Update standalone.sh for node name &amp; tcp group port"
cd $JBOSS_HOME$n/bin
pwd
cp -r -p standalone.sh   standalone.sh.$tz
ls -larth standalone.sh
sed -i "s|iamnode1|iamnode$n|g"  standalone.sh


case "$MAX" in

1)  echo "Creating JGroups for one node with port offset of $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ;;
2)  echo "Creating JGroups for two nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###################
    ;;
3)  echo "Creating JGroups for three nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###################
    ;;
4)  echo "Creating JGroups for four nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###########################
    ;;
5)  echo "Creating JGroups for five nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\],caim-srv-01\[8100\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node5_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node5_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 5 ]
        then
    sed -i '684s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node5_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node5_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###########################
    ;;
6)  echo "Creating JGroups for six nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\],caim-srv-01\[8100\],caim-srv-01\[8200\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node6_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node6_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 5 ]
        then
    sed -i '684s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 6 ]
        then
    sed -i '684s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node6_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node6_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    sed -i '682s|<journal-max-io&gt;5000</journal-max-io&gt;|<journal-max-io&gt;4000</journal-max-io&gt;|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '901s|<journal-max-io&gt;5000</journal-max-io&gt;|<journal-max-io&gt;4000</journal-max-io&gt;|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml

    ###########################
    ;;
7)  echo "Creating JGroups for seven nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\],caim-srv-01\[8100\],caim-srv-01\[8200\],caim-srv-01\[8300\]|g" $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node7_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node7_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 5 ]
        then
    sed -i '684s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 6 ]
        then
    sed -i '684s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node7|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 7 ]
        then
    sed -i '684s|node1|node7|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node7_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node7_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###########################
    sed -i '682s|<journal-max-io&gt;5000</journal-max-io&gt;|<journal-max-io&gt;3300</journal-max-io&gt;|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '901s|<journal-max-io&gt;5000</journal-max-io&gt;|<journal-max-io&gt;3300</journal-max-io&gt;|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    ;;
8)  echo "Creating JGroups for eight nodes with port offset of 100 - $o"
    sed -i "s|caim-srv-01\[7600\]|caim-srv-01\[7700\],caim-srv-01\[7800\],caim-srv-01\[7900\],caim-srv-01\[8000\],caim-srv-01\[8100\],caim-srv-01\[8200\],caim-srv-01\[8300\],caim-srv-01\[8400\]|g"  $JBOSS_HOME$n/bin/standalone.sh
    ###################
    if [ $counter -eq 1 ]
       then
    sed -i '684s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node8_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node8_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 2 ]
        then
    sed -i '684s|node1|node2|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node1_live_to_node2_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 3 ]
        then
    sed -i '684s|node1|node3|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node2_live_to_node3_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 4 ]
        then
    sed -i '684s|node1|node4|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node3_live_to_node4_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 5 ]
        then
    sed -i '684s|node1|node5|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node4_live_to_node5_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 6 ]
        then
    sed -i '684s|node1|node6|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node7|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node5_live_to_node6_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 7 ]
        then
    sed -i '684s|node1|node7|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node7_live_to_node8_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node7_live_to_node8_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node8|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node6_live_to_node7_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    elif [ $counter -eq 8 ]
        then
    sed -i '684s|node1|node8|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '738s|node1_live_to_node1_backup|node8_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '753s|node1_live_to_node1_backup|node8_live_to_node1_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '902s|node1|node1|'                                             $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '943s|node1_live_to_node1_backup|node7_live_to_node8_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '953s|node1_live_to_node1_backup|node7_live_to_node8_backup|'   $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    fi
    ###########################
    sed -i '682s|<journal-max-io&gt;5000</journal-max-io&gt;|<journal-max-io&gt;3000</journal-max-io&gt;|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    sed -i '901s|<journal-max-io&gt;5000</journal-max-io&gt;|<journal-max-io&gt;3000</journal-max-io&gt;|' $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
    ;;
esac

ls -lart $JBOSS_HOME$n/bin/standalone.sh
grep caim-srv $JBOSS_HOME$n/bin/standalone.sh
echo ""
echo "For Node: $n"
echo ""
grep node $JBOSS_HOME$n/standalone/configuration/ca-standalone-full-ha.xml
echo ""
echo ""
echo ""


echo ""
echo ""
echo "Update CA APM / Wily Information / Agent for this instance"
cp -r -p /opt/CA/VirtualAppliance/custom/apm/wily_im $JBOSS_HOME$n/standalone/wily_im
chown -R wildfly:wildfly $JBOSS_HOME$n/standalone/wily_im
echo "JAVA_OPTS=\"\$JAVA_OPTS -Dcom.wily.introscope.agent.jmx.enable=true -Dcom.wily.introscope.agent.agentManager.url.1=localhost:5001 -Djboss.modules.system.pkgs=com.wily,com.wily.*,org.jboss.byteman,org.jboss.logmanager -Xbootclasspath/p:$JBOSS_HOME$n/modules/system/layers/base/org/jboss/logging/main/jboss-logging-3.1.4.GA.jar:$JBOSS_HOME$n/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.0.Final.jar:$JBOSS_HOME$n/modules/system/layers/base/org/jboss/as/logging/main/wildfly-logging-8.2.0.Final.jar:$JBOSS_HOME$n/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.2.Final.jar\""  &gt;&gt; standalone.conf
echo "JAVA_OPTS=\"\$JAVA_OPTS -Dcom.wily.introscope.agent.agentName=iamnode$n  -Dcom.wily.introscope.agentProfile=$JBOSS_HOME$n/standalone/wily_im/core/config/IntroscopeAgent.profile -javaagent:$JBOSS_HOME$n/standalone/wily_im/Agent.jar    \""  &gt;&gt; standalone.conf
echo ""
echo ""

 counter=$(( $counter + 00001 ))
done






counter=1
while [ $counter -le $MAX ]
do
  echo "Reset ownership permissions for $JBOSS_HOME$n to wildfly userID"
  chown -R wildfly:wildfly $JBOSS_HOME$n
  echo "Start up node: $n of $MAX Wildfly cluster"
  n=$((100+counter)); n=${n#1}


  if [ "$(whoami)" != "wildfly" ]; then
       echo "Run this process under the wildfly userid to avoid permissions issue with root"
       su - wildfly -c "$JBOSS_HOME$n/bin/standalone.sh &amp;"
       chown -R wildfly:wildfly $JBOSS_HOME$n
  else
  $JBOSS_HOME$n/bin/standalone.sh &amp;
  fi

  counter=$(( $counter + 00001 ))
done


Reduce log duplication: Avoid nohup.out

If you plan on starting your J2EE services manually, and wish to keep them running after you log out, a common method is to use nohup ./command.sh &.

The challenge with the above process, is it will create its own output file nohup.out in the folder that the command was executed in.

Additionally, this nohup.out would be a 2nd I/O operation that would recreate the server.log file for the J2EE service.

To avoid this 2nd I/O operation, review leveraging a redirection of the nohup to /dev/null or determine if this J2EE service can be enabled as a RC/init.d or systemd service.

Example to update the wildfly .profile to allow an “alias” using a bash shell function, to start up the wildfly service; and avoid the creation of the nohup.out file.

echo "Enable alias (or function)  to start and stop wildfly"

#Example of function - Use this to avoid double I/O for nohup process (nohup.out file)
function start_im01 () {
     echo "Starting IM 01 node with nohup process"
     cd /opt/CA/wildfly-idm01/bin/
     pwd
     nohup ./standalone.sh  >/dev/null 2>&1 &
     sleep 1
     /bin/ps -ef | grep wildfly-idm01 | grep -v grep
}
export -f start_im01

function stop_im01 () {
     echo "Stopping IM 01 node"
     echo "This may take 30-120 seconds"
     cd /opt/CA/wildfly-idm01/bin/
     pwd
     ./jboss-cli.sh --connect  --command=":shutdown"
     sleep 5
     /bin/kill -9 `/bin/ps -ef | grep wildfly-idm01 | grep -v grep | awk '{print $2}'` >/dev/null 2>&1
}
export -f stop_im01

You may now start and stop your J2EE Wildfly service with the new “aliases” of start_im01 and stop_im01

You may note that stop_im01 attempts to cleanly stop the Wildfly service via the JBOSS/Wildfly management console port ; and if that fails, we will search and kill the associated java service. If you did “kill” a service, and have startup issues suggest removing the $JBOSS_HOME/standalone/tmp & /data folders before restart.

DevOps – Lower project risk

We all wish to be out of the “install” business. How many times has your projects require installing or re-installing OS, solutions, and databases, & then perform post-install configurations, where it seems the same challenges are repeated.

Fortunately, with the SaaS services, virtualization of OS platforms, and solution appliances, the effort/time to install a component is on its way to being a small line item within a project plan.

However, in the enterprise software world, there are still components that have not been fully deployed in one of the above models, or the business/technical requirements do not fit the above models.

The value of DevOps to an enterprise project, is in two (2) areas:

  1. Automating deployment & re-deployment of solution components
    • First effort is always a wash for initial deployments, but value gained on the next environment(s)
    • Utilizing automated deployments ensure quality between peer members of components, e.g. install exactly the same way
    • Avoids the “too-many-cooks-in-the-kitchen” challenge when more than two (2) resource deploy in differ ways.
    • Allows rapid scaling of the solution with horizontally cluster integration.
  2. Automating deployment of business content from a dev to test to qa to pre-prod to eventually the production environment.
    • Hands on keyboard once
    • Workflow Approval/Reject of changes
    • Avoid downtime for business release cycles.
    • Allow for rollback of business content.
    • Lower project cost due to large maintenance windows.

DevOps does not have to be overwhelming. While customers’ may invest in enterprise DevOps solutions or open-source solution, like Chef, the proof-of-concept process that many follow first is using the embedded command line offerings in solutions to install solution components using the “silent” or “non-interactive” switches. For DevOPS with business content migration, the use of APIs via SOAP/REST or again CLI (command line interfaces) are used with script languages that customer staffs are familiar with.

Example of a command line to install Oracle Java JDK, that was then migrated to an enterprise DevOps solution:

A view of the architecture methodology we use for DEV-OPS of any vendor solution, e.g. CA Technologies, Oracle, RedHat, Microsoft, etc.

DevOps Architecture Methodology:  Bottom-Up Approach 

  • To meet the expected use of the CLI processes for migration into a commercial DevOps solution, the approach will attempt to emulate the same behavior.
  • Server(s) Acquisition –The servers are assumed to be compatible with the solution’s support matrices and to have the supported OS.
    • Most solution(s) are able to adequate run on 2-4 vCPU with 8-32 GB RAM.
      • Disk space may be from 10-100 GB.
    • A sandbox environment should attempt to run all solutions within a 2 vCPU with 8 GB RAM on 80 GB disk (20 GB for OS and 60 GB for data)
      • OS Mount Point
        • The majority of the solutions will deploy under /opt/<subfolder> on a Linux/UNIX OS.
        • This mount point will be chosen for all vendors and 3rd party solutions.
      • OS Libraries
        • OS libraries that are identified from literature, the installers, debug steps will be pre-loaded as a separate script.
      • OS Entropy
        • OS Entropy will be requested to be deployed prior to any installation via the OS RGND service or 3rd party RGND solutions. {Do NOT miss this step}

  • 3rd Party & vendor solution Installations
    • Any software declared as a predecessor solution, will be installed in the correct order to avoid re-work effort.
    • Any software solution that allow peer and/or cluster setup; will be deployed as a cluster member of 1; to allow future “stacks” to be integrated rapidly; and to allow a solution to scale horizontally.
    • Any co-location of software components, on the same server, will be isolated by folder, network ports, and JBOSS/Wildfly/J2EE instances.
    • Any software installation that require pre-installation steps will be identified and a process will be built via CLI to manage the general use-case of deployment and integration.
    • Any software installation that require input for service accounts; passwords; or other variables will be address with an input properties file and/or script variables that will be defined at the header of the script.
    • Any software installation that requires non-root account to install, will be identified and/or updated to use sudo access to execute as the non-root account.
    • Any software wizard installation that is not clear on “changes” or delta to the install base, will be execute several times to capture the deltas.
      • Process to capture install deltas (file based)
        • Install 1st time with interactive console mode. §tar/zip up the install folder
        • Install 2nd time with interactive console mode
          • Tar/zip up the install folder
        • Copy both files to a workstation/laptop and use a file/folder compare tool (WinMerge/Beyond Compare) to identify the file /folder deltas
    • Update software installation silent install scripts to use variables to manage the deltas.

Finally, we arrive at the DevOps Architecture Methodology:  Installation Processes

  • All solutions will be installed with CLI processes that may be executed as root or a non-root service account, as needed.
  • Interaction with Web Server/Web Application Servers will be managed with the CLI process of CuRL/MS Powershell/PDI/etc..
  • Bottom-Up approach will be used, where assumption of pre-installed components will be declared.
  • Installations scripts will follow the methodology:
    • Declare properties file and/or variables to be used.
    • Uninstall any prior installation
      • Execute shutdown script/process (if exists)
      • Execute OS kill command (search for running processes)
      • Execute uninstall script (provided by solution)
      • Remove installation folder
    • Install solution
      • Update silent install input file based on properties and/or script variables
      • Install solution via silent install input file
    • Perform base validation checks (query on files/folders/running processes)

Now we reach this section: DevOps Architecture Methodology:  Post Install / Integration

  • Realized & set/manage the expectation that there is a point-of-diminished returns for value of automating installations.
    • 1st Question:  Do we understand task A intent (requirements/business logic)?
    • 2nd Question:  Can task A be automated.  
    • 3rd Question:  Should task A be automated.
  • The post install and/or migration integration steps between components where the components reside on separate servers.
    • Otherwise the dev-ops process needs to be aware of the remote IP/hostnames of the cluster members.
      • May be pre-defined in  3rd Party & Vendor solution Installations
      • Note:  If “dummy hostnames” are to be used, ensure they are mapped to the current hostname to have them resolvable by DNS lookup.
  • Business logic unique to each customer may be pre-defined by use-case
    • The assumption is that well-defined business logic unique to each solution has been pre-built for consumption.
  • Service Accounts’ Passwords that were hard-coded as part of the silent install scripts will need to be immediately changed to new secure passwords prior to any production roll-out.

The project effort for DevOps above can be considered a “wash” with the expected manual installation effort for the first environment (dev) for OS & a vendors solution. The value will be realized / gained for project timelines due to deployment of the next 2-5 environments, when resources change, re-deployment is required, when there is a need to horizontally scale the solution, or if a disaster recovery site is required. Project timeline risk will be minimized, and the knowledge gained by the resources that build the DevOps processes will lower business risk during future triages of technical challenges.

The second value of DevOps is around business release process, that we will cover in another blog entry, to promote business logic/content from one environment to the next environment, until finally deployment to the production environment.

Avoid Data Quality Issues during Testing (TDM)

Why do we see data quality challenge in lower environments (Test, Dev, QA) that we do not see in Production Environments?

If the project team was asked to set up lower environments for any new solution, it might be that the TDM (test-data-management) methodology is not a formal corporate process.

TDM may be simply described as capturing non-PII (sensitive) production data and coping a full or limited set of the data to the non-production environments. This non-PII data may be 1:1 or masked during this process.

A TDM (test-data-management) process for a new environment may be a challenge if there is no current production environment or that the current production environment is from a prior solution or M&A (merge/acquisitions).

While there are formal paid tools/solutions for TDM, a project team may wish to leverage CLI (command-line) and/or scripts to create this sub-set of non-PII production data for the lower environments.

This process may be as simple as deciding to export the full DIT (directory structure/directory information tree) of an LDAP store with all its current group names, but replace the userID/Full Name/sensitive data with “dummy/masked” data. This exported data would be loaded with the near-Production data, to allow for full use-case and negative use-case testing in the lower environments.

The Goal? Avoid show-stopper or high-level issues due to missed data quality concerns during a Go-Live or Business Release Cycle. This is very important when we have a small maintenance window to add new functionality.

Let us help with the knowledge transfer and building of representatives environments. We see this challenge often for the IAM solutions that manage 1000’s of endpoints, where even the basic Active Directory representation is missing the same DIT structure and group objects as the project AD domains, especially for M&A business projects.

Writing Successful Test Plans

One of the challenges we see is that project team members dislike writing.

Documentation that is very visible business owners/team leads, e.g. business/technical requirements, design, or project management, will not be greatly impacted due to the maturity of the senior resources.

However, one area seems to suffer and does have an impact for project timelines & future go-live estimates. Documentation for test plans may be very simplistic or detailed.

Project suffer timeline challenges when test plans & tests scripts are too simplistic.

The business QA resources assigned to execute the test plan/test scripts can NOT be assumed to have the in-depth background/knowledge of the solution. If the initial conditions and final output are not clearly called out (or how to reset them), then we have seen project timeline is drawn out as they are pushed into a seemingly never-ending cycle of QA testing.

To ensure your project is successful, demand that the test scripts for the test plans are written out as if to be executed by your great-grandparents. This includes which hyperlinks to use, which browser to use, which initial conditions to reset, which tool to reset to initial conditions, which steps to follow, how to record the final answer, where to capture the results, screenshot to be captured where and how.

The above methodology ensures that we do not have a “black box” of a solution, e.g. something-goes-in and we-hope-that-something-good-comes-out.

With the above process, the QA team lead can then scale out their team as needed.

When expected input/output information is captured, automated testing can be introduced with enhanced reporting and validation. This becomes exponentially valuable for IAM solution that manages 100’s of endpoints from legacy [AS/400, HP-NONSTOP NSK, Mainframe (ACF2/TSS/RACF/TSO)] to SaaS Cloud solutions.

So don’t contemplate, spend the time and reap the values. Make your grandparents proud!

Transparency through Automated Testing

One of the challenges that businesses have for projects is an awareness of the true status of tasks.

Project Methodology continues to advance with concepts of Agile Project Management which work well for larger projects. One of the value statements from Agile is the question to project resources when they can complete a task. This question provides a view into the mindset of the resource’s skill set and confidence to meet the task goal. If the resource is a junior resource or has limited skill in the task, then the effort provided to the team will be high. With Agile methodology using this process, it becomes very easy for resources, while they frantically research, to inadvertently drain the project bucket of effort, e.g., a 4-hour task that turns into a week duration.

Another area that has great success with enforcing transparency is automated testing. Automated testing may be used for unit, integration, use-case, performance, and scale testing. However, for project transparency, to lower business risk and project cost overrun, we would state the value of automated testing is from use-case & regression testing.

After technical and business requirements are complete, ensure that a project scheduled or WBS (work-breakdown-structure) has a defined milestone to migrate ALL manual use-case testing to automation. The effort to convert from manual use-case testing to automate testing will be considered by a few to have little value. However, when the final parts of a project are to meet a go-live over a weekend or to add in new business release with adjusted business logic. What would you trust to reach your goals 100%?

Below are two (2) common scenarios:

  1. Solution Upgrade Go-Live over a weekend. You have to be allocated 48 hours to backup of solution data & all platforms, perform a data snapshot, migrate data, integrate with newer solution components (possible new agents), combine with production data, and validate all use-cases for all business logic. And allow time for roll-back if, during triage of issues, the business team determines that show-stopper issues will not be addressed in the period. If you fail, you may be allowed one more attempt on another of your weekends, with all 2-20 people.
  2. Solution Business Release Cycle – Over a weekend or business day. You have the option to deploy new business logic to your solution. You can lower business risk to deploy during a business day but will require additional use-case and regression testing. If you have no automation, you will leverage a QA team of 2-10 people to exercise the use-cases; and sometimes negative use-cases.

Math: Assume your solution has twenty (20) use-cases & sub-use-cases where each use-case may have twenty (20) test scripts. Assume that you have an excellent QA/business/technical resources that have adequate capture the initial conditions (that must be reset every time) for each test script & they are checking for data quality challenges as well. Assume each test script takes about ten (10) minutes to execute, where your QA team resource (not the same skill set) will follow exactly and record success/failure. Perhaps you have trained them to use QA tools to screen capture your failure messages, and assign a technical project team resource to address.

20 use-case x 20 scripts/use-case x 10 min/script = 4000 minutes for one QA resource. Well, we have 1440 minutes in a day, so 4000/1440 = 2.78 days or 66.7 hours. Assume we add ten (10) QA business resources, while we have lower the QA cycle from 66.7 hours to 6.7 hours; we will be required to “freeze” any additional updates during this QA cycle; and likely impact our maintenance window for remediation of “found” issues for either scenario above.

Be aware of the “smoke” testing follies. This type of testing still leaves issues “burning.”

Enforce transparency for project owners, project managers, and team members.

Ensure that the effort to build the automated testing is kept for future regression when the new business logic phase is implemented. Prove to yourselves that prior business logic will NOT be impacted.

Many tools can be leveraged for automation, e.g., Open Source Jmeter (used by many customers), Selenium, or paid tools (Broadcom/CA Technologies Blazemeter), SOAPUI

Let us help.

We firmly believe, encourage, and perform knowledge transfer to our customers to help them succeed, and ensure that the introduction of automated testing lowers TCO of any solution. We can train your staff very quickly to leverage Jmeter from their desktop/servers to automate any written testing plans for solutions. These JMeter process can then be shared with all project team members.