Upgrade CA API Gateway via docker “in-place”

CA API Gateway (ssg) is used to manage SaaS endpoints/applications for the CA/Symantec Identity Suite solution. One of the challenges of appliances and Docker containers is the underlying 3rd party libraries may get dated, and require updates.

Most vendors will not allow post-updates or direct updates to their containers libraries, as this has an impact on the support model. So we must rely on the support process and push vendors to release additional updates to stay ahead of any security concerns.


The CA API Gateway (ssg) when deployed on docker, has a streamlined process for updating in place, as long as you have backed-up the MySQL database when the docker images are updated.

We wanted to capture the process to upgrade from CA API Gateway 9.4 (ssg94) to Gateway 10.0 (ssg10). Fortunately, the MySQL 8.0 database has the same structure, tables, and routines as the MySQL 5.7 database for CA API Gateway 9.4.

The challenge we have is the documented process to upgrade is difficult to implement on the same host OS; and there was a lost opportunity to manage the license file from 9.4 to license 10.0 during the re-import of the MySQL database.


The below diagram, from the CA API Gateway 10.0 upgrade process can be adjusted to streamline the upgrade process.

expedited_scenario_1
Ref: https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-0/install-configure-upgrade/upgrade-the-gateway/upgrade-an-appliance-gateway/manual-expedited-appliance-upgrade.html

The above-documented process outlines dropping the MySQL database ssg completely, and then create a new clean db. Using this documented process, we can slightly adjust it, to avoid an unnecessary step to re-import the license file after the restart of the gateway container. We also wish to add additional validation steps to show what is changing.

Proposal for modifications:

  1. Create a clean CA API Gateway 10.0 (ssg10) Docker deployment on the same Host OS. May use docker-compose with REST service enabled and use different TCP listening ports to allow two (2) docker containers to run simultaneously during the testing cycle. After testing may keep the default TCP license ports of 8443 & 9443.
  2. Allow the CA API Gateway 10.0 container to start cleanly with MySQL 8.0 DB and with the correct license file for version 10. Then export the MySQL database table that contains the updated license table.
  3. Import the prior backup MySQL file to the new CA API Gateway deployment. Then before startup, import the ssg10 license mysql file as well. This will replace the ssg94 license information.
  4. Restart the CA API Gateway container, and monitor the logs for any errors and ensure the new license file is used
  5. If REST API was enabled (via the docker-compose file & touch a file name “restman”), then use CURL to validate all REST services are available, and list all prior API Gateway Policy Services are displayed.

A visual example of this process using the prior diagram.

Note: The official documentation uses sed to replace a string “NO_AUTO_CREATE_USER”; but the documentation shows two examples. One with a comma & one without. We have included the one with the comma, but we did not see this line in the MySQL sql export, so it was deemed low value, but still included in our process.

Example of upgrade process and validation of using REST

Note the two (2) running CA API Gateway container of 9.4 (with MySQL 5.7) and 10.0 (with MySQL 8.0) with different TCP listen services; and validation of REST services for ssg10

Below are the above steps called out with additional validations steps, and the use of the “time” command to monitor the export of the files.

# Pre-Step 1:  On Test System:  Prepare SSG10 docker compose yml file and correct license.xml & confirm startup.
time docker-compose -p ssg10 -f ./docker-compose-ssg10-0.yml up -d      {Wait 90-120 seconds}
docker ps -a
docker logs ssg10 -f --tail 100
docker exec -it mysql-ssg10   mysql --user=root --password=7layer -e "show databases;"


# Step 2:  On PROD HOST OS: Stop SSG94 and export the current MySQL 5.7 database with routines (aka stored procedures) & remove unwanted lines
docker stop ssg94
time docker exec -tt mysql-ssg  /usr/bin/mysqldump -h 127.0.0.1 -u root --password=7layer  ssg --routines > ssg94.backup.before.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.sql
time docker exec -tt mysql-ssg  /usr/bin/mysqldump -h 127.0.0.1 -u root --password=7layer  ssg --routines > ssg94.backup.updated.for.mysql8.sql
sed -i "s/NO_AUTO_CREATE_USER,//g"   			ssg94.backup.updated.for.mysql8.sql
sed -i "/Using a password on the command/d" 	ssg94.backup.updated.for.mysql8.sql


# Step 3: On PROD HOST OS: Deploy SSG10 with docker compose yml file & correct license xml file & export db table license_document
time docker-compose -p ssg10 -f ./docker-compose-ssg10-0.yml up -d      {Wait 90-120 seconds}  
docker ps -a
docker logs ssg10 -f --tail 100     
docker stop ssg10
time docker exec -tt  mysql-ssg10  /usr/bin/mysqldump -h 127.0.0.1 -u root --password=7layer  ssg --routines license_document  > ssg10.license.export.sql
sed -i "/Using a password on the command/d" 	ssg10.license.export.sql

# Step 4: On PROD HOST OS: Drop the SSG10 MySQL 8.0 ssg database and rebuilt with imports of SQL files.
time docker exec -it -u root -e term=xterm mysql-ssg10 /usr/bin/mysqladmin --user=root --password=7layer drop ssg
docker exec -it mysql-ssg10   mysql --user=root --password=7layer -e "show databases;"
time docker exec -it -u root -e term=xterm mysql-ssg10 /usr/bin/mysqladmin --user=root --password=7layer create ssg
docker exec -it mysql-ssg10   mysql --user=root --password=7layer -e "show databases;"
time docker exec -i  mysql-ssg10  /usr/bin/mysql -u root --password=7layer ssg    <  ssg94.backup.updated.for.mysql8.sql
time docker exec -i  mysql-ssg10  /usr/bin/mysql -u root --password=7layer ssg    <  ssg10.license.export.sql
docker exec -it mysql-ssg10  mysql --user=root --password=7layer ssg  -e "SELECT * FROM license_document;" | grep -A 12 -e "<license "

# Step 5: On PROD HOST OS:  Start SSG10 and validate no errors 
docker start ssg10       {Wait 90-120 seconds} 
docker ps -a

# Step 6:  Validate license    
docker logs ssg10 -f --tail 100  
docker logs ssg10 -f 2>&1  | grep -i license

# Step 7:  Validate REST services enabled and we can see all services
curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/rest.wadl
curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/services
# Example to validate ServiceNow REST service to CA APIGW
curl --insecure --user  admin:gwALPtteR5R1  --compressed --header "Accept: application/json" --header "url: https://dev101846.service-now.com"  "https://localhost:9443/ServiceNow/v1/Users?filter=userName+eq+%22ztestalan10340%22&attributes=userName"
# Example validate ServiceNow REST service via LB to CA APIGW
curl --insecure --user  admin:gwALPtteR5R1  --compressed --header "Accept: application/json" --header "url: https://dev101846.service-now.com"  "https://192.168.242.135/ServiceNow/v1/Users?filter=userName+eq+%22ztestalan10340%22&attributes=userName"
# Direct REST service to ServiceNow to validate development instance is available.
curl --user  admin:gwALPtteR5R1  --compressed --header "Accept: application/json"  'https://dev101846.service-now.com/api/now/table/sys_user?sysparm_query=user_name=testalan13095'

# Step 8:  Certs required for IM JCS Tier to avoid typical cert issues.
a. Ensure the CA API Gateway public root CA cert or self-signed cert is imported to each JCS keystore
b. If using a LoadBalancer, e.g. httpd, ensure this public root CA cert or self-signed cert is imported to each JCS keystore.


docker commands collected to assist with RCA efforts for Operation Teams

# Extra commands to assist RCA efforts or OPS teams
#
# Validate routing is enabled within the CAAPIGW (ssg) container
#   docker exec -it ssg  bash -c "curl -L www.google.com"
#   docker exec -it -u root -e term=xterm ssg /bin/bash -c "curl -vk --tlsv1.2  https://www.service-now.com"

# Interactive Session with mysql>  prompt
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "show databases;"
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "SELECT User,Password,authentication_string FROM mysql.user;"
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "SELECT name,login,password,enabled,expiration,password_expiry FROM internal_user;"


#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "truncate logon_info;"
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "delete from logon_info where login ='ssgadmin';"
# If MySQL root password is random, find via logs  (use redirect to switch from JSON to text to use grep)


#  docker logs mysql-ssg 2>&1 | grep -i "Generated root password"
#  docker logs mysql-ssg -f       {Used to tail the logs}
#  Limit the logs to see
#  docker logs ssg10 -f --tail 100

# Commands to install additional packages for vul scans (ps from procps) & update passwords (mkpasswd from whois)
#   docker exec -it -u root -e term=xterm   mysql-ssg   /bin/bash -c "apt-get update -y && apt-get upgrade -y && apt-get install -y procps && apt-get install -y whois"
#   docker exec -it  mysql-ssg ps aux

#  Update password process
# Generate SSHA512 Password (use one of the below methods)
#   docker exec -it -u root -e term=xterm   mysql-ssg   /bin/bash -c "mkpasswd -m sha-512 7layer"
#   python -c 'import crypt; print(crypt.crypt("7layer", crypt.mksalt(crypt.METHOD_SHA512)))'
#   perl -le 'print crypt "7layer", "\$6\$customSalt\$"'

# Update password via command line (escape any $ characters)
#  docker exec -it -u root -e term=xterm  mysql-ssg mysql  --user=root --password=7layer ssg -e "UPDATE internal_user SET password='\$6\$SzW/q9xVM9\$Ed/LjCDVpIYNTq94CsqO2stR0h4KniPOl/7iQDv1SEXNu9ftv//6hohlJxNeizmac/V9cEb6WmJfdHQCFwpoc0' WHERE name='pmadmin'; "

# View user and password hash in DB
#   docker exec -it -u root -e term=xterm  mysql-ssg mysql  --user=root --password=7layer ssg -e "select * from internal_user \G;"

# View if account is active
#   docker exec -it -u root -e term=xterm  mysql-ssg mysql  --user=root --password=7layer ssg -e "select * from logon_info \G;"

# Reset if account is NOT active
#   docker exec -it -u root -e term=xterm  mysql-ssg mysql  --user=root --password=7layer ssg -e "UPDATE logon_info set state='ACTIVE', fail_count=0 where login='pmadmin';"

# REST WEB SERVICES
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/doc/home.html
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/services
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/rest.wadl
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/gateway-management.xsd
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/folders
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/folders/template
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/folders?name=My%20Service
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/doc/restDoc.html
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/emailListeners?sort=host&order=desc
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/doc/authentication.html
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/passwords/template
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/policies/template
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/doc/migration.html
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/ssgconnectors?enabled=true
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/clusterProperties/template
#  curl --insecure --user pmadmin:7layer  https://localhost:9443/restman/1.0/rest.wadl

Change the pmadmin password at the docker command line

Process flows collect for the CA API Gateway docker deployment

Example of the docker-compose yml file for CA API Gateway with REST web services and license xml file.

We attempt to keep useful notes/hints included in the yml file to allow for future reference. The example below redirect ports to TCP 18443 and 19443 from the standard ports of 8443 and 9443 for the CA API Gateway; and MySQL from 3306 to 23306 for testing protocols in non-Production enviornments.

# docker-compose-ssg10-0-mysql8-0_with_rest_and_external_mysql_volume.yml
# Startup:  docker-compose -p ssg -f ./docker-compose-ssg10-0-mysql8.yml up  -d
# Stop:     docker-compose -p ssg -f ./docker-compose-ssg10-0-mysql8.yml down
#
#
# Ensure Host OS Network allows IPv4 forwarding:   sysctl -a | grep ipv4.ip_forward
# Validate docker network has access with curl:  curl -vk --tlsv1.2  https://www.service-now.com
# Note:  Do NOT use TABS in this file
# Monitor startup of containers with:  docker logs ssg10 -f --tail 100   AND   docker logs mysql-ssg10 -f  --tail 100
# https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-0/using-the-container-gateway/getting-started-with-the-container-gateway/run-the-container-gateway-on-docker-engine/sample-docker-compose-deployment-file.html
version: "2.2"
services:
   ssg10:
     container_name: ssg10
     # Ref: https://hub.docker.com/r/caapim/gateway/tags
     #image: caapim/gateway:latest
     image: caapim/gateway:10.0.00_20200428
     mem_limit: 10048m
     volumes:
        # Ensure ssg_license.xml is a valid SSG license file for 9.4 or 10.0
        - ./ssg_license_10.xml:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/license/license.xml
        # https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-0/apis-and-toolkits/rest-management-api.html
        # Touch the file restman to auto-start rest webservices
        # Validate REST API with curl
        # curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/rest.wadl
        # curl --insecure --user pmadmin:7layer  https://localhost:18443/restman/1.0/rest.wadl
        - ./restman:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/services/restman
     ports:
       - "58443:8443"
       - "59443:9443"
     environment:
        ACCEPT_LICENSE: "true"
        SSG_CLUSTER_COMMAND: "create"
        SSG_CLUSTER_HOST: "localhost"
        SSG_CLUSTER_PASSWORD: "7layer"
        SSG_DATABASE_TYPE: "mysql"
        SSG_DATABASE_HOST: "mysql-ssg"
        SSG_DATABASE_PORT: "3306"
        SSG_DATABASE_NAME: "ssg"
        SSG_DATABASE_USER: "gateway"
        SSG_DATABASE_PASSWORD: "7layer"
        SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql-ssg10:3306/ssg?useSSL=false"
        SSG_DATABASE_ADMIN_USER: "root"
        SSG_DATABASE_ADMIN_PASS: "7layer"
        SSG_ADMIN_USERNAME: "pmadmin"
        SSG_ADMIN_PASSWORD: "7layer"
        EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=false -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false"
        SSG_INTERNAL_SERVICES: "restman wsman"
     links:
        - mysql-ssg10
   mysql-ssg10:
     container_name: mysql-ssg10
     # Ref https://hub.docker.com/_/mysql?tab=tags
     image: mysql:8.0.20
     #image: mysql:latest
     # SSG 10.0 requires MySQL 8.x per documentation
     #https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-0/install-configure-upgrade/using-mysql-8_0-with-gateway-10.html
     mem_limit: 1048m
     restart: always
     ports:
        - "23306:3306"
     environment:
        - MYSQL_ROOT_PASSWORD=7layer
        #- MYSQL_RANDOM_ROOT_PASSWORD=yes
        - MYSQL_USER=gateway
        - MYSQL_PASSWORD=7layer
        - MYSQL_DATABASE=ssg
     command:
       - "--character-set-server=utf8mb3"
       - "--log-bin-trust-function-creators=1"
       - "--default-authentication-plugin=mysql_native_password"
       - "--innodb_log_buffer_size=32M"
       - "--innodb_log_file_size=80M"
       - "--max_allowed_packet=8M"
#     volumes:
#       - mysql_db8:/var/lib/mysql
# Persist SSG MySQL DB Data
# Validate after shutdown with:  docker volume ls  &  docker volume inspect ssg_mysql_db
# Note:  Important - Random Root Password will not work for persist MySQL - Password must be known for 1st time
#   volumes:
#     mysql_db8:
#
# Extra commands to assist RCA efforts or OPS teams
#
# Validate routing is enabled within the CAAPIGW (ssg) container
#   docker exec -it ssg  bash -c "curl -L www.google.com"
#   docker exec -it -u root -e term=xterm ssg /bin/bash -c "curl -vk --tlsv1.2  https://www.service-now.com"
# Interactive Session with mysql>  prompt
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "show databases;"
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "SELECT User,Password,authentication_string FROM mysql.user;"
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "SELECT name,login,password,enabled,expiration,password_expiry FROM internal_user;"
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "truncate logon_info;"
#  docker exec -it mysql-ssg   mysql --user=root --password=7layer -e "delete from logon_info where login ='ssgadmin';"
# If MySQL root password is random, find via logs  (use redirect to switch from JSON to text to use grep)
#  docker logs mysql-ssg 2>&1 | grep -i "Generated root password"
#  docker logs mysql-ssg -f       {Used to tail the logs}
#  Limit the logs to see
#  docker logs ssg10 -f --tail 100
# Commands to install additional packages for vul scans (ps from procps) & update passwords (mkpasswd from whois)
#   docker exec -it -u root -e term=xterm   mysql-ssg   /bin/bash -c "apt-get update -y && apt-get upgrade -y && apt-get install -y procps && apt-get install -y whois"
#   docker exec -it  mysql-ssg ps aux
#  Update password process
# Generate SSHA512 Password (use one of the below methods)
#   docker exec -it -u root -e term=xterm   mysql-ssg   /bin/bash -c "mkpasswd -m sha-512 7layer"
#   python -c 'import crypt; print(crypt.crypt("7layer", crypt.mksalt(crypt.METHOD_SHA512)))'
#   perl -le 'print crypt "7layer", "\$6\$customSalt\$"'
# Update password via command line (escape any $ characters)
#  docker exec -it -u root -e term=xterm  mysql-ssg mysql  --user=root --password=7layer ssg -e "UPDATE internal_user SET password='\$6\$SzW/q9xVM9\$Ed/LjCDVpIYNTq94CsqO2stR0h4KniPOl/7iQDv1SEXNu9ftv//6hohlJxNeizmac/V9cEb6WmJfdHQCFwpoc0' WHERE name='pmadmin'; "
# View user and password hash in DB
#   docker exec -it -u root -e term=xterm  mysql-ssg mysql  --user=root --password=7layer ssg -e "select * from internal_user \G;"
# View if account is active
#   docker exec -it -u root -e term=xterm  mysql-ssg mysql  --user=root --password=7layer ssg -e "select * from logon_info \G;"
# Reset if account is NOT active
#   docker exec -it -u root -e term=xterm  mysql-ssg mysql  --user=root --password=7layer ssg -e "UPDATE logon_info set state='ACTIVE', fail_count=0 where login='pmadmin';"
# REST WEB SERVICES
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/doc/home.html
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/services
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/rest.wadl
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/gateway-management.xsd
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/folders
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/folders/template
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/folders?name=My%20Service
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/doc/restDoc.html
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/emailListeners?sort=host&order=desc
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/doc/authentication.html
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/passwords/template
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/policies/template
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/doc/migration.html
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/ssgconnectors?enabled=true
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/clusterProperties/template
#  curl --insecure --user pmadmin:7layer  https://localhost:19443/restman/1.0/rest.wadl

API Gateway and Docker Lab

While assisting a site with their upgrade process from CA API Gateway 9.2 (docker) to the latest CA API Gateway 9.4 image, we needed to clarify the steps. In this blog entry, we have capture our validation processes of the documented and undocumented features of API Gateway docker deployment ( https://hub.docker.com/r/caapim/gateway/ ), pedantic verbose steps to assist with training of staff resources; and enhanced the external checks for a DAR (disaster and recovery) scenario using docker & docker-compose tools.

Please use this lab to jump start your knowledge of the tools: ‘docker’, ‘docker-compose’ and the API Gateway. We have added many checks and the use of bash shell to view the contents of the API Gateway containers. If you have additional notes/tips, please leave a comment.

To lower business risk during this exercise, we made the follow decisions:

1) Avoid use of default naming conventions, to prevent accidental deletion of the supporting MySQL database for CA API Gateway. The default ‘docker-compose.yml’ was renamed as appropriate for each API Gateway version.

2) Instead of using different folders to host configuration files, we defined project names as part of the startup process for docker-compose.

3) Any docker container updates would reference the BASH shell directly instead of a soft-link, to avoid different behaviors between the API GW container and the MySQL container.

Challenges:

Challenge #1: Both the API Gateway 9.2 and 9.4 docker container have defects with regards to using the standardized ‘docker stop/start containerID‘ process. API Gateway 9.2 would not restart cleanly; and API Gateway 9.4 container would not update the embedded health check process, e.g. docker ps -a OR docker inspect containerID

Resolution #1: Both challenges were addressed in the enclosed testing scripts. Docker-compose is used exclusively for API Gateway 9.2 container, and touching an internal file in the API Gateway 9.4 container.

Challenge #2: The docker parameters between API Gateway 9.2 and API Gateway 9.4 had changed.

Resolution #2: Identify the missing parameters with ‘docker logs containerID’ and review of the embedded deployment script of ‘entrypoint.sh’

Infrastructure: Seven (7) files were used for this lab on CentOS 7.x (/opt/docker/api)

  1. ssg_license.xml (required from Broadcom/CA Sales Team – ask for 90 day trial if a current one is not available)
  2. docker-compose-ssg94.yml (the primary install configuration file for API GW 9.4)
  3. docker-compose-ssg92.yml (the primary install configuration file for API GW 9.2)
  4. docker-compose-ssg94-join-db.xml (the restart configuration file – use as needed)
  5. docker-compose-ssg92-join-db.xml (the restart configuration file – use as needed)
  6. 01_create_both_ssg92_and_ssg94_docker_deployments.sh (The installation of ‘docker’ and ‘docker-compose’ with the deployment of API GW 9.2 [with MySQL 5.5] and API GW 9.4 [with MySQL 5.7] ; with some additional updates)
  7. 02_backup_and_migrate_mysql_ssg_data_ from_ssg92_to_ssg94_db.sh (The export/import process from API GW 9.2 to API GW 9.4 and some additional checks)

Example of the seven (7) lab files’ contents:

  1. ssg_license.xml ( a view of the header only )
<?xml version="1.0" encoding="UTF-8"?>
<license Id="5774266080443298199" xmlns="http://l7tech.com/license">
    <description>LIC-PRODUCTION</description>
    <licenseAttributes/>
    <valid>2018-12-10T19:32:31.000Z</valid>
    <expires>2019-12-11T19:32:31.000Z</expires>
    <host name=""/>
    <ip address=""/>
    <product name="Layer 7 SecureSpan Suite">
        <version major="9" minor=""/>
        <featureset name="set:Profile:EnterpriseGateway"/>
    </product>

2. docker-compose-ssg94.yml

version: "2.2"
services:
    ssg94:
      container_name: ssg94
      image: caapim/gateway:latest
      mem_limit: 4g
      volumes:
         - /opt/docker/api/ssg_license.xml:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/license/license.xml
      expose:
      - "8777"
      - "2142"
      ports:
        - "8443:8443"
        - "9443:9443"
      environment:
        ACCEPT_LICENSE: "true"
        SSG_CLUSTER_COMMAND: "create"
        SSG_CLUSTER_HOST: "localhost"
        SSG_CLUSTER_PASSWORD: "7layer"
        SSG_DATABASE_TYPE: "mysql"
        SSG_DATABASE_HOST: "mysql57"
        SSG_DATABASE_PORT: "3306"
        SSG_DATABASE_NAME: "ssg"
        SSG_DATABASE_USER: "gateway"
        SSG_DATABASE_PASSWORD: "7layer"
        SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql57:3306/ssg?useSSL=false"
        SSG_DATABASE_WAIT_TIMEOUT: "120"
        SSG_DATABASE_ADMIN_USER: "root"
        SSG_DATABASE_ADMIN_PASS: "7layer"
        SSG_ADMIN_USERNAME: "pmadmin"
        SSG_ADMIN_PASSWORD: "7layer"
        SSG_INTERNAL_SERVICES: "restman wsman"
        EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=false -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false  -Dcom.l7tech.service.metrics.enabled=false -Dcom.l7tech.server.disableFileLogsinks=false "
      links:
        - mysql57
    mysql57:
      container_name: ssg94_mysql57
      image: mysql:5.7
      restart: always
      mem_limit: 2g
      ports:
       - "3306:3306"
      environment:
         - MYSQL_ROOT_PASSWORD=7layer
         - MYSQL_USER=gateway
         - MYSQL_PASSWORD=7layer
         - MYSQL_DATABASE=ssg

3. docker-compose-ssg92.yml

version: "2.2"
services:
    ssg92:
      container_name: ssg92
      image: caapim/gateway:9.2.00-9087_CR10
      mem_limit: 4g
      expose:
      - "8778"
      - "2143"
      ports:
        - "8444:8443"
        - "9444:9443"
      environment:
        SKIP_CONFIG_SERVER_CHECK: "true"
        ACCEPT_LICENSE: "true"
        SSG_CLUSTER_COMMAND: "create"
        SSG_CLUSTER_HOST: "localhost"
        SSG_CLUSTER_PASSWORD: "7layer"
        SSG_DATABASE_TYPE: "mysql"
        SSG_DATABASE_HOST: "mysql55"
        SSG_DATABASE_PORT: "3306"
        SSG_DATABASE_NAME: "ssg"
        SSG_DATABASE_USER: "root"
        SSG_DATABASE_PASSWORD: "7layer"
        SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql55:3306/ssg?useSSL=false"
        SSG_DATABASE_WAIT_TIMEOUT: "120"
        SSG_DATABASE_ADMIN_USER: "root"
        SSG_DATABASE_ADMIN_PASS: "7layer"
        SSG_ADMIN_USERNAME: "pmadmin"
        SSG_ADMIN_PASSWORD: "7layer"
        SSG_ADMIN_USER: "pmadmin"
        SSG_ADMIN_PASS: "7layer"
        SSG_INTERNAL_SERVICES: "restman wsman"
        EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=true -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false  -Dcom.l7tech.service.metrics.enabled=false "
        SSG_LICENSE: "$SSG_LICENSE_ENV"
      links:
        - mysql55
    mysql55:
      container_name: ssg92_mysql55
      image: mysql:5.5
      restart: always
      mem_limit: 2g
      ports:
      - "3307:3306"
      environment:
        - MYSQL_ROOT_PASSWORD=7layer

4. docker-compose-ssg94-join-db.yml

version: "2.2"
services:
    ssg94:
      container_name: ssg94
      image: caapim/gateway:latest
      mem_limit: 4g
      volumes:
         - /opt/docker/api/ssg_license.xml:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/license/license.xml
      expose:
      - "8777"
      - "2142"
      ports:
        - "8443:8443"
        - "9443:9443"
      environment:
        ACCEPT_LICENSE: "true"
        #SSG_CLUSTER_COMMAND: "create"
        SSG_CLUSTER_COMMAND: "join"
        SSG_CLUSTER_HOST: "localhost"
        SSG_CLUSTER_PASSWORD: "7layer"
        SSG_DATABASE_TYPE: "mysql"
        SSG_DATABASE_HOST: "mysql57"
        SSG_DATABASE_PORT: "3306"
        SSG_DATABASE_NAME: "ssg"
        SSG_DATABASE_USER: "gateway"
        SSG_DATABASE_PASSWORD: "7layer"
        SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql57:3306/ssg?useSSL=false"
        SSG_DATABASE_WAIT_TIMEOUT: "120"
        SSG_DATABASE_ADMIN_USER: "root"
        SSG_DATABASE_ADMIN_PASS: "7layer"
        SSG_ADMIN_USERNAME: "pmadmin"
        SSG_ADMIN_PASSWORD: "7layer"
        SSG_INTERNAL_SERVICES: "restman wsman"
        EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=false -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false  -Dcom.l7tech.service.metrics.enabled=false -Dcom.l7tech.server.disableFileLogsinks=false "
      links:
        - mysql57
    mysql57:
      container_name: ssg94_mysql57
      image: mysql:5.7
      restart: always
      mem_limit: 2g
      ports:
       - "3306:3306"
      environment:
         - MYSQL_ROOT_PASSWORD=7layer
         - MYSQL_USER=gateway
         - MYSQL_PASSWORD=7layer
         - MYSQL_DATABASE=ssg

5. docker-compose-ssg92-join-db.yml

version: "2.2"
services:
    ssg92:
      container_name: ssg92
      image: caapim/gateway:9.2.00-9087_CR10
      mem_limit: 4g
      expose:
      - "8778"
      - "2143"
      ports:
        - "8444:8443"
        - "9444:9443"
      environment:
        SKIP_CONFIG_SERVER_CHECK: "true"
        ACCEPT_LICENSE: "true"
        SSG_CLUSTER_COMMAND: "join"
        SSG_CLUSTER_HOST: "localhost"
        SSG_CLUSTER_PASSWORD: "7layer"
        SSG_DATABASE_TYPE: "mysql"
        SSG_DATABASE_HOST: "mysql55"
        SSG_DATABASE_PORT: "3306"
        SSG_DATABASE_NAME: "ssg"
        SSG_DATABASE_USER: "root"
        SSG_DATABASE_PASSWORD: "7layer"
        SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql55:3306/ssg?useSSL=false"
        SSG_DATABASE_WAIT_TIMEOUT: "120"
        SSG_DATABASE_ADMIN_USER: "root"
        SSG_DATABASE_ADMIN_PASS: "7layer"
        SSG_ADMIN_USERNAME: "pmadmin"
        SSG_ADMIN_PASSWORD: "7layer"
        SSG_ADMIN_USER: "pmadmin"
        SSG_ADMIN_PASS: "7layer"
        SSG_INTERNAL_SERVICES: "restman wsman"
        EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=true -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false  -Dcom.l7tech.service.metrics.enabled=false "
        SSG_LICENSE: "$SSG_LICENSE_ENV"
      links:
        - mysql55
    mysql55:
      container_name: ssg92_mysql55
      image: mysql:5.5
      restart: always
      mem_limit: 2g
      ports:
      - "3307:3306"
      environment:
        - MYSQL_ROOT_PASSWORD=7layer

6. 01_create_both_ssg92_and_ssg94_docker_deployments.sh

#!/bin/bash
##################################################################
#
# Script to validate upgrade process from CA API GW 9.2 to 9.4 with docker
#  - Avoid using default of 'docker-compose.yml'
#  - Define different project names for API GW 9.2 and 9.4 to avoid conflict
#  - Explictly use bash shell  /bin/bash  instead of soft-link
#
# 1. Use docker with docker-compose to download & start
#      CA API GW 9.4 (with MySQL 5.7) &
#      CA API GW 9.2 (with MySQL 5.5)
#
# 2. Configure CA API GW 9.4 with TCP 8443/9443
#              CA API GW 9.2 with TCP 8444/9444 (redirect to 8443/9443)
#
# 3. Configure MySQL 5.7 to be externally exposed on TCP 3306
#              MySQL 5.5 to be externally exposed on TCP 3307
#  - Adjust 'grant' token on MySQL configuration file for root account
#
# 4. Validate authentication credentials to the above web services with curl
#
#
# 5. Add network modules via yum to API GW 9.4 container
#   - To assist with troubleshooting / debug exercises
#
# 6. Enable system to use API GW GUI to perform final validation
#   - Appears to be an issue to use browers to access the API GW UI TCP 8443/8444
#
#
# Alan Baugher, ANA, 10/19
#
##################################################################


echo ""
echo ""
echo "################################"
echo "Install docker and docker-compose via yum if missing"
echo "Watch for message:  Nothing to do "
echo ""
echo "yum -y install docker docker-compose "
yum -y install docker docker-compose
echo "################################"
echo ""


echo "################################"
echo "Shut down any prior docker container running for API GW 9.2 and 9.4"
cd /opt/docker/api
pwd
echo "Issue this command if script fails:  docker stop \$(docker ps -a -q)  && docker rm \$(docker ps -a -q)   "
echo "################################"
echo ""


echo "################################"
export SSG_LICENSE_ENV=$(cat ./ssg_license.xml | gzip | base64 --wrap=0)
echo "Execute  'docker-compose down'  to ensure no prior data or containers for API GW 9.4"
docker-compose -p ssg94 -f /opt/docker/api/docker-compose-ssg94.yml down
echo "################################"
echo "Execute  'docker-compose down'  to ensure no prior data or containers for API GW 9.2"
docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92.yml down
echo "################################"
echo ""


echo "################################"
echo "Execute  'docker ps -a'   to validate no running docker containers for API GW 9.2 nor 9.4"
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "################################"
echo ""


echo "################################"
echo "Change folder to execute docker-compose script for API GW 9.4 with MySql 5.7 with TCP 8443/9443"
echo "Execute  'docker-compose up -d'  to start docker containers for API GW 9.4 with MySql 5.7 with TCP 8443/9443"
docker-compose -p ssg94 -f /opt/docker/api/docker-compose-ssg94.yml up -d
echo "################################"
echo "Change folder to execute docker-compose script for API GW 9.2 with MySql 5.5 with TCP 8444/9444"
echo "Execute  'docker-compose up -d'  to start docker containers for API GW 9.2 with MySql 5.5 with TCP 8444/9444"
docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92.yml up -d
echo "################################"
echo ""


echo "################################"
echo "Backup current API GW 9.4 running container for future analysis"
echo "docker export ssg94 > ssg94.export.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar "
docker export ssg94 > ssg94.export.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar
echo "################################"
echo ""


echo "################################"
echo "Update API GW 9.4 running container with additional supporting tools with yum"
echo "docker exec -it -u root -e TERM=xterm ssg94 /bin/sh -c \"yum install -y -q net-tools iproute unzip vi --nogpgcheck\" "
docker exec -it -u root -e TERM=xterm ssg94 /bin/sh -c "yum install -y -q net-tools iproute unzip vi --nogpgcheck "
echo "Export API GW 9.4 running container after supporting tools are added"
echo "docker export ssg94 > ssg94.export.tools.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar "
docker export ssg94 > ssg94.export.tools.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar
echo "################################"
echo ""


echo "################################"
echo "Validate network ports are exposed for API GW Manager UI "
netstat -anpeW | grep -e docker -e "Local" | grep -e "tcp" -e "Local"
echo "################################"
echo ""

echo "################################"
echo "Sleep 70 seconds for both API GW to be ready"
echo "################################"
sleep 70
echo ""


echo ""
echo "################################"
echo "Extra:  Open TCP 3306 for mysql remote access "
docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /bin/bash -c "echo -e '\0041includedir /etc/mysql/conf.d/\n\0041includedir /etc/mysql/mysql.conf.d/\n[mysqld]\nskip-grant-tables' > /etc/mysql/mysql.cnf && cat /etc/mysql/mysql.cnf "
#docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /bin/bash -c "/etc/init.d/mysql restart"
#docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /bin/bash -c "/etc/init.d/mysql status && echo -n"
echo "################################"
docker restart ssg94_mysql57
echo ""



echo "################################"
echo "Execute  'docker ps -a'   to validate running docker containers for API GW 9.2 and 9.4 with their correct ports"
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "################################"
echo ""


echo "################################"
echo "Test authentication with the SSG backup URL for API 9.2 TCP 8444 - should see six (6) lines"
echo "curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8444/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "#########           ############"
curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8444/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""


echo "################################"
echo "Test authentication with the SSG backup URL for API 9.4 TCP 8443 - should see six (6) lines"
echo "curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8443/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "#########           ############"
curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8443/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""


echo "################################"
echo "Next Steps:"
echo "       Open the API GW UI for 9.2 and create a new entry in the lower left panel"
echo ""
echo "Example: "
echo "       Right click on hostname entry and select 'Publish RESTful Service Proxy with WADL' "
echo "       Select Manual Entry, then click Next"
echo "       Enter data for two (2) fields:"
echo "                  Service Name:  Alan "
echo "                  Resource Base URL:  http://www.anapartner.com/alan "
echo "       Then select Finish Button "
echo "################################"
echo ""

7. 02_backup_and_migrate_mysql_ssg_data_from_ssg92_to_ssg94_db.sh

#!/bin/bash
#######################################################################
#
# Script to validate upgrade process from CA API 9.2 to 9.4 with docker
#  - Avoid using default of 'docker-compose.yml'
#  - Define different project names for API GW 9.2 and 9.4 to avoid conflict
#  - Explictly use bash shell  /bin/bash  instead of soft-link /bin/sh
#
# 1. Stop docker containers for CA API GW 9.2 & 9.4 (leave mysql containers running)
#    - To prevent any updates to mysql db during migration
#
# 2. Use mysqldump command to export CA API 9.2 MySQL 5.5 ssg database with stored procedures (aka routines)
#   - Review excluding the audit tables to avoid carrying over excessive data
#
# 3. Use mysql command to import sql file to CA API 9.4 MySQL 5.7 ssg database
#   - Review if dropping / recreate the ssg database will avoid any install issues
#   - Keep eye on table cluster_info {as this has the Gateway1 defination with the host IP address}
#
# 4. Restart CA API GW 9.2 & 9.4 containers
#
#    - Challenge 1: CA API GW 9.2 docker image has issue with docker stop/start process
#    the reference /root/entrypoint.sh will loop with creation of a license folder
#    - Addressed with custom docker-compose file to recreate image to join existing MySQL 5.5 container
#
#    - Challenge 2: CA API GW 9.4 docker image has issue with docker stop/start process
#    The new heathcheck.sh process calls a base.sh script that compare the date-time stamp for two files
#    , the datestamp for one file is not updated correctly upon docker start process.
#    - Addressed with custom docker bash script to "touch" the primary file to allow date stamp to be updated.  Validate with: docker logs ssg94
#      WARNING 1      com.l7tech.server.boot.GatewayBoot: Unable to touch /opt/SecureSpan/Gateway/node/default/var/started:
#                  /opt/SecureSpan/Gateway/node/default/var/started (Permission denied)
#
#    - Challenge 3: CA API GW 9.4 docker image appears to have similar issue for hazelcast startup
#    The container may hold for 300 seconds due to hazelcast configuration not completing correctly
#     SEVERE  1      com.hazelcast.instance.Node: [172.17.0.3]:8777 [gateway] [3.10.2] Could not join cluster in 300000 ms. Shutting down now!
#     Unable to start the server: Error starting server : Lifecycle error: Could not initialize Hazelcast cluster
#     WARNING 107    com.hazelcast.cluster.impl.TcpIpJoiner: [172.17.0.3]:8777 [gateway] [3.10.2] Error during join check!
#    - Addessed with different project names to avoid conflict between API GW 9.2 broadcast to API GW 9.4
#
#    - Challenge 4: CA API GW 9.2 appears to be stuck in a while loop for /opt/docker/entrypoint.sh
#      apim-provisioning: INFO: waiting for the shutdown file at "/userdata/shutdown" to be created
#    - Addressed:  Does not seem to have impact with current testing.  Ignore.  Validate with:  docker logs ssg92
#
# 5. Important Note: Ensure that the SSG_CLUSTER_HOST and SSG_CLUSTER_PASSWORD values for CA API GW 9.4 docker-compose file
#    match those set in the configured MySQL database.
#    After CA API GW 9.4 container connects to the existing Gateway database, the Container Gateway will automatically upgrades
#    the ssg database if the ssg database version is lower than the version of the Container Gateway.
#    - Ensure the jdbc hostname
#
#    - Ref:  https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/9-4/other-gateway-form-factors/using-the-container-gateway/getting-started-with-the-container-gateway/connect-the-container-gateway-to-an-existing-mysql-database.html
#
#
# Alan Baugher, ANA, 10/19
#
#######################################################################

echo ""
echo "####################################"
echo "Early check: Address a file permission issue with the API GW 9.4 container"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c 'chmod 666 /opt/SecureSpan/Gateway/node/default/var/started' "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "chmod 666 /opt/SecureSpan/Gateway/node/default/var/started"
echo "May validate issue with:  docker logs ssg94 "
echo "####################################"


echo ""
echo "####################################"
echo "Temporarily shutdown the API GW containers for 9.2 and 9.4 to avoid any updates to the mysql db during export & import"
echo "docker stop ssg92 ssg94 "
docker stop ssg92 ssg94
echo "####################################"
echo ""


echo "####################################"
echo "Validate API GW container are down and the MySQL db containers are up and working"
echo "Pause for 5 seconds:  sleep 5"
sleep 5
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "####################################"
echo ""


echo "####################################"
echo "Export the API GW 9.2 MySQL 5.5 ssg db with stored procedures (aka routines)"
echo "time docker exec -i `docker ps -a | grep mysql:5.5 | awk '{print $1}'` mysqldump -u root --password=7layer ssg  --routines >  ssg92.backup_with_routines.sql  2> /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.5 | awk '{print $1}'` mysqldump -u root --password=7layer ssg  --routines >  ssg92.backup_with_routines.sql  2> /dev/null
echo "View the size of the MySQL 5.5. ssg db for API GW 9.2"
ls -lart | grep ssg92.backup_with_routines.sql
echo "####################################"
echo ""


echo "####################################"
echo "Export the API GW 9.4 MySQL 5.7 ssg db with stored procedures (aka routines) as a 'before' reference file"
echo "time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines >  ssg94.before.backup_with_routines.sql  2> /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines >  ssg94.before.backup_with_routines.sql  2> /dev/null
echo "View the size of the MySQL 5.7. ssg db for API GW 9.4 as the 'before' reference file"
ls -lart | grep ssg94.before.backup_with_routines.sql
echo "####################################"
echo ""


echo "####################################"
echo "Import the MySQL 5.5 ssg db with stored procedures (aka routines) into MySQL 5.7 for API GW 9.4"
echo "time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysql -u root --password=7layer ssg    < ssg92.backup_with_routines.sql 2> /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysql -u root --password=7layer ssg    < ssg92.backup_with_routines.sql 2> /dev/null
echo "####################################"
echo ""


echo "####################################"
echo "Export the API GW 9.4 MySQL 5.7 ssg db wht stored procedures (aka routines) as a 'after' import reference file"
echo "time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines >  ssg94.after.backup_with_routines.sql 2> /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines >  ssg94.after.backup_with_routines.sql 2> /dev/null
echo "View the size of the MySQL 5.7. ssg db for API GW 9.4 as the 'after' reference file"
ls -lart | grep ssg94.after.backup_with_routines.sql
echo "####################################"
echo ""


echo "####################################"
echo "Restart the API GW containers for 9.2 and 9.4 "
# Note: Restart of the ssg94 container will 'auto' upgrade the ssg database to 9.4 tags
echo "docker restart ssg94 "
docker restart ssg94
#docker rm ssg94
#docker-compose -p ssg94 -f /opt/docker/api/docker-compose-ssg94-join-db.yml up -d
echo "####################################"
# Note:  API GW 9.2 docker image was not designed for stop/start correctly; rm then redeploy
export SSG_LICENSE_ENV=$(cat ssg_license.xml | gzip | base64 --wrap=0)
echo "Remove the API GW 9.2 container via:  docker rm ssg92"
docker rm ssg92
echo "Redeploy the API GW 9.2 container "
echo "docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92-join-db.yml up -d "
docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92-join-db.yml up -d
echo "####################################"
echo ""



echo "####################################"
echo "Validate API GW container are up and the mysql db containers are working"
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "####################################"
echo ""


echo "####################################"
echo "Export the API GW 9.4 MySQL 5.7 ssg db after import & after the 'auto' upgrade as an 'after' auto upgrade reference file"
docker stop ssg94
echo "time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines >  ssg94.auto.after.backup_with_routines.sql 2> /dev/null "
time docker exec -i `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /usr/bin/mysqldump -u root --password=7layer ssg  --routines >  ssg94.auto.after.backup_with_routines.sql 2> /dev/null
echo "View all the exported MySQL files to compare process flow"
ls -lart ssg*.sql
docker start ssg94
echo "View the auto upgrade from version 9.2 to version 9.4 with a delta compare of the exported sql files"
echo "diff ssg94.after.backup_with_routines.sql  ssg94.before.backup_with_routines.sql  | grep -i \"INSERT INTO .ssg_version.\" "
diff ssg94.after.backup_with_routines.sql  ssg94.before.backup_with_routines.sql  | grep -i "INSERT INTO .ssg_version."
echo "####################################"
echo ""


echo "####################################"
echo "Execute  'docker ps -a'   to validate running docker containers for API GW 9.4 and 9.2"
echo "docker ps --format \"table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}\" "
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "####################################"
echo ""


echo "####################################"
echo "Show current API GW 9.4 MySQL 5.7 databases"
echo "Validate that 'ssg' database exists "
echo "docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  -e \"show databases;\" "
docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  -e "show databases;"
echo "####################################"
echo ""


echo "####################################"
echo "Review for any delta of the MySQL ssg database after import"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.5 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e \"show tables;\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.5 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e "show tables;" > ssg92.tables.txt
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e \"show tables;\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e "show tables;" > ssg94.tables.txt
echo "Observer for any delta from the below command"
echo "diff ssg92.tables.txt ssg94.tables.txt"
diff ssg92.tables.txt ssg94.tables.txt
echo "####################################"
echo ""


echo "####################################"
echo "Show current API GW 9.4 admin user in the MySQL 5.7 ssg database"
echo "docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer ssg -e \"SELECT name,login,password,enabled,expiration,password_expiry FROM internal_user;\" "
docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer ssg -e "SELECT name,login,password,enabled,expiration,password_expiry FROM internal_user;"
echo "####################################"
echo ""


echo "####################################"
echo "Show current API GW 9.4 admin user in the intermediate configuration file on the AIP GW 9.4 container"
echo "docker exec -it -u root -e TERM=xterm ssg94 /bin/bash -c \"grep -i -e l7.login -e l7.password /opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle/001_update_admin_user.xml.req.bundle\" "
docker exec -it -u root -e TERM=xterm ssg94 /bin/bash -c "grep -i -e l7.login -e l7.password /opt/SecureSpan/Gateway/node/default/etc/bootstrap/bundle/001_update_admin_user.xml.req.bundle"
echo "####################################"
echo ""


echo "####################################"
echo "Show all 'new' files created or linked in API GW 9.4 container with mtime of 1 day. Excluding lock (LCK) files"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c \"find /opt -type f -mtime -1 -ls | grep -i -v -e '.LCK'\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "find /opt -type f -mtime -1 -ls | grep -i -v -e '.LCK'"
echo "####################################"
echo ""


echo "####################################"
echo " View the license.xml file that was copied to the API GW 9.4 container bootstrap folder before copied to the MySQL 5.7 ssg db table "
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c \"ls -lart  /opt/SecureSpan/Gateway/node/default/etc/bootstrap/license \" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "ls -lart  /opt/SecureSpan/Gateway/node/default/etc/bootstrap/license "
echo "####################################"
echo ""


echo "####################################"
echo "View logon count for the API GW 9.4 admin user via MySQL 5.7 ssg db"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=gateway --password=7layer ssg -e \"select hex(goid), version, hex(provider_goid), login, fail_count, last_attempted, last_activity, state from logon_info;\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=gateway --password=7layer ssg -e "select hex(goid), version, hex(provider_goid), login, fail_count, last_attempted, last_activity, state from logon_info;"
echo "####################################"
echo ""


echo "####################################"
echo "View the API GW 9.4 MySQL 5.7 mysql.user table"
### docker logs `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  2>&1 | grep  "GENERATED ROOT PASSWORD"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e \"SELECT User,account_locked,password_expired,password_last_changed,authentication_string FROM mysql.user;\" "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer  ssg -e "SELECT User,account_locked,password_expired,password_last_changed,authentication_string FROM mysql.user;"
echo "####################################"
echo ""


echo "####################################"
echo "To remove any locked account (including pmadmin SSG Admin User ID) from the MySQL 5.7 ssg logon_info table  {or any account}"
echo "docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer ssg -e \"delete from logon_info where login ='pmadmin';\" "
echo "docker exec -it -u root -e TERM=xterm  `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=7layer ssg -e \"truncate logon_info;\"  "
echo "####################################"
echo ""


echo "####################################"
echo "To change root password for MySQL 5.7 mysql.user db"
echo "docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'`  mysql --user=root --password=OLD_PASSWORD -e  \"SET PASSWORD FOR 'root'@'localhost' = PASSWORD('7layer');\" "
echo "####################################"
echo ""


echo "####################################"
echo "Sleep 30 seconds to address restart health check time-stamp issue with API GW 9.4"
sleep 30
echo "####################################"
echo ""


echo "####################################"
echo "Address API GW 9.4 container health check upon stop/start or restart gap.  (base.sh script)"
echo "docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c \"date +%s -r /opt/SecureSpan/Gateway/node/default/var/started  && date +%s -r /opt/SecureSpan/Gateway/node/default/var/preboot\" "
docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "date +%s -r /opt/SecureSpan/Gateway/node/default/var/started  && date +%s -r /opt/SecureSpan/Gateway/node/default/var/preboot"
echo "Touch to update date-time stamp for one file: /opt/SecureSpan/Gateway/node/default/var/started"
echo "docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c \"touch /opt/SecureSpan/Gateway/node/default/var/started\" "
docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "touch /opt/SecureSpan/Gateway/node/default/var/started"
docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c "date +%s -r /opt/SecureSpan/Gateway/node/default/var/started  && date +%s -r /opt/SecureSpan/Gateway/node/default/var/preboot"
echo "####################################"
echo ""


echo "####################################"
echo "Sleep 30 seconds to allow health check status to update for API GW 9.4"
echo "May also monitor health and overall status with:   docker inspect ssg94 "
sleep 30
echo "####################################"
echo ""


echo "####################################"
echo "Execute  'docker ps -a'   to validate running docker containers for API GW 9.4 and 9.2"
echo "docker ps --format \"table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}\" "
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "####################################"
echo ""


echo "################################"
echo "Test authentication with the SSG backup URL for API 9.2 TCP 8444 - should see minimal of six (6) lines"
echo "curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8444/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "#########           ############"
curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8444/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""


echo "################################"
echo "Test authentication with the SSG backup URL for API 9.4 TCP 8443 - should see minimal of six (6) lines"
echo "curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8443/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "#########           ############"
curl -s --insecure  -u pmadmin:7layer  https://$(hostname -s):8443/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""

View of the API Gateway via the MS Windows API GW UI for both API GW 9.2 (using the 9.3 UI) and API 9.4 (using the 9.4 UI). The API GW Policies will be migrated from API 9.2 to API 9.4 via the export/import of MySQL ssg database. After import, the API GW 9.4 docker image will ‘auto’ upgrade the ssg database to the 9.4 version.

Interesting view of the API GW 9.4 MySQL database ‘ssg’ after import and a restart (that will ‘auto’ upgrade the ssg database version). Note multiple Gateway “nodes” that will appear after each ‘docker restart containerID’