Written by CA API Gateway (ssg) is used to manage SaaS endpoints/applications for the CA/Symantec Identity Suite solution. One of the challenges of appliances and Docker containers is the underlying 3rd party libraries may get dated, and require updates.
Most vendors will not allow post-updates or direct updates to their containers libraries, as this has an impact on the support model. So we must rely on the support process and push vendors to release additional updates to stay ahead of any security concerns.
The CA API Gateway (ssg) when deployed on docker, has a streamlined process for updating in place, as long as you have backed-up the MySQL database when the docker images are updated.
We wanted to capture the process to upgrade from CA API Gateway 9.4 (ssg94) to Gateway 10.0 (ssg10). Fortunately, the MySQL 8.0 database has the same structure, tables, and routines as the MySQL 5.7 database for CA API Gateway 9.4.
The challenge we have is the documented process to upgrade is difficult to implement on the same host OS; and there was a lost opportunity to manage the license file from 9.4 to license 10.0 during the re-import of the MySQL database.
The below diagram, from the CA API Gateway 10.0 upgrade process can be adjusted to streamline the upgrade process.
The above-documented process outlines dropping the MySQL database ssg completely, and then create a new clean db. Using this documented process, we can slightly adjust it, to avoid an unnecessary step to re-import the license file after the restart of the gateway container. We also wish to add additional validation steps to show what is changing.
Proposal for modifications:
- Create a clean CA API Gateway 10.0 (ssg10) Docker deployment on the same Host OS. May use docker-compose with REST service enabled and use different TCP listening ports to allow two (2) docker containers to run simultaneously during the testing cycle. After testing may keep the default TCP license ports of 8443 & 9443.
- Allow the CA API Gateway 10.0 container to start cleanly with MySQL 8.0 DB and with the correct license file for version 10. Then export the MySQL database table that contains the updated license table.
- Import the prior backup MySQL file to the new CA API Gateway deployment. Then before startup, import the ssg10 license mysql file as well. This will replace the ssg94 license information.
- Restart the CA API Gateway container, and monitor the logs for any errors and ensure the new license file is used
- If REST API was enabled (via the docker-compose file & touch a file name “restman”), then use CURL to validate all REST services are available, and list all prior API Gateway Policy Services are displayed.
A visual example of this process using the prior diagram.
Note: The official documentation uses sed to replace a string “NO_AUTO_CREATE_USER”; but the documentation shows two examples. One with a comma & one without. We have included the one with the comma, but we did not see this line in the MySQL sql export, so it was deemed low value, but still included in our process.
Example of upgrade process and validation of using REST
Note the two (2) running CA API Gateway container of 9.4 (with MySQL 5.7) and 10.0 (with MySQL 8.0) with different TCP listen services; and validation of REST services for ssg10
Below are the above steps called out with additional validations steps, and the use of the “time” command to monitor the export of the files.
# Pre-Step 1: On Test System: Prepare SSG10 docker compose yml file and correct license.xml & confirm startup.
time docker-compose -p ssg10 -f ./docker-compose-ssg10-0.yml up -d {Wait 90-120 seconds}
docker ps -a
docker logs ssg10 -f --tail 100
docker exec -it mysql-ssg10 mysql --user=root --password=7layer -e "show databases;"
# Step 2: On PROD HOST OS: Stop SSG94 and export the current MySQL 5.7 database with routines (aka stored procedures) & remove unwanted lines
docker stop ssg94
time docker exec -tt mysql-ssg /usr/bin/mysqldump -h 127.0.0.1 -u root --password=7layer ssg --routines > ssg94.backup.before.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.sql
time docker exec -tt mysql-ssg /usr/bin/mysqldump -h 127.0.0.1 -u root --password=7layer ssg --routines > ssg94.backup.updated.for.mysql8.sql
sed -i "s/NO_AUTO_CREATE_USER,//g" ssg94.backup.updated.for.mysql8.sql
sed -i "/Using a password on the command/d" ssg94.backup.updated.for.mysql8.sql
# Step 3: On PROD HOST OS: Deploy SSG10 with docker compose yml file & correct license xml file & export db table license_document
time docker-compose -p ssg10 -f ./docker-compose-ssg10-0.yml up -d {Wait 90-120 seconds}
docker ps -a
docker logs ssg10 -f --tail 100
docker stop ssg10
time docker exec -tt mysql-ssg10 /usr/bin/mysqldump -h 127.0.0.1 -u root --password=7layer ssg --routines license_document > ssg10.license.export.sql
sed -i "/Using a password on the command/d" ssg10.license.export.sql
# Step 4: On PROD HOST OS: Drop the SSG10 MySQL 8.0 ssg database and rebuilt with imports of SQL files.
time docker exec -it -u root -e term=xterm mysql-ssg10 /usr/bin/mysqladmin --user=root --password=7layer drop ssg
docker exec -it mysql-ssg10 mysql --user=root --password=7layer -e "show databases;"
time docker exec -it -u root -e term=xterm mysql-ssg10 /usr/bin/mysqladmin --user=root --password=7layer create ssg
docker exec -it mysql-ssg10 mysql --user=root --password=7layer -e "show databases;"
time docker exec -i mysql-ssg10 /usr/bin/mysql -u root --password=7layer ssg < ssg94.backup.updated.for.mysql8.sql
time docker exec -i mysql-ssg10 /usr/bin/mysql -u root --password=7layer ssg < ssg10.license.export.sql
docker exec -it mysql-ssg10 mysql --user=root --password=7layer ssg -e "SELECT * FROM license_document;" | grep -A 12 -e "<license "
# Step 5: On PROD HOST OS: Start SSG10 and validate no errors
docker start ssg10 {Wait 90-120 seconds}
docker ps -a
# Step 6: Validate license
docker logs ssg10 -f --tail 100
docker logs ssg10 -f 2>&1 | grep -i license
# Step 7: Validate REST services enabled and we can see all services
curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/rest.wadl
curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/services
# Example to validate ServiceNow REST service to CA APIGW
curl --insecure --user admin:gwALPtteR5R1 --compressed --header "Accept: application/json" --header "url: https://dev101846.service-now.com" "https://localhost:9443/ServiceNow/v1/Users?filter=userName+eq+%22ztestalan10340%22&attributes=userName"
# Example validate ServiceNow REST service via LB to CA APIGW
curl --insecure --user admin:gwALPtteR5R1 --compressed --header "Accept: application/json" --header "url: https://dev101846.service-now.com" "https://192.168.242.135/ServiceNow/v1/Users?filter=userName+eq+%22ztestalan10340%22&attributes=userName"
# Direct REST service to ServiceNow to validate development instance is available.
curl --user admin:gwALPtteR5R1 --compressed --header "Accept: application/json" 'https://dev101846.service-now.com/api/now/table/sys_user?sysparm_query=user_name=testalan13095'
# Step 8: Certs required for IM JCS Tier to avoid typical cert issues.
a. Ensure the CA API Gateway public root CA cert or self-signed cert is imported to each JCS keystore
b. If using a LoadBalancer, e.g. httpd, ensure this public root CA cert or self-signed cert is imported to each JCS keystore.
docker commands collected to assist with RCA efforts for Operation Teams
# Extra commands to assist RCA efforts or OPS teams
#
# Validate routing is enabled within the CAAPIGW (ssg) container
# docker exec -it ssg bash -c "curl -L www.google.com"
# docker exec -it -u root -e term=xterm ssg /bin/bash -c "curl -vk --tlsv1.2 https://www.service-now.com"
# Interactive Session with mysql> prompt
# docker exec -it mysql-ssg mysql --user=root --password=7layer
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "show databases;"
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "SELECT User,Password,authentication_string FROM mysql.user;"
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "SELECT name,login,password,enabled,expiration,password_expiry FROM internal_user;"
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "truncate logon_info;"
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "delete from logon_info where login ='ssgadmin';"
# If MySQL root password is random, find via logs (use redirect to switch from JSON to text to use grep)
# docker logs mysql-ssg 2>&1 | grep -i "Generated root password"
# docker logs mysql-ssg -f {Used to tail the logs}
# Limit the logs to see
# docker logs ssg10 -f --tail 100
# Commands to install additional packages for vul scans (ps from procps) & update passwords (mkpasswd from whois)
# docker exec -it -u root -e term=xterm mysql-ssg /bin/bash -c "apt-get update -y && apt-get upgrade -y && apt-get install -y procps && apt-get install -y whois"
# docker exec -it mysql-ssg ps aux
# Update password process
# Generate SSHA512 Password (use one of the below methods)
# docker exec -it -u root -e term=xterm mysql-ssg /bin/bash -c "mkpasswd -m sha-512 7layer"
# python -c 'import crypt; print(crypt.crypt("7layer", crypt.mksalt(crypt.METHOD_SHA512)))'
# perl -le 'print crypt "7layer", "\$6\$customSalt\$"'
# Update password via command line (escape any $ characters)
# docker exec -it -u root -e term=xterm mysql-ssg mysql --user=root --password=7layer ssg -e "UPDATE internal_user SET password='\$6\$SzW/q9xVM9\$Ed/LjCDVpIYNTq94CsqO2stR0h4KniPOl/7iQDv1SEXNu9ftv//6hohlJxNeizmac/V9cEb6WmJfdHQCFwpoc0' WHERE name='pmadmin'; "
# View user and password hash in DB
# docker exec -it -u root -e term=xterm mysql-ssg mysql --user=root --password=7layer ssg -e "select * from internal_user \G;"
# View if account is active
# docker exec -it -u root -e term=xterm mysql-ssg mysql --user=root --password=7layer ssg -e "select * from logon_info \G;"
# Reset if account is NOT active
# docker exec -it -u root -e term=xterm mysql-ssg mysql --user=root --password=7layer ssg -e "UPDATE logon_info set state='ACTIVE', fail_count=0 where login='pmadmin';"
# REST WEB SERVICES
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/doc/home.html
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/services
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/rest.wadl
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/gateway-management.xsd
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/folders
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/folders/template
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/folders?name=My%20Service
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/doc/restDoc.html
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/emailListeners?sort=host&order=desc
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/doc/authentication.html
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/passwords/template
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/policies/template
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/doc/migration.html
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/ssgconnectors?enabled=true
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/clusterProperties/template
# curl --insecure --user pmadmin:7layer https://localhost:9443/restman/1.0/rest.wadl
Change the pmadmin password at the docker command line
Process flows collect for the CA API Gateway docker deployment
Example of the docker-compose yml file for CA API Gateway with REST web services and license xml file.
We attempt to keep useful notes/hints included in the yml file to allow for future reference. The example below redirect ports to TCP 18443 and 19443 from the standard ports of 8443 and 9443 for the CA API Gateway; and MySQL from 3306 to 23306 for testing protocols in non-Production enviornments.
# docker-compose-ssg10-0-mysql8-0_with_rest_and_external_mysql_volume.yml
# Startup: docker-compose -p ssg -f ./docker-compose-ssg10-0-mysql8.yml up -d
# Stop: docker-compose -p ssg -f ./docker-compose-ssg10-0-mysql8.yml down
#
#
# Ensure Host OS Network allows IPv4 forwarding: sysctl -a | grep ipv4.ip_forward
# Validate docker network has access with curl: curl -vk --tlsv1.2 https://www.service-now.com
# Note: Do NOT use TABS in this file
# Monitor startup of containers with: docker logs ssg10 -f --tail 100 AND docker logs mysql-ssg10 -f --tail 100
# https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-0/using-the-container-gateway/getting-started-with-the-container-gateway/run-the-container-gateway-on-docker-engine/sample-docker-compose-deployment-file.html
version: "2.2"
services:
ssg10:
container_name: ssg10
# Ref: https://hub.docker.com/r/caapim/gateway/tags
#image: caapim/gateway:latest
image: caapim/gateway:10.0.00_20200428
mem_limit: 10048m
volumes:
# Ensure ssg_license.xml is a valid SSG license file for 9.4 or 10.0
- ./ssg_license_10.xml:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/license/license.xml
# https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-0/apis-and-toolkits/rest-management-api.html
# Touch the file restman to auto-start rest webservices
# Validate REST API with curl
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/rest.wadl
# curl --insecure --user pmadmin:7layer https://localhost:18443/restman/1.0/rest.wadl
- ./restman:/opt/SecureSpan/Gateway/node/default/etc/bootstrap/services/restman
ports:
- "58443:8443"
- "59443:9443"
environment:
ACCEPT_LICENSE: "true"
SSG_CLUSTER_COMMAND: "create"
SSG_CLUSTER_HOST: "localhost"
SSG_CLUSTER_PASSWORD: "7layer"
SSG_DATABASE_TYPE: "mysql"
SSG_DATABASE_HOST: "mysql-ssg"
SSG_DATABASE_PORT: "3306"
SSG_DATABASE_NAME: "ssg"
SSG_DATABASE_USER: "gateway"
SSG_DATABASE_PASSWORD: "7layer"
SSG_DATABASE_JDBC_URL: "jdbc:mysql://mysql-ssg10:3306/ssg?useSSL=false"
SSG_DATABASE_ADMIN_USER: "root"
SSG_DATABASE_ADMIN_PASS: "7layer"
SSG_ADMIN_USERNAME: "pmadmin"
SSG_ADMIN_PASSWORD: "7layer"
EXTRA_JAVA_ARGS: "-Dcom.l7tech.bootstrap.env.license.enable=false -Dcom.l7tech.bootstrap.autoTrustSslKey=trustAnchor,TrustedFor.SSL,TrustedFor.SAML_ISSUER -Dcom.l7tech.server.transport.jms.topicMasterOnly=false"
SSG_INTERNAL_SERVICES: "restman wsman"
links:
- mysql-ssg10
mysql-ssg10:
container_name: mysql-ssg10
# Ref https://hub.docker.com/_/mysql?tab=tags
image: mysql:8.0.20
#image: mysql:latest
# SSG 10.0 requires MySQL 8.x per documentation
#https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-gateway/10-0/install-configure-upgrade/using-mysql-8_0-with-gateway-10.html
mem_limit: 1048m
restart: always
ports:
- "23306:3306"
environment:
- MYSQL_ROOT_PASSWORD=7layer
#- MYSQL_RANDOM_ROOT_PASSWORD=yes
- MYSQL_USER=gateway
- MYSQL_PASSWORD=7layer
- MYSQL_DATABASE=ssg
command:
- "--character-set-server=utf8mb3"
- "--log-bin-trust-function-creators=1"
- "--default-authentication-plugin=mysql_native_password"
- "--innodb_log_buffer_size=32M"
- "--innodb_log_file_size=80M"
- "--max_allowed_packet=8M"
# volumes:
# - mysql_db8:/var/lib/mysql
# Persist SSG MySQL DB Data
# Validate after shutdown with: docker volume ls & docker volume inspect ssg_mysql_db
# Note: Important - Random Root Password will not work for persist MySQL - Password must be known for 1st time
# volumes:
# mysql_db8:
#
# Extra commands to assist RCA efforts or OPS teams
#
# Validate routing is enabled within the CAAPIGW (ssg) container
# docker exec -it ssg bash -c "curl -L www.google.com"
# docker exec -it -u root -e term=xterm ssg /bin/bash -c "curl -vk --tlsv1.2 https://www.service-now.com"
# Interactive Session with mysql> prompt
# docker exec -it mysql-ssg mysql --user=root --password=7layer
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "show databases;"
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "SELECT User,Password,authentication_string FROM mysql.user;"
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "SELECT name,login,password,enabled,expiration,password_expiry FROM internal_user;"
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "truncate logon_info;"
# docker exec -it mysql-ssg mysql --user=root --password=7layer -e "delete from logon_info where login ='ssgadmin';"
# If MySQL root password is random, find via logs (use redirect to switch from JSON to text to use grep)
# docker logs mysql-ssg 2>&1 | grep -i "Generated root password"
# docker logs mysql-ssg -f {Used to tail the logs}
# Limit the logs to see
# docker logs ssg10 -f --tail 100
# Commands to install additional packages for vul scans (ps from procps) & update passwords (mkpasswd from whois)
# docker exec -it -u root -e term=xterm mysql-ssg /bin/bash -c "apt-get update -y && apt-get upgrade -y && apt-get install -y procps && apt-get install -y whois"
# docker exec -it mysql-ssg ps aux
# Update password process
# Generate SSHA512 Password (use one of the below methods)
# docker exec -it -u root -e term=xterm mysql-ssg /bin/bash -c "mkpasswd -m sha-512 7layer"
# python -c 'import crypt; print(crypt.crypt("7layer", crypt.mksalt(crypt.METHOD_SHA512)))'
# perl -le 'print crypt "7layer", "\$6\$customSalt\$"'
# Update password via command line (escape any $ characters)
# docker exec -it -u root -e term=xterm mysql-ssg mysql --user=root --password=7layer ssg -e "UPDATE internal_user SET password='\$6\$SzW/q9xVM9\$Ed/LjCDVpIYNTq94CsqO2stR0h4KniPOl/7iQDv1SEXNu9ftv//6hohlJxNeizmac/V9cEb6WmJfdHQCFwpoc0' WHERE name='pmadmin'; "
# View user and password hash in DB
# docker exec -it -u root -e term=xterm mysql-ssg mysql --user=root --password=7layer ssg -e "select * from internal_user \G;"
# View if account is active
# docker exec -it -u root -e term=xterm mysql-ssg mysql --user=root --password=7layer ssg -e "select * from logon_info \G;"
# Reset if account is NOT active
# docker exec -it -u root -e term=xterm mysql-ssg mysql --user=root --password=7layer ssg -e "UPDATE logon_info set state='ACTIVE', fail_count=0 where login='pmadmin';"
# REST WEB SERVICES
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/doc/home.html
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/services
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/rest.wadl
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/gateway-management.xsd
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/folders
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/folders/template
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/folders?name=My%20Service
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/doc/restDoc.html
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/emailListeners?sort=host&order=desc
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/doc/authentication.html
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/passwords/template
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/policies/template
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/doc/migration.html
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/ssgconnectors?enabled=true
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/clusterProperties/template
# curl --insecure --user pmadmin:7layer https://localhost:19443/restman/1.0/rest.wadl