Docker Image(s) provide fantastic value, as this platform-as-a-service methodology gets us all out of the painful “install-business”. We may focus effort on the business value that a solution provides.
However, the associated docker containers may provide some challenges.
Business Risks:
For the CA API Gateway solution, we have two (2) business risks to address:
- The docker container of the API Gateway solution may be ephemeral & replaced with newer releases.
- Example: Any API Gateway application logs that reside on the docker container, as a file, may be lost when the container version is updated or redeployed.
- The docker container of the associated MySQL Database may have growth concerns with the default OOTB API Gateway Audit Event process.
- Example: The MySQL Database of ibdata1 may continue to grow and be impacted by current disk constraints. To reduce MySQL Database size and remove the low value audit data, it will be necessary to declare an outage window to export (w/o audit data)/resize mysql db/import the data (w/o audit data).
Resolution(s):
To address the above risks, we may leverage the syslog feature set provided by the API Gateway:
- The API Gateway documentation does allow for syslog configuration for the primary Audit Event and the application logs.
This blog entry will review the use of syslog, create individual syslog files for each API Gateway application, avoid common mis-configurations, and how to validate the processes.
Before we start this process, to provide justification, lets review how “large” is the API Gateway Audit Events in the MySQL Database & that API Gateway logs are no longer retained on the container (as of r9.4 release)
Validation of API Gateway Database Growth & Application Logs:
Pre-Step 00: Review the current MySQL ibdata1 file size (via docker command)
docker exec -it -u root -e TERM=XTERM ssg94_mysql57 /bin/bash -c "find /var -type f -mtime -1 -ls | head -5"
Pre-Step 01: Review the audit tables sizes in the ‘ssg’ MySQL database with the below query:
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'` mysql --user=gateway --password=7layer ssg -e "SELECT TABLE_NAME, table_rows, data_length, index_length, round(((data_length + index_length) / 1024 / 1024),2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema = 'ssg' ORDER BY (data_length + index_length) DESC; "
Pre-Step 02: Validate no API Gateway application files reside and primary ssg file is redirect to /dev/null (r9.4)
docker exec -it -u root -e TERM=XTERM `docker ps -a | grep caapim/gateway:latest | awk '{print $1}'` /bin/bash -c 'ls -larth /opt/SecureSpan/Gateway/node/default/var/logs'
Enabling Syslog for API Gateway Applications
Step 1: Enable & update the remote rsyslogd service on the docker host (or remote host) [/etc/rsyslog.conf]
- Enable/Allow UDP 514
- Un-comment two (2) lines
- Define unique syslog facilities for each API GW application & the primary Audit Event
- Note: Use the mapping of names from syslog parameter to facilities # •https://tools.ietf.org/html/rfc5424 [Page 10]
- See example(s) in the image below
- Exclude duplicate logging to /var/messages
- Add a semi-colon with the facility.none on this line
- ;local3.none;local4.none;local5.none;local6.none
- Restart the updated rsyslog.service
- systemctl restart rsyslog.service
- Validate UDP 514 is available.
- netstat -an | grep :514
Step 2: Validate Syslog is functioning correctly with the facilities (syslog naming convention) using the OS command:
- logger -s -p local3.warn Testing for ServiceNow SysLog
- logger -s -p local4.warn Testing for Google GCP SysLog
- logger -s -p local5.warn Testing for OpenStack SysLog
- logger -s -p local6.warn Testing for SSG Audit Event SysLog
Step 3: Enable syslog for each API Application with a unique log file associated to a unique facility number.
a. Review the associated API Gateway Applications in the API Gateway Policy Manager and their associated URI strings,e.g. /gcp/*, /servicenow/*, etc.
b. Open the API Gateway Policy Manager / Tasks / Logging and Auditing / Manage Log/Audit Sinks Window
c. Create new Log Sinks for every API Gateway Application. Configure these for ‘syslog’, with the correct facility number, and ensure the FILTER is set correctly with the Category & Services URI string match (otherwise you may not see any data). Suggest Category=Traffic Log (as your first iteration). Reference for facility number (0-23) to match syslog naming conventions: https://tools.ietf.org/html/rfc5424 [page 10]
Step 4: Validate the API Gateway Application Syslog with remote web service call via curl to the various application URI strings.
curl -s --insecure -u pmadmin:7layer https://$(hostname -s):8443/gcp/apple cat /var/log/google-gcp.log | tail -1 curl -s --insecure -u pmadmin:7layer https://$(hostname -s):8443/servicenow/pear cat /var/log/servicenow.log | tail -1 curl -s --insecure -u pmadmin:7layer https://$(hostname -s):8443/openstack/banana cat /var/log/openstack.log | tail -1
Enabling Syslog for API Gateway Audit Event
The current documentation for enabling syslog for the API Gateway Audit Event process is clear, but administrators may have a challenge if the FILTER is not set correctly. To address this common mis-configuration, the follow steps are provided to run through the configuration, and clarify where to avoid this challenge.
Step 1AE: Review the prior OOTB configuration / view. The API Gateway Audit Events are stored in the MySQL database, and the built-in view tool allows for queries to be performed. However, unless the “File/Delete Old Audit Events” is executed, growth will continue to impact MySQL db.
Step 2AE: Configure the API Gateway Audit Events to not use the MySQL database. The below process will create a “[Internal Audit Sink Policy]” that we will use to redirect.
Step 3AE: Important Step: Disable “all” assertions in the newly created “[Internal Audit Sink Policy]”.
- Double-click on the lower-left panel item ‘[Internal Audit Sink Policy]”.
- The right-upper panel will displays nine (9) line items.
- Use mouse to select all lines, then click on the RED button on the middle column to disable all assertions (lines).
- In the upper-left panel, search for the string “continue”
- The object “Continue Processing” will display
- Use mouse to drag-n-drop this item to the upper-right panel.
- Click “Save and Activate” selection on the upper-right panel, as shown in the image below.
Step 4AE: Now create the new API Gateway Log Sink for Audit Events. Ensure Filters are broad to the following Categories: Audits and Gateway Log. Avoid any additional filters. Ensure the facility number matches the syslog naming convention.
Step 5AE: Confirm that audit events are being sent to the syslog defined.
Step 6AE: Confirm that API Gateway configuration for syslog is retained after “destroying” the docker container, and rebuilding it to connect to the existing MySQL database.
a. Destroy and rebuild API Gateway container (r9.4)
docker stop ssg94 docker rm ssg94 docker-compose -p ssg94 -f ./docker-compose-ssg94-join-db.yml up -d cat /var/log/ssg-audit-events.log | tail -5
b. Wait one minute and try again
cat /var/log/ssg-audit-events.log | tail -5
Additional Notes/Recommendations:
The primary API Gateway startup ‘ssg’ logs, that are accessible via ‘docker logs containerID’ may be converted to syslog, but this process will impact the JSON formatting that that ‘docker logs containerID’ uses.
A review of the OOTB syslog for /var/log/messages shows that these docker messages are also already forwarded here. Recommend skipping this unnecessary configuration step.
cat /var/log/messages | grep -i l7tech | tail -2 docker logs ssg94 2>&1 | tail -2
Reference Table from rfc5424 ( https://tools.ietf.org/html/rfc5424 ). Facilities #16-22 are open and not predefined for other applications.
View of API Gateway “Manage Log Sinks” Window
View of the “View Logs” Window for “Log Sinks” may display, but no data will be returned, as there are no local files. See prior screen shot where ssg_0_0.log is redirected to /dev/null.
If the above screen does not load correctly (r9.4), then in the docker-compose file, add in an additional JVM switch:
EXTRA_JAVA_ARGS: " -Dcom.l7tech.server.disableFileLogsinks=false "
If you wish to know more or need assistance, please contact us.