While preparing to enable a feature within the Identity Suite Virtual Appliance for TLS encryption for the Provisioning Tier to send notification events, we noticed some challenges that we wish to clarify.
The Identity Suite Virtual Appliance has four (4) web services that use pre-built self-signed certificates when first deployed. Documentation is provided to change these certificates/key using aliases or soft-links.
One of the challenges we discovered is the Provisioning Tier may be using an older version of libcurl & OpenSSL that have constraints that need to be managed. These libraries are used during the web submission to the IME ETACALLBACK webservice. We will review the processes to capture these error messages and how to address them.
We will introduce the use of Let’s Encrypt wildcard certificates into the four (4) web services and the Provisioning Server’s ETACALLBACK use of a valid public root certificate.
The Apache HTTPD service is used for both a forward proxy (TCP 443) to the three (3) Wildfly Services and service for the vApp Management Console (TCP 10443). The Apache HTTPD service SSL certs use the path /etc/pki/tls/certs/localhost.crt for a self-signed certificate. A soft-link is used to redirect this to a location that the ‘config’ service ID has access to modify. The same is true for the private key.
A view of the Apache HTTPD SSL self-signed certificate and key.
The three (3) Wildfly services are deployed for the Identity Manager, Identity Governance and Identity Portal components. The configuration for TLS security is defined within the primary Wildfly configuration file of standalone.xml. The current configuration is already setup with the paths to PKCS12 keystore files of:
A view of the three (3) Wildfly PKCS12 keystore files and view of the self-signed cert/key with the pseudo hostname of the vApp host.
Provisioning Server process for TLS enablement for IME ETACALLBACK process.
Step 1. Ensure that the Provisioning Server is enabled to send data/notification events to the IME.
Step 2. Within the IME Management Console, there is a baseURL parameter. This string is sent down to the Provisioning Server upon restart of the IME, and appended to a list. This list is viewable and manageable within the Provisioning Manager UI under [System/Identity Manager Setup]. The URL string will be appended with the string ETACALLBACK/?env=identityEnv. Within this Provisioning Server, we can manage which URLs have priority in the list. This list is a failover list and not load-balancing. We have the opportunity to introduce an F5 or similar load balancer URL, but we should enable TLS security prior.
Step 3. Added the public root CA Cert or CA chain certs to the following location. [System/Domain Configuration/Identity Manager Server/Trusted CA Bundle]. This PEM file may be placed in the Provisioning Server bin folder with no path or may use a fully qualified path to the PEM file. Note: The Provisioning Server is using a version of openssl/libcurl that will report errors that can be managed with wildcard certificates. We will show the common errors in this blog entry.
Let’sEncrypt Certificates offers a free service to build wildcard certificates. We are fond of using their DNS method to request a wildcard certificate.
sudo certbot certonly --manual --preferred-challenges dns -d *.aks.iam.anapartner.dev --register-unsafely-without-email
Let’s Encrypt will provide four (4) files to be used. [certN.pem, privkeyN.pem, chainN.pem, fullchainN.pem]
cert1.pem [The primary server side wildcard cert]
privkey1.pem [The primary server side private key associated with the wildcard cert]
chain1.pem [The intermediate chain certs that are needed to validate the cert1 cert]
fullchain1.pem [two files together in the correct order of cert1.pem and chain1.pem.]
NOTE: fullchain1.pem is the file you typically would use as the cert for a solution, so the solution will also have the intermediate CA chain certs for validation]
Important Note: One of the root public certs was cross-signed by another root public cert that expired. Most solutions are able to manage this challenge, but the provisioning service ETACALLBACK has a challenge with an expired certificate, but there are replacements for this expired certificate that we will walk through. Ref: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/
Create a new CA chain PEM files for LE (Let’s Encrypt) validation to use with the Provisioning Server.
Validate with browsers and view the HTTPS lock symbol to view the certificate
Test with an update to a Provisioning Global User’s attribute [Note: No need to sync to accounts]. Ensure that the Identity Manager Setup Log Level = DEBUG to monitor this submission with the Provisioning Server etanotifyXXXXXXX.log.
A view of the submission for updating the Global User’s Description via IMPS (IM Provisioning Server) etanotifyXXXXXXX.log. The configuration will be loaded for using the URLs defined. Then we can monitor for the submission of the update.
Finally, a view using the IME VST (View Submitted Tasks) for the ETACALLBACK process using the task Provisioning Modify User.
Common TLS errors seen with the Provisioning Server ETACALLBACK
Ensure that the configuration is enabled for debug log level, so we may view these errors to correct them. [rc=77] will occur if the PEM file does not exist or is not in the correct path. [rc=51] will occur if the URL defined does not match the exact server-side certificate (this is a good reason to use a wildcard certificate or adjust your URL FQDN to match the cert subject (CN=XXXX) value. [rc=60] will occur if the remote web service is using a self-signed certificate or if the certificate has any expiration dates within the certificate or chain or the public root CA cert.
Other Error messages (curl)
If you see an error message with Apache HTTPD (TCP 443) with curl about “curl: (60) Peer certificate cannot be authenticated with known CA certificates”, please ignore this, as the vApp does not have the “ca-bundle.crt” configuration enabled. See RedHat note: https://access.redhat.com/solutions/523823
Kubernetes was designed for the deployment of applications to cloud architecture with containers. Another way of thinking about Kubernetes; it gets us “out-of-the-install-binaries” business and focuses our efforts on the business value of a solution. We have documented our process of how we train our resources and partners. This process will help your team to excel and gain confidence with cloud technologies.
One of the business challenges of Kubernetes in the cloud architecture is the ongoing cost ($300-$600/month per resource) during the learning or development process. To lower this ongoing cost per resource, we focused on a method to use on-prem Kubernetes deployments.
We have found examples online of using minikube and Oracle Virtualbox to assist with keeping costs low while using an on-prem deployment but did not find many examples of using Vmware Workstation to our satisfaction. Our goal was to utilize a solution that we are very familiar with and has the supporting capabilities for rollback via snapshots.
We have used Vmware Workstation for many years while working on service projects. We cannot overstate its usefulness to offer a “play-ground” and development environment independent of a client’s environment. The features of snapshots allow for negative use-case testing or “what-if” scenarios to destroy or impact solutions being tested with minimal impact.
In this entry, we will discuss the use of Vmware Workstation and CentOS (or Ubuntu) as the primary Kubernetes Nodes. Both CentOS and/or Ubuntu OS are used by the cloud providers as their Kubernetes nodes, so this on-prem process will translate well.
Some of our team members run the Kubernetes environment from their laptop, a collection of individual servers, or a larger server that may scale to the number of vCPU/RAM required for the Kubernetes solution.
Decision 1: Choose an OS to be used.
Either CentOS or Ubuntu OS is acceptable to use for on-prem. When we checked the OSes used by the cloud providers, we noted they used one of these two (2) OS for Linux OS. We decided on CentOS 7, as iptables for routing are used within Kubernetes; and iptables are used by default in CentOS 7. You may find that other OSes will work fine as well.
Decision 2: Build a reference image
Identify all expected binaries to be used within this image. This reference image will be cloned for the Kubernetes control plane node (1) and the worker nodes (3-4). We will also use this image to build a supporting node (non-Kubernetes) for SiteMinder integration and a docker repository for the Kubernetes docker images. For a total of six (6) nodes.
Decision 3: DNS and Certificates
Recommendation: Please do not attempt to deploy a Kubernetes solution on-prem without having purchased a DNS domain/site and use wild card certificates tied to the DNS domain.
Without these two (2) supporting components, it is a challenge to have a working Kubernetes solution that reflects what you will experience in a cloud deployment.
For example, we purchased a domain for $12/year, and then created several “A” records that will host the IP addresses we may use to redirect to cloud or on-prem. Using sub-domains “A” records, we can have as many cloud addresses as we wish.
DNS "A" Records Example:
aks.iam.anapartner.net (MS Azure),
eks.iam.anapartner.net (Amazon),
gke.iam.anapartner.net (Google).
DNS "CNAME" Records Example:
alertmanager.aks.iam.anapartner.net,
grafana.aks.iam.anapartner.net,
jaeger.aks.iam.anapartner.net,
kibana.aks.iam.anapartner.net,
mgmt-ssp.aks.iam.anapartner.net,
sm.aks.iam.anapartner.net,
ssp.aks.iam.anapartner.net.
Example of using Synology DNS Server for Kubernetes cluster’s application. With “A” and “CNAME” records.
Finally, we prefer to use wildcard certificates for these domains to avoid challenges within our Kubernetes deployment. There are several services out there offering free certificates.
We chose Let’sEncrypt https://letsencrypt.org/. While Let’sEncrypt has automated processes to renew their certs, we chose to use their DNS validation process with a CertBot solution. We can renew these certificates every 90 days for on-prem usage. The DNS validation process requires a unique string generated by the Let’sEncrypt process to be populated in a DNS “TXT” record like so: _acme-challenge.aks.iam.anapartner.net . See the example at the bottom of this blog entry on this process.
Decision 4: Supporting Components: Storage, Load-Balancing, DNS Resolution (Local)
The last step required for on-prem deployment is where will you decide to place persistence storage for your Kubernetes cluster. We chose to use an NFS share.
We first tested using the control-plane node, then decided to move the NFS share to a Synology NAS solution. Similar for the DNS resolution option, at first we used a DNS service on the control-plane node and then moved to to the Synology NAS solution.
For Load-Balancing, Kubernetes has a service option of NodePort and LoadBalancing. The LoadBalancing service if not deployed in the cloud, will default to NodePort behavior. To introduce load balancing for on-prem, we introduced the HA-proxy service on the control-plane node, along with Kubernetes NodePort service to meet this goal.
After the decisions have been made, we can now walk through the steps to set up a Vmware environment for Kubernetes.
Reference Image
Step 1: Download the OS DVD ISO image for deployment on Vmware Workstation (Centos 7 / Ubuntu ).
Determine specs for the future solution to be deployed on Kubernetes. Some solutions have pods that may require minimal memory/disc space. For the solution we decided on deploying, we confirmed that we need 16 GBRAM and 4vCPU minimal. We have confirmed these specs were required by previously deploying the solution in a cloud environment.
Without these memory/cpu specs, the solution that we chose would pause the deployment of Kubernetes pods to the nodes. You may or may not see error messages in the deployment of pods stating that the nodes did not have enough resources for all or some of the pods.
For disc size, we selected 100 GB to future-proof the solution during testing. For networking, please select BRIDGED mode, to allow the Vmware images to have minimal network issues when routing within your local network. Please avoid double NAT’ing the deployment to reduce your headaches.
Step 2: Install useful base packages and disable any UI tools. Please install an Entropy Daemon to avoid delays due to certificates usage of /dev/random and low entropy.
### UI Update for CentOS7 was stopping yum deployment - not required for our solution to be tested (e.g. VIP Auth Hub)
# su to root to run the below commands. We will add sudo access later.
su -
systemctl disable packagekit; systemctl stop packagekit; systemctl status packagekit
### Installed base useful packages.
yum -y install dnf epel-release yum-utils nfs-utils
### Install useful 2nd tools.
yum -y install openldap-clients jq python3-pip tree
pip3 install yq
yum -y upgrade
### Install Entropy process (epel repo)
dnf -y install haveged
systemctl enable haveged --now
Step 3: Install docker and update the docker configuration for use with Kubernetes. Update the path & storage-driver for the docker images for initial deployment.
### Install Docker repo & docker package
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf -y install docker-ce
docker version
systemctl enable docker --now
docker version
### Update docker image info after deployment and restart service
cat << EOF > /etc/docker/daemon.json
{
"debug": false,
"data-root": "/home/docker-images",
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
### Restart docker to load updated image info.
systemctl restart docker; systemctl status docker; docker version
Step 4: Deploy the three (3) primary Kubernetes & the HELM binaries.
Ensure you select a Kubernetes version that matches what solution you wish to deploy and work with. This can be a gotcha if the Kubernetes binaries update during a dnf / yum upgrade process and your solution has not been vetted for the newer release of Kubernetes. See the reference link below on how to upgrade Kubernetes binaries.
### Stop FirewallD - May add ports later for security
systemctl stop firewalld;systemctl disable firewalld; iptables -F
### Update OS Parameters for kubernetes
setenforce 0
sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
modprobe br_netfilter
cat << EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
### Note: IP forwarding is enabled by default.
sysctl -a | grep -i forward
### Note: Update /etc/fstab to comment out swap line with # character
### Warning: kubectl init will fail if swap is left on cp or any worker node.
swapoff -a
sed -i 's|UUID\=\(.*\)-\(.*\)-\(.*\)-\(.*\)-\(.*\) swap|#UUID\=\1-\2-\3-\4-\5 swap|g' /etc/fstab
cat /etc/fstab
Step 6: Create SSH key for root or other services IDs to allow remote script updates from CP to Worker Nodes
### Create SSH key for root to allow remote script updates from CP to Worker Nodes - Enter a Blank/Null PASSWORD.
su -
rm -rf ~/.ssh; echo y | ssh-keygen -b 4096 -C $USER -f ~/.ssh/id_rsa
### Copy the public rsa key to authorized keys to avoid password between cp/worker nodes for remote ssh commands.
cp -r -p ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys;chmod 600 ~/.ssh/authorized_keys;ls -lart .ssh
### Test for remote connection with no password:
ssh -i ~/.ssh/id_rsa root@localhost
### Copy the id_rsa key to your host system for ease of testing.
### Add your local non-root user to sudo wheel group. Change vip to your user ID.
LOCALUSER=vip
gpasswd -a $LOCALUSER wheel
### Update sudoers file to allow wheel group with no-password
sed -i 's|# %wheel|%wheel|g' /etc/sudoers
### View update wheel group.
grep "%wheel" /etc/sudoers
# Example of return query.
# %wheel ALL=(ALL) ALL
# %wheel ALL=(ALL) NOPASSWD: ALL
Step 7: Stop or adjust the OS network manager, shutdown the reference image, and create a Vmware Snapshot
### Adjust or Disable the OS NetworkManager (to avoid overwriting /etc/resolv.conf)
### Important when using an internal DNS server.
systemctl disable NetworkManager;systemctl stop NetworkManager
### reboot CentOS7 Image and validate no issues upon reboot.
reboot
### Shutdown image and manually create snapshot called "base"
Vmware Workstation Cloning
Step 8: Now that we have a reference image, we can now make clone images for the control-plane (1), the worker nodes (4), and the supporting node (1). This is a fairly quick process.
export BASE=/home/me/vmware/kub
export REF=/home/me/vmware/kub/CentOS7/CentOS7.vmx
VM=cp;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=worker01;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=worker02;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=worker03;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=worker04;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
VM=sm;mkdir -p $BASE/$VM; time vmrun -T ws clone $REF $BASE/$VM/$VM.vmx -cloneName=$VM -snapshot=base full
Step 9: Start the clone images and remotely assign new hostname/IP addresses to the images
Step 12: Copy the root .ssh public cert to your main host, rename it to a useful name and these test your newly deployed clone images for DSN resolution using ssh. Please confirm this step is successful prior to continuing with the configuration of the control plane and worker nodes.
Step 13a: Copy files to CP Node from Vmware Workstation host and configure the CP node for dedicated CP usage. Recommend using two terminals/sessions to speed up the process. Install HAproxy for Load Balancing, copy the Let’s Encrypt wild card certificates, and copy the Kubernetes solution you will be deploying (scripts/yaml).
### Open Terminal 1 to CP host.
### Add bash completion to have better use of TAB to view parameters.
CP=192.168.2.60
ssh -tt -i ~/vip_kub_root_id_rsa root@$CP
dnf -y install bash-completion
echo 'export KUBECONFIG=/etc/kubernetes/admin.conf' >>~/.bashrc
kubectl completion bash >/etc/bash_completion.d/kubectl
echo "alias k=kubectl | complete -F __start_kubectl k" >>~/.bashrc
### Install HAProxy and replace the haproxy.cfg file.
dnf -y install haproxy
systemctl enable haproxy --now
netstat -anp | grep -i -e haproxy
### Open Terminal 2 to host and push files to CP node.
### Copy HAProxy configuration, certs, and scripts
scp -i ~/vip_kub_root_id_rsa haproxy.cfg root@$CP:/etc/haproxy/haproxy.cfg
scp -i ~/vip_kub_root_id_rsa cloud-certs-aks-eks-gke_exp-202X-01-12.tar root@$CP:
scp -i ~/vip_kub_root_id_rsa 202X-11-03_vip_auth_hub_working_centos7_v2.tar root@$CP:
### On Terminal 1 - on CP host - Restart to use new haproxy configuration file.
systemctl restart haproxy
netstat -anp | grep -i -e haproxy
### Extract CERTS to root home folder
tar -xvf cloud-certs-aks-eks-gke_exp-202X-01-12.tar
### Extract Working Scripts
tar -xvf 202X-11-03_vip_auth_hub_working_centos7_v2.tar
### Update env variables for unique environment within step00 file.
vi step00_kubernetes_env.sh
### Add the env variables to the .bashrc file
echo ". ./step00_kubernetes_env.sh"
Step 13b: Example of /etc/haproxy/haproxy.cfg configuration for Kubernetes Load Balancing functionality for on-prem worker nodes. HAproxy deployed on control plane (CP) node. The example configuration file will route TCP 80/443/389 to one (1) of the four (4) worker nodes. If a Kubernetes NodePort service is enabled for TCP 389 (31888) ports, then this load balancer will function correctly and route the traffic for LDAP traffic as well.
[root@cp ~]# cat /etc/haproxy/haproxy.cfg
global
user haproxy
group haproxy
chroot /var/lib/haproxy
log /dev/log local0
log /dev/log local1 notice
defaults
mode http
log global
retries 2
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 10m
timeout server 10m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
frontend ingress
bind *:80
option tcplog
mode http
option forwardfor
option http-server-close
default_backend kubernetes-ingress-nodes
backend kubernetes-ingress-nodes
mode http
balance roundrobin
server k8s-ingress-0 worker01.aks.iam.anapartner.net:80 check fall 3 rise 2 send-proxy-v2
server k8s-ingress-1 worker02.aks.iam.anapartner.net:80 check fall 3 rise 2 send-proxy-v2
server k8s-ingress-2 worker03.aks.iam.anapartner.net:80 check fall 3 rise 2 send-proxy-v2
server k8s-ingress-2 worker04.aks.iam.anapartner.net:80 check fall 3 rise 2 send-proxy-v2
frontend ingress-https
bind *:443
option tcplog
mode tcp
option forwardfor
option http-server-close
default_backend kubernetes-ingress-nodes-https
backend kubernetes-ingress-nodes-https
mode tcp
balance roundrobin
server k8s-ingress-0 worker01.aks.iam.anapartner.net:443 check fall 3 rise 2 send-proxy-v2
server k8s-ingress-1 worker02.aks.iam.anapartner.net:443 check fall 3 rise 2 send-proxy-v2
server k8s-ingress-2 worker03.aks.iam.anapartner.net:443 check fall 3 rise 2 send-proxy-v2
server k8s-ingress-2 worker04.aks.iam.anapartner.net:443 check fall 3 rise 2 send-proxy-v2
frontend ldap
bind *:389
option tcplog
mode tcp
default_backend kubernetes-nodes-ldap
backend kubernetes-nodes-ldap
mode tcp
balance roundrobin
server k8s-ldap-0 worker01.aks.iam.anapartner.net:31888 check fall 3 rise 2
server k8s-ldap-1 worker02.aks.iam.anapartner.net:31888 check fall 3 rise 2
server k8s-ldap-2 worker03.aks.iam.anapartner.net:31888 check fall 3 rise 2
server k8s-ldap-2 worker04.aks.iam.anapartner.net:31888 check fall 3 rise 2
Deploy Solution on Kubernetes
Step 14: Validate that DNS and Storage are ready before deploying any solution or if you wish to have a base Kubernetes environment to use with the control-plane and four (4). worker nodes.
### Step: Setup NFS Share either on-prem remote server or Synology NFS
### Use version 4.x checkbox for Synology.
### Example of lines on remote Linux Host with NFS share.
yum -y install nfs-utils
systemctl enable --now nfs-server rpcbind
mkdir -p /export/nfsshare ; chown nobody /export/nfsshare ; chmod -R 777 /export/nfsshare
echo "/export/nfsshare *(rw,sync,no_root_squash,insecure)" >> /etc/exports
exportfs -rav; exportfs -v
firewall-cmd --add-service=nfs --permanent
firewall-cmd --add-service={nfs3,mountd,rpc-bind} --permanent
firewall-cmd --reload
#### Setup DNS entries (A and CNAME) for twelve (12) items ( May be on-prem DNS or Synology DNS)
ns.aks.iam.anapartner.net A IP_ADDRESS (192.168.2.60)
aks.iam.anapartner.net NS ns.aks.iam.anapartner.net
cp.aks.iam.anapartner.net A IP_ADDRESS (192.168.2.60)
worker01.aks.iam.anapartner.net A IP_ADDRESS (192.168.2.61)
worker02.aks.iam.anapartner.net A IP_ADDRESS (192.168.2.62)
worker03.aks.iam.anapartner.net A IP_ADDRESS (192.168.2.63)
worker04.aks.iam.anapartner.net A IP_ADDRESS (192.168.2.64)
sm.aks.iam.anapartner.net A IP_ADDRESS (192.168.2.65)
kibana CNAME cp.aks.iam.anapartner.net
grafana CNAME cp.aks.iam.anapartner.net
jaeger CNAME cp.aks.iam.anapartner.net
alertmanager CNAME cp.aks.iam.anapartner.net
ssp CNAME cp.aks.iam.anapartner.net
ssp-mgmt CNAME cp.aks.iam.anapartner.net
### Pre-Step: Enable DNS resolution for external IP addresses
### Enable forwarding to external h/w router and 8.8.8.8
Step 15: Recommendation. Deploy your solution in steps using Kubernetes yaml or Helm charts to assist with debugging any deployment issues. Do not forget to use kubectl logs, and kubectl describe to isolate startup or cert issues.
### Run scripts one-by-one. They will have a watch command in each that will
### provide feedback on the startup processes.
### Total startup from scratch to final with VIP Sample App is about 15-20 minutes.
### Note: Step04 has a different chart variables for on-prem for Symantec Directory.
### Note: /step00_kubernetes_env.sh is called by each script.
./step01_kubernetes_cluster_init_with_worker_nodes.sh
./step02_kubernetes_cluster_with_ingress_and_other_charts.sh
./step03_kubernetes_cluster_with_vip_auth_hub_charts.sh
./step04_kubernetes_cluster_with_vip_auth_hub_sample_app.sh
Docker Registry for On-Prem
There are two (2) types of docker registries we have found useful.
a. The standard Mirror method will capture all docker images from “docker.io” site to a local mirror. When Kubernetes or Helm deployments are used, the docker configuration file can be adjusted to check the local mirror without updating Kubernetes yaml files or Helm charts.
b. The second method is a full query of all images after they have been deployed once, and using the docker push process into a local registry. The challenge of the second method is that the Kubernetes yaml files and/or Helm charts do have to be updated to use this local registry.
Either method will help lower bandwidth cost to re-download the same docker images, if you use a docker prune method to keep your worker nodes disc size “clean”. If the docker prune process is not used, you may notice that the worker nodes may run out of disc space due to temporary docker images/containers that did not clean up properly.
#!/bin/bash
#################################################################################
# Create a local docker mirror registry for docker-ios
# and local docker non-mirror registry for all other images
# to minimize download impact
# during restart of the kubernetes solution
#
# All registry iamges will be placed on NFS share
# mount -v -t nfs 192.168.2.30:/volume1/nfs /mnt &>/dev/null
#
# Certs will be provided by Let's Encrypt every 90 days
#
# For docker-io mirror registry, all clients must have the following line in
# /etc/docker/daemon.json {Note: Use commas as needed}
#
# "registry-mirrors":
# [
# "https://sm.aks.iam.anapartner.net:444"
# ],
#
#
#
# ANA 11/2021
#
#################################################################################
# To remove all containers - to allow restart of process
docker rm -f `docker ps -a | grep -v -e CONTAINER | awk '{print $1}'` ; docker image rm `docker image ls | grep -v -e REPOSITORY | grep -e minutes -e hour -e days -e '2 weeks'| awk '{print $3}'` &>/dev/null
#################################################################################
# Update HOST name for local server for docker image
HOST=sm.aks.iam.anapartner.net
NFS_SERVER=192.168.2.30
NFS_SHARE=/volume1/nfs
#################################################################################
function start_registry {
local_port=$1
remote_registry_name=$2
if [ "$3" == "" ]; then
remote_registry_url=$remote_registry_name
else
remote_registry_url=$3
fi
echo -e "$local_port $remote_registry_name $remote_registry_url"
mount -v -t nfs $NFS_SERVER:$NFS_SHARE /mnt &>/dev/null
mkdir -p /mnt/registry/${remote_registry_name} &>/dev/null
docker run -d --name registry-${remote_registry_name}-mirror \
-p $local_port:443 \
--restart=always \
-e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
-e REGISTRY_PROXY_REMOTEURL="https://${remote_registry_url}/" \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/fullchain.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/privkey.pem \
-e REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED=true \
-v /mnt/registry/certs:/certs \
-v /mnt/registry/${remote_registry_name}:/var/lib/registry \
registry:latest
sleep 1
echo "#################################################################################"
curl -s -X GET https://$HOST:$local_port/v2/_catalog | jq
echo "#################################################################################"
}
#################################################################################
# start_registry <local_port> <remote_registry_name> <remote_registry_url>
#################################################################################
start_registry 444 docker-io registry-1.docker.io
#################################################################################
# Non-Proxy configuration to allow 'docker tag & docker push' for all other images
#################################################################################
remote_registry_name=all
local_port=455
mkdir -p /var/lib/docker/registry/${remote_registry_name} &>/dev/null
docker run -d --name registry-${remote_registry_name}-mirror \
-p $local_port:443 \
--restart=always \
-e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
-e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/fullchain.pem \
-e REGISTRY_HTTP_TLS_KEY=/certs/privkey.pem \
-e REGISTRY_COMPATIBILITY_SCHEMA1_ENABLED=true \
-v /mnt/registry/certs:/certs \
-v /mnt/registry/${remote_registry_name}:/var/lib/registry \
registry:latest
sleep 1
echo "#################################################################################"
curl -s -X GET https://$HOST:$local_port/v2/_catalog | jq
echo "#################################################################################"
docker ps -a
echo "#################################################################################"
echo "##### To tail the log of the docker-io container - useful for monitoring helm deployments #####"
echo "docker logs `docker ps -a --no-trunc | grep -v NAMES | grep 'docker-io' | awk '{print $1}'` -f "
echo "#################################################################################"
echo "##### To tail the log of the ALL container - useful for monitoring helm deployments #####"
echo "docker logs `docker ps -a --no-trunc | grep -v NAMES | grep 'all' | awk '{print $1}'` -f "
echo "#################################################################################"
echo "##### Location of Registry Files on NFS share #####"
echo "ls -lart /mnt/registry/docker-io/docker/registry/v2/repositories"
echo "ls -lart /mnt/registry/all/docker/registry/v2/repositories"
echo "#################################################################################"
Example of the /etc/docker/daemon.json configuration file to use a local mirror for docker.io. See the parameter of “registry-mirrors”. Unfortunately, we were unable to use this process for the other docker registries.
Use Let’sEncrypt Certbox and manual DNS validation, to create our 90-day wild card certificates. Manual DNS validation allows us to avoid setting up a public-facing component for our internal labs.
# Step 1: Install SNAP service for Certbot usage on your host OS
cat /etc/redhat-release
Red Hat Enterprise Linux release 8.3 (Ootpa)
sudo yum install -y snapd
Updating Subscription Management repositories.
Package snapd-2.49-2.el8.x86_64 is already installed.
systemctl enable --now snapd.socket
### Wait 1 min
snap install core; sudo snap refresh core
# Step 2: Remove prior certbot (if installed by yum/dnf)
yum remove -y certbot.
# Step 3: Install new "classic" Certbot
sudo snap install --classic certbot
certbot 1.17.0 from Certbot Project (certbot-eff✓) installed
sudo ln -s /snap/bin/certbot /usr/bin/certbot
# Step 4: Issue certbot command with wildcard cert & update your DNS TXT record with the string provided.
sudo certbot certonly --manual --preferred-challenges dns -d *.aks.iam.anapartner.org --register-unsafely-without-email
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.
Requesting a certificate for *.aks.iam.anapartner.org
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please deploy a DNS TXT record under the name:
_acme-challenge.iam.anapartner.org.
with the following value:
u2cXXXXXXXXXXXXXXXXXXXXc
Before continuing, verify the TXT record has been deployed. Depending on the DNS
provider, this may take some time, from a few seconds to multiple minutes. You can
check if it has finished deploying with aid of online tools, such as the Google
Admin Toolbox: https://toolbox.googleapps.com/apps/dig/#TXT/_acme-challenge.iam.anapartner.org.
Look for one or more bolded line(s) below the line ';ANSWER'. It should show the
value(s) you've just added.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Step 5: In a 2nd terminal, validate that the DNS record has been updated and can be seen by a standard DNS query. Have the 2nd console window open to test the DNS record, prior to <ENTER> key on verification request
# Example:
nslookup -type=txt _acme-challenge.aks.iam.anapartner.org
Non-authoritative answer:
_acme-challenge.aks.iam.anapartner.org text = "u2cXXXXXXXXXXXXXXXXXXXXc"
# Step 6: Press <ENTER> after you have validated the TXT record.
Press Enter to Continue
Waiting for verification...
Cleaning up challenges
Subscribe to the EFF mailing list (email: nala@baugher.us).
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/aks.iam.anapartner.org/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/aks.iam.anapartner.org/privkey.pem
# Step 7: View certs of fullchain.pem & privkey.pem
cat /etc/letsencrypt/live/aks.iam.anapartner.org/fullchain.pem
-----BEGIN CERTIFICATE-----
<REMOVED>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<REMOVED>
-----END CERTIFICATE-----
cat /etc/letsencrypt/live/aks.iam.anapartner.org/privkey.pem
-----BEGIN PRIVATE KEY-----
<REMOVED>
-----END PRIVATE KEY-----
# Step 8: Use the two files for your kubernetes solution
# Step 9: Ensure domain on host OS, cp, worker nodes in /etc/resolv.conf is set correctly to aks.iam.anapartner.org to allow the certs to be resolved correctly.
# Step 10: Ensure Synology NAS DNS service is configurated with all alias
# Step 11: Optional: Validate certs with openssl
# Show the kubernetes self-signed cert
true | openssl s_client -connect kibana.aks.iam.anapartner.org:443 2>/dev/null | openssl x509 -inform pem -noout -text
# Show the new wildcard cert for same hostname & port
curl -vvI https://kibana.aks.iam.anapartner.org/app/home#/curl -vvI https://kibana.aks.iam.anapartner.org/app/home#/ 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
nmap -p 443 --script ssl-cert kibana.aks.iam.anapartner.org
Kubernetes Side Note: Let's Encrypt certs do NOT show up within the Kubernetes cluster certs check process.
kubeadm certs check-expiration
View of the DNS TXT records to be updated with your DNS service provider. The Let’sEncrypt Certbot will need to be able to query these records for it to assign you wildcard certificates. Create the _acme-challenge hostname entry as a TXT type, and paste in the string provided by the Let’sEncrypt Certbot process. Wait 5 minutes or test the TXT record with nslookup, then upon positive validation, continue the Let’sEncrypt Certbot process.
View your kubernetes cluster / nodes for any constraints
After your cluster is created and you have worker nodes joined to the cluster, you may wish to monitor for any constraints of your on-prem deployment. Kubectl command with the action verb of describe or top is very useful for this goal.
kubectl describe nodes worker01kubectl top node / kubectl top pod
Kubernetes Training (Formal)
If you are new to Kubernetes, we recommend the following class. You may need to dedicate 4-8 weeks to complete the course and then take the CKA exam via the Linux Foundation.
On Linux OS, there are two (2) device drivers that provide entropy “noise” for components that require encryption, e.g. the /dev/random and the /dev/urandom device drivers. The /dev/random is a “blocking” device driver. When the “noise” is low, any component that relies on this driver will be “stalled” until enough entropy is returned. We can measure the entropy from a range of 0-4096. Where a value over 1000 is excellent. Any value in the double or single digits will impact the performance of the OS and solutions with delays. The root cause of these delays is not evident during troubleshooting, and typically there are no warning nor error messages related to entropy.
The Symantec Identity Suite solution, when deployed on Linux OS is typically deployed with the JVM switch -Djava.security.egd=file:/dev/./urandom for any component that uses Java (Oracle or AdoptOpenJDK), e.g. Wildfly (IM/IG/IP) and IAMCS (JCS). This JVM variable is sufficient for most use-cases to manage the encryption/hash needs of the solution.
However, for any component that does not provide a mechanism to use the alternative of /dev/urandom driver, the Linux OS vendors offer tools such as the “rng-tools” package. We can review what OS RNGD service is available using package tools, e.g.
dnf list installed | grep -i rng
If the Symantec Identity Suite or other solutions are deployed as standalone components, then we may adjust the Linux OS as we need with no restrictions to add the RNGD daemon as we wish. One favorite is the HAVEGED daemon over the default OS RNGD.
See prior notes on value and testing for Entropy on Linux OS (standalone deployments):
The challenge for Virtual Appliances is that we are limited to what functionality the Symantec Product Team provides for us to leverage. The RNGD service was available on the vApp r14.3, but was disabled for OS challenges with 100% utilization with CentOS 6.4. The service is still installed, but the actual binary is non-executable.
A new Virtual Appliance patch would be required to re-enable this RNGD on vApp r14.3cp2. We have access via sudo, to /sbin/chkconfig, /sbin/service to re-enable this service, but as the binary is not executable, we cannot progress any further. We can see the alias in the documentation still exist, but the OS alias was removed in the cp2 update.
However, since vApp r14.4 was release, we can focus on this Virtual Appliance which is running Centos 8 stream. The RNGD service here is disabled (masked) but can be re-enabled for our use with the sudo command. There is no current documented method for RNGD on vApp r14.4 at this time, but the steps below will show an approved way using the ‘config’ userID and sudo commands.
Confirm that the “rng-tools” package is installed and that the RNGD binary is executable. We can also see that the RNGD service is “masked”. Masked services are prevented from starting manually or automatically as an extra safety measure when we wish for tighter control over our systems.
If we test OS entropy for this vApp r14.4 server without RNGD, we can monitor how a simple BASH shell script that emulates a password being generated will impact the “entropy” of /dev/random. The below script will reduce the entropy to low numbers. This process will now impact the OS itself and any components that reference /dev/random. We can observe with “lsof /dev/random” that the java programs will still reference /dev/random; even though most activity is going to /dev/urandom.
Using the time command in the BASH shell script, we can see that the response is rapid for the first 20+ iterations, but as soon as the entropy is depleted, each execution is delayed by 10-30x times.
Now let’s see what RNGD service will do for us when it is enabled. Let’s follow the steps below to unmask, enable, and start the RNGD service as the ‘config’ userID. We have access to sudo to the Centos 8 Stream command of /sbin/systemctl.
After the RNGD service is enabled, test again with the same prior BASH shell script but bump the loops to 1000 or higher. Note using the time command we can see that each loop finishes within a fraction of a second.
Aim to keep the solution footprint small and the right-sized to solve the business’ needs. Do not accept the default performance; avoid over-purchasing to scale to your expected growth.
Use the JVM switch wherever there is a java process, e.g. BLC or home-grown ETL (extract-transform-load) processes.
-Djava.security.egd=file:/dev/./urandom
If you suspect a dependence may impact the OS or other processes on /dev/random, then enable the OS RNGD and perform your testing. Monitor with the top command to ensure RNGD service is providing value and not impacting the solution.
One business risk to manage when new business logic is being promoted to production environments is how to plan for a rollback process, where prior state data is restored, especially for an application/endpoint that is critical for a business; and as important to users as their login credentials and access.
In this entry, we showcase how to use CA Directory to snapshot an endpoint on a scheduled basis (daily/hourly) and have the process prepare a rollback delta file for user’s entitlements.
Understanding how queries may be direct to an endpoint/application or via the CA Identity Manager provisioning tier, we can speed up this process rapidly for sites that have millions of identities in an endpoint.
#!/bin/bash
##############################################################################
#
# POC to demostrate process to snapshot endpoint data on a daily basis
# and to allow a format for roll back
#
# 1. Review ADS with dxsearch/dxmodify
# 2. Create ADS representative Router DSA with CA Directory
# 3. Create ldif delta of snapshot data
# 4. Convert 'replace' to 'add' to ensure Roll back process is a 'merge'
# and NOT an 'overwrite' of entitlements
#
#
#
# A. Baugher, ANA, 11/2019
#
##############################################################################
########## Secure password for script ########
FILE=/tmp/.ads.hash.pwd
#rm -rf $FILE $FILE.salt
[[ -f $FILE ]]
echo "Check if $FILE exists: $?"
[[ -s $FILE ]]
echo "Check if $FILE is populated: $?"
if [[ ! -s $FILE && ! -s $FILE.salt ]]
then
# File did not have any data
# Run script once with pwd then replace with junk data in script
SALT=$RANDOM$RANDOM$RANDOM
PWD=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ENCPWD=$(echo $PWD | openssl enc -aes-256-cbc -a -salt -pass pass:$SALT)
echo $ENCPWD > $FILE
echo $SALT > $FILE.salt
chmod 600 $FILE $FILE.salt
fi
if [[ -s $FILE && -s $FILE.salt ]]
then
ENCPWD=`cat $FILE`
SALT=`cat $FILE.salt`
echo "$PWD and $SALT for $ENCPWD"
MYPWD=$(echo "$ENCPWD" | openssl enc -aes-256-cbc -a -d -salt -pass pass:$SALT)
echo "$PWD and $SALT for $MYPWD"
else
echo "Missing password encrypted data and salt"
exit 1
fi
#exit
echo ""
echo "##############################################################################"
echo "Step 0 # Remove prior ads schema files"
echo "##############################################################################"
ADS_SCHEMA=ads_schema
ADS_SUFFIX="dc=exchange,dc=lab"
RANDOM_PORT=50389
rm -rf $DXHOME/config/knowledge/$ADS_SCHEMA.dxc
rm -rf $DXHOME/config/servers/$ADS_SCHEMA.dxi
rm -rf $DXHOME/config/schema/$ADS_SCHEMA.dxc
echo ""
echo "##############################################################################"
echo "Step 1 # Create new router DSA"
echo "##############################################################################"
echo "dxnewdsa -t router $ADS_SCHEMA $RANDOM_PORT $ADS_SUFFIX"
dxnewdsa -t router $ADS_SCHEMA $RANDOM_PORT $ADS_SUFFIX
echo""
echo "##############################################################################"
echo "Step 2 # Create temporary LDIF file of ADS schema"
echo "##############################################################################"
cd $DXHOME/config/schema
ADS_BIND_DN="CN=Administrator,CN=Users,DC=exchange,DC=lab"
ADS_BIND_PWD=$MYPWD
ADS_PASSFILE=/tmp/.ads.pwd
echo -n $MYPWD > $ADS_PASSFILE
chmod 600 $ADS_PASSFILE
ADS_SERVER=dc2016.exchange.lab
ADS_PORT=389
echo "dxschemaldif -v -D $ADS_BIND_DN -w ADS_BIND_PASSWORD_HERE $ADS_SERVER:$ADS_PORT > $ADS_SCHEMA.ldif"
dxschemaldif -v -D $ADS_BIND_DN -w $ADS_BIND_PWD $ADS_SERVER:$ADS_PORT > $ADS_SCHEMA.ldif
echo ""
echo "##############################################################################"
echo "Step 3 # Replace unknown SYNTAX with closely related SYNTAX known by CA Directory r12.6.5"
echo "##############################################################################"
echo "sed -i 's|1.2.840.113556.1.4.1221|1.3.6.1.4.1.1466.115.121.1.26|g' $ADS_SCHEMA.ldif"
sed -i 's|1.2.840.113556.1.4.1221|1.3.6.1.4.1.1466.115.121.1.26|g' $ADS_SCHEMA.ldif
echo ""
echo "##############################################################################"
echo "Step 4 - # Create CA Directory Schema DXC File from LDIF Schema File"
echo "##############################################################################"
echo "ldif2dxc -f $ADS_SCHEMA.ldif -b bad.ldif -x default.dxg -v $ADS_SCHEMA.dxc"
ldif2dxc -f $ADS_SCHEMA.ldif -b bad.ldif -x default.dxg -v $ADS_SCHEMA.dxc
echo ""
echo "##############################################################################"
echo "Step 5 - # Update router DSA schema reference"
echo "##############################################################################"
echo "sed -i \"s|source \"../schema/default.dxg\";|source \"../schema/default.dxg\";\nsource \"../schema/$ADS_SCHEMA.dxc\"; |g\" $DXHOME/config /servers/$ADS_SCHEMA.dxi"
sed -i "s|source \"../schema/default.dxg\";|source \"../schema/default.dxg\";\nsource \"../schema/$ADS_SCHEMA.dxc\"; |g" $DXHOME/config/servers /$ADS_SCHEMA.dxi
echo ""
echo "##############################################################################"
echo "Step 6 - # Query ADS endpoint for snapshot 1 "
echo "##############################################################################"
echo "dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX '(objectClass=User)' memberOf > snapshot_1_ $ADS_SCHEMA.ldif "
echo "ldifsort snapshot_1_$ADS_SCHEMA.ldif snapshot_1_sorted_$ADS_SCHEMA.ldif "
dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX "(objectClass=User)" memberOf | perl -p00e 's/\r?\ n //g' > snapshot_1_$ADS_SCHEMA.ldif
ldifsort snapshot_1_$ADS_SCHEMA.ldif snapshot_1_sorted_$ADS_SCHEMA.ldif
echo ""
echo "##############################################################################"
echo "Step 7 - # Query ADS endpoint for snapshot 2"
echo "##############################################################################"
echo "dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX '(objectClass=User)' memberOf > snapshot_2_ $ADS_SCHEMA.ldif "
echo "ldifsort snapshot_2_$ADS_SCHEMA.ldif snapshot_2_sorted_$ADS_SCHEMA.ldif "
dxsearch -LLL -h $ADS_SERVER -p $ADS_PORT -x -D $ADS_BIND_DN -y $ADS_PASSFILE -b $ADS_SUFFIX "(objectClass=User)" memberOf | perl -p00e 's/\r?\ n //g' > snapshot_2_$ADS_SCHEMA.ldif
ldifsort snapshot_2_$ADS_SCHEMA.ldif snapshot_2_sorted_$ADS_SCHEMA.ldif
echo ""
echo "##############################################################################"
echo "Step 8 - # Find the delta for any removed objects"
echo "##############################################################################"
echo "ldifdelta -x -S $ADS_SCHEMA snapshot_2_sorted_$ADS_SCHEMA.ldif snapshot_1_sorted_$ADS_SCHEMA.ldif"
ldifdelta -x -S $ADS_SCHEMA snapshot_2_sorted_$ADS_SCHEMA.ldif snapshot_1_sorted_$ADS_SCHEMA.ldif
echo ""
echo "##############################################################################"
echo "Step 9a: Convert from User ldapmodify syntax of 'overwrite' of 'replace' "
echo "##############################################################################"
ldifdelta -S $ADS_SCHEMA snapshot_2_sorted_$ADS_SCHEMA.ldif snapshot_1_sorted_$ADS_SCHEMA.ldif user_mod_syntax_input.ldif >/dev/null 2>&1
cat user_mod_syntax_input.ldif | perl -p00e 's/\r?\n //g' > user_mod_syntax.ldif
cat user_mod_syntax.ldif
echo "##############################################################################"
echo "Step 9b: Convert to ADS Group ldapmodify syntax with a 'merge' of 'add' for the group objects"
echo "##############################################################################"
perl /opt/CA/Directory/dxserver/samples/dxsoak/convert.pl user_mod_syntax.ldif > group_mod_syntax_input.ldif
cat group_mod_syntax_input.ldif | perl -p00e 's/\r?\n //g' > group_mod_syntax.ldif
cat group_mod_syntax.ldif
echo "##############################################################################"
Example of output from above script:
[dsa@vapp0001]$ ./active_directory_user_delta_via_ca_dir_tools-lab.sh
Check if /tmp/.ads.hash.pwd exists: 0
Check if /tmp/.ads.hash.pwd is populated: 0
/opt/CA/Directory/dxserver/samples/dxsoak and 31936904511291 for U2FsdGVkX195Ti6A8GdFTG6Kmrf6xDcOhrd2aPWVezc=
/opt/CA/Directory/dxserver/samples/dxsoak and 31936904511291 for CAdemo123
20200427150345,505.0Z = Current OS UTC time stamp
##############################################################################
Step 0 # Remove prior ads schema files
##############################################################################
20200427150345,509.0Z = Current OS UTC time stamp
##############################################################################
Step 1 # Create new router DSA
##############################################################################
dxnewdsa -t router ads_schema 50389 dc=exchange,dc=lab
Writing the knowledge file...
knowledge file written
Writing the initialization file...
Initialization file written
Starting the DSA 'ads_schema'...
ads_schema starting
ads_schema started
20200427150345,513.0Z = Current OS UTC time stamp
##############################################################################
Step 2 # Create temporary LDIF file of ADS schema
##############################################################################
dxschemaldif -v -D CN=Administrator,CN=Users,DC=exchange,DC=lab -w ADS_BIND_PASSWORD_HERE dc2016.exchange.lab:389 > ads_schema.ldif
>> Issuing LDAP v3 synchronous bind to 'dc2016.exchange.lab:389'...
>> Fetching root DSE 'subschemaSubentry' attribute...
>> Downloading schema from 'CN=Aggregate,CN=Schema,CN=Configuration,DC=exchange,DC=lab'...
>> Received (4527) values
>> Done.
20200427150345,539.0Z = Current OS UTC time stamp
##############################################################################
Step 3 # Replace unknown SYNTAX with closely related SYNTAX known by CA Directory r12.6.5
##############################################################################
sed -i 's|1.2.840.113556.1.4.1221|1.3.6.1.4.1.1466.115.121.1.26|g' ads_schema.ldif
20200427150345,560.0Z = Current OS UTC time stamp
##############################################################################
Step 4 - # Create CA Directory Schema DXC File from LDIF Schema File
##############################################################################
ldif2dxc -f ads_schema.ldif -b bad.ldif -x default.dxg -v ads_schema.dxc
>> Opening input file 'ads_schema.ldif' ...
>> Opening existing dxserver schema file '/opt/CA/Directory/dxserver/config/schema/default.dxg' ...
>> Opening bad file 'bad.ldif' ...
>> Opening output file '/opt/CA/Directory/dxserver/config/schema/ads_schema.dxc' ...
>> Processing dxserver schema group file '/opt/CA/Directory/dxserver/config/schema/default.dxg'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/x500.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/cosine.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/umich.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/inetop.dxc'...
>> Processing dxserver schema config file '/opt/CA/Directory/dxserver/config/schema/dxserver.dxc'...
>> Loaded (248) existing dxserver schema entries
>> Loading LDIF records...
>> Loading LDIF record number (1)...
>> Skipping attr: 'objectClass'
>> Skipping attr: 'objectClass'
>> Processing loaded LDIF records...
>> Moving objectClasses to end of list...
>> Sorting attrs/objectClasses so parents precede their children...
>> Processing attributeTypes...
>> Defaulting 'directoryString' syntax without any (required) matching rules to 'caseIgnoreString'...
[Remove repeating lines x 1000]
>> Processing objectClasses...
>> Skipping existing schema entry 'top' with oid '2.5.6.0'...
>> Skipping existing schema entry 'locality' with oid '2.5.6.3'...
>> Skipping existing schema entry 'device' with oid '2.5.6.14'...
>> Skipping existing schema entry 'certificationAuthority' with oid '2.5.6.16'...
>> Skipping existing schema entry 'groupOfNames' with oid '2.5.6.9'...
>> Skipping existing schema entry 'organizationalRole' with oid '2.5.6.8'...
>> Skipping existing schema entry 'organizationalUnit' with oid '2.5.6.5'...
>> Skipping existing schema entry 'domain' with oid '1.2.840.113556.1.5.66'...
>> Skipping existing schema entry 'rFC822LocalPart' with oid '0.9.2342.19200300.100.4.14'...
>> Skipping existing schema entry 'applicationProcess' with oid '2.5.6.11'...
>> Skipping existing schema entry 'document' with oid '0.9.2342.19200300.100.4.6'...
>> Skipping existing schema entry 'room' with oid '0.9.2342.19200300.100.4.7'...
>> Skipping existing schema entry 'domainRelatedObject' with oid '0.9.2342.19200300.100.4.17'...
>> Skipping existing schema entry 'country' with oid '2.5.6.2'...
>> Skipping existing schema entry 'friendlyCountry' with oid '0.9.2342.19200300.100.4.18'...
>> Skipping existing schema entry 'groupOfUniqueNames' with oid '2.5.6.17'...
>> Skipping existing schema entry 'organization' with oid '2.5.6.4'...
>> Skipping existing schema entry 'simpleSecurityObject' with oid '0.9.2342.19200300.100.4.19'...
>> Skipping existing schema entry 'person' with oid '2.5.6.6'...
>> Skipping existing schema entry 'organizationalPerson' with oid '2.5.6.7'...
>> Skipping existing schema entry 'inetOrgPerson' with oid '2.16.840.1.113730.3.2.2'...
>> Skipping existing schema entry 'residentialPerson' with oid '2.5.6.10'...
>> Skipping existing schema entry 'applicationEntity' with oid '2.5.6.12'...
>> Skipping existing schema entry 'dSA' with oid '2.5.6.13'...
>> Skipping existing schema entry 'cRLDistributionPoint' with oid '2.5.6.19'...
>> Skipping existing schema entry 'documentSeries' with oid '0.9.2342.19200300.100.4.9'...
>> Skipping existing schema entry 'account' with oid '0.9.2342.19200300.100.4.5'...
>> Converting LDIF records to DXserver schema format...
>> Converted (4398) of (4525) schema records
20200427150345,894.0Z = Current OS UTC time stamp
##############################################################################
Step 5 - # Update router DSA schema reference
##############################################################################
sed -i "s|source "../schema/default.dxg";|source "../schema/default.dxg";\nsource "../schema/ads_schema.dxc"; |g" /opt/CA/Directory/dxserver/config/servers/ads_schema.dxi
20200427150345,897.0Z = Current OS UTC time stamp
##############################################################################
step 6 - # Update an ADS account with memberOf for testing with initial conditions
##############################################################################
dxmodify -c -H ldap://dc2016.exchange.lab:389 -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd << EOF >/dev/null 2>&1
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
ldap_modify: Already exists (68)
additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
ldap_modify: Already exists (68)
additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0
adding new entry CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
20200427150345,909.0Z = Current OS UTC time stamp
##############################################################################
Step 7 - # Query ADS endpoint for snapshot 1
##############################################################################
dxsearch -LLL -h dc2016.exchange.lab -p 389 -x -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd -b dc=exchange,dc=lab '(&(objectClass=User)(memberOf=*))' memberOf | perl -p00e 's/\r?\n //g' > snapshot_1_ads_schema.ldif
ldifsort snapshot_1_ads_schema.ldif snapshot_1_sorted_ads_schema.ldif
creating buckets
creating sort cluster 1 of size 200
sorting 0 records
creating sort cluster 2 of size 200
sorting 200 records
creating sort cluster 3 of size 200
sorting 400 records
3 buckets created
sorting 588 records
588 records sorted, 0 bad records
20200427150345,940.0Z = Current OS UTC time stamp
##############################################################################
Step 8 - # Update an ADS account with memberOf for testing after snapshot
##############################################################################
dxmodify -c -H ldap://dc2016.exchange.lab:389 -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd << EOF
Ignore the error msg: DSID-031A1254, problem 5003 (WILL_NOT_PERFORM)
This error will occur if a non-existant value is removed from the group's member attribute
##############################################################################
ldap_initialize( ldap://dc2016.exchange.lab:389 )
delete member:
CN=Test User 001,CN=Users,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
delete member:
CN=eeeee,CN=Users,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
delete member:
CN=Test User 001,CN=Users,DC=exchange,DC=lab
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
ldap_modify: Server is unwilling to perform (53)
additional info: 00000561: SvcErr: DSID-031A1254, problem 5003 (WILL_NOT_PERFORM), data 0
delete member:
CN=alantest,CN=Users,DC=exchange,DC=lab
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
delete member:
CN=eeeee,CN=Users,DC=exchange,DC=lab
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
ldap_modify: Server is unwilling to perform (53)
additional info: 00000561: SvcErr: DSID-031A1254, problem 5003 (WILL_NOT_PERFORM), data 0
add member:
CN=alantest,CN=Users,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modify complete
deleting entry "CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab"
delete complete
20200427150345,954.0Z = Current OS UTC time stamp
##############################################################################
Step 9 - # Query ADS endpoint for snapshot 2
##############################################################################
dxsearch -LLL -h dc2016.exchange.lab -p 389 -x -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd -b dc=exchange,dc=lab '(&(objectClass=User)(memberOf=*))' memberOf | perl -p00e 's/\r?\n //g' > snapshot_2_ads_schema.ldif
ldifsort snapshot_2_ads_schema.ldif snapshot_2_sorted_ads_schema.ldif
creating buckets
creating sort cluster 1 of size 200
sorting 0 records
creating sort cluster 2 of size 200
sorting 200 records
creating sort cluster 3 of size 200
sorting 400 records
3 buckets created
sorting 587 records
587 records sorted, 0 bad records
20200427150345,985.0Z = Current OS UTC time stamp
##############################################################################
Step 10 - # Find the delta for any removed objects
##############################################################################
ldifdelta -x -S ads_schema snapshot_2_sorted_ads_schema.ldif snapshot_1_sorted_ads_schema.ldif
dn: CN=eeeee,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-
dn: CN=alantest,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=la
b
-
dn: CN=Test User 001,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-
dn: CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab
changetype: add
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
ldifdelta summary:
587 entries in old file
588 entries in new file
Produced:
1 add entry records
0 delete entry records
3 modify entry records
20200427150346,070.0Z = Current OS UTC time stamp
##############################################################################
Step 11a: Convert from User ldapmodify syntax of 'overwrite' of 'replace'
##############################################################################
dn: CN=eeeee,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-
dn: CN=alantest,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
memberOf: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
-
dn: CN=Test User 001,CN=Users,DC=exchange,DC=lab
changetype: modify
replace: memberOf
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
-
dn: CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab
changetype: add
memberOf: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
20200427150346,163.0Z = Current OS UTC time stamp
##############################################################################
Step 11b: Convert to ADS Group ldapmodify syntax with a 'merge' of 'add' for the group objects
##############################################################################
dn: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=alantest,CN=Users,DC=exchange,DC=lab
dn: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=eeeee,CN=Users,DC=exchange,DC=lab
member: CN=Test User 001,CN=Users,DC=exchange,DC=lab
dn: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=alantest,CN=Users,DC=exchange,DC=lab
dn: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
changetype: modify
add: member
member: CN=alantest,CN=Users,DC=exchange,DC=lab
# Ignoring Users: [CN=alan-del-scenario,OU=o365,DC=exchange,DC=lab <-> CN=Account Operators,CN=Builtin,DC=exchange,DC=lab] Reason: User NOT present in the latest Snapshot! Cannot add to group.
20200427150346,172.0Z = Current OS UTC time stamp
##############################################################################
Step 11c: Query ADS Group member(s) before Roll back process
##############################################################################
dn: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab
dn: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
20200427150346,185.0Z = Current OS UTC time stamp
##############################################################################
Step 12: Roll back change to ADS User membershipOf to ADS
##############################################################################
Ignore the false positive warning message of: (ENTRY_EXISTS) - This is the 'merge' process
##############################################################################
dxmodify -c -H ldap://dc2016.exchange.lab:389 -D CN=Administrator,CN=Users,DC=exchange,DC=lab -y /tmp/.ads.pwd -f group_mod_syntax.ldif
modifying entry CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
modifying entry CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
modifying entry CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
ldap_modify: Already exists (68)
additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0
modifying entry CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
ldap_modify: Already exists (68)
additional info: 00000562: UpdErr: DSID-031A11E2, problem 6005 (ENTRY_EXISTS), data 0
20200427150346,194.0Z = Current OS UTC time stamp
##############################################################################
Step 13: Query ADS Group member after Roll back process
##############################################################################
dn: CN=Account Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=eeeee,CN=Users,DC=exchange,DC=lab
member: CN=Test User 001,CN=Users,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab
dn: CN=Access Control Assistance Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab
dn: CN=Backup Operators,CN=Builtin,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab
dn: CN=Help Desk,OU=Microsoft Exchange Security Groups,DC=exchange,DC=lab
member: CN=alantest,CN=Users,DC=exchange,DC=lab
The Symantec (CA/Broadcom) Directory solution provides a mechanism for routing LDAPv3 traffic to other solutions. This routing mechanism allows Symantec Directory to act as a virtual directory service for other directories, e.g., MS Active Directory, SunOne, Novell eDirectory, etc.
The Symantec Identity Suite solution uses the LDAP protocol for its mid-tier and connector-tier components. The Provisioning Server is exposed on TCP 20389/20390, the JCS (Java Connector Server) is exposed on TCP 20410/20411, and the CCS (C++ Connector Server) is exposed on TCP 20402/20403.
We wished to isolate provisioning data challenges within the Symantec Identity Management solution that was not fully viewable using the existing debugging logs & features of the provisioning tier & connector tiers. Using Symantec Directory, we can leverage the routing mechanism to build a MITM (man-in-the-middle) methodology to track all LDAP traffic through the Symantec Identity Manager connector tier.
We focused on the final leg of provisioning and created a process to track the JCS -> CCS LDAP traffic. We wanted to understand what and how the data was being sent from the JCS to the CCS to isolate issues to the CCS service and MS Active Directory. Using the trace level of Symantec Directory, we can capture all LDAP traffic, including binds/queries/add/modify actions.
The below steps showcase how to use Symantec Directory as an approved MITM process for troubleshooting exercises. We found this process more valuable than deploying Wireshark on the JCS/CCS Server and decoding the encrypted traffic for LDAP.
Background:
Symantec Directory documentation on routing. Please note the concept / feature of “set transparent-routing = true;” to avoid schema challenges when routing to other directory/ldap solutions.
The Symantec Identity Management connector tier may be deployed on MS Windows or Linux OS. If the CCS service is being used, then MS Windows OS is required for this MS Visual C++ component/service. As we are focused on the CCS service, we will introduce the Symantec Directory solution on the same MS Windows OS.
NOTE: We will keep the MITM process contained on a single host, and will not redirect the network traffic beyond the host.
Step 1: Deploy the latest Symantec Directory solution on MS Windows OS. This deployment is a blank slate for the next steps to follow.
Step 2: Copy the folders of schema, limits, and ssld from an existing Symantec Directory deployment of the Symantec Identity Manager solution. Using the existing schema files, references, and certificates will allow us to avoid any challenges during startup of the Router DSA due to the pre-defined provisioning/connector tier configurations. Please note when copying from a Linux OS version of Symantec Directory, we will need to update the path from Linux format to MS Windows format in the SSLD impd.dxc file for “cert-dir” and “ca-file” parameters.
Step 3: Create a new Router DSA DXI configuration file. This is the primary configuration file for Symantec Directory DSA. It will referenced the schema, knowledge, limits, and certificates for the DSA. Note the parameters for “transparent-routing” to avoid schema challenges with other solutions. Note the trace level used to trace the LDAP traffic in the Symantec Directory Router DSA trace log.
# DXserver/config/servers/admin_router_ccs_30402.dxi
# logging and tracing
close summary-log;
close trace-log;
source "../logging/default.dxc";
# schema
clear schema;
source "../schema/impd.dxg";
# access controls
clear access;
# source "../access/";
# ssld
source "../ssld/impd.dxc";
# knowledge
clear dsas;
source "../knowledge/admin_router_ccs_group.dxg";
# operational settings
source "../settings/default.dxc";
# service limits
source "../limits/impd.dxc";
# database - none - transparent router
set transparent-routing=TRUE;
# tunnel through eAdmin server error code and messages
set route-non-compliant-ldap-error-codes = true;
set trace=ldap,time,stats;
#set trace=dsa,time;
Step 4: Create the three (3) knowledge files. The “group” knowledge file will be used to redirect to the other two (2) knowledge files of the router DSA and the re-direct DSA to the CCS service.
# DXserver/config/knowledge/admin_router_ccs_group.dxg
# The admin_router_ccs_30402.dxc PORT 30402
# will be used for the IAMCS (JCS) CCS port override configuration file
# server_ccs.properties via proxyConnectionConfig.proxyServerPort=30402
source "admin_router_ccs_30402.dxc";
source "admin_ccs_server_01.dxc";
# DXserver/config/knowledge/admin_ccs_server_01.dxc
# This file is sourced by admin_router_ccs_group.dxg.
set dsa admin_ccs_server_01 =
{
prefix = <dc etasa>
dsa-name = <dc etasa><cn admin_ccs_server_01>
dsa-password = "secret"
address = ipv4 localhost port 20402
auth-levels = clear-password
dsp-idle-time = 100000
dsa-flags = load-share
trust-flags = allow-check-password, no-server-credentials, trust-conveyed-originator
link-flags = dsp-ldap
#link-flags = dsp-ldap, ssl-encryption
# Note: ssl will require update to /etc/hosts with: <IP_Address> eta_server
};
Step 5: Update the JCS configuration file that contains the TCP port that we will be redirecting to. In this example, we will declare TCP 30402 to be the new port.
Overview of all files updated and their relationship to each other.
Validation
Start up the solution in the following order. Ensure that the new Symantec Directory Router DSA is starting with no issue. If there are any syntax issues, isolate them with the command: dxserver -d start DSA_NAME.
Start the Router DSA first, then restart the im_jcs (JCS) service. The im_ccs (CCS) service will be auto-started by the JCS service. Wait one (1) minute, then check that both TCP Ports 20402 (CCS) and 30402 (Router DSA) are both in the LISTEN state. If we do not see these both ports, please stop and restart these services.
May use MS Sysinternals ProcessExplorer to monitor both services and using the TCP/IP tab, to view which ports are being used.
A view of the im_ccs.exe and dxserver.exe services and which TCP ports they are listening on.
Use a 3rd party LDAP client tool, such as Jxplorer to authenticate to both the CCS and the Router DSA ports, with the embedded service ID of “cn=root,dc=etasa”. We should see exactly the SAME data.
Use the IME or IMPS to perform a query to MS Active Directory (or any other endpoint that uses the CCS connector tier). We should now see the “cache” on the CCS service be populated with the endpoint information, and the base DN structure. We can now track all LDAP traffic through the Router DSA MITM process.
View of trace logs
We can monitor when the JCS first binds to the CCS service.
We can monitor when the IMPS via the JCS queries if the CCS is aware of the ADS endpoint.
Finally, we can view when the IMPS service decrypt its stored information on the Active Directory endpoint, and push this information to the CCS cache, to allow communication to MS Active Directory. Using Notepad++ we can tail the trace log.
Please note, this is a secure LDAP/S tunnel from the IMPS -> JCS -> CCS -> MS ADS.
We can now view how this data is pushed via this secure tunnel with the MITM process.
We can now monitor all traffic and assist with troubleshooting any CCS/MS-ADS challenges.
This same MITM methodology/process may also be used for the IMPS (TCP 20389/2039) and the JCS (TCP 20410/20411) services. We have used this process to capture the IME (JIAM) LDAP traffic to the IMPS Service, to isolate multiple queries for Child Provisioning Roles. Which has been used by the product team to enhance the solution to lower startup durations of the IME in the latest releases.
Binds/queries/add/modification all work with this approach, but we do see an issue with OID for IMPS ADS endpoint “explore process” on ADS OU object. We are reviewing how to address this last challenge that states “critical extension is unavailable” for a LDAP control property of the OU object. The OIDs captured appear to be related to SunOne/Iplanet.
Based on recent requests, we wished to revisit this “hidden” gem to expand the CA Identity Suite Provisioning schema to meet unique business requirements. Enable 100’s of SaaS and onPrem applications/endpoints for custom business logic to user’s endpoint accounts’ attributes.
Since early days of the CA Identity Suite solution (eAdmin r8.1sp2), there has been a provisioning SDK that provided an approved process to extend the CA Identity Manager’s IMPD (provisioning directory) schema from the default of 99 user custom fields to 900 additional user custom fields. To compare, typically, the default 99 user custom fields are used with the standard 40-50 default user profile fields, e.g. givenName (First Name), sn (LastName), userID, telephone #, etc. to meet most business use-cases.
Unfortunately, this extended schema process is not well known.
The only known documentation is an embedded readme.txt within a compressed package. Occasionally there will be support tickets or community notes that request this feature as an “enhancement”.
This package is included in the Provisioning SDK download; for IM r14.3, the file name is:
CA Identity Suite (Identity Manager) Provisioning Tier does NOT attempt to be a meta-directory, but act as a virtual directory to the 1000’s of managed endpoints/userstores/applications. As long as the “explore” operation was successful, there will be a “pointer” object that references the correct location of the endpoint accounts. And when a “correlation” operation occurs, this endpoint account “pointer” object is attached (via inclusion referential objects), to the associated global user ID.
By using this “virtual directory” architecture, it is possible for IM business rules or 3rd party tools to directly view the 1000’s of managed endpoints “real data” and not a “stored” representation of this data.
However, some clients do wish to “collect” the native data, and store this within the IMPD provisioning store, as SNAPSHOT data, to monitor for non-approved / OOB (out-of-band) access. If some fields are dedicated to select endpoints, the default of 99 custom fields may quickly run out.
Tackling Case-insensitivity Requirement:
Adjusting the IMPD schema for case-insensitivity; this would allow for case-insensitive correlation rules, and if the new fields are exposed to the IME, case-insensitive comparisions for business rules (PX).
Challenge:
The above Provisioning SDK process will build the extended eTCustomField100-999 and eTCustomFieldName100-999 attributes with case=sensitive. Interestingly, we did not identify a requirement for case sensitivity with the default custom fields, but it does appear this was a decision when the SDK was created. Please note the observation of the OOTB etrust_admin.schema file (for the IMPS data). This OOTB schema for the default custom fields displays a mix of case sensitivity for the eTCustomField00-99 and eTCustomFieldName00-99.
Proposal:
To address this new requirement; and to clarify there are three (3) possible deployments to enable this extended schema. We will review the pro/cons of each possible deployment choice.
Supporting Note 1:
eTCustomFieldXXX is the attribute that will contain a value.
eTCustomFieldNameXXX is the attribute that will contain a business name for this custom field.
Supporting Note 2:
The CA IM Provisioning Tier was/is developed with early x86 MS VC++ code. We attempted to use later release of the MS Visual Studio VC solution for this process but it failed to generate the output files.
Phase 1 Steps: Enhance the IM Provisioning Tier with 900 new custom fields with case = insensitive.
Download & install MS Visual Studio VC 2010 Express, to have access to the ‘nmake’ executable.
Update OS PATH variables to reference this MS VC 2010 bin folder
Execute the nmake binary, to ensure it is working fine
where make & make /?
Download & install CA IM Provisioning SDK on the same server/ workstation as ‘nmake’ binary.
IM r14.3 GEN50000000002780.zip 200 MB
Open a command line window; and then change folder to the Provisioning SDK’s COSX Samples folder
cd “C:\Program Files (x86)\CA\Identity Manager\Provisioning SDK\admin\samples\COSX”
Execute the gencosx.bat batch file to generate the additional schema for N attributes.
gencosx.bat 900 { Max allowed value is 900; which will generate 100-999 attributes}
The output text file: cosxparse.pty
**** The above steps only need to be executed ONCE on a workstation. After the output text file is generated, we should only need & retain this file for future updates. ****
Use Notepad++ to search and replace a string in the following file, cosxparse.pty
“case=sensitive” to “case=insensitive”
{We may be selective and only replace a few attributes instead of all additional 900 attributes.}
Execute the following commands to generate the binary file.
Use batch files to set environmental values for the nmake program. “C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\vcvarsall.bat“
Execute ‘nmake’ nmake
The new output file (binary) will be: C:\Program Files (x86)\CA\Identity Manager\Provisioning SDK\admin\data\ cosxparse.ptt
Before overwriting existing files; backup the three (3) prior files of IMPS/CCS data folder & IMPD schema folder for: etrust_cosx.schema etrust_cosx.dxc cosxparse.ptt
Copy the file, cosxparse.ptt, to the IMPS server data folder
Stop IMPS service: su – imps & imps stop
Execute the follow command: schemagen -n COSX
This process will create two (2) new output files:
etrust_cosx.dxc
etrust_cosx.schema
Validate the two (2) new generated files have case-insensitivity set.
Copy etrust_cosx.dxc to all CA Directory schema folders; including DX routers (on IMPS servers).
Validate this file is reference in the IMPD group knowledge schema file: etrust_admin.dxg
Copy etrust_cosx.schema & cosxparse.ptt to all CA IMPS Servers, the CCS Servers’ data folders, & the CA IMPS GUI data folder.
Validate the file, etrust_cosx.schema, is reference in the IMPS configuration file: etrust_admin.conf
Restart CA Directory and IMPS/CCS Services.
dxserver stop all / dxserver start all
imps stop / imps start
net stop im_jcs / net start im_jcs {this will also restart the im_ccs service}
With the IMPS GUI
Assign a ‘business name’ to the newly created eTCustomField100+ under SYSTEM/GLOBAL PROPERTIES/CUSTOM USER FIELDS {If you do not see these newly created fields, then the IMPS GUI data folder was not updated per step 11.}
Validate that E&C Correlation Rules will now work for these extended fields with case-insensitivity. SYSTEM/DOMAIN CONFIGURATIONS/EXPLORE AND CORRELATE/CORRELATION ATTRIBUTE/
Validate the custom fields are viewable for each Global User.
We may now STOP HERE if we do NOT need to expose these new custom fields to the IME.
Pro: Able to use customfields for account templates and correlations rules.
Con: Not exposed to IME for 1:1 mapping nor exposed for PX Business Rules.
Update the CA IMPS directory.xml as needed for some or all 900 fields.
<ImsManagedObjectAttr physicalname="eTCustomField100" description="Custom Field 100" displayname="Custom Field 100" valuetype="String" multivalued="true" wellknown="%CUSTOM_FIELD_100%" maxlength="0"/>
Update the IME’s IMCD to IMPS 1:1 mappings.
identityEnv_environment_settings.xml
We may now stop here. The next advance configuration is only required if we wish to manage the various Endpoint Mapping Tab with the IM UI; instead of the IMPS GUI. We would consider the next Phase 3 Steps, to be low value for the effort; as this configuration is typically set once and done in the IMPS GUI.
Pro: Able to use customfields for account templates and correlations rules. Also able to map these files 1:1 in the IME for IMCD attributes to be mapped to the IMPS extended custom attributes. These IMPS extended custom attributes will now be exposed for PX Business Rules.
Con: Not exposed to IM UI to update Endpoint’s Mapping TAB for ADS and DYN endpoints.
Phase 3 Steps – IME Advanced
If planning on exposing these new custom fields in both the Endpoint’ Mapping Attribute Screen & Endpoint Account Templates via the IME, follow these additional steps:
Replace commonobjects.xml in ..\Identity Manager\IAM Suite\Identity Manager\tools\RoleDefinitionGenerator\lib\roledefgen.jar by following the steps given below:
rename roledefgen.jar as roledefgen.zip
Open roledefgen.zip
open com\ca\iam\roledefgen\commonobjects.xml and replace the contents with the attached/provided commonobjects.xml file
save the zip
rename the zip to jar
Now roledefgen.jar will contain the commonobjects.xml file with extended custom attributes
execute the below RoleDefGenerator.bat to generate jars for all the required java/Dyn endpoints
open the generated endpoint jars one by one and modify them by following below steps:
rename the original .jar as .zip
open framework.xml and increase the version “version=” (2nd line)
rename the generated .jar as .zip
open and copy the contents of -RoleDef.xml
paste the copied content in step 4 to the file -RoleDef.xml in original .zip (step 1)
save the original .zip and rename it to .jar
replace the save .jar in ..\wildfly-8.2.0.Final\standalone\deployments\iam_im.ear\ user_console.war\WEB-INF\lib
restart IM and test the Custom attributes in IM web-UI
Post Update Note:
Validate if we may need to rebuild the IMPD DSAs (4) for existing users that may have already had these extended attributes but with case=sensitive set previously.
This step is not required if this is the first time the extended attributes have been deployed or if the case=sensitive has not been changed.
Process: Export the IMPD LDIFs, rebuild the IMPD DSA and then re-import LDIFs.
I was interested to see an intersection between Docker, VMware, and an application (Home Assistant) that users may wish to run on their laptops and/or workstations.
The Home Assistant application seemed especially valuable to business travelers/road warriors that would like a simple and flexible dashboard to keep an eye out for activity at home.
I have put together the following steps to be completed in thirty (30) minutes or less using community and/or non-commercial licenses.
This lab will cover the following solutions/applications: VMware player (free personal license), home assistant (open-source home automation platform ), Docker (automation of application on a prebuilt os), Ring Door Bell (ring.com) and Fast.com (monitor of download speeds)
Please review and see if this lab may have value to your project team(s) to increase their awareness of docker and still have value for home use.
Ring Door Bell (ring.com) & Fast.com
The above Dashboard image is the goal of this lab; to take advantage of the community tools for home automation, and enable your Ring.com credentials to allow viewing/monitoring while on the road or at home. Additionally, we have added Fast.com configuration to allow for bandwidth monitoring of download speed using the Netflix’s sponsored site.
Step 1: Create a single folder for download(s) and installation
Avoid clutter from VMware configuration and data files if allowed to use defaults. Otherwise, we may have files in two (2) different folders.
Step 2a: Download the Home Assistant VMDK bootable disk image
We wish to pre-download this bootable image to be ready to be consumed by VMware Player (Note: If you already have VMware workstation, you may use it as well instead of VMware Player)
See the link below in the next step.
Step 2b: Download the Home Assistant VMDK bootable disk image
The pre-built vmdk compressed file may be accessed under “Getting Started” and “Software Requirements”
Select the “VMDK (VMWare Workstation) link to download this file.
Step 2c: Copy and Extract the VMDK from the compressed gz file
Suggest a copy be made of the vmdk file, as future steps will modify this file. The file is compressed with gzip, but you may use 7zip ( https://www.7-zip.org/ ) or other 3rd party tools to extract. The MS Windows built-in zip tool will not likely extract this file.
Step 3a: Download a free, personal license copy of Vmware Player
If you already have VMware Workstation, you may skip these series of steps; or you may wish to install this VMware Player package along with your existing VMware workstation installation.
Step 3b: Install Vmware Player, and designate for personal use aka “non-commercial use” when asked for license key.
During installation, when asked for a license, select “non-commercial use” for personal use on your home laptop/workstation.
Step 4a: Start Vmware Player, and select “Create a New Virtual Machine”
Now we are ready to create our first Virtual Machine on our laptop/workstation. We will use a default boot-strap configuration to build the initial settings, then modify them for the Home Assistant pre-built bootable disk image.
Step 4b: Select the following configurations to jump start VMDK
Choose a generic Linux Operating System and Version configuration. I selected “Other Linux 5.x or later kernel 64-bit”. Next, select the folder where the Home Assistance vmdk file was extracted. Rename your VM as you wish. I kept it as “homeassistant”.
Step 4c: Allow discovery of the VMDK for Home Assistant
VMware player will recognize that a pre-existing vmdk file exists in this folder, and will warn you of this fact. Click Continue to accept this warning message.
On the next screen, select “Store virtual disk as a single file” to avoid the clutter of temporary files.
Step 4d: Create the new Virtual Machine
We are now ready to complete the new Virtual Machine with default configurations.
Note: When this step is complete, please do NOT start/play the VM yet; as that will define default OS configuration settings; which we do not require.
Step 5a: Edit the new Virtual Machine Settings
Now we are ready to adjust the default configurations to enable the use of the pre-built Home Assistant VMDK bootable disk file.
Step 5c: Add correct Hard Drive Type (IDE) for bootable VMDK
Select “Add” button, to re-add a “Hard Drive” with Type = IDE. Select “Use an existing virtual disk”. This “existing virtual disk” will be the Home Assistant VMDK file.
Select Next button.
Step 5d: Select the “hassos_ova-2.xx.vmdk” file for the bootable existing disk
Select the Home Assistant VMDK file that was extracted. Ensure that you do NOT select the temporary file that was created prior with the name “homeassistant.vmdk”
Select Finish button.
Step 5e: Allow vmdk disk to be imported
You may convert or allow the VMDK to remain in its prior “format”. We have tested with both selections; and have not observed any impact with either selection.
After import, observe that the Hard Drive now has IDE as the connection configuration.
We will now expand this Hard Drive from the default of 6 GB (maximum size) in the next step.
Step 5f: Expand VMDK from 6 GB (default) to 32 GB for max disk size
Select “Hard Drive”, then in the right sub-panel, select “Expand disk capacity”
Update the value from 6.0 to 32.0 for maximum disk size in GB.
Click OK and observe the update on both panel windows for the hard drive.
Click OK to close edit windows. Reminder: Do NOT start/play the image yet.
Step 6a: Convert “BIOS” (default) to “EFI” type for new Virtual Machine
Last step before we start the image. The Home Assistant bootable VMDK disk was designed and configured for the boot-loader of EFI, instead of the older legacy “BIOS” boot-loader.
If you have VMware workstation/ ESXi server, you may have access to a GUI entry to adjust this virtual firmware bootloader configuration.
However, VMware Player does not expose this setting in the GUI. To address this challenge, we will use VMware documented method to directly update the configuration file for our new Virtual machine for one (1) setting. https://communities.vmware.com/docs/DOC-28494
Navigate to the folder where the VMDK was extracted. You will now see several other files, include the primary configuration file for our new Virtual Machine. Its name will be “homeassistant.vmx” . The “*.vmx” filename extension/suffix will contain hardware configuration for booting the VWmare VM server image.
Step 6b: Edit configuration file for new Virtual Machine
Use either MS Windows notepad.exe or Notepad++ or similar tool to edit the configuration file.
If the VM image was not started, we will NOT find a key:value pair with the string “firmware”. Note: If the VM image was started before we add in our entry, then startup issues will occur. (If this happens, please restart the lab from Step 4a.)
Append the following string to the bottom of the file & save the file.
firmware = “efi“
Step 7a: Start the new Virtual Machine
We are now ready to start our image and begin to use the Home Assistance application. Select our new Virtual Machine & click “Play virtual machine”.
Observe the screen for “boot-loader” information related to EFI. This will be confirmation that we did configure the VMDK hard drive image to load correctly and will have no unexpected issues.
Step 7b: Click within Virtual Machine window to “active” and then <enter>
The VM will boot fairly quickly, and you may notice the text will appear to stop.
Click within the VM window with your mouse, then press the <ENTER> key to see the login prompt.
Enter the login userID: root
Note: If you wish to re-focus your mouse/keyboard outside of VMware Player, press the keys <CNTRL> and <ALT> together, to redirect focus. Click back into the VMware Player window anytime to enter new text.
Step 7c: Discover IP address of homeassistant docker application
Now we get to play with some basic shell and docker commands to get our IP address and validate a port.
At the hassio > prompt, enter the text: login
This will give us a root shell account. To find our current dynamic IP address, that the VMplayer installation created for us, issue the following command:
ip addr | grep dynamic
To view the three (3) docker containers, issue the following command:
docker ps
This will display the status of each container. After 1 minute uptime, we can use the Home Assistant application.
To validate the actual TCP Port used (8123), issue the following docker command:
We will use the IP address and TCP port (8123) within a browser window (IE/Chrome/Firefox/Opera/etc.) on the laptop/workstation to access the Home Assistant application.
Step 8a: Login to Home Assistant Application with a Browser
When we first start the Home Assistant Application, it will ask for a primary account to be created. Use either your name or admin or any value.
If you plan to eventually expose this application to the internet from your home system, we would recommend a complex password; and perhaps storage in a key safe like LastPass https://www.lastpass.com/ or locally in Key Pass https://keepass.info/ file.
Step 8b: Use detect to re-assign default location to your area
Adjust the defaults to your location if you wish. Use the “detect” feature to reset values, then click next. May use a mouse to assist with refinement of location on the embedded map feature.
Step 8c: Home Assistant Landing Page
Click Finish to skip the question about early integration.
Now we are at the Landing Page for Home Assistant. Congratulations with the setup of Home Assistant.
We now will configure two (2) items that have value to home users.
Step 9a: Enable the Home Assistance Configuration Tool
Before we add-on new features, we need to make it easy for us to adjust the Home Assistance configuration file.
Select the MENU item (three lines in the upper left window – Next to HOME string)
You will see a side panel of selection items. Select “Hass.io“
Step 9b: Select Add-On Store & Configurator Tool
Select the “ADD-ON STORE” displayed at the top of the window. Scroll down till you view the item “Configurator” under the section “Official add-ons”
Select the item “Configurator”
Step 9c: Install and Start the Configurator Tool
Select “Install” and “Start” of the “Configurator” Tool
Step 9d: Open the Web UI to use the Configurator Tool
Select the “Open Web UI” link. You may wish to save this URL link in your favorites or remember how to re-access this URL with additional updates.
After the landing page for the “Configurator” tool has loaded, select the FOLDER ICON in the upper left of the window. This will allow you to access the various configuration files.
Step 9e: Select primary Home Assistant configuration file (configuration.yaml)
Now select configuration.yaml from the left panel. The default configuration file will load with minimal information.
This is where we will make most of the updates to enable our home applications of Ring Doorbell and Fast.com (download monitor).
Step 10a: Add fast.com & Ring Door Bell Add-On (with sensors/camera)
We are now ready to add in as many integrations as we wish.
There are 100’s of prebuilt configurations that can be reviewed on the Home Assistant site.
For Ring Doorbell (ring.com) and Fast.com, we have already identified the configurations we need, and these can be pasted to the primary configuration file. We have also enclosed the references for each configuration.
# Download speed test for home use
# Ref: https://www.home-assistant.io/integrations/fastdotcom/
fastdotcom:
scan_interval:
minutes: 30
# Ring Doorbell
# Ref: https://www.home-assistant.io/integrations/ring/
# Ref: http://automation.moebius.site/2019/01/hassio-home-assistant-installing-a-ring-doorbell-and-simple-automations/
# Ref: https://www.ivobeerens.nl/2019/01/15/install-home-assistant-hass-io-in-vmware-workstation/
sensor:
- platform: ring
ring:
username: !secret ring_username
password: !secret ring_password
camera:
- platform: ring
binary_sensor:
- platform: ring
Step 10b: Save configuration.yaml file & confirm no syntax errors
Click save, and validate that you have a GREEN checkbox (this is used for syntax checking of the configuration files for spacing and formatting).
After saving, click the FOLDER ICON in the upper left.
We will now add the Ring.com credentials to the secrets.yaml file.
Step 10c: Select “secrets.yaml” to host the Ring.com credentials
From the side panel, select the “secrets.yaml configuration file to add the Ring.com credentials.
Step 10d: Enter Ring.com credentials & save this file
Enter Ring.com credentials in the following format.
# Enter your ring.com credentials here to keep them separate
# from the default configuration file.
ring_username: email_address_used_for_ring.com_here@email.com
ring_password: password_used_for_ring.com_here
Step 11a: Restart Home Assistance Application
Configurations are done. Restart the Home Assistance Application to use the configurations for Ring.com and Fast.com
Select “Configuration” from the left panel menu, then scroll down in right panel to select “Server Controls”
Step 11b: Restart Home Assistant Application
Select “Restart” and accept the warning message with OK. The connection will drop for 30-60 seconds, then the browser may reload with the prior screen. (If you saved your credentials in the browser password management section when “asked” by the browser). If not, re-authenticate with your Home Assistant credentials.
Step 11c: Extra – Monitor for Error Messages in Notification Logs
This section is ONLY needed if you see an error message in the Notification Logs, e.g. missing data in the secrets.yaml and/or incorrect credentials for Ring.com.
Step 12: Done – Site 1 & Site 2
Below example for one (1) site with just one (1) Door Bell Ring device and integrated with Fast.com
Example with many devices integrated with Ring.com
We hope this lab was of value, and that others take advantage of this prebuilt appliance with docker and vmware. Please share with others to allow them to to gain awareness of docker processes.
Extra of interest: AWS and Ring.com Mp4 Videos
There are additional configurations that will allow auto-downloading of the mp4 videos from the AWS hosted site for Ring.com. Note the Video_URL for camera.front_door.
A view of the many pre-built integrations for Home Assistant
Docker Image(s) provide fantastic value, as this platform-as-a-service methodology gets us all out of the painful “install-business”. We may focus effort on the business value that a solution provides.
However, the associated docker containers may provide some challenges.
Business Risks:
For the CA API Gateway solution, we have two (2) business risks to address:
The docker container of the API Gateway solution may be ephemeral & replaced with newer releases.
Example: Any API Gateway application logs that reside on the docker container, as a file, may be lost when the container version is updated or redeployed.
The docker container of the associated MySQL Database may have growth concerns with the default OOTB API Gateway Audit Event process.
Example: The MySQL Database of ibdata1 may continue to grow and be impacted by current disk constraints. To reduce MySQL Database size and remove the low value audit data, it will be necessary to declare an outage window to export (w/o audit data)/resize mysql db/import the data (w/o audit data).
Resolution(s):
To address the above risks, we may leverage the syslog feature set provided by the API Gateway:
The API Gateway documentation does allow for syslog configuration for the primary Audit Event and the application logs.
This blog entry will review the use of syslog, create individual syslog files for each API Gateway application, avoid common mis-configurations, and how to validate the processes.
Before we start this process, to provide justification, lets review how “large” is the API Gateway Audit Events in the MySQL Database & that API Gateway logs are no longer retained on the container (as of r9.4 release)
Validation of API Gateway Database Growth & Application Logs:
Pre-Step 00: Review the current MySQL ibdata1 file size (via docker command)
docker exec -it -u root -e TERM=XTERM ssg94_mysql57 /bin/bash -c "find /var -type f -mtime -1 -ls | head -5"
Pre-Step 01: Review the audit tables sizes in the ‘ssg’ MySQL database with the below query:
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'` mysql --user=gateway --password=7layer ssg -e "SELECT TABLE_NAME, table_rows, data_length, index_length, round(((data_length + index_length) / 1024 / 1024),2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema = 'ssg' ORDER BY (data_length + index_length) DESC; "
Pre-Step 02: Validate no API Gateway application files reside and primary ssg file is redirect to /dev/null (r9.4)
Add a semi-colon with the facility.none on this line
;local3.none;local4.none;local5.none;local6.none
Restart the updated rsyslog.service
systemctl restart rsyslog.service
Validate UDP 514 is available.
netstat -an | grep :514
Step 2: Validate Syslog is functioning correctly with the facilities (syslog naming convention) using the OS command:
logger -s -p local3.warn Testing for ServiceNow SysLog
logger -s -p local4.warn Testing for Google GCP SysLog
logger -s -p local5.warn Testing for OpenStack SysLog
logger -s -p local6.warn Testing for SSG Audit Event SysLog
Step 3: Enable syslog for each API Application with a unique log file associated to a unique facility number.
a. Review the associated API Gateway Applications in the API Gateway Policy Manager and their associated URI strings,e.g. /gcp/*, /servicenow/*, etc.
b. Open the API Gateway Policy Manager / Tasks / Logging and Auditing / Manage Log/Audit Sinks Window
c. Create new Log Sinks for every API Gateway Application. Configure these for ‘syslog’, with the correct facility number, and ensure the FILTER is set correctly with the Category & Services URI string match (otherwise you may not see any data). Suggest Category=Traffic Log (as your first iteration). Reference for facility number (0-23) to match syslog naming conventions: https://tools.ietf.org/html/rfc5424 [page 10]
Step 4: Validate the API Gateway Application Syslog with remote web service call via curl to the various application URI strings.
The current documentation for enabling syslog for the API Gateway Audit Event process is clear, but administrators may have a challenge if the FILTER is not set correctly. To address this common mis-configuration, the follow steps are provided to run through the configuration, and clarify where to avoid this challenge.
Step 1AE: Review the prior OOTB configuration / view. The API Gateway Audit Events are stored in the MySQL database, and the built-in view tool allows for queries to be performed. However, unless the “File/Delete Old Audit Events” is executed, growth will continue to impact MySQL db.
Step 2AE: Configure the API Gateway Audit Events to not use the MySQL database. The below process will create a “[Internal Audit Sink Policy]” that we will use to redirect.
Step 3AE: Important Step: Disable “all” assertions in the newly created “[Internal Audit Sink Policy]”.
Double-click on the lower-left panel item ‘[Internal Audit Sink Policy]”.
The right-upper panel will displays nine (9) line items.
Use mouse to select all lines, then click on the RED button on the middle column to disable all assertions (lines).
In the upper-left panel, search for the string “continue”
The object “Continue Processing” will display
Use mouse to drag-n-drop this item to the upper-right panel.
Click “Save and Activate” selection on the upper-right panel, as shown in the image below.
Step 4AE: Now create the new API Gateway Log Sink for Audit Events. Ensure Filters are broad to the following Categories: Audits and Gateway Log. Avoid any additional filters. Ensure the facility number matches the syslog naming convention.
Step 5AE: Confirm that audit events are being sent to the syslog defined.
Step 6AE: Confirm that API Gateway configuration for syslog is retained after “destroying” the docker container, and rebuilding it to connect to the existing MySQL database.
a. Destroy and rebuild API Gateway container (r9.4)
The primary API Gateway startup ‘ssg’ logs, that are accessible via ‘docker logs containerID’ may be converted to syslog, but this process will impact the JSON formatting that that ‘docker logs containerID’ uses.
A review of the OOTB syslog for /var/log/messages shows that these docker messages are also already forwarded here. Recommend skipping this unnecessary configuration step.
View of the “View Logs” Window for “Log Sinks” may display, but no data will be returned, as there are no local files. See prior screen shot where ssg_0_0.log is redirected to /dev/null.
If the above screen does not load correctly (r9.4), then in the docker-compose file, add in an additional JVM switch:
The CA Directory solution provides a mechanism to automate daily on-line backups, via one simple parameter:
dump dxgrid-db period 0 86400;
Where the first number is the offset from GMT/UTC (in seconds) and the second number is how often to run the backup (in seconds), e.g. Once a day = 86400 sec = 24 hr x 60 min/hr x 60 sec/min
Two Gaps/Challenge(s):
History: The automated backup process will overwrite the existing offline file(s) (*.zdb) for the Data DSA. Any requirement or need to perform a RCA is lost due to this fact. What was the data like 10 days ago? With the current state process, only the CA Directory or IM logs would be of assistance.
Size: The automated backup will create an offline file (*.zdb) footprint of the same size as the data (*.db) file. If your Data DSA (*.db) is 10 GB, then your offline (*.zdb) will be 10 GB. The Identity Provisioning User store has four (4) Data DSAs, that would multiple this number , e.g. four (4) db files + four (4) offline zdb files at 10 GB each, will require minimal of 80 GB disk space free. If we attempt to retain a history of these files for fourteen (14) days, this would be four (4) db + fourteen (14) zdb = eighteen (18) x 10 GB = 180 GB disk space required.
Resolutions:
Leverage the CA Directory tool (dxdumpdb) to convert from the binary data (*.db/*.zdb) to LDIF and the OS crontab for the ‘dsa’ account to automate a post ‘online backup’ export and conversion process.
Step 1: Validate the ‘dsa’ user ID has access to crontab (to avoid using root for this effort). cat /etc/cron.allow
If access is missing, append the ‘dsa’ user ID to this file.
Step 2: Validate that online backup process have been scheduled for your Data DSA. Use a find command to identify the offline files (*.zdb ). Note the size of the offline Data DSA files (*.zdb).
Step 3: Identify the online backup process start time, as defined in the Data DSA settings DXC file or perhaps DXI file. Convert this GMT offset time to the local time on the CA Directory server. (See references to assist)
Step 4: Use crontab -e as ‘dsa’ user ID, to create a new entry: (may use crontab -l to view any entries). Use the dxdumpdb -z switch with the DSA_NAME to create the exported LDIF file. Redirect this output to gzip to automatically bypass any need for temporary files. Note: Crontab has limited variable expansion, and any % characters must be escaped.
Example of the crontab for ‘dsa’ to run 30 minutes after (at 2 am CST) the online backup process is scheduled (at 1:30 am CST).
# Goal: Export and compress the daily DSA offline backup to ldif.gz at 2 AM every day
# - Ensure this crontab runs AFTER the daily automated backup (zdb) of the CA Directory Data DSAs
# - Review these two (2) tokens for DATA DSAs: ($DXHOME/config/settings/impd.dxc or ./impd_backup.dxc)
# a) Location: set dxgrid-backup-location = "/opt/CA/Directory/dxserver/backup/";
# b) Online Backup Period: dump dxgrid-db period 0 86400;
#
# Note1: The 'N' start time of the 'dump dxgrid-db period N M' is the offset in seconds from midnight of UTC
# For 24 hr clock, 0130 (AM) CST calculate the following in UTC/GMT => 0130 CST + 6 hours = 0730 UTC
# Due to the six (6) hour difference between CST and UTC TZ: 7.5 * 3600 = 27000 seconds
# Example(s):
# dump dxgrid-db period 19800 86400; [Once a day at 2330 CST]
# dump dxgrid-db period 27000 86400; [Once a day at 0130 CST]
#
# Note2: Alternatively, may force an online backup using this line:
# dump dxgrid-db;
# & issuing this command: dxserver init all
#
#####################################################################
# 1 2 3 4 5 6
# min hr d-o-m month d-o-w command(s)
#####################################################################
#####
##### Testing Backup Every Five (5) Minutes ####
#*/5 * * * * . $HOME/.profile && dxdumpdb -z `dxserver status | grep "impd-main" | awk "{print $1}"` | gzip -9 > /tmp/`hostname`_`dxserver status | grep "impd-main" | awk '{print $1}'`_`/bin/date --utc +\%Y\%m\%d\%H\%M\%S.0Z`.ldif.gz
#####
##### Backup daily at 2 AM CST - 30 minutes after the online backup at 1:30 AM CST #####
#####
0 2 * * * . $HOME/.profile && dxdumpdb -z `dxserver status | grep "impd-main" | awk "{print $1}"` | gzip -9 > /tmp/`hostname`_`dxserver status | grep "impd-main" | awk '{print $1}'`_`/bin/date --utc +\%Y\%m\%d\%H\%M\%S.0Z`.ldif.gz
0 2 * * * . $HOME/.profile && dxdumpdb -z `dxserver status | grep "impd-co" | awk "{print $1}"` | gzip -9 > /tmp/`hostname`_`dxserver status | grep "impd-co" | awk '{print $1}'`_`/bin/date --utc +\%Y\%m\%d\%H\%M\%S.0Z`.ldif.gz
0 2 * * * . $HOME/.profile && dxdumpdb -z `dxserver status | grep "impd-inc" | awk "{print $1}"` | gzip -9 > /tmp/`hostname`_`dxserver status | grep "impd-inc" | awk '{print $1}'`_`/bin/date --utc +\%Y\%m\%d\%H\%M\%S.0Z`.ldif.gz
0 2 * * * . $HOME/.profile && dxdumpdb -z `dxserver status | grep "impd-notify" | awk "{print $1}"` | gzip -9 > /tmp/`hostname`_`dxserver status | grep "impd-notify" | awk '{print $1}'`_`/bin/date --utc +\%Y\%m\%d\%H\%M\%S.0Z`.ldif.gz
Example of the above lines that can be placed in a bash shell, instead of called directly via crontab. Note: Able to use variables and no need to escape the `date % characters `
Monitor with tail -f /var/log/cron (or syslog depending on your OS version), when the crontab is executed for your ‘dsa’ account
View the output folder for the newly created gzip LDIF files. The files may be extracted back to LDIF format, via gzip -d file.ldif.gz. Compare these file sizes with the original (*.zdb) files of 2GB.
Recommendation(s):
Implement a similar process and retain this data for fourteen (14) days, to assist with any RCA or similar analysis that may be needed for historical data. Avoid copied the (*.db or *.zdb) files for backup, unless using this process to force a clean sync between peer MW Data DSAs.
The Data DSAs may be reloaded (dxloadb) from these LDIF snapshots; the LDIF files do not have the same file size impact as the binary db files; and as LDIF files, they may be quickly search for prior data using standard tools such as grep “text string” filename.ldif.
This process will assist in site preparation for a DAR (disaster and recovery) scenario. Protect your data.
While assisting a site with their upgrade process from CA API Gateway 9.2 (docker) to the latest CA API Gateway 9.4 image, we needed to clarify the steps. In this blog entry, we have capture our validation processes of the documented and undocumented features of API Gateway docker deployment ( https://hub.docker.com/r/caapim/gateway/ ), pedantic verbose steps to assist with training of staff resources; and enhanced the external checks for a DAR (disaster and recovery) scenario using docker & docker-compose tools.
Please use this lab to jump start your knowledge of the tools: ‘docker’, ‘docker-compose’ and the API Gateway. We have added many checks and the use of bash shell to view the contents of the API Gateway containers. If you have additional notes/tips, please leave a comment.
To lower business risk during this exercise, we made the follow decisions:
1) Avoid use of default naming conventions, to prevent accidental deletion of the supporting MySQL database for CA API Gateway. The default ‘docker-compose.yml’ was renamed as appropriate for each API Gateway version.
2) Instead of using different folders to host configuration files, we defined project names as part of the startup process for docker-compose.
3) Any docker container updates would reference the BASH shell directly instead of a soft-link, to avoid different behaviors between the API GW container and the MySQL container.
Challenges:
Challenge #1: Both the API Gateway 9.2 and 9.4 docker container have defects with regards to using the standardized ‘docker stop/start containerID‘ process. API Gateway 9.2 would not restart cleanly; and API Gateway 9.4 container would not update the embedded health check process, e.g. docker ps -a OR docker inspect containerID
Resolution #1: Both challenges were addressed in the enclosed testing scripts. Docker-compose is used exclusively for API Gateway 9.2 container, and touching an internal file in the API Gateway 9.4 container.
Challenge #2: The docker parameters between API Gateway 9.2 and API Gateway 9.4 had changed.
Resolution #2: Identify the missing parameters with ‘docker logs containerID’ and review of the embedded deployment script of ‘entrypoint.sh’
Infrastructure: Seven (7) files were used for this lab on CentOS 7.x (/opt/docker/api)
ssg_license.xml (required from Broadcom/CA Sales Team – ask for 90 day trial if a current one is not available)
docker-compose-ssg94.yml (the primary install configuration file for API GW 9.4)
docker-compose-ssg92.yml (the primary install configuration file for API GW 9.2)
docker-compose-ssg94-join-db.xml (the restart configuration file – use as needed)
docker-compose-ssg92-join-db.xml (the restart configuration file – use as needed)
01_create_both_ssg92_and_ssg94_docker_deployments.sh (The installation of ‘docker’ and ‘docker-compose’ with the deployment of API GW 9.2 [with MySQL 5.5] and API GW 9.4 [with MySQL 5.7] ; with some additional updates)
02_backup_and_migrate_mysql_ssg_data_ from_ssg92_to_ssg94_db.sh (The export/import process from API GW 9.2 to API GW 9.4 and some additional checks)
#!/bin/bash
##################################################################
#
# Script to validate upgrade process from CA API GW 9.2 to 9.4 with docker
# - Avoid using default of 'docker-compose.yml'
# - Define different project names for API GW 9.2 and 9.4 to avoid conflict
# - Explictly use bash shell /bin/bash instead of soft-link
#
# 1. Use docker with docker-compose to download & start
# CA API GW 9.4 (with MySQL 5.7) &
# CA API GW 9.2 (with MySQL 5.5)
#
# 2. Configure CA API GW 9.4 with TCP 8443/9443
# CA API GW 9.2 with TCP 8444/9444 (redirect to 8443/9443)
#
# 3. Configure MySQL 5.7 to be externally exposed on TCP 3306
# MySQL 5.5 to be externally exposed on TCP 3307
# - Adjust 'grant' token on MySQL configuration file for root account
#
# 4. Validate authentication credentials to the above web services with curl
#
#
# 5. Add network modules via yum to API GW 9.4 container
# - To assist with troubleshooting / debug exercises
#
# 6. Enable system to use API GW GUI to perform final validation
# - Appears to be an issue to use browers to access the API GW UI TCP 8443/8444
#
#
# Alan Baugher, ANA, 10/19
#
##################################################################
echo ""
echo ""
echo "################################"
echo "Install docker and docker-compose via yum if missing"
echo "Watch for message: Nothing to do "
echo ""
echo "yum -y install docker docker-compose "
yum -y install docker docker-compose
echo "################################"
echo ""
echo "################################"
echo "Shut down any prior docker container running for API GW 9.2 and 9.4"
cd /opt/docker/api
pwd
echo "Issue this command if script fails: docker stop \$(docker ps -a -q) && docker rm \$(docker ps -a -q) "
echo "################################"
echo ""
echo "################################"
export SSG_LICENSE_ENV=$(cat ./ssg_license.xml | gzip | base64 --wrap=0)
echo "Execute 'docker-compose down' to ensure no prior data or containers for API GW 9.4"
docker-compose -p ssg94 -f /opt/docker/api/docker-compose-ssg94.yml down
echo "################################"
echo "Execute 'docker-compose down' to ensure no prior data or containers for API GW 9.2"
docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92.yml down
echo "################################"
echo ""
echo "################################"
echo "Execute 'docker ps -a' to validate no running docker containers for API GW 9.2 nor 9.4"
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "################################"
echo ""
echo "################################"
echo "Change folder to execute docker-compose script for API GW 9.4 with MySql 5.7 with TCP 8443/9443"
echo "Execute 'docker-compose up -d' to start docker containers for API GW 9.4 with MySql 5.7 with TCP 8443/9443"
docker-compose -p ssg94 -f /opt/docker/api/docker-compose-ssg94.yml up -d
echo "################################"
echo "Change folder to execute docker-compose script for API GW 9.2 with MySql 5.5 with TCP 8444/9444"
echo "Execute 'docker-compose up -d' to start docker containers for API GW 9.2 with MySql 5.5 with TCP 8444/9444"
docker-compose -p ssg92 -f /opt/docker/api/docker-compose-ssg92.yml up -d
echo "################################"
echo ""
echo "################################"
echo "Backup current API GW 9.4 running container for future analysis"
echo "docker export ssg94 > ssg94.export.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar "
docker export ssg94 > ssg94.export.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar
echo "################################"
echo ""
echo "################################"
echo "Update API GW 9.4 running container with additional supporting tools with yum"
echo "docker exec -it -u root -e TERM=xterm ssg94 /bin/sh -c \"yum install -y -q net-tools iproute unzip vi --nogpgcheck\" "
docker exec -it -u root -e TERM=xterm ssg94 /bin/sh -c "yum install -y -q net-tools iproute unzip vi --nogpgcheck "
echo "Export API GW 9.4 running container after supporting tools are added"
echo "docker export ssg94 > ssg94.export.tools.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar "
docker export ssg94 > ssg94.export.tools.`/bin/date --utc +%Y%m%d%H%M%S.0Z`.tar
echo "################################"
echo ""
echo "################################"
echo "Validate network ports are exposed for API GW Manager UI "
netstat -anpeW | grep -e docker -e "Local" | grep -e "tcp" -e "Local"
echo "################################"
echo ""
echo "################################"
echo "Sleep 70 seconds for both API GW to be ready"
echo "################################"
sleep 70
echo ""
echo ""
echo "################################"
echo "Extra: Open TCP 3306 for mysql remote access "
docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /bin/bash -c "echo -e '\0041includedir /etc/mysql/conf.d/\n\0041includedir /etc/mysql/mysql.conf.d/\n[mysqld]\nskip-grant-tables' > /etc/mysql/mysql.cnf && cat /etc/mysql/mysql.cnf "
#docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /bin/bash -c "/etc/init.d/mysql restart"
#docker exec -it -u root -e TERM=xterm `docker ps -a | grep mysql:5.7 | awk '{print $1}'` /bin/bash -c "/etc/init.d/mysql status && echo -n"
echo "################################"
docker restart ssg94_mysql57
echo ""
echo "################################"
echo "Execute 'docker ps -a' to validate running docker containers for API GW 9.2 and 9.4 with their correct ports"
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.RunningFor}}\t{{.Status}}\t{{.Ports}}"
echo "################################"
echo ""
echo "################################"
echo "Test authentication with the SSG backup URL for API 9.2 TCP 8444 - should see six (6) lines"
echo "curl -s --insecure -u pmadmin:7layer https://$(hostname -s):8444/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "######### ############"
curl -s --insecure -u pmadmin:7layer https://$(hostname -s):8444/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""
echo "################################"
echo "Test authentication with the SSG backup URL for API 9.4 TCP 8443 - should see six (6) lines"
echo "curl -s --insecure -u pmadmin:7layer https://$(hostname -s):8443/ssg/backup | grep -e 'title' -e 'Gateway node' -e 'input' -e 'form action' "
echo "######### ############"
curl -s --insecure -u pmadmin:7layer https://$(hostname -s):8443/ssg/backup | grep -e "title" -e "Gateway node" -e "input" -e "form action"
echo "################################"
echo ""
echo "################################"
echo "Next Steps:"
echo " Open the API GW UI for 9.2 and create a new entry in the lower left panel"
echo ""
echo "Example: "
echo " Right click on hostname entry and select 'Publish RESTful Service Proxy with WADL' "
echo " Select Manual Entry, then click Next"
echo " Enter data for two (2) fields:"
echo " Service Name: Alan "
echo " Resource Base URL: http://www.anapartner.com/alan "
echo " Then select Finish Button "
echo "################################"
echo ""
View of the API Gateway via the MS Windows API GW UI for both API GW 9.2 (using the 9.3 UI) and API 9.4 (using the 9.4 UI). The API GW Policies will be migrated from API 9.2 to API 9.4 via the export/import of MySQL ssg database. After import, the API GW 9.4 docker image will ‘auto’ upgrade the ssg database to the 9.4 version.
Interesting view of the API GW 9.4 MySQL database ‘ssg’ after import and a restart (that will ‘auto’ upgrade the ssg database version). Note multiple Gateway “nodes” that will appear after each ‘docker restart containerID’
